Skip to content

feat(btree): refactor GlobalIndexScan & add inte test for btree global index#262

Open
lszskye wants to merge 1 commit intoalibaba:mainfrom
lszskye:global_index_eval
Open

feat(btree): refactor GlobalIndexScan & add inte test for btree global index#262
lszskye wants to merge 1 commit intoalibaba:mainfrom
lszskye:global_index_eval

Conversation

@lszskye
Copy link
Copy Markdown
Collaborator

@lszskye lszskye commented May 6, 2026

Purpose

Linked issue: #38

Introduce:

  • OffsetGlobalIndexReader: wraps any GlobalIndexReader and rewrites
    the row ids in its results by adding a fixed offset. This is how local
    row ids become global row ids.
  • UnionGlobalIndexReader: wraps a list of GlobalIndexReaders and
    unions their results via GlobalIndexResult::Or. It can run sub-readers
    in parallel through an Executor, or sequentially when no executor is
    given.

Add inte test for btree global index

Tests

GlobalIndexTest

Comment thread include/paimon/global_index/global_index_result.h
auto offset_reader = std::make_shared<OffsetGlobalIndexReader>(fake_reader, 10);
// FakeGlobalIndexReader returns error for VectorSearch
ASSERT_NOK_WITH_MSG(offset_reader->VisitVectorSearch(nullptr),
"FakeGlobalIndexReader does not support vector search");
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test vector search in OffsetGlobalIndexReaderTest, as it has different logic.
Also test for nullptr inner result.

Comment thread src/paimon/common/global_index/union_global_index_reader.cpp
Comment thread src/paimon/common/global_index/union_global_index_reader_test.cpp
ASSERT_OK_AND_ASSIGN(auto result, union_reader.VisitVectorSearch(nullptr));
ASSERT_FALSE(result);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add normal case for vector search.

Comment thread src/paimon/core/global_index/global_index_evaluator.h
}
// TODO(lisizhuo.lsz): add executor in UnionGlobalIndexReader
readers.push_back(std::make_shared<UnionGlobalIndexReader>(std::move(union_readers),
/*executor=*/nullptr));
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add executor for UnionGlobalIndexReader. May add param in GlobalIndexScan::Create().

Comment thread src/paimon/core/operation/data_evolution_file_store_scan_test.cpp
Comment thread include/paimon/global_index/global_index_scan.h
Comment thread include/paimon/global_index/global_index_reader.h
Comment thread test/inte/global_index_test.cpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants