Co-authored-by: rayx <rui.xing@alibaba-inc.com>
Co-authored-by: Jalin Wang <wangjianning.wjn@alibaba-inc.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* feat(query): add VectorViewClause zero-copy path and unify validate
- Add VectorViewClause (string_view-based) as zero-copy counterpart to
VectorClause; variant now holds VectorClause | VectorViewClause | FtsClause
- Add QueryTarget::get_vector_view() unified accessor via std::visit,
returns optional<VectorViewClause> regardless of which variant is held
- Split validate_and_sanitize into QueryTarget::validate (read-only) +
sanitize_sparse_vector (mutate); validate handles both VectorClause and
VectorViewClause via get_vector_view()
- Collection::Query passes original request directly to sqlengine when no
sparse sanitization is needed; only copies when sort is required
- Change build_query_info/BuildSQLInfoFromSearchQuery to take const
SearchQuery& so VectorMatrixNode string_views point to caller's data
* refactor: make Reranker stateless with std::variant value semantics (#461)
Replace class hierarchy (Reranker/ScoreBasedReranker/RrfReranker/
WeightedReranker/CallbackReranker) with std::variant<RrfParams,
WeightedParams, CallbackParams> value type and a stateless free function
reranker::rerank().
Key changes:
- reranker.h: define RerankParams variant + reranker::rerank() API
- query.h: MultiQuery::reranker (shared_ptr) -> MultiQuery::rerank (value)
- schema.h: add CollectionSchema::get_field_ptr() returning FieldSchema::Ptr
- collection.cc: push field lookup to caller, pass vector<FieldSchema::Ptr>
- c_api: remove opaque zvec_reranker_t, add zvec_multi_query_set_rerank_*
- python binding: expose _RrfParams/_WeightedParams/_CallbackParams + setters
- python layer: WeightedReRanker(list[float]), remove Python rerank logic
- all tests updated to new interface
Benefits:
- Thread-safe by design: no mutable state, safe to share across threads
- Collection-decoupled: no bind_schema(), field info passed as parameter
- Simpler lifecycle: value semantics, no shared_ptr management
Closes#461
* chore: remove nightly_build.yml unrelated to reranker refactor
* chore: remove uv.lock unrelated to reranker refactor
* fix: raise ValueError when multi-query has no reranker
After the reranker stateless refactor the C++ MultiQuery rerank
strategy uses a std::variant with a default value, so the implicit
'reranker required' validation no longer triggered. Restore the
check in QueryExecutor._execute_multi_query so that a hybrid
(multi-query) request without a reranker raises ValueError.
* fix(reranker): use index_type FTS check for non-vector normalization
Replace dynamic_cast nullptr check with explicit IndexType::FTS check
and map FTS/BM25 positive scores to (0.0, 1.0) via 2*atan(score)/pi.
* refactor(reranker): move Params types into reranker namespace and qualify usages
Move RrfParams, WeightedParams, CallbackParams and RerankParams into the
zvec::reranker namespace, and add explicit reranker:: qualification at all
usage sites outside the reranker module (query.h, python/c bindings, tests).
* refactor(query): drop unused PendingQuery wrapper, use std::vector<SearchQuery> directly
* refactor(reranker): make _to_cpp_params non-abstract with default NotImplementedError
Remove @abstractmethod from RerankFunction._to_cpp_params and provide a
default implementation raising NotImplementedError. Drop the redundant
_to_cpp_params overrides from Qwen and Sentence rerankers since they use
the Python rerank path and don't need the C++ conversion.
* refactor: change rerank interface from map-based to vector-based (#452)
- Define QueryResult = list[Doc] type alias in doc.py
- Change C++ Reranker::rerank() signature from map<string, DocPtrList> to vector<DocPtrList>
- Extend bind_schema() to accept field_names for index-based field lookup
- Update ScoreBasedReranker/WeightedReranker/CallbackReranker implementations
- Adapt collection.cc MultiQuery path to use vector<DocPtrList>
- Update Python binding to expose rerank() and use vector<double> weights
- Refactor Python RerankFunction interface to list[QueryResult] -> QueryResult
- Remove Python-layer rerank logic from RrfReRanker/WeightedReRanker (delegate to C++)
- Update query_executor to return list[list[Doc]] instead of dict
- Update all related unit tests (C++ and Python)
* refactor: replace list[Doc] with QueryResult type alias in executor and rerank functions
* refactor: replace list[list[Doc]] with list[QueryResult] in query_executor
* fix: remove unused Doc import in rerank_function.py (ruff F401)
* refactor(query_executor): merge duplicate rerank return paths
* refactor: RrfReRanker/WeightedReRanker.rerank() directly call C++ reranker
* refactor: simplify QueryExecutor into unified class, remove Factory/subclasses/validation/concurrency
* refactor: rename _VectorQuery to _SearchQuery, from_vector_query to from_search_query
* refactor(query_executor): split execute into single/multi paths, rename core_vector to search_query, drop unused core_vectors
* style: apply ruff formatter to test_reranker.py and query_executor.py
* refactor: make rescore() private in ScoreBasedReranker hierarchy
* style: apply clang-format to reranker.h
* style: apply clang-format to all modified C++ files
* refactor: rename private methods in QueryExecutor for clearer semantics
* refactor: rename mvq to multi_query for clarity
* fix: make BasicRRF test order-independent for equal scores
* fix: update collection_test to use vector-based reranker interface
* fix: update reranker tests to expect TypeError instead of NotImplementedError
* refactor: remove PendingQuery wrapper, use SearchQuery directly in MultiQuery path
* refactor: simplify MultiQuery path - remove seen_fields, merge field_names into main loop
* fix: address review comments - defensive checks and remove fields param from C API
- ScoreBasedReranker::rerank(): early return empty list when topn <= 0
- WeightedReranker::rescore(): null-check schema_ before use
- CallbackReranker::rerank(): check callback_ is not empty before invoke
- C API zvec_reranker_create_weighted(): remove unused fields parameter
* fix: remove duplicate field name test (check was intentionally removed)
* fix: address egolearner review comments
- Rename QueryResult to DocList for clarity (见名知义)
- Change docstring to #: comment for type alias
- Fix output_fields check: use 'is not None' instead of truthy check
(None means unset, [] means explicit empty list - different semantics)
- Raise ValueError when search-by-id finds no document
* refactor: remove redundant output_fields assignment in _build_search_query
* refactor: address egolearner review comments (C++ refactoring)
- c_api.cc: simplify weighted reranker creation with inline vector ctor
- python_reranker.cc: refactor unwrap_rerank_result - take by value,
early error return, move semantics
- Rename C API functions for consistent naming:
zvec_reranker_create_rrf -> zvec_create_rrf_reranker
zvec_reranker_create_weighted -> zvec_create_weighted_reranker
zvec_reranker_destroy -> zvec_destroy_reranker
zvec_reranker_get_rank_constant -> zvec_get_reranker_rank_constant
- reranker.h/cc: bind_schema returns Result<void>, caches
vector<const FieldSchema*> to avoid repeated schema lookups in rescore
- python_param.cc: rename py::arg vector_query to search_query
* revert: rollback bind_schema refactoring due to thread-safety concern
The field_schemas_ caching approach introduces a data race when the same
WeightedReranker instance is shared across concurrent queries: bind_schema()
writes field_schemas_ while rerank() reads it concurrently.
Revert to storing schema_ + field_names_ and looking up fields in rescore().
Add @note thread-safety warning to WeightedReranker class documentation.
* fix: unify error message format in collection.cc
Change 'Vector field not found: X' to 'Invalid query: field X not found'
for consistent error formatting as suggested by zhourrr.
* fix: sort __all__ and remove duplicates in __init__.pyi
Fix RUF022 lint error: sort __all__ alphabetically and remove duplicate
entries (DenseEmbeddingFunction, ReRanker).
* style: format query_executor.py with ruff formatter
* fix: resolve Python test failures after FTS rebase integration
- test_query_executor.py: update method names to match refactored API
(_do_build -> _build_queries, _do_merge_rerank_results -> _merge_and_rerank)
- test_reranker.py: fix expected exception type (TypeError from pybind11)
- test_collection_fts.py: update error message match patterns
- test_collection_fts_vector_hybrid.py: remove obsolete 'metrics' param,
update weights from dict to positional list, adapt validation tests
for multi-vector queries (now supported with reranker)
- test_collection_dql.py: remove 'metrics' param, update weights format
- collection.cc: distinguish FTS vs vector fields in MultiQuery path
using get_fts_clause() to route field lookup correctly
- reranker.cc: use get_field() instead of get_vector_field() in rescore
to support FTS+vector hybrid weighted reranking
* refactor: pass topn as rerank() parameter, move rerank_field to model rerankers
* fix: address review comments - rename test functions and restore duplicate field check
* refactor: simplify MultiQuery field lookup, let validate_and_sanitize handle type check
* feat: add FTS support for Collection::CreateIndex/DropIndex
Enable dynamic creation and removal of FTS indexes on existing STRING
columns through the standard CreateIndex/DropIndex API, matching the
lifecycle model already used by vector and scalar (invert) indexes.
Key changes:
- New FtsIndexer class (fts_indexer.h/cc) encapsulating per-segment FTS
RocksDB management: multi-field lifecycle, snapshot, insert, seal
- New BlockType::FTS_INDEX with block_id-based directory naming
(fts.<block_id>.rocksdb) for crash-safe snapshot-and-swap
- Segment::create_fts_index builds FTS index on a snapshot copy by
scanning forward store, then outputs new SegmentMeta + FtsIndexer
for atomic reload (same pattern as create_scalar_index)
- Segment::drop_fts_index snapshots, removes field CFs, outputs updated
meta (or nullptr when last FTS field is removed)
- Collection layer wires FTS into the existing task dispatch, version
update, and reload loops alongside vector/invert paths
- CreateIndex/DropIndex reject unsupported index types explicitly
instead of falling through to the wrong branch
* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex
fix: address review comments for FTS CreateIndex/DropIndex
- Reject CreateIndex when column already has a different index type
(e.g. FTS on an INVERT-indexed column) at both Collection and
Segment layers
- Allow same-type different-params CreateIndex to rebuild the index
(remove old + create new + replay data), aligned with INVERT behavior
- Return OK when CreateIndex is called with identical params
- Rename operator[] to get() in both FtsIndexer and InvertedIndexer
- Add test cases: create→drop→create→drop cycle, and params-change
rebuild with case-sensitivity verification
* android skip Feature_CreateOrDropFtsIndex
* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex
Add a fast path that copies the first segment's index file as the merge base and only merges the tail segments into it. Limited to streaming indexes (HNSW, HNSW_RABITQ, FLAT) with matching index_type + quantize_type and no filter; IVF/VAMANA always rebuild (their Merge is dump-then-reopen and would drop the base docs).
Also, fix the incorrect concurrency of the compaction task.
Windows.h defines ERROR as a macro which collides with LogLevel::ERROR.
Rename all LogLevel enum values to kDebug/kInfo/kWarn/kError/kFatal style,
and remove the now-unnecessary #undef ERROR workaround in jieba_tokenizer.
* feat: migrate multi-vector query and reranker logic to C++
- Add Reranker base class with RrfReRanker and WeightedReRanker implementations
- Add Collection::MultiQuery interface for multi-vector queries with reranking
- Add MultiVectorQuery struct in doc.h with forward declaration for Reranker
- Add C API bindings for reranker and MultiQuery (zvec_reranker_*, zvec_multi_vector_query_*, zvec_collection_multi_query)
- Add Python binding for reranker classes with py::function bridge for callback
- Validate duplicate field names in multi-vector queries (C++ and Python consistent)
- Remove TODO comment about concurrent execution (SQLEngine is not thread-safe)
- Update collection.h MultiQuery doc comment from concurrently to sequentially
- Add C++ collection tests (6 MultiQuery test cases)
- Add C API tests (reranker functions + multi_vector_query end-to-end)
- Implement Python test cases (11 previously skipped tests now active)
- Simplify Python query_executor validation for unified duplicate field check
* style: format Python files with ruff
* fix: adapt to main branch API changes (VectorQuery->Query rename, validate_and_sanitize)
* fix: multi_vector tests now use multiple same-type vector fields (dense2, sparse2)
* fix: suppress RET501 for intentional default return None in RerankFunction._get_object
* style: ruff format test_collection.py
* refact multi vector query
* format code
* fix(multi-vector): expose SubVectorQuery in Python binding, fix tests
- Register _SubVectorQuery in pybind11 with from_vector_query() factory
- Convert _VectorQuery to _SubVectorQuery in MultiVectorQueryExecutor
- Relax RRF/Weighted score assertion tolerance from 1e-10 to 1e-6
- Fix WeightedReRanker test metric to IP (matching HnswIndexParam default)
* style: ruff format query_executor.py
* fix: define _USE_MATH_DEFINES for M_PI on Windows (MSVC)
* refactor: include reranker.h directly in query.h instead of forward declaration
* refactor(reranker): move topn from member variable to rerank() parameter
* refact code
* style(python): fix ruff UP035/UP037 in multi_vector_reranker
- import Callable from collections.abc instead of typing (UP035)
- remove redundant quotes around MetricType annotations (UP037)
* chore: trigger PR sync
* refact code
* fix(examples): restore CMakeLists.txt formatting broken by clang-format
* refactor(reranker): remove redundant metrics_ map by querying schema directly, and use insert return value to avoid duplicate set lookup
* refactor(reranker): defer schema binding to query time and remove C API callback reranker
Extract the shared train+attach logic into
SegmentHelper::PrepareQuantizeField, used by both
SegmentImpl::create_vector_index and SegmentHelper::ReduceVectorIndex.
When add_column is called on a multi-segment collection with a nullable
field and no expression, segment.cc previously sliced an Arrow ChunkedArray
with an offset that exceeded the array length, triggering SIGABRT in
Arrow's chunked_array.cc:170 assertion.
Fix the slicing logic in segment.cc to materialize null values per segment.
Add comprehensive tests in collection_test.cc and segment_test.cc covering
multi-segment add_column scenarios (nullable/non-nullable, with/without
expression, with/without unflushed data, drop+re-add).
Add modernize-use-override to .clang-tidy and apply fixes across
src/ and tests/: replace redundant virtual with override, annotate
missing override on derived methods, and drop redundant virtual on
already-overridden methods.
When a nullable scalar field has no inverted index, the forward filter path
fails to handle null values from Arrow's filter evaluation:
1. get_forward_bit(): BooleanArray::operator[] returns nullopt for null entries,
which is_filtered() treats as "no filter" (not filtered), letting null docs
through. Fix: use value_or(false) to treat null as "not matched".
2. is_matched_by_forward_filter(): reads BooleanScalar.value without checking
is_valid, which is UB for null scalars. Fix: check is_valid first.
Closes#409
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Set unique WORKING_DIRECTORY per test binary via cc_test()/cuda_test() to prevent filesystem path conflicts when running tests in parallel. Each test runs in ${CMAKE_BINARY_DIR}/test_tmp/${test_name}/.
- Enable parallel ctest execution in the unittest target with ProcessorCount-based --parallel flag (defaults to NPROC - 1).
- Set TEST_BINARY_DIR environment variable for crash recovery tests so they can locate helper binaries from isolated working directories.
- Update LocateDataGenerator() and LocateOptimizeGenerator() to search TEST_BINARY_DIR and TEST_BINARY_DIR/bin for helper executables.
Rewrite file/path handling to use std::filesystem and UTF-8-safe helpers.
Switch Windows file open/create paths to wide-char APIs, replace manual
separator concatenation with PathJoin, and enable RocksDB UTF-8 filenames.
Also add UTF-8 path coverage for file IO, version manager recovery, and
collection open/flush/reopen flows.
Yin Yongqi
egolearner
luoxiaojian
Zhuanglin Zheng
feihongxu0824
mrcs64
Jalin Wang
lichen2015
Cuiys
Qinren Zhou
rayx
ZeFeng Yin