refactor: change rerank interface from map-based to vector-based (#458)

* refactor: change rerank interface from map-based to vector-based (#452)

- Define QueryResult = list[Doc] type alias in doc.py
- Change C++ Reranker::rerank() signature from map<string, DocPtrList> to vector<DocPtrList>
- Extend bind_schema() to accept field_names for index-based field lookup
- Update ScoreBasedReranker/WeightedReranker/CallbackReranker implementations
- Adapt collection.cc MultiQuery path to use vector<DocPtrList>
- Update Python binding to expose rerank() and use vector<double> weights
- Refactor Python RerankFunction interface to list[QueryResult] -> QueryResult
- Remove Python-layer rerank logic from RrfReRanker/WeightedReRanker (delegate to C++)
- Update query_executor to return list[list[Doc]] instead of dict
- Update all related unit tests (C++ and Python)

* refactor: replace list[Doc] with QueryResult type alias in executor and rerank functions

* refactor: replace list[list[Doc]] with list[QueryResult] in query_executor

* fix: remove unused Doc import in rerank_function.py (ruff F401)

* refactor(query_executor): merge duplicate rerank return paths

* refactor: RrfReRanker/WeightedReRanker.rerank() directly call C++ reranker

* refactor: simplify QueryExecutor into unified class, remove Factory/subclasses/validation/concurrency

* refactor: rename _VectorQuery to _SearchQuery, from_vector_query to from_search_query

* refactor(query_executor): split execute into single/multi paths, rename core_vector to search_query, drop unused core_vectors

* style: apply ruff formatter to test_reranker.py and query_executor.py

* refactor: make rescore() private in ScoreBasedReranker hierarchy

* style: apply clang-format to reranker.h

* style: apply clang-format to all modified C++ files

* refactor: rename private methods in QueryExecutor for clearer semantics

* refactor: rename mvq to multi_query for clarity

* fix: make BasicRRF test order-independent for equal scores

* fix: update collection_test to use vector-based reranker interface

* fix: update reranker tests to expect TypeError instead of NotImplementedError

* refactor: remove PendingQuery wrapper, use SearchQuery directly in MultiQuery path

* refactor: simplify MultiQuery path - remove seen_fields, merge field_names into main loop

* fix: address review comments - defensive checks and remove fields param from C API

- ScoreBasedReranker::rerank(): early return empty list when topn <= 0
- WeightedReranker::rescore(): null-check schema_ before use
- CallbackReranker::rerank(): check callback_ is not empty before invoke
- C API zvec_reranker_create_weighted(): remove unused fields parameter

* fix: remove duplicate field name test (check was intentionally removed)

* fix: address egolearner review comments

- Rename QueryResult to DocList for clarity (见名知义)
- Change docstring to #: comment for type alias
- Fix output_fields check: use 'is not None' instead of truthy check
  (None means unset, [] means explicit empty list - different semantics)
- Raise ValueError when search-by-id finds no document

* refactor: remove redundant output_fields assignment in _build_search_query

* refactor: address egolearner review comments (C++ refactoring)

- c_api.cc: simplify weighted reranker creation with inline vector ctor
- python_reranker.cc: refactor unwrap_rerank_result - take by value,
  early error return, move semantics
- Rename C API functions for consistent naming:
  zvec_reranker_create_rrf -> zvec_create_rrf_reranker
  zvec_reranker_create_weighted -> zvec_create_weighted_reranker
  zvec_reranker_destroy -> zvec_destroy_reranker
  zvec_reranker_get_rank_constant -> zvec_get_reranker_rank_constant
- reranker.h/cc: bind_schema returns Result<void>, caches
  vector<const FieldSchema*> to avoid repeated schema lookups in rescore
- python_param.cc: rename py::arg vector_query to search_query

* revert: rollback bind_schema refactoring due to thread-safety concern

The field_schemas_ caching approach introduces a data race when the same
WeightedReranker instance is shared across concurrent queries: bind_schema()
writes field_schemas_ while rerank() reads it concurrently.

Revert to storing schema_ + field_names_ and looking up fields in rescore().
Add @note thread-safety warning to WeightedReranker class documentation.

* fix: unify error message format in collection.cc

Change 'Vector field not found: X' to 'Invalid query: field X not found'
for consistent error formatting as suggested by zhourrr.

* fix: sort __all__ and remove duplicates in __init__.pyi

Fix RUF022 lint error: sort __all__ alphabetically and remove duplicate
entries (DenseEmbeddingFunction, ReRanker).

* style: format query_executor.py with ruff formatter

* fix: resolve Python test failures after FTS rebase integration

- test_query_executor.py: update method names to match refactored API
  (_do_build -> _build_queries, _do_merge_rerank_results -> _merge_and_rerank)
- test_reranker.py: fix expected exception type (TypeError from pybind11)
- test_collection_fts.py: update error message match patterns
- test_collection_fts_vector_hybrid.py: remove obsolete 'metrics' param,
  update weights from dict to positional list, adapt validation tests
  for multi-vector queries (now supported with reranker)
- test_collection_dql.py: remove 'metrics' param, update weights format
- collection.cc: distinguish FTS vs vector fields in MultiQuery path
  using get_fts_clause() to route field lookup correctly
- reranker.cc: use get_field() instead of get_vector_field() in rescore
  to support FTS+vector hybrid weighted reranking

* refactor: pass topn as rerank() parameter, move rerank_field to model rerankers

* fix: address review comments - rename test functions and restore duplicate field check

* refactor: simplify MultiQuery field lookup, let validate_and_sanitize handle type check
This commit is contained in:
Cuiys
2026-06-04 16:03:28 +08:00
committed by GitHub
parent f562bdd636
commit c46efe1241
29 changed files with 704 additions and 1098 deletions
+10 -52
View File
@@ -976,18 +976,6 @@ class TestCollectionQuery:
result = collection_with_multiple_docs.query(filter="id in (1)", topk=100)
assert len(result) == 1
def test_collection_query_with_vector_and_id(
self, collection_with_single_doc: Collection, single_doc: Doc
):
with pytest.raises(ValueError):
collection_with_single_doc.query(
Query(
field_name="dense",
id=single_doc.id,
vector=single_doc.vector("dense"),
)
)
def test_collection_query_with_filter_not_in(
self, collection_with_multiple_docs: Collection, multiple_docs
):
@@ -1013,30 +1001,6 @@ class TestCollectionQuery:
)
assert len(result) == 10
def test_collection_query_multi_vector_with_same_field(
self, collection_with_multiple_docs: Collection, multiple_docs
):
# Multi-vector query on same field without reranker should raise ValueError
with pytest.raises(ValueError, match="Reranker is required"):
collection_with_multiple_docs.query(
[
Query(field_name="dense", vector=multiple_docs[0].vector("dense")),
Query(field_name="dense", vector=multiple_docs[1].vector("dense")),
]
)
# Same field name with reranker should also raise ValueError
reranker = RrfReRanker(topn=10, rank_constant=60)
with pytest.raises(ValueError, match="appears more than once"):
collection_with_multiple_docs.query(
[
Query(field_name="dense", vector=multiple_docs[0].vector("dense")),
Query(field_name="dense", vector=multiple_docs[1].vector("dense")),
],
topk=10,
reranker=reranker,
)
def test_collection_query_by_dense_vector(
self, collection_with_multiple_docs: Collection, multiple_docs
):
@@ -1087,7 +1051,7 @@ class TestCollectionQuery:
self, collection_with_multiple_docs: Collection, multiple_docs
):
"""Test multi-vector query with RRF reranker on multiple dense vectors."""
reranker = RrfReRanker(topn=10, rank_constant=60)
reranker = RrfReRanker(rank_constant=60)
result = collection_with_multiple_docs.query(
[
Query(field_name="dense", vector=multiple_docs[0].vector("dense")),
@@ -1106,7 +1070,7 @@ class TestCollectionQuery:
self, collection_with_multiple_docs: Collection, multiple_docs
):
"""Test multi-vector query with RRF reranker on multiple sparse vectors."""
reranker = RrfReRanker(topn=10, rank_constant=60)
reranker = RrfReRanker(rank_constant=60)
result = collection_with_multiple_docs.query(
[
Query(field_name="sparse", vector=multiple_docs[0].vector("sparse")),
@@ -1125,7 +1089,7 @@ class TestCollectionQuery:
self, collection_with_multiple_docs: Collection, multiple_docs
):
"""Test multi-vector query with RRF reranker combining dense + sparse."""
reranker = RrfReRanker(topn=10, rank_constant=60)
reranker = RrfReRanker(rank_constant=60)
result = collection_with_multiple_docs.query(
[
Query(field_name="dense", vector=multiple_docs[0].vector("dense")),
@@ -1141,9 +1105,7 @@ class TestCollectionQuery:
self, collection_with_multiple_docs: Collection, multiple_docs
):
"""Test multi-vector query with Weighted reranker on multiple dense vectors."""
metrics = {"dense": MetricType.IP, "dense2": MetricType.IP}
weights = {"dense": 0.6, "dense2": 0.4}
reranker = WeightedReRanker(topn=10, metrics=metrics, weights=weights)
reranker = WeightedReRanker(weights=[0.6, 0.4])
result = collection_with_multiple_docs.query(
[
Query(field_name="dense", vector=multiple_docs[0].vector("dense")),
@@ -1159,9 +1121,7 @@ class TestCollectionQuery:
self, collection_with_multiple_docs: Collection, multiple_docs
):
"""Test multi-vector query with Weighted reranker on multiple sparse vectors."""
metrics = {"sparse": MetricType.IP, "sparse2": MetricType.IP}
weights = {"sparse": 0.6, "sparse2": 0.4}
reranker = WeightedReRanker(topn=10, metrics=metrics, weights=weights)
reranker = WeightedReRanker(weights=[0.6, 0.4])
result = collection_with_multiple_docs.query(
[
Query(field_name="sparse", vector=multiple_docs[0].vector("sparse")),
@@ -1180,9 +1140,7 @@ class TestCollectionQuery:
self, collection_with_multiple_docs: Collection, multiple_docs
):
"""Test multi-vector query with Weighted reranker combining dense + sparse."""
metrics = {"dense": MetricType.IP, "sparse": MetricType.IP}
weights = {"dense": 0.7, "sparse": 0.3}
reranker = WeightedReRanker(topn=10, metrics=metrics, weights=weights)
reranker = WeightedReRanker(weights=[0.7, 0.3])
result = collection_with_multiple_docs.query(
[
Query(field_name="dense", vector=multiple_docs[0].vector("dense")),
@@ -1203,7 +1161,7 @@ class TestCollectionQuery:
def my_rerank_callback(query_results, topn):
callback_invoked.append(True)
all_docs = []
for docs in query_results.values():
for docs in query_results:
all_docs.extend(docs)
seen = set()
unique_docs = []
@@ -1214,7 +1172,7 @@ class TestCollectionQuery:
unique_docs.sort(key=lambda d: d.score(), reverse=True)
return unique_docs[:topn]
reranker = CallbackReRanker(callback=my_rerank_callback, topn=10)
reranker = CallbackReRanker(callback=my_rerank_callback)
result = collection_with_multiple_docs.query(
[
Query(field_name="dense", vector=multiple_docs[0].vector("dense")),
@@ -1234,7 +1192,7 @@ class TestCollectionQuery:
def my_rerank_callback(query_results, topn):
all_docs = []
for docs in query_results.values():
for docs in query_results:
all_docs.extend(docs)
seen = set()
unique_docs = []
@@ -1245,7 +1203,7 @@ class TestCollectionQuery:
unique_docs.sort(key=lambda d: d.score(), reverse=True)
return unique_docs[:topn]
reranker = CallbackReRanker(callback=my_rerank_callback, topn=5)
reranker = CallbackReRanker(callback=my_rerank_callback)
result = collection_with_multiple_docs.query(
[
Query(field_name="dense", vector=multiple_docs[0].vector("dense")),