* refactor: change rerank interface from map-based to vector-based (#452) - Define QueryResult = list[Doc] type alias in doc.py - Change C++ Reranker::rerank() signature from map<string, DocPtrList> to vector<DocPtrList> - Extend bind_schema() to accept field_names for index-based field lookup - Update ScoreBasedReranker/WeightedReranker/CallbackReranker implementations - Adapt collection.cc MultiQuery path to use vector<DocPtrList> - Update Python binding to expose rerank() and use vector<double> weights - Refactor Python RerankFunction interface to list[QueryResult] -> QueryResult - Remove Python-layer rerank logic from RrfReRanker/WeightedReRanker (delegate to C++) - Update query_executor to return list[list[Doc]] instead of dict - Update all related unit tests (C++ and Python) * refactor: replace list[Doc] with QueryResult type alias in executor and rerank functions * refactor: replace list[list[Doc]] with list[QueryResult] in query_executor * fix: remove unused Doc import in rerank_function.py (ruff F401) * refactor(query_executor): merge duplicate rerank return paths * refactor: RrfReRanker/WeightedReRanker.rerank() directly call C++ reranker * refactor: simplify QueryExecutor into unified class, remove Factory/subclasses/validation/concurrency * refactor: rename _VectorQuery to _SearchQuery, from_vector_query to from_search_query * refactor(query_executor): split execute into single/multi paths, rename core_vector to search_query, drop unused core_vectors * style: apply ruff formatter to test_reranker.py and query_executor.py * refactor: make rescore() private in ScoreBasedReranker hierarchy * style: apply clang-format to reranker.h * style: apply clang-format to all modified C++ files * refactor: rename private methods in QueryExecutor for clearer semantics * refactor: rename mvq to multi_query for clarity * fix: make BasicRRF test order-independent for equal scores * fix: update collection_test to use vector-based reranker interface * fix: update reranker tests to expect TypeError instead of NotImplementedError * refactor: remove PendingQuery wrapper, use SearchQuery directly in MultiQuery path * refactor: simplify MultiQuery path - remove seen_fields, merge field_names into main loop * fix: address review comments - defensive checks and remove fields param from C API - ScoreBasedReranker::rerank(): early return empty list when topn <= 0 - WeightedReranker::rescore(): null-check schema_ before use - CallbackReranker::rerank(): check callback_ is not empty before invoke - C API zvec_reranker_create_weighted(): remove unused fields parameter * fix: remove duplicate field name test (check was intentionally removed) * fix: address egolearner review comments - Rename QueryResult to DocList for clarity (见名知义) - Change docstring to #: comment for type alias - Fix output_fields check: use 'is not None' instead of truthy check (None means unset, [] means explicit empty list - different semantics) - Raise ValueError when search-by-id finds no document * refactor: remove redundant output_fields assignment in _build_search_query * refactor: address egolearner review comments (C++ refactoring) - c_api.cc: simplify weighted reranker creation with inline vector ctor - python_reranker.cc: refactor unwrap_rerank_result - take by value, early error return, move semantics - Rename C API functions for consistent naming: zvec_reranker_create_rrf -> zvec_create_rrf_reranker zvec_reranker_create_weighted -> zvec_create_weighted_reranker zvec_reranker_destroy -> zvec_destroy_reranker zvec_reranker_get_rank_constant -> zvec_get_reranker_rank_constant - reranker.h/cc: bind_schema returns Result<void>, caches vector<const FieldSchema*> to avoid repeated schema lookups in rescore - python_param.cc: rename py::arg vector_query to search_query * revert: rollback bind_schema refactoring due to thread-safety concern The field_schemas_ caching approach introduces a data race when the same WeightedReranker instance is shared across concurrent queries: bind_schema() writes field_schemas_ while rerank() reads it concurrently. Revert to storing schema_ + field_names_ and looking up fields in rescore(). Add @note thread-safety warning to WeightedReranker class documentation. * fix: unify error message format in collection.cc Change 'Vector field not found: X' to 'Invalid query: field X not found' for consistent error formatting as suggested by zhourrr. * fix: sort __all__ and remove duplicates in __init__.pyi Fix RUF022 lint error: sort __all__ alphabetically and remove duplicate entries (DenseEmbeddingFunction, ReRanker). * style: format query_executor.py with ruff formatter * fix: resolve Python test failures after FTS rebase integration - test_query_executor.py: update method names to match refactored API (_do_build -> _build_queries, _do_merge_rerank_results -> _merge_and_rerank) - test_reranker.py: fix expected exception type (TypeError from pybind11) - test_collection_fts.py: update error message match patterns - test_collection_fts_vector_hybrid.py: remove obsolete 'metrics' param, update weights from dict to positional list, adapt validation tests for multi-vector queries (now supported with reranker) - test_collection_dql.py: remove 'metrics' param, update weights format - collection.cc: distinguish FTS vs vector fields in MultiQuery path using get_fts_clause() to route field lookup correctly - reranker.cc: use get_field() instead of get_vector_field() in rescore to support FTS+vector hybrid weighted reranking * refactor: pass topn as rerank() parameter, move rerank_field to model rerankers * fix: address review comments - rename test functions and restore duplicate field check * refactor: simplify MultiQuery field lookup, let validate_and_sanitize handle type check
159 lines
5.2 KiB
Python
159 lines
5.2 KiB
Python
# Copyright 2025-present the zvec project
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
"""Tests for FTS (Full-Text Search) query support in the Python SDK."""
|
|
|
|
import pickle
|
|
|
|
import pytest
|
|
|
|
from zvec.model.param.query import Fts, Query
|
|
|
|
|
|
class TestFtsQueryValidation:
|
|
"""Test FTS parameter validation in Query dataclass."""
|
|
|
|
def test_fts_query_string_only(self):
|
|
"""Query with only query_string in Fts should be valid."""
|
|
q = Query(
|
|
field_name="content", fts=Fts(query_string='+hello -world "exact phrase"')
|
|
)
|
|
q._validate()
|
|
assert q.fts.query_string == '+hello -world "exact phrase"'
|
|
assert q.fts.match_string is None
|
|
assert q.has_fts() is True
|
|
|
|
def test_fts_match_string_only(self):
|
|
"""Query with only match_string in Fts should be valid."""
|
|
q = Query(field_name="content", fts=Fts(match_string="machine learning"))
|
|
q._validate()
|
|
assert q.fts.match_string == "machine learning"
|
|
assert q.fts.query_string is None
|
|
assert q.has_fts() is True
|
|
|
|
def test_fts_query_string_and_match_string_mutually_exclusive(self):
|
|
"""Cannot provide both query_string and match_string in Fts."""
|
|
q = Query(
|
|
field_name="content",
|
|
fts=Fts(query_string="+hello", match_string="hello world"),
|
|
)
|
|
with pytest.raises(ValueError, match="mutually exclusive"):
|
|
q._validate()
|
|
|
|
def test_no_fts(self):
|
|
"""Query without FTS fields should have has_fts() == False."""
|
|
q = Query(field_name="embedding", vector=[0.1, 0.2, 0.3])
|
|
assert q.has_fts() is False
|
|
|
|
def test_vector_and_fts_mutually_exclusive(self):
|
|
"""Cannot combine vector search with FTS in a single Query."""
|
|
q = Query(
|
|
field_name="embedding",
|
|
vector=[0.1, 0.2, 0.3],
|
|
fts=Fts(match_string="deep learning"),
|
|
)
|
|
with pytest.raises(ValueError, match="Cannot combine fts with vector search"):
|
|
q._validate()
|
|
|
|
def test_fts_without_vector_or_id(self):
|
|
"""Query with only FTS (no vector, no id) should be valid."""
|
|
q = Query(field_name="content", fts=Fts(query_string="hello"))
|
|
q._validate()
|
|
assert q.has_vector() is False
|
|
assert q.has_id() is False
|
|
assert q.has_fts() is True
|
|
|
|
|
|
class TestFtsQueryBinding:
|
|
"""Test FTS binding layer (_Fts)."""
|
|
|
|
def test_import_fts_query(self):
|
|
"""_Fts should be importable from _zvec.param."""
|
|
from _zvec.param import _Fts
|
|
|
|
fts = _Fts()
|
|
assert fts.query_string == ""
|
|
assert fts.match_string == ""
|
|
|
|
def test_fts_query_set_fields(self):
|
|
"""Setting fields on _Fts should work."""
|
|
from _zvec.param import _Fts
|
|
|
|
fts = _Fts()
|
|
fts.query_string = "+hello -world"
|
|
assert fts.query_string == "+hello -world"
|
|
|
|
fts2 = _Fts()
|
|
fts2.match_string = "machine learning"
|
|
assert fts2.match_string == "machine learning"
|
|
|
|
def test_fts_query_pickle(self):
|
|
"""_Fts should support pickling."""
|
|
from _zvec.param import _Fts
|
|
|
|
fts = _Fts()
|
|
fts.query_string = "+vector search"
|
|
fts.match_string = ""
|
|
|
|
data = pickle.dumps(fts)
|
|
restored = pickle.loads(data)
|
|
assert restored.query_string == "+vector search"
|
|
assert restored.match_string == ""
|
|
|
|
def test_search_query_fts_field(self):
|
|
"""_SearchQuery should have fts field."""
|
|
from _zvec.param import _Fts, _SearchQuery
|
|
|
|
vq = _SearchQuery()
|
|
# fts should be None by default (optional)
|
|
assert vq.fts is None
|
|
|
|
# set fts
|
|
fts = _Fts()
|
|
fts.query_string = "hello"
|
|
vq.fts = fts
|
|
assert vq.fts is not None
|
|
assert vq.fts.query_string == "hello"
|
|
|
|
def test_search_query_pickle_with_fts(self):
|
|
"""_SearchQuery with fts should survive pickling."""
|
|
from _zvec.param import _Fts, _SearchQuery
|
|
|
|
vq = _SearchQuery()
|
|
vq.topk = 10
|
|
vq.field_name = "embedding"
|
|
fts = _Fts()
|
|
fts.match_string = "test query"
|
|
vq.fts = fts
|
|
|
|
data = pickle.dumps(vq)
|
|
restored = pickle.loads(data)
|
|
assert restored.topk == 10
|
|
assert restored.field_name == "embedding"
|
|
assert restored.fts is not None
|
|
assert restored.fts.match_string == "test query"
|
|
|
|
def test_search_query_pickle_without_fts(self):
|
|
"""_SearchQuery without fts should survive pickling."""
|
|
from _zvec.param import _SearchQuery
|
|
|
|
vq = _SearchQuery()
|
|
vq.topk = 5
|
|
vq.field_name = "vec"
|
|
|
|
data = pickle.dumps(vq)
|
|
restored = pickle.loads(data)
|
|
assert restored.topk == 5
|
|
assert restored.field_name == "vec"
|
|
assert restored.fts is None
|