Commit Graph

21 Commits

Author SHA1 Message Date
egolearner 439dd10f5e
feat(query): support FTS + vector hybrid retrieval in MultiQuery (#459)
Previously MultiQuery only accepted vector sub-queries by using
get_vector_field() to look up each sub-query's field. FTS sub-queries
(whose field_name points to an FTS-indexed string column) would fail
with "Vector field not found".

Changes:
- collection.cc: use get_field() uniformly in MultiQuery path; let
  validate_and_sanitize() check type compatibility internally, which
  is consistent with the single-query path.
- query_executor.py: allow SingleVectorQueryExecutor to accept
  multi-query when it contains an FTS query (with reranker), and route
  to C++ MultiQuery fast path.
- Add test_collection_fts_vector_hybrid.py covering hybrid retrieval
  ranking, scoring, filter, validation, and edge cases.
2026-06-03 21:17:49 +08:00
ihb2032 df447a54e2
Refactor CPU feature detection to use an explicit x86 whitelist (#258)
* fix(cpu_features): refactor architecture detection to explicit x86 whitelist

Currently, `cpu_features.cc` assumes any non-ARM architecture is x86/x64, which leads to a fatal missing `<cpuid.h>` error on architectures like RISC-V.
This commit refactors the preprocessor macros to explicitly whitelist x86 architectures (`__x86_64__`, `__i386__`, `_M_X64`, `_M_IX86`). All other architectures (RISC-V, ARM, etc.) will now safely fall back to the default zero-initialization, allowing cross-compilation to succeed.

Signed-off-by: ihb2032 <hebome@foxmail.com>

* ci: Add RISE RISC-V runner

Introduce the RISC-V CI runner provided by the RISE project.
This enables automated testing and building for the RISC-V architecture.

Signed-off-by: ihb2032 <hebome@foxmail.com>

* ci: use python 3.12 for RISC-V64

Signed-off-by: ihb2032 <hebome@foxmail.com>

* ci: add RISC-V numpy dependencies

Signed-off-by: ihb2032 <hebome@foxmail.com>

* ci: add RISC-V wheel cache

Signed-off-by: ihb2032 <hebome@foxmail.com>

* ci: use pre-built RISE numpy wheel to speed up riscv builds

Signed-off-by: ihb2032 <hebome@foxmail.com>

* ci: use pre-built RISE cmake wheel to speed up riscv builds

Signed-off-by: ihb2032 <hebome@foxmail.com>

* ci: split RISC-V build and test into separate jobs

Signed-off-by: ihb2032 <hebome@foxmail.com>

* ci: fix

Signed-off-by: ihb2032 <hebome@foxmail.com>

* ci: fix

Signed-off-by: ihb2032 <hebome@foxmail.com>

* Update hnsw_streamer_test.cc

* ci: add cache

Signed-off-by: ihb2032 <hebome@foxmail.com>

* Update hnsw_streamer_test.cc

* Update test_gil_release.py

* ci: schedule workflow to run overnight

---------

Signed-off-by: ihb2032 <hebome@foxmail.com>
Co-authored-by: ZeFeng Yin <yinzefeng.yzf@alibaba-inc.com>
2026-06-02 11:51:25 +08:00
egolearner 02bfb31cf5
feat: add fts support (#408)
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
  query_string and match_string syntax, phrase queries, boolean operators
  (AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.

  ## Core
  - BitPacked posting format with block-max WAND pruning
  - Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
    lowercase, with extensible pipeline composition
  - Query parser: boolean operators, phrase queries, field scoping,
    boost, MUST(+) modifier inside OR (ES query_string semantics)
  - AST rewriter: dedup repeated terms with linear boost aggregation,
    flatten same-type composites, canonicalize OR-with-must_not into AND
    wrapper, empty-node propagation, contradiction detection
  - FTS reduce/merge integrated into Optimize compaction
  - Multi-segment score-descending sort
  - Auto-register bundled jieba dict on SDK import

  ## Performance
  - Block-max WAND with cached block_max_info_for (single binary search)
  - AVX2/SSE bitpacked encoding with cross-arch scalar fallback
  - MultiGet for batch posting retrieval and phrase position verification
  - HashSkipList memtable for posting writes
  - PinnableSlice zero-copy reads
  - Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
  - Candidate-driven (brute-force) evaluation for selective invert filters
  - Precomputed BM25 IDF weights, cached SIMD dispatch pointers
  - Shortest-list anchor for phrase position matching
  - Single-open per-term posting iterator

  ## Query
  - Tokenize query terms through the same pipeline as indexing
  - EmptyNode for zero-token queries (all stop-words / punctuation)
  - Backslash unescape after lexing in query parser
  - Schema allows collections without vector fields (FTS-only use case)
  - Create/Drop Index validates supported index types
  - FTS fields disallowed in SQL filter expressions

  ## Bindings
  - C API: fts query params, brute-force ratio config
  - Python SDK: FTS search, jieba dict auto-registration

  ## Internals
  - Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
  - Hide tokenizer pipeline from public header (Pimpl-style FtsState)
  - ListColumnFamilies to avoid double-open on segment load
  - Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
2026-06-01 15:02:54 +08:00
lichen2015 de8fb760ef
fix: sync querier schema after column DDL to fix empty query fields (#429)
* fix: sync querier schema after column DDL to fix empty query fields (#426)

* fix: sync querier schema after create_index/drop_index
2026-05-29 17:36:30 +08:00
lichen2015 f539580138
feat: migrate multi-vector query and reranker logic to C++ (#405)
* feat: migrate multi-vector query and reranker logic to C++

- Add Reranker base class with RrfReRanker and WeightedReRanker implementations
- Add Collection::MultiQuery interface for multi-vector queries with reranking
- Add MultiVectorQuery struct in doc.h with forward declaration for Reranker
- Add C API bindings for reranker and MultiQuery (zvec_reranker_*, zvec_multi_vector_query_*, zvec_collection_multi_query)
- Add Python binding for reranker classes with py::function bridge for callback
- Validate duplicate field names in multi-vector queries (C++ and Python consistent)
- Remove TODO comment about concurrent execution (SQLEngine is not thread-safe)
- Update collection.h MultiQuery doc comment from concurrently to sequentially
- Add C++ collection tests (6 MultiQuery test cases)
- Add C API tests (reranker functions + multi_vector_query end-to-end)
- Implement Python test cases (11 previously skipped tests now active)
- Simplify Python query_executor validation for unified duplicate field check

* style: format Python files with ruff

* fix: adapt to main branch API changes (VectorQuery->Query rename, validate_and_sanitize)

* fix: multi_vector tests now use multiple same-type vector fields (dense2, sparse2)

* fix: suppress RET501 for intentional default return None in RerankFunction._get_object

* style: ruff format test_collection.py

* refact multi vector query

* format code

* fix(multi-vector): expose SubVectorQuery in Python binding, fix tests

- Register _SubVectorQuery in pybind11 with from_vector_query() factory
- Convert _VectorQuery to _SubVectorQuery in MultiVectorQueryExecutor
- Relax RRF/Weighted score assertion tolerance from 1e-10 to 1e-6
- Fix WeightedReRanker test metric to IP (matching HnswIndexParam default)

* style: ruff format query_executor.py

* fix: define _USE_MATH_DEFINES for M_PI on Windows (MSVC)

* refactor: include reranker.h directly in query.h instead of forward declaration

* refactor(reranker): move topn from member variable to rerank() parameter

* refact code

* style(python): fix ruff UP035/UP037 in multi_vector_reranker

- import Callable from collections.abc instead of typing (UP035)
- remove redundant quotes around MetricType annotations (UP037)

* chore: trigger PR sync

* refact code

* fix(examples): restore CMakeLists.txt formatting broken by clang-format

* refactor(reranker): remove redundant metrics_ map by querying schema directly, and use insert return value to avoid duplicate set lookup

* refactor(reranker): defer schema binding to query time and remove C API callback reranker
2026-05-29 10:10:41 +08:00
lichen2015 e0ba23179b
feat: fetch() add output_fields param (#358) 2026-05-27 22:58:49 +08:00
feihongxu0824 273ffc2ad8
enhance: release gil lock in collection bindings (#363) 2026-05-14 10:15:49 +08:00
Qinren Zhou e8eaf0aa57
fix: python ut (#401) 2026-05-13 15:56:58 +08:00
egolearner d0489da401
ut: fix some uts (#394) 2026-05-12 19:04:33 +08:00
egolearner ce468af29b
deprecate python VectorQuery (#267)
* deprecate python VectorQuery

* remove import
2026-05-12 14:55:49 +08:00
luoxiaojian efab064676
feat: refac entity and impl Vamana. (#371) 2026-04-30 10:22:25 +08:00
Qinren Zhou c17bd8876e
fix: validate query_params type (#351) 2026-04-20 10:54:06 +08:00
Qinren Zhou f602ed30ce
feat: enlarge topK limit (#348) 2026-04-17 17:51:07 +08:00
Jalin Wang 95d092e9b3
fix: relax collection path restriction (#340) 2026-04-16 15:26:15 +08:00
liututu12 4e8829704a
test: Add test cases for field values in Chinese scenarios to open and dml (#264) 2026-03-26 19:42:27 +08:00
egolearner e5ba11b6fe
feat: add hnsw-rabitq support (#69)
* feat: add hnsw-rabitq

* undefine transform

* support config rotator type

* support sample_count

* fix ut

* refactor: update interface

* refactor: update interface

* update rabitq index params

* fix interface update

* fix searcher test

* fix streamer test

* streamer support bf and add more ut

* rm env check

* add collection ut

* add schema check

* add rabitq query param binding

* add integration test

* fix local_builder

* cleanup dist calculator

* add RaBitQ-Library submodule

* cleanup rabitq converter/reformer

* add files

* disable build on mac

* disable Feature_Optimize_HNSW_RABITQ

* disable python/tests/test_collection_hnsw_rabitq.py:13

* check avx2/avx512

* fix refine

* fix mac ci

* add dimension check

* fix mac ci

* fix ci

* add missing lib

* fix centroids selection for Cosine/InnerProduct

* rename hnsw-rabitq to hnsw_rabitq

* address comments

* fix compile

* fix rabitqlib name

* fix mac compile

* check avx2/avx512 support

* runtime check again compile-time

* check AUTO_DETECT_ARCH

* rm debug log

* fix

* rabitq use avx2 by default

* add missing file

* fix search_bf/group_by dist

* address comments

* fix typo

* address comments

* rabitq support disable id_map

* address comments
2026-03-19 17:21:42 +08:00
liututu12 3bd1999604
tests: add tests for recall (#196)
* add recall cases

* add test_collection_crash_recovery_insertdoc.py

* add test_collection_crash_recovery_updatedoc.py

* add test_collection_crash_recovery_upsertdoc.py

* add test_collection_crash_recovery_deletedoc.py

* add test_collection_crash_recovery_createindex.py

* add test_collection_crash_recovery_deleteindex.py

* test_collection_crash_recovery_addcolumn.py

* test_collection_crash_recovery_addcolumn.py

* add dropcolumn updatecolumn cases

* update test_collection_crash_recovery_altercolumn.py

* update test_collection_recall

* update test_collection_crash_recovery_createindex.py

* test_collection_recall.py

* test_collection_recall.py

* feat: add recall test cases and optimize DML/DQL test cases

* upd test case

* fix doc_num

---------

Co-authored-by: iaojnh <yinzefeng.yzf@alibaba-inc.com>
2026-03-05 20:13:45 +08:00
Cuiys 1fa15ce96b
feat: support ai extension (#88) 2026-02-12 18:33:41 +08:00
Qinren Zhou 0b9bd34354
minor: subsequent initialization should fail silently (#79) 2026-02-06 21:16:08 +08:00
egolearner 89f9c378ac
fix: fix default m (#20)
* fix: fix default m

* fix doc str

* fix ut
2026-01-12 10:22:58 +08:00
sanyi 6524fabd80 Initial commit 2025-12-30 11:02:17 +08:00