4679a8f39d
feat: support zvec core-only build ( #481 )
2026-06-09 23:45:42 +08:00
443500dc45
feat: add FTS support for Collection::CreateIndex/DropIndex ( #445 )
...
* feat: add FTS support for Collection::CreateIndex/DropIndex
Enable dynamic creation and removal of FTS indexes on existing STRING
columns through the standard CreateIndex/DropIndex API, matching the
lifecycle model already used by vector and scalar (invert) indexes.
Key changes:
- New FtsIndexer class (fts_indexer.h/cc) encapsulating per-segment FTS
RocksDB management: multi-field lifecycle, snapshot, insert, seal
- New BlockType::FTS_INDEX with block_id-based directory naming
(fts.<block_id>.rocksdb) for crash-safe snapshot-and-swap
- Segment::create_fts_index builds FTS index on a snapshot copy by
scanning forward store, then outputs new SegmentMeta + FtsIndexer
for atomic reload (same pattern as create_scalar_index)
- Segment::drop_fts_index snapshots, removes field CFs, outputs updated
meta (or nullptr when last FTS field is removed)
- Collection layer wires FTS into the existing task dispatch, version
update, and reload loops alongside vector/invert paths
- CreateIndex/DropIndex reject unsupported index types explicitly
instead of falling through to the wrong branch
* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex
fix: address review comments for FTS CreateIndex/DropIndex
- Reject CreateIndex when column already has a different index type
(e.g. FTS on an INVERT-indexed column) at both Collection and
Segment layers
- Allow same-type different-params CreateIndex to rebuild the index
(remove old + create new + replay data), aligned with INVERT behavior
- Return OK when CreateIndex is called with identical params
- Rename operator[] to get() in both FtsIndexer and InvertedIndexer
- Add test cases: create→drop→create→drop cycle, and params-change
rebuild with case-sensitivity verification
* android skip Feature_CreateOrDropFtsIndex
* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex
2026-06-03 17:56:56 +08:00
dbea635019
refactor: clarify segment-local row ID handling and tests and fixes bugs ( #432 )
2026-06-03 15:20:45 +08:00
8e8bb81db0
refactor: tidy query APIs and execution ( #431 )
2026-06-01 21:07:35 +08:00
02bfb31cf5
feat: add fts support ( #408 )
...
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
query_string and match_string syntax, phrase queries, boolean operators
(AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.
## Core
- BitPacked posting format with block-max WAND pruning
- Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
lowercase, with extensible pipeline composition
- Query parser: boolean operators, phrase queries, field scoping,
boost, MUST(+) modifier inside OR (ES query_string semantics)
- AST rewriter: dedup repeated terms with linear boost aggregation,
flatten same-type composites, canonicalize OR-with-must_not into AND
wrapper, empty-node propagation, contradiction detection
- FTS reduce/merge integrated into Optimize compaction
- Multi-segment score-descending sort
- Auto-register bundled jieba dict on SDK import
## Performance
- Block-max WAND with cached block_max_info_for (single binary search)
- AVX2/SSE bitpacked encoding with cross-arch scalar fallback
- MultiGet for batch posting retrieval and phrase position verification
- HashSkipList memtable for posting writes
- PinnableSlice zero-copy reads
- Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
- Candidate-driven (brute-force) evaluation for selective invert filters
- Precomputed BM25 IDF weights, cached SIMD dispatch pointers
- Shortest-list anchor for phrase position matching
- Single-open per-term posting iterator
## Query
- Tokenize query terms through the same pipeline as indexing
- EmptyNode for zero-token queries (all stop-words / punctuation)
- Backslash unescape after lexing in query parser
- Schema allows collections without vector fields (FTS-only use case)
- Create/Drop Index validates supported index types
- FTS fields disallowed in SQL filter expressions
## Bindings
- C API: fts query params, brute-force ratio config
- Python SDK: FTS search, jieba dict auto-registration
## Internals
- Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
- Hide tokenizer pipeline from public header (Pimpl-style FtsState)
- ListColumnFamilies to avoid double-open on segment load
- Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
2026-06-01 15:02:54 +08:00
8dcb6cbd7f
refactor: drop VectorQuery, unify single-target query on SearchQuery ( #428 )
2026-05-29 16:36:09 +08:00
e0ba23179b
feat: fetch() add output_fields param ( #358 )
2026-05-27 22:58:49 +08:00
1e25294c07
feat(ci): integrate clang-tidy for changed C/C++ files ( #116 )
...
* feat(ci): add clang-tidy checks for changed cpp files
* fix(ci): correct clang-tidy workflow heredoc indentation
* test: stabilize fp16 euclidean matrix comparison
* test: fix fp16 matrix CI and clang-tidy warnings
* ci: scope clang-tidy PR and filter compile-db files
* check nullptr only
* fix nullptr
* fix: format
* chore: ignore c
* fix: tests files
* fix: workflow
* fix: specify clang-tidy version
* fix: filter regex
* fix: more files
* fix: update version
* add thirdparty cache
* add pipeline dependency
* fix clang-tidy trigger
---------
Co-authored-by: kgeg401 <kgeg401@users.noreply.github.com>
Co-authored-by: jiliang.ljl <jiliang.ljl@alibaba-inc.com>
2026-04-20 20:14:21 +08:00
e5ba11b6fe
feat: add hnsw-rabitq support ( #69 )
...
* feat: add hnsw-rabitq
* undefine transform
* support config rotator type
* support sample_count
* fix ut
* refactor: update interface
* refactor: update interface
* update rabitq index params
* fix interface update
* fix searcher test
* fix streamer test
* streamer support bf and add more ut
* rm env check
* add collection ut
* add schema check
* add rabitq query param binding
* add integration test
* fix local_builder
* cleanup dist calculator
* add RaBitQ-Library submodule
* cleanup rabitq converter/reformer
* add files
* disable build on mac
* disable Feature_Optimize_HNSW_RABITQ
* disable python/tests/test_collection_hnsw_rabitq.py:13
* check avx2/avx512
* fix refine
* fix mac ci
* add dimension check
* fix mac ci
* fix ci
* add missing lib
* fix centroids selection for Cosine/InnerProduct
* rename hnsw-rabitq to hnsw_rabitq
* address comments
* fix compile
* fix rabitqlib name
* fix mac compile
* check avx2/avx512 support
* runtime check again compile-time
* check AUTO_DETECT_ARCH
* rm debug log
* fix
* rabitq use avx2 by default
* add missing file
* fix search_bf/group_by dist
* address comments
* fix typo
* address comments
* rabitq support disable id_map
* address comments
2026-03-19 17:21:42 +08:00
9314cb14a6
fix: use per-block filter instead of per-segment filter during query ( #223 )
...
* fix: id mismatch
* fix: remove debug string
* fix: use protected
* chore(comment): add comments
* feat: more ut
* Update vector_recall_test.cc
* fix: resolve comments
2026-03-16 10:55:51 +08:00
fb0b900a80
fix(build): replace CMAKE_SOURCE_DIR with PROJECT_ROOT_DIR for subproject builds ( #195 )
2026-03-04 20:01:52 +08:00
f1349cc91a
fix: remove unnecessary column_name param from the AddColumn API ( #59 )
2026-02-04 14:44:13 +08:00
0ebf5621a6
feat: support core cpp sdk ( #30 )
...
* support core cpp sdk
* fix
* fix
* fix
* fix cr
2026-01-26 20:01:44 +08:00
e957442355
feat: refact cpp sdk ( #27 )
...
* refact cpp sdk
---------
Co-authored-by: zhouqinren.zqr <zhouqinren.zqr@alibaba-inc.com>
2026-01-21 20:05:42 +08:00
6524fabd80
Initial commit
2025-12-30 11:02:17 +08:00
feihongxu0824
egolearner
Qinren Zhou
lichen2015
kgeg401
sanyi