feat: add fts support (#408)
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
query_string and match_string syntax, phrase queries, boolean operators
(AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.
## Core
- BitPacked posting format with block-max WAND pruning
- Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
lowercase, with extensible pipeline composition
- Query parser: boolean operators, phrase queries, field scoping,
boost, MUST(+) modifier inside OR (ES query_string semantics)
- AST rewriter: dedup repeated terms with linear boost aggregation,
flatten same-type composites, canonicalize OR-with-must_not into AND
wrapper, empty-node propagation, contradiction detection
- FTS reduce/merge integrated into Optimize compaction
- Multi-segment score-descending sort
- Auto-register bundled jieba dict on SDK import
## Performance
- Block-max WAND with cached block_max_info_for (single binary search)
- AVX2/SSE bitpacked encoding with cross-arch scalar fallback
- MultiGet for batch posting retrieval and phrase position verification
- HashSkipList memtable for posting writes
- PinnableSlice zero-copy reads
- Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
- Candidate-driven (brute-force) evaluation for selective invert filters
- Precomputed BM25 IDF weights, cached SIMD dispatch pointers
- Shortest-list anchor for phrase position matching
- Single-open per-term posting iterator
## Query
- Tokenize query terms through the same pipeline as indexing
- EmptyNode for zero-token queries (all stop-words / punctuation)
- Backslash unescape after lexing in query parser
- Schema allows collections without vector fields (FTS-only use case)
- Create/Drop Index validates supported index types
- FTS fields disallowed in SQL filter expressions
## Bindings
- C API: fts query params, brute-force ratio config
- Python SDK: FTS search, jieba dict auto-registration
## Internals
- Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
- Hide tokenizer pipeline from public header (Pimpl-style FtsState)
- ListColumnFamilies to avoid double-open on segment load
- Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
This commit is contained in:
@@ -40,3 +40,12 @@
|
||||
[submodule "thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]
|
||||
path = thirdparty/RaBitQ-Library/RaBitQ-Library-0.1
|
||||
url = https://github.com/VectorDB-NTU/RaBitQ-Library.git
|
||||
[submodule "thirdparty/cppjieba/cppjieba-5.6.7"]
|
||||
path = thirdparty/cppjieba/cppjieba-5.6.7
|
||||
url = https://github.com/yanyiwu/cppjieba.git
|
||||
[submodule "thirdparty/FastPFOR/FastPFOR-0.4.0"]
|
||||
path = thirdparty/FastPFOR/FastPFOR-0.4.0
|
||||
url = https://github.com/fast-pack/FastPFOR.git
|
||||
[submodule "thirdparty/limonp/limonp-v1.0.2"]
|
||||
path = thirdparty/limonp/limonp-v1.0.2
|
||||
url = https://github.com/yanyiwu/limonp.git
|
||||
|
||||
Reference in New Issue
Block a user