Add BM25-based full-text search with CJK (jieba) tokenization, supporting
query_string and match_string syntax, phrase queries, boolean operators
(AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.
## Core
- BitPacked posting format with block-max WAND pruning
- Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
lowercase, with extensible pipeline composition
- Query parser: boolean operators, phrase queries, field scoping,
boost, MUST(+) modifier inside OR (ES query_string semantics)
- AST rewriter: dedup repeated terms with linear boost aggregation,
flatten same-type composites, canonicalize OR-with-must_not into AND
wrapper, empty-node propagation, contradiction detection
- FTS reduce/merge integrated into Optimize compaction
- Multi-segment score-descending sort
- Auto-register bundled jieba dict on SDK import
## Performance
- Block-max WAND with cached block_max_info_for (single binary search)
- AVX2/SSE bitpacked encoding with cross-arch scalar fallback
- MultiGet for batch posting retrieval and phrase position verification
- HashSkipList memtable for posting writes
- PinnableSlice zero-copy reads
- Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
- Candidate-driven (brute-force) evaluation for selective invert filters
- Precomputed BM25 IDF weights, cached SIMD dispatch pointers
- Shortest-list anchor for phrase position matching
- Single-open per-term posting iterator
## Query
- Tokenize query terms through the same pipeline as indexing
- EmptyNode for zero-token queries (all stop-words / punctuation)
- Backslash unescape after lexing in query parser
- Schema allows collections without vector fields (FTS-only use case)
- Create/Drop Index validates supported index types
- FTS fields disallowed in SQL filter expressions
## Bindings
- C API: fts query params, brute-force ratio config
- Python SDK: FTS search, jieba dict auto-registration
## Internals
- Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
- Hide tokenizer pipeline from public header (Pimpl-style FtsState)
- ListColumnFamilies to avoid double-open on segment load
- Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
52 lines
2.1 KiB
Plaintext
52 lines
2.1 KiB
Plaintext
[submodule "thirdparty/googletest/googletest-1.10.0"]
|
|
path = thirdparty/googletest/googletest-1.10.0
|
|
url = https://github.com/google/googletest.git
|
|
[submodule "thirdparty/sparsehash/sparsehash-2.0.4"]
|
|
path = thirdparty/sparsehash/sparsehash-2.0.4
|
|
url = https://github.com/sparsehash/sparsehash.git
|
|
ignore = untracked
|
|
[submodule "thirdparty/gflags/gflags-2.2.2"]
|
|
path = thirdparty/gflags/gflags-2.2.2
|
|
url = https://github.com/gflags/gflags.git
|
|
[submodule "thirdparty/rocksdb/rocksdb-8.1.1"]
|
|
path = thirdparty/rocksdb/rocksdb-8.1.1
|
|
url = https://github.com/facebook/rocksdb.git
|
|
ignore = all
|
|
[submodule "thirdparty/yaml-cpp/yaml-cpp-0.6.3"]
|
|
path = thirdparty/yaml-cpp/yaml-cpp-0.6.3
|
|
url = https://github.com/jbeder/yaml-cpp.git
|
|
[submodule "thirdparty/arrow/apache-arrow-21.0.0"]
|
|
path = thirdparty/arrow/apache-arrow-21.0.0
|
|
url = https://github.com/apache/arrow.git
|
|
ignore = all
|
|
[submodule "thirdparty/glog/glog-0.5.0"]
|
|
path = thirdparty/glog/glog-0.5.0
|
|
url = https://github.com/google/glog.git
|
|
ignore = all
|
|
[submodule "thirdparty/protobuf/protobuf-3.21.12"]
|
|
path = thirdparty/protobuf/protobuf-3.21.12
|
|
url = https://github.com/protocolbuffers/protobuf.git
|
|
[submodule "thirdparty/lz4/lz4-1.9.4"]
|
|
path = thirdparty/lz4/lz4-1.9.4
|
|
url = https://github.com/lz4/lz4.git
|
|
[submodule "thirdparty/antlr/antlr4"]
|
|
path = thirdparty/antlr/antlr4
|
|
url = https://github.com/antlr/antlr4.git
|
|
ignore = all
|
|
[submodule "thirdparty/magic_enum/magic_enum-0.9.7"]
|
|
path = thirdparty/magic_enum/magic_enum-0.9.7
|
|
url = https://github.com/Neargye/magic_enum.git
|
|
ignore = all
|
|
[submodule "thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]
|
|
path = thirdparty/RaBitQ-Library/RaBitQ-Library-0.1
|
|
url = https://github.com/VectorDB-NTU/RaBitQ-Library.git
|
|
[submodule "thirdparty/cppjieba/cppjieba-5.6.7"]
|
|
path = thirdparty/cppjieba/cppjieba-5.6.7
|
|
url = https://github.com/yanyiwu/cppjieba.git
|
|
[submodule "thirdparty/FastPFOR/FastPFOR-0.4.0"]
|
|
path = thirdparty/FastPFOR/FastPFOR-0.4.0
|
|
url = https://github.com/fast-pack/FastPFOR.git
|
|
[submodule "thirdparty/limonp/limonp-v1.0.2"]
|
|
path = thirdparty/limonp/limonp-v1.0.2
|
|
url = https://github.com/yanyiwu/limonp.git
|