Files
zvec/.gitmodules
T
egolearner 02bfb31cf5 feat: add fts support (#408)
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
  query_string and match_string syntax, phrase queries, boolean operators
  (AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.

  ## Core
  - BitPacked posting format with block-max WAND pruning
  - Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
    lowercase, with extensible pipeline composition
  - Query parser: boolean operators, phrase queries, field scoping,
    boost, MUST(+) modifier inside OR (ES query_string semantics)
  - AST rewriter: dedup repeated terms with linear boost aggregation,
    flatten same-type composites, canonicalize OR-with-must_not into AND
    wrapper, empty-node propagation, contradiction detection
  - FTS reduce/merge integrated into Optimize compaction
  - Multi-segment score-descending sort
  - Auto-register bundled jieba dict on SDK import

  ## Performance
  - Block-max WAND with cached block_max_info_for (single binary search)
  - AVX2/SSE bitpacked encoding with cross-arch scalar fallback
  - MultiGet for batch posting retrieval and phrase position verification
  - HashSkipList memtable for posting writes
  - PinnableSlice zero-copy reads
  - Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
  - Candidate-driven (brute-force) evaluation for selective invert filters
  - Precomputed BM25 IDF weights, cached SIMD dispatch pointers
  - Shortest-list anchor for phrase position matching
  - Single-open per-term posting iterator

  ## Query
  - Tokenize query terms through the same pipeline as indexing
  - EmptyNode for zero-token queries (all stop-words / punctuation)
  - Backslash unescape after lexing in query parser
  - Schema allows collections without vector fields (FTS-only use case)
  - Create/Drop Index validates supported index types
  - FTS fields disallowed in SQL filter expressions

  ## Bindings
  - C API: fts query params, brute-force ratio config
  - Python SDK: FTS search, jieba dict auto-registration

  ## Internals
  - Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
  - Hide tokenizer pipeline from public header (Pimpl-style FtsState)
  - ListColumnFamilies to avoid double-open on segment load
  - Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
2026-06-01 15:02:54 +08:00

52 lines
2.1 KiB
Plaintext

[submodule "thirdparty/googletest/googletest-1.10.0"]
path = thirdparty/googletest/googletest-1.10.0
url = https://github.com/google/googletest.git
[submodule "thirdparty/sparsehash/sparsehash-2.0.4"]
path = thirdparty/sparsehash/sparsehash-2.0.4
url = https://github.com/sparsehash/sparsehash.git
ignore = untracked
[submodule "thirdparty/gflags/gflags-2.2.2"]
path = thirdparty/gflags/gflags-2.2.2
url = https://github.com/gflags/gflags.git
[submodule "thirdparty/rocksdb/rocksdb-8.1.1"]
path = thirdparty/rocksdb/rocksdb-8.1.1
url = https://github.com/facebook/rocksdb.git
ignore = all
[submodule "thirdparty/yaml-cpp/yaml-cpp-0.6.3"]
path = thirdparty/yaml-cpp/yaml-cpp-0.6.3
url = https://github.com/jbeder/yaml-cpp.git
[submodule "thirdparty/arrow/apache-arrow-21.0.0"]
path = thirdparty/arrow/apache-arrow-21.0.0
url = https://github.com/apache/arrow.git
ignore = all
[submodule "thirdparty/glog/glog-0.5.0"]
path = thirdparty/glog/glog-0.5.0
url = https://github.com/google/glog.git
ignore = all
[submodule "thirdparty/protobuf/protobuf-3.21.12"]
path = thirdparty/protobuf/protobuf-3.21.12
url = https://github.com/protocolbuffers/protobuf.git
[submodule "thirdparty/lz4/lz4-1.9.4"]
path = thirdparty/lz4/lz4-1.9.4
url = https://github.com/lz4/lz4.git
[submodule "thirdparty/antlr/antlr4"]
path = thirdparty/antlr/antlr4
url = https://github.com/antlr/antlr4.git
ignore = all
[submodule "thirdparty/magic_enum/magic_enum-0.9.7"]
path = thirdparty/magic_enum/magic_enum-0.9.7
url = https://github.com/Neargye/magic_enum.git
ignore = all
[submodule "thirdparty/RaBitQ-Library/RaBitQ-Library-0.1"]
path = thirdparty/RaBitQ-Library/RaBitQ-Library-0.1
url = https://github.com/VectorDB-NTU/RaBitQ-Library.git
[submodule "thirdparty/cppjieba/cppjieba-5.6.7"]
path = thirdparty/cppjieba/cppjieba-5.6.7
url = https://github.com/yanyiwu/cppjieba.git
[submodule "thirdparty/FastPFOR/FastPFOR-0.4.0"]
path = thirdparty/FastPFOR/FastPFOR-0.4.0
url = https://github.com/fast-pack/FastPFOR.git
[submodule "thirdparty/limonp/limonp-v1.0.2"]
path = thirdparty/limonp/limonp-v1.0.2
url = https://github.com/yanyiwu/limonp.git