Commit Graph

7 Commits

Author SHA1 Message Date
egolearner 443500dc45
feat: add FTS support for Collection::CreateIndex/DropIndex (#445)
* feat: add FTS support for Collection::CreateIndex/DropIndex

Enable dynamic creation and removal of FTS indexes on existing STRING
columns through the standard CreateIndex/DropIndex API, matching the
lifecycle model already used by vector and scalar (invert) indexes.

Key changes:
- New FtsIndexer class (fts_indexer.h/cc) encapsulating per-segment FTS
  RocksDB management: multi-field lifecycle, snapshot, insert, seal
- New BlockType::FTS_INDEX with block_id-based directory naming
  (fts.<block_id>.rocksdb) for crash-safe snapshot-and-swap
- Segment::create_fts_index builds FTS index on a snapshot copy by
  scanning forward store, then outputs new SegmentMeta + FtsIndexer
  for atomic reload (same pattern as create_scalar_index)
- Segment::drop_fts_index snapshots, removes field CFs, outputs updated
  meta (or nullptr when last FTS field is removed)
- Collection layer wires FTS into the existing task dispatch, version
  update, and reload loops alongside vector/invert paths
- CreateIndex/DropIndex reject unsupported index types explicitly
  instead of falling through to the wrong branch

* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex

    fix: address review comments for FTS CreateIndex/DropIndex

    - Reject CreateIndex when column already has a different index type
      (e.g. FTS on an INVERT-indexed column) at both Collection and
      Segment layers
    - Allow same-type different-params CreateIndex to rebuild the index
      (remove old + create new + replay data), aligned with INVERT behavior
    - Return OK when CreateIndex is called with identical params
    - Rename operator[] to get() in both FtsIndexer and InvertedIndexer
    - Add test cases: create→drop→create→drop cycle, and params-change
      rebuild with case-sensitivity verification

* android skip Feature_CreateOrDropFtsIndex

* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex
2026-06-03 17:56:56 +08:00
Qinren Zhou dbea635019
refactor: clarify segment-local row ID handling and tests and fixes bugs (#432) 2026-06-03 15:20:45 +08:00
egolearner 02bfb31cf5
feat: add fts support (#408)
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
  query_string and match_string syntax, phrase queries, boolean operators
  (AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.

  ## Core
  - BitPacked posting format with block-max WAND pruning
  - Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
    lowercase, with extensible pipeline composition
  - Query parser: boolean operators, phrase queries, field scoping,
    boost, MUST(+) modifier inside OR (ES query_string semantics)
  - AST rewriter: dedup repeated terms with linear boost aggregation,
    flatten same-type composites, canonicalize OR-with-must_not into AND
    wrapper, empty-node propagation, contradiction detection
  - FTS reduce/merge integrated into Optimize compaction
  - Multi-segment score-descending sort
  - Auto-register bundled jieba dict on SDK import

  ## Performance
  - Block-max WAND with cached block_max_info_for (single binary search)
  - AVX2/SSE bitpacked encoding with cross-arch scalar fallback
  - MultiGet for batch posting retrieval and phrase position verification
  - HashSkipList memtable for posting writes
  - PinnableSlice zero-copy reads
  - Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
  - Candidate-driven (brute-force) evaluation for selective invert filters
  - Precomputed BM25 IDF weights, cached SIMD dispatch pointers
  - Shortest-list anchor for phrase position matching
  - Single-open per-term posting iterator

  ## Query
  - Tokenize query terms through the same pipeline as indexing
  - EmptyNode for zero-token queries (all stop-words / punctuation)
  - Backslash unescape after lexing in query parser
  - Schema allows collections without vector fields (FTS-only use case)
  - Create/Drop Index validates supported index types
  - FTS fields disallowed in SQL filter expressions

  ## Bindings
  - C API: fts query params, brute-force ratio config
  - Python SDK: FTS search, jieba dict auto-registration

  ## Internals
  - Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
  - Hide tokenizer pipeline from public header (Pimpl-style FtsState)
  - ListColumnFamilies to avoid double-open on segment load
  - Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
2026-06-01 15:02:54 +08:00
lichen2015 e0ba23179b
feat: fetch() add output_fields param (#358) 2026-05-27 22:58:49 +08:00
lichen2015 f1349cc91a
fix: remove unnecessary column_name param from the AddColumn API (#59) 2026-02-04 14:44:13 +08:00
feihongxu0824 0ebf5621a6
feat: support core cpp sdk (#30)
* support core cpp sdk

* fix

* fix

* fix

* fix cr
2026-01-26 20:01:44 +08:00
sanyi 6524fabd80 Initial commit 2025-12-30 11:02:17 +08:00