02bfb31cf5
feat: add fts support ( #408 )
...
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
query_string and match_string syntax, phrase queries, boolean operators
(AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.
## Core
- BitPacked posting format with block-max WAND pruning
- Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
lowercase, with extensible pipeline composition
- Query parser: boolean operators, phrase queries, field scoping,
boost, MUST(+) modifier inside OR (ES query_string semantics)
- AST rewriter: dedup repeated terms with linear boost aggregation,
flatten same-type composites, canonicalize OR-with-must_not into AND
wrapper, empty-node propagation, contradiction detection
- FTS reduce/merge integrated into Optimize compaction
- Multi-segment score-descending sort
- Auto-register bundled jieba dict on SDK import
## Performance
- Block-max WAND with cached block_max_info_for (single binary search)
- AVX2/SSE bitpacked encoding with cross-arch scalar fallback
- MultiGet for batch posting retrieval and phrase position verification
- HashSkipList memtable for posting writes
- PinnableSlice zero-copy reads
- Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
- Candidate-driven (brute-force) evaluation for selective invert filters
- Precomputed BM25 IDF weights, cached SIMD dispatch pointers
- Shortest-list anchor for phrase position matching
- Single-open per-term posting iterator
## Query
- Tokenize query terms through the same pipeline as indexing
- EmptyNode for zero-token queries (all stop-words / punctuation)
- Backslash unescape after lexing in query parser
- Schema allows collections without vector fields (FTS-only use case)
- Create/Drop Index validates supported index types
- FTS fields disallowed in SQL filter expressions
## Bindings
- C API: fts query params, brute-force ratio config
- Python SDK: FTS search, jieba dict auto-registration
## Internals
- Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
- Hide tokenizer pipeline from public header (Pimpl-style FtsState)
- ListColumnFamilies to avoid double-open on segment load
- Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
2026-06-01 15:02:54 +08:00
9bfc4b556a
chore: fix more warnings ( #418 )
2026-05-21 11:15:41 +08:00
51c6d9e0ec
chore(thirdparty): mark third-party include paths as SYSTEM ( #413 )
...
Add SYSTEM to target_include_directories in vendored CMake wrappers and
set INTERFACE_SYSTEM_INCLUDE_DIRECTORIES on imported / sub-project
targets so warnings from third-party headers no longer surface in our
build output. Introduces a mark_target_includes_system() helper in
cmake/utils.cmake (resolves aliases, skips missing targets) used by the
upstream-add_subdirectory wrappers (glog, gflags, googletest, yaml-cpp,
protobuf, antlr).
2026-05-19 15:03:07 +08:00
269c611f78
build(thirdparty): replace CRoaring submodule with amalgamation (~292M -> ~1MB) ( #381 )
2026-05-12 14:55:08 +08:00
1d4ae0b1b5
refactor: support UTF-8 file paths via std::filesystem ( #359 )
...
Rewrite file/path handling to use std::filesystem and UTF-8-safe helpers.
Switch Windows file open/create paths to wide-char APIs, replace manual
separator concatenation with PathJoin, and enable RocksDB UTF-8 filenames.
Also add UTF-8 path coverage for file IO, version manager recovery, and
collection open/flush/reopen flows.
2026-05-08 17:44:22 +08:00
30a20e14b0
feat: add iOS build support ( #321 )
...
* support ios build
* fix: ci
* fix: update build_ios.sh
* fix: pr
* fix: cmake minimum version
* fix: cmake minimum version
* fix: cmake minimum version in ci
* fix: add all test in ci
* fix: ci
2026-04-09 16:14:26 +08:00
ca4b893755
fix: misc fix for Windows(BUILD_SHARED_LIBS, CI) ( #296 )
2026-04-02 09:23:35 +08:00
8182cfff93
feat: windows support ( #216 )
...
Preliminary support for Windows with remaining TODOs
2026-03-30 20:47:34 +08:00
c8df06ab3e
fix: build on mac ( #272 )
2026-03-27 20:11:22 +08:00
e5ba11b6fe
feat: add hnsw-rabitq support ( #69 )
...
* feat: add hnsw-rabitq
* undefine transform
* support config rotator type
* support sample_count
* fix ut
* refactor: update interface
* refactor: update interface
* update rabitq index params
* fix interface update
* fix searcher test
* fix streamer test
* streamer support bf and add more ut
* rm env check
* add collection ut
* add schema check
* add rabitq query param binding
* add integration test
* fix local_builder
* cleanup dist calculator
* add RaBitQ-Library submodule
* cleanup rabitq converter/reformer
* add files
* disable build on mac
* disable Feature_Optimize_HNSW_RABITQ
* disable python/tests/test_collection_hnsw_rabitq.py:13
* check avx2/avx512
* fix refine
* fix mac ci
* add dimension check
* fix mac ci
* fix ci
* add missing lib
* fix centroids selection for Cosine/InnerProduct
* rename hnsw-rabitq to hnsw_rabitq
* address comments
* fix compile
* fix rabitqlib name
* fix mac compile
* check avx2/avx512 support
* runtime check again compile-time
* check AUTO_DETECT_ARCH
* rm debug log
* fix
* rabitq use avx2 by default
* add missing file
* fix search_bf/group_by dist
* address comments
* fix typo
* address comments
* rabitq support disable id_map
* address comments
2026-03-19 17:21:42 +08:00
b10c3fc316
chore: default not use oss and use beijing-oss ( #207 )
2026-03-09 17:10:57 +08:00
43e3eeac40
feat: support android platform cross build ( #90 )
2026-03-04 16:42:23 +08:00
83c03ed0e4
chore(build): accelerate third-party downloads using OSS mirror URLs ( #194 )
2026-03-04 11:19:10 +08:00
5e637bcfdf
build: disable rocksdb -arch=native ( #183 )
2026-02-28 11:19:44 +08:00
0d84e30dd3
fix: update antlr4 CMAKE_POLICY from OLD to NEW for CMake 3.31+ compatibility ( #145 )
...
CMake 3.31+ has removed support for the OLD behavior of policies
CMP0042, CMP0045, CMP0054, and CMP0059. This causes a hard
configuration error when building from source with newer CMake versions.
Changing these to NEW resolves the build failure. The NEW behavior has
been the default since CMake 3.0/3.3 and does not affect functionality.
Fixes #144
2026-02-27 14:35:55 +08:00
e957442355
feat: refact cpp sdk ( #27 )
...
* refact cpp sdk
---------
Co-authored-by: zhouqinren.zqr <zhouqinren.zqr@alibaba-inc.com>
2026-01-21 20:05:42 +08:00
b3287fea3c
chore: support build arrow with debug ( #11 )
...
* chore: support build arrow with debug
* align
---------
Co-authored-by: feihongxu0824 <xufeihong.xfh@alibaba-inc.com>
2026-01-12 11:49:24 +08:00
6524fabd80
Initial commit
2025-12-30 11:02:17 +08:00
egolearner
luoxiaojian
Jalin Wang
feihongxu0824
Qinren Zhou
Aashish Thapa
sanyi