9ff5db33db
refactor: remove redundant IndexLogger ( #652 )
2026-08-05 10:51:18 +08:00
d15a37e425
fix: remove redundant virtual ( #631 )
2026-07-29 19:23:16 +08:00
1ad6df2539
refactor(thread-pool): make CPU affinity opt-in ( #623 )
2026-07-29 15:30:04 +08:00
e2ea49d4ac
fix: limit Windows all-in-one DLL exports ( #611 )
2026-07-28 11:51:44 +08:00
ec8a78ee08
refactor(diskann): decouple from libaio via dlopen ( #532 )
...
Co-authored-by: Zefeng Yin <yinzefeng.yzf@alibaba-inc.com>
2026-07-16 17:58:25 +08:00
c54f5e16e9
rotate: add an optional random rotation feature in INT8/INT4 quantization method ( #483 )
...
Co-authored-by: rayx <rui.xing@alibaba-inc.com>
Co-authored-by: Jalin Wang <wangjianning.wjn@alibaba-inc.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-06-26 19:16:19 +08:00
823b5e6824
fix: resolve compiler warnings and enable -Werror across all CI platforms ( #460 )
2026-06-05 14:51:37 +08:00
e720c1fd20
feat: add diskann index ( #369 )
2026-06-04 20:52:43 +08:00
02bfb31cf5
feat: add fts support ( #408 )
...
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
query_string and match_string syntax, phrase queries, boolean operators
(AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.
## Core
- BitPacked posting format with block-max WAND pruning
- Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
lowercase, with extensible pipeline composition
- Query parser: boolean operators, phrase queries, field scoping,
boost, MUST(+) modifier inside OR (ES query_string semantics)
- AST rewriter: dedup repeated terms with linear boost aggregation,
flatten same-type composites, canonicalize OR-with-must_not into AND
wrapper, empty-node propagation, contradiction detection
- FTS reduce/merge integrated into Optimize compaction
- Multi-segment score-descending sort
- Auto-register bundled jieba dict on SDK import
## Performance
- Block-max WAND with cached block_max_info_for (single binary search)
- AVX2/SSE bitpacked encoding with cross-arch scalar fallback
- MultiGet for batch posting retrieval and phrase position verification
- HashSkipList memtable for posting writes
- PinnableSlice zero-copy reads
- Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
- Candidate-driven (brute-force) evaluation for selective invert filters
- Precomputed BM25 IDF weights, cached SIMD dispatch pointers
- Shortest-list anchor for phrase position matching
- Single-open per-term posting iterator
## Query
- Tokenize query terms through the same pipeline as indexing
- EmptyNode for zero-token queries (all stop-words / punctuation)
- Backslash unescape after lexing in query parser
- Schema allows collections without vector fields (FTS-only use case)
- Create/Drop Index validates supported index types
- FTS fields disallowed in SQL filter expressions
## Bindings
- C API: fts query params, brute-force ratio config
- Python SDK: FTS search, jieba dict auto-registration
## Internals
- Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
- Hide tokenizer pipeline from public header (Pimpl-style FtsState)
- ListColumnFamilies to avoid double-open on segment load
- Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
2026-06-01 15:02:54 +08:00
9bfc4b556a
chore: fix more warnings ( #418 )
2026-05-21 11:15:41 +08:00
3e02031eb8
refactor: remove hamming metric ( #365 )
...
* refactor: remove binary metric
* fix: fix windows
2026-05-18 19:31:20 +08:00
efab064676
feat: refac entity and impl Vamana. ( #371 )
2026-04-30 10:22:25 +08:00
2b16ed39cd
chore: rm hnsw-rabitq searcher/streamer ( #367 )
...
* chore: rm hnsw-rabitq searcher/streamer
* fix clang-tidy
* fix
2026-04-23 10:20:46 +08:00
8182cfff93
feat: windows support ( #216 )
...
Preliminary support for Windows with remaining TODOs
2026-03-30 20:47:34 +08:00
e5ba11b6fe
feat: add hnsw-rabitq support ( #69 )
...
* feat: add hnsw-rabitq
* undefine transform
* support config rotator type
* support sample_count
* fix ut
* refactor: update interface
* refactor: update interface
* update rabitq index params
* fix interface update
* fix searcher test
* fix streamer test
* streamer support bf and add more ut
* rm env check
* add collection ut
* add schema check
* add rabitq query param binding
* add integration test
* fix local_builder
* cleanup dist calculator
* add RaBitQ-Library submodule
* cleanup rabitq converter/reformer
* add files
* disable build on mac
* disable Feature_Optimize_HNSW_RABITQ
* disable python/tests/test_collection_hnsw_rabitq.py:13
* check avx2/avx512
* fix refine
* fix mac ci
* add dimension check
* fix mac ci
* fix ci
* add missing lib
* fix centroids selection for Cosine/InnerProduct
* rename hnsw-rabitq to hnsw_rabitq
* address comments
* fix compile
* fix rabitqlib name
* fix mac compile
* check avx2/avx512 support
* runtime check again compile-time
* check AUTO_DETECT_ARCH
* rm debug log
* fix
* rabitq use avx2 by default
* add missing file
* fix search_bf/group_by dist
* address comments
* fix typo
* address comments
* rabitq support disable id_map
* address comments
2026-03-19 17:21:42 +08:00
e7c4a2f2b8
feat: local_builder simplify disable idmap ( #239 )
2026-03-18 14:42:00 +08:00
fb0b900a80
fix(build): replace CMAKE_SOURCE_DIR with PROJECT_ROOT_DIR for subproject builds ( #195 )
2026-03-04 20:01:52 +08:00
2bfeac5034
chore: switch core tools to LOG_ERROR instead of cerr for better traceability when errors occur ( #80 )
2026-02-09 19:36:14 +08:00
ec26525b38
feat(docs): update bench tool readme.md ( #77 )
2026-02-06 17:31:46 +08:00
78f08bd0cc
feat: auto scalable segment meta section of mmap files ( #67 )
...
The underly core module only support a fixed-size segment due to the index file format where the section of segments meta limit the max num of data segments. This commit makes the meta section dynamically allocated and organized in a link-table manner to support much more segments.
* revert local_builder's meta cap setting
* feat: auto scalable segment meta section of mmap files
* Revert "feat: auto scalable segment meta section of mmap files"
This reverts commit 82c11e451544b912b901be33fae79e964a487b1f.
* v2: don't change to absolute offset
2026-02-06 14:58:56 +08:00
705c5d5547
hotfix: ci will build tools & fix local_builder ( #39 )
...
* hotfix: ci add make_all & fix local_builder
* try to fix pythonbinding error
* mac ci doesn't need activation
* fix linux
* sync mac ci
2026-01-27 17:34:18 +08:00
0ebf5621a6
feat: support core cpp sdk ( #30 )
...
* support core cpp sdk
* fix
* fix
* fix
* fix cr
2026-01-26 20:01:44 +08:00
5e2492b2cb
hot fix: local builder should use new flat param string ( #37 )
2026-01-26 17:28:38 +08:00
e36a7de157
fix: local_builder DisableIdMap=true auto set streamer params ( #34 )
2026-01-26 16:23:10 +08:00
e957442355
feat: refact cpp sdk ( #27 )
...
* refact cpp sdk
---------
Co-authored-by: zhouqinren.zqr <zhouqinren.zqr@alibaba-inc.com>
2026-01-21 20:05:42 +08:00
3335c1f6d1
upd mmap segment_meta_capacity ( #24 )
...
* upd mmap segment_meta_capacity
* clang format
2026-01-15 11:36:46 +08:00
220a9aba11
chore:add git commit msg and branch name in pre-commit and modify org ( #1 )
...
* chore: add git commit msg and branch name in pre-commit
* docs: modify org to alibaba
* ci: mac build with make
2025-12-30 15:39:05 +08:00
6524fabd80
Initial commit
2025-12-30 11:02:17 +08:00
ZeFeng Yin
rayx
luoxiaojian
feihongxu0824
Zhuanglin Zheng
egolearner
Jalin Wang
iaojnh
Cuiys
sanyi