Commit Graph
60 Commits
Author SHA1 Message Date
Jalin Wang b1af0e4ebf feat(index): expose is_dirty status through core Index interface (#488)
Adds is_dirty() at each layer: IndexMapping → IndexStorage (virtual) → MMapFileStorage / BufferStorage → Index::IsDirty().
2026-06-17 11:20:40 +08:00
feihongxu0824 841b1ed3f9 fix: refactor parquet buffer cache ownership (#492) 2026-06-16 10:47:07 +08:00
luoxiaojian 7a5f84e648 feat: pass prefetch config (PO and PL) as search params (#482) 2026-06-12 16:33:41 +08:00
egolearner 7c4571b8b9 fix: fix missing include in block_heap.h (#491) 2026-06-11 13:34:53 +08:00
rayx 719e1cdb78 feat: support to pass in list size (#487) 2026-06-10 16:51:48 +08:00
feihongxu0824 4679a8f39d feat: support zvec core-only build (#481) 2026-06-09 23:45:42 +08:00
luoxiaojian da39a33feb feat(quantizer): introduce UniformInt8 quantizer with global scale/bias (#474) 2026-06-09 20:10:53 +08:00
luoxiaojian dd4d8117df feat(entity/search): add LinearPool/BlockHeap, refac entity layout and access. (#450)
* feat(entity/search): add LinearPool/BlockHeap, split contiguous entity layout and add direct-pointer fast search path

Introduce single-heap candidate structures (LinearPool/BlockHeap) and
refactor greedy search to dispatch between pool and dual-heap paths.
Split node layout into separate vector/graph arrays for better cache
locality, add zero-copy get_vector_ptr() on the hot path, and extract
huge-page allocation into MemoryHelper. Also adds pyglass attribution.
2026-06-08 13:51:46 +08:00
feihongxu0824 faadf62bd5 fix: use project root for diskann cmake paths (#464) 2026-06-08 09:54:16 +08:00
luoxiaojian 823b5e6824 fix: resolve compiler warnings and enable -Werror across all CI platforms (#460) 2026-06-05 14:51:37 +08:00
rayx e720c1fd20 feat: add diskann index (#369) 2026-06-04 20:52:43 +08:00
Jalin Wang 95e5ad5105 perf(segment): reuse first vector index file as merge base during compaction (#440)
Add a fast path that copies the first segment's index file as the merge base and only merges the tail segments into it. Limited to streaming indexes (HNSW, HNSW_RABITQ, FLAT) with matching index_type + quantize_type and no filter; IVF/VAMANA always rebuild (their Merge is dump-then-reopen and would drop the base docs).
Also, fix the incorrect concurrency of the compaction task.
2026-06-03 14:22:07 +08:00
luoxiaojian 45a11212d2 fix(vamana): medoid entry, concurrent build crash and prune quality. (#433)
* fix(vamana): medoid entry point, concurrent build crash and prune quality

- Add DiskANN-standard medoid (centroid-closest point) as the persisted
  entry point, computed at dump time, replacing the fixed node-0 start.
- Fix data races on node_chunks_ / dist_chunks_ / node_chunk_bases_ during
  concurrent build: add mutexes with double-checked locking and pre-reserve
  capacity so the lock-free read fast path never hits reallocation.
- Fix RobustPrune for metrics with signed internal distance (quantized int8
  cosine): introduce IndexMetric::build_distance_offset() to shift distances
  into a non-negative range, restoring a geometrically meaningful
  occlude_factor and improving graph quality / low-ef recall.
2026-06-02 16:15:57 +08:00
egolearner 3d6906ca20 refactor: apply modernize-use-override across codebase (#442)
- Remove redundant `virtual` keyword from methods already marked `override`
- Replace `virtual ~Derived()` with `~Derived() override` for derived class destructors
- Add missing `override` on methods that override base class virtuals
2026-06-02 13:49:39 +08:00
egolearner 23a1ef815e fix: validate reformer/entity ex_bits consistency in HnswRabitqStreamer (#436)
When a RabitqReformer trained with one total_bits is reused for an entity
configured with a different total_bits, the quantized data layout (ex_code
size) silently mismatches. This causes get_full_est() to interpret binary
quantization codes as floating-point factors, producing garbage distances
that sporadically drop search results.

Add an ex_bits() accessor to RabitqReformer and validate it against the
entity's ex_bits in HnswRabitqStreamer::open(), returning IndexError_Mismatch
on inconsistency. Fix the HNSWRabitqGeneral test to re-create a converter
and reformer with matching total_bits=2 for the third invocation. Add
TestExBitsMismatch to verify the mismatch is correctly rejected.
2026-06-02 11:12:15 +08:00
ZeFeng Yin 0807adeec5 fix(buffer_storage): vector block leak (#434) 2026-06-02 10:19:38 +08:00
ZeFeng Yin 74beb2a828 feat: buffer storage write (#414) 2026-06-01 10:40:49 +08:00
ZeFeng Yin 0cb2d8830e feat: small block(4K) read for 2M chunk size (#406) 2026-05-26 11:27:06 +08:00
ZeFeng Yin d9b0920ac7 fix: ivf provider sorted by local id (#422) 2026-05-26 10:33:44 +08:00
egolearner 9aae7494ad chore: enable modernize-use-override and fix existing violations (#419)
Add modernize-use-override to .clang-tidy and apply fixes across
src/ and tests/: replace redundant virtual with override, annotate
missing override on derived methods, and drop redundant virtual on
already-overridden methods.
2026-05-21 19:05:32 +08:00
ZeFeng Yin bae0e1b763 chore: fix nightly coverage run-error (#417) 2026-05-21 11:47:49 +08:00
egolearner 9bfc4b556a chore: fix more warnings (#418) 2026-05-21 11:15:41 +08:00
Jasonandrayx d580c08187 fix: correct format specifiers for uint64_t types (#374)
Signed-off-by: syaojun <libevent@yeah.net>
Co-authored-by: rayx <rui.xing@alibaba-inc.com>
2026-05-19 13:45:32 +08:00
rayx 3e02031eb8 refactor: remove hamming metric (#365)
* refactor: remove binary metric

* fix: fix windows
2026-05-18 19:31:20 +08:00
ZeFeng Yin ee40e5e0b7 fix hnsw context fetch (#386) 2026-05-11 11:17:40 +08:00
ZeFeng Yin d02ae2b74e fix: rollback hnsw ChunkSize to 2M (#383) 2026-05-07 11:00:00 +08:00
luoxiaojian efab064676 feat: refac entity and impl Vamana. (#371) 2026-04-30 10:22:25 +08:00
ZeFeng Yin 005680522b feat: merge vector arrow buffer (#320) 2026-04-29 19:24:50 +08:00
ZeFeng Yin 07e986d3d5 fix: hnsw chunk size init (#372) 2026-04-29 10:50:16 +08:00
egolearner 2b16ed39cd chore: rm hnsw-rabitq searcher/streamer (#367)
* chore: rm hnsw-rabitq searcher/streamer

* fix clang-tidy

* fix
2026-04-23 10:20:46 +08:00
ZeFeng Yin 4a4c02c1a5 chore: rm hnsw builder/searcher (#356) 2026-04-21 10:44:06 +08:00
Jalin Wang 1492175906 fix: add GetLastErrorString() helper for accurate OS error messages (#341) 2026-04-16 15:30:47 +08:00
Zihao Wangandrayx d29bffca39 fix(quantizer): use rounded int8 values for SQ8 metadata to fix recall drop (#329)
fixes #328

Problem:
SQ8 metadata (squared_sum, sum) was computed from pre-rounded float
values, causing mismatch with actual stored int8 values. On asymmetric
datasets (e.g. OpenAI 1536D where |x_min| >> x_max), this leads to
severe recall drop.

Solution:
Move std::round before accumulating squared_sum and sum.

Co-authored-by: rayx <rui.xing@alibaba-inc.com>
2026-04-16 10:07:14 +08:00
Jalin Wang 8182cfff93 feat: windows support (#216)
Preliminary support for Windows with remaining TODOs
2026-03-30 20:47:34 +08:00
Qinren Zhou ae345ad070 fix: recovery from crash during optimization (#246) 2026-03-24 10:53:16 +08:00
rayx 31aaa2febb feat: add euclidean one2many (#188)
* add euclidean one2many implementation
2026-03-23 20:20:48 +08:00
egolearner e5ba11b6fe feat: add hnsw-rabitq support (#69)
* feat: add hnsw-rabitq

* undefine transform

* support config rotator type

* support sample_count

* fix ut

* refactor: update interface

* refactor: update interface

* update rabitq index params

* fix interface update

* fix searcher test

* fix streamer test

* streamer support bf and add more ut

* rm env check

* add collection ut

* add schema check

* add rabitq query param binding

* add integration test

* fix local_builder

* cleanup dist calculator

* add RaBitQ-Library submodule

* cleanup rabitq converter/reformer

* add files

* disable build on mac

* disable Feature_Optimize_HNSW_RABITQ

* disable python/tests/test_collection_hnsw_rabitq.py:13

* check avx2/avx512

* fix refine

* fix mac ci

* add dimension check

* fix mac ci

* fix ci

* add missing lib

* fix centroids selection for Cosine/InnerProduct

* rename hnsw-rabitq to hnsw_rabitq

* address comments

* fix compile

* fix rabitqlib name

* fix mac compile

* check avx2/avx512 support

* runtime check again compile-time

* check AUTO_DETECT_ARCH

* rm debug log

* fix

* rabitq use avx2 by default

* add missing file

* fix search_bf/group_by dist

* address comments

* fix typo

* address comments

* rabitq support disable id_map

* address comments
2026-03-19 17:21:42 +08:00
ZeFeng Yin 432de52c5f [enhance] buffer storage (#83) 2026-03-19 15:24:46 +08:00
rayx d4ffbf06af fix: fix metric batch (#245) 2026-03-19 13:39:10 +08:00
luoxiaojian 840bcdad26 feat: Introduce turbo. (#243) 2026-03-18 20:43:18 +08:00
Qinren Zhou 3c1241f7c6 feat: enlarge indice size limit for sparse vectors (#229) 2026-03-16 17:12:24 +08:00
luoxiaojian 79387c237f feat: enable icelake and l2 batch distance for int8 quantization. (#213) 2026-03-13 17:26:22 +08:00
feihongxu0824 24d877da5e fix: Replace VLAs with std::vector for MSVC compatibility and stack safety (#190) 2026-03-03 10:07:21 +08:00
longwayandZeFeng Yin 4e420311df chore(flat): replace std::map with std::unordered_map for key_id_mapping_ (#153)
Reduces key lookup in get_id() from O(log N) to O(1) average, and index
load time from O(N log N) to O(N). Also lowers per-entry memory usage by
eliminating the three tree-node pointers (parent/left/right).

Co-authored-by: ZeFeng Yin <yinzefeng.yzf@alibaba-inc.com>
2026-02-28 13:10:38 +08:00
ZeFeng Yin ba6acbf48f fix: ivf optimize (#169) 2026-02-27 17:35:52 +08:00
lichen2015andyinzefeng.yzf e7ad7cc31e fix: combined indexer should use key instead of index (#87)
Co-authored-by: yinzefeng.yzf <yinzefeng.yzf@alibaba-inc.com>
2026-02-12 10:38:05 +08:00
Jalin Wang 78f08bd0cc feat: auto scalable segment meta section of mmap files (#67)
The underly core module only support a fixed-size segment due to the index file format where the section of segments meta limit the max num of data segments. This commit makes the meta section dynamically allocated and organized in a link-table manner to support much more segments.


* revert local_builder's meta cap setting

* feat: auto scalable segment meta section of mmap files

* Revert "feat: auto scalable segment meta section of mmap files"

This reverts commit 82c11e451544b912b901be33fae79e964a487b1f.

* v2: don't change to absolute offset
2026-02-06 14:58:56 +08:00
richyreachyandfeihongxu0824 a20d484e9e refactor(core): unify search invoke (#15)
Co-authored-by: feihongxu0824 <xufeihong.xfh@alibaba-inc.com>
2026-02-03 12:40:23 +08:00
姚军 ea9d957893 fix: some typos in hnsw index (#50) 2026-02-03 10:41:21 +08:00
iaojnh 8c1819ca26 change hnsw single dist to batch dist (#40) 2026-01-29 10:18:57 +08:00