Commit Graph
72 Commits
Author SHA1 Message Date
rayxandZefeng Yin ec8a78ee08 refactor(diskann): decouple from libaio via dlopen (#532)
Co-authored-by: Zefeng Yin <yinzefeng.yzf@alibaba-inc.com>
2026-07-16 17:58:25 +08:00
rayx 8693cf9de7 refactor: turbo quantizer (#546) 2026-07-15 19:54:07 +08:00
ZeFeng Yin f9ef15a362 chore: some optimize in ivf index (#528) 2026-07-14 10:55:51 +08:00
Jalin Wangandjiliang.ljl 1afdea8dc5 feat(python): expose group-by search to Python API (#561)
Co-authored-by: jiliang.ljl <jiliang.ljl@alibaba-inc.com>
2026-07-13 20:22:07 +08:00
Jalin Wang 468c565f86 feat(core, indexer): support group_by searching and fix hnsw sparse (#527) 2026-07-07 17:15:12 +08:00
egolearner 54340bb705 fix: IVF nprobe selected list scanning (#544) 2026-07-03 17:17:37 +08:00
c54f5e16e9 rotate: add an optional random rotation feature in INT8/INT4 quantization method (#483)
Co-authored-by: rayx <rui.xing@alibaba-inc.com>
Co-authored-by: Jalin Wang <wangjianning.wjn@alibaba-inc.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-06-26 19:16:19 +08:00
Jalin Wang cdef8686c7 fix(diskann, hnsw sparse builder): lost-wakeup race in builder progress loops (#530) 2026-06-26 16:35:24 +08:00
ZeFeng Yin cb0d3dbdb3 fix: some bugs in ivf index (#524) 2026-06-24 21:39:27 +08:00
Jalin Wang 85cd08d383 feat(storage): expose copy-on-write mmap flag and fix MMAP_POPULATE flag placement
This PR exposes a copy-on-write mmap option through the public StorageOptions API and fixes a bug where the MAP_POPULATE flag was applied to the wrong mmap() argument.
2026-06-17 14:18:12 +08:00
ZeFeng Yin e12bad5e37 feat: external vector source support (#490) 2026-06-17 14:15:52 +08:00
luoxiaojian 6df823f804 fix(hnsw): restore persisted info on reopen for UniformInt8 quantizer (#496) 2026-06-17 11:28:36 +08:00
Jalin Wang b1af0e4ebf feat(index): expose is_dirty status through core Index interface (#488)
Adds is_dirty() at each layer: IndexMapping → IndexStorage (virtual) → MMapFileStorage / BufferStorage → Index::IsDirty().
2026-06-17 11:20:40 +08:00
feihongxu0824 841b1ed3f9 fix: refactor parquet buffer cache ownership (#492) 2026-06-16 10:47:07 +08:00
luoxiaojian 7a5f84e648 feat: pass prefetch config (PO and PL) as search params (#482) 2026-06-12 16:33:41 +08:00
egolearner 7c4571b8b9 fix: fix missing include in block_heap.h (#491) 2026-06-11 13:34:53 +08:00
rayx 719e1cdb78 feat: support to pass in list size (#487) 2026-06-10 16:51:48 +08:00
feihongxu0824 4679a8f39d feat: support zvec core-only build (#481) 2026-06-09 23:45:42 +08:00
luoxiaojian da39a33feb feat(quantizer): introduce UniformInt8 quantizer with global scale/bias (#474) 2026-06-09 20:10:53 +08:00
luoxiaojian dd4d8117df feat(entity/search): add LinearPool/BlockHeap, refac entity layout and access. (#450)
* feat(entity/search): add LinearPool/BlockHeap, split contiguous entity layout and add direct-pointer fast search path

Introduce single-heap candidate structures (LinearPool/BlockHeap) and
refactor greedy search to dispatch between pool and dual-heap paths.
Split node layout into separate vector/graph arrays for better cache
locality, add zero-copy get_vector_ptr() on the hot path, and extract
huge-page allocation into MemoryHelper. Also adds pyglass attribution.
2026-06-08 13:51:46 +08:00
feihongxu0824 faadf62bd5 fix: use project root for diskann cmake paths (#464) 2026-06-08 09:54:16 +08:00
luoxiaojian 823b5e6824 fix: resolve compiler warnings and enable -Werror across all CI platforms (#460) 2026-06-05 14:51:37 +08:00
rayx e720c1fd20 feat: add diskann index (#369) 2026-06-04 20:52:43 +08:00
Jalin Wang 95e5ad5105 perf(segment): reuse first vector index file as merge base during compaction (#440)
Add a fast path that copies the first segment's index file as the merge base and only merges the tail segments into it. Limited to streaming indexes (HNSW, HNSW_RABITQ, FLAT) with matching index_type + quantize_type and no filter; IVF/VAMANA always rebuild (their Merge is dump-then-reopen and would drop the base docs).
Also, fix the incorrect concurrency of the compaction task.
2026-06-03 14:22:07 +08:00
luoxiaojian 45a11212d2 fix(vamana): medoid entry, concurrent build crash and prune quality. (#433)
* fix(vamana): medoid entry point, concurrent build crash and prune quality

- Add DiskANN-standard medoid (centroid-closest point) as the persisted
  entry point, computed at dump time, replacing the fixed node-0 start.
- Fix data races on node_chunks_ / dist_chunks_ / node_chunk_bases_ during
  concurrent build: add mutexes with double-checked locking and pre-reserve
  capacity so the lock-free read fast path never hits reallocation.
- Fix RobustPrune for metrics with signed internal distance (quantized int8
  cosine): introduce IndexMetric::build_distance_offset() to shift distances
  into a non-negative range, restoring a geometrically meaningful
  occlude_factor and improving graph quality / low-ef recall.
2026-06-02 16:15:57 +08:00
egolearner 3d6906ca20 refactor: apply modernize-use-override across codebase (#442)
- Remove redundant `virtual` keyword from methods already marked `override`
- Replace `virtual ~Derived()` with `~Derived() override` for derived class destructors
- Add missing `override` on methods that override base class virtuals
2026-06-02 13:49:39 +08:00
egolearner 23a1ef815e fix: validate reformer/entity ex_bits consistency in HnswRabitqStreamer (#436)
When a RabitqReformer trained with one total_bits is reused for an entity
configured with a different total_bits, the quantized data layout (ex_code
size) silently mismatches. This causes get_full_est() to interpret binary
quantization codes as floating-point factors, producing garbage distances
that sporadically drop search results.

Add an ex_bits() accessor to RabitqReformer and validate it against the
entity's ex_bits in HnswRabitqStreamer::open(), returning IndexError_Mismatch
on inconsistency. Fix the HNSWRabitqGeneral test to re-create a converter
and reformer with matching total_bits=2 for the third invocation. Add
TestExBitsMismatch to verify the mismatch is correctly rejected.
2026-06-02 11:12:15 +08:00
ZeFeng Yin 0807adeec5 fix(buffer_storage): vector block leak (#434) 2026-06-02 10:19:38 +08:00
ZeFeng Yin 74beb2a828 feat: buffer storage write (#414) 2026-06-01 10:40:49 +08:00
ZeFeng Yin 0cb2d8830e feat: small block(4K) read for 2M chunk size (#406) 2026-05-26 11:27:06 +08:00
ZeFeng Yin d9b0920ac7 fix: ivf provider sorted by local id (#422) 2026-05-26 10:33:44 +08:00
egolearner 9aae7494ad chore: enable modernize-use-override and fix existing violations (#419)
Add modernize-use-override to .clang-tidy and apply fixes across
src/ and tests/: replace redundant virtual with override, annotate
missing override on derived methods, and drop redundant virtual on
already-overridden methods.
2026-05-21 19:05:32 +08:00
ZeFeng Yin bae0e1b763 chore: fix nightly coverage run-error (#417) 2026-05-21 11:47:49 +08:00
egolearner 9bfc4b556a chore: fix more warnings (#418) 2026-05-21 11:15:41 +08:00
Jasonandrayx d580c08187 fix: correct format specifiers for uint64_t types (#374)
Signed-off-by: syaojun <libevent@yeah.net>
Co-authored-by: rayx <rui.xing@alibaba-inc.com>
2026-05-19 13:45:32 +08:00
rayx 3e02031eb8 refactor: remove hamming metric (#365)
* refactor: remove binary metric

* fix: fix windows
2026-05-18 19:31:20 +08:00
ZeFeng Yin ee40e5e0b7 fix hnsw context fetch (#386) 2026-05-11 11:17:40 +08:00
ZeFeng Yin d02ae2b74e fix: rollback hnsw ChunkSize to 2M (#383) 2026-05-07 11:00:00 +08:00
luoxiaojian efab064676 feat: refac entity and impl Vamana. (#371) 2026-04-30 10:22:25 +08:00
ZeFeng Yin 005680522b feat: merge vector arrow buffer (#320) 2026-04-29 19:24:50 +08:00
ZeFeng Yin 07e986d3d5 fix: hnsw chunk size init (#372) 2026-04-29 10:50:16 +08:00
egolearner 2b16ed39cd chore: rm hnsw-rabitq searcher/streamer (#367)
* chore: rm hnsw-rabitq searcher/streamer

* fix clang-tidy

* fix
2026-04-23 10:20:46 +08:00
ZeFeng Yin 4a4c02c1a5 chore: rm hnsw builder/searcher (#356) 2026-04-21 10:44:06 +08:00
Jalin Wang 1492175906 fix: add GetLastErrorString() helper for accurate OS error messages (#341) 2026-04-16 15:30:47 +08:00
Zihao Wangandrayx d29bffca39 fix(quantizer): use rounded int8 values for SQ8 metadata to fix recall drop (#329)
fixes #328

Problem:
SQ8 metadata (squared_sum, sum) was computed from pre-rounded float
values, causing mismatch with actual stored int8 values. On asymmetric
datasets (e.g. OpenAI 1536D where |x_min| >> x_max), this leads to
severe recall drop.

Solution:
Move std::round before accumulating squared_sum and sum.

Co-authored-by: rayx <rui.xing@alibaba-inc.com>
2026-04-16 10:07:14 +08:00
Jalin Wang 8182cfff93 feat: windows support (#216)
Preliminary support for Windows with remaining TODOs
2026-03-30 20:47:34 +08:00
Qinren Zhou ae345ad070 fix: recovery from crash during optimization (#246) 2026-03-24 10:53:16 +08:00
rayx 31aaa2febb feat: add euclidean one2many (#188)
* add euclidean one2many implementation
2026-03-23 20:20:48 +08:00
egolearner e5ba11b6fe feat: add hnsw-rabitq support (#69)
* feat: add hnsw-rabitq

* undefine transform

* support config rotator type

* support sample_count

* fix ut

* refactor: update interface

* refactor: update interface

* update rabitq index params

* fix interface update

* fix searcher test

* fix streamer test

* streamer support bf and add more ut

* rm env check

* add collection ut

* add schema check

* add rabitq query param binding

* add integration test

* fix local_builder

* cleanup dist calculator

* add RaBitQ-Library submodule

* cleanup rabitq converter/reformer

* add files

* disable build on mac

* disable Feature_Optimize_HNSW_RABITQ

* disable python/tests/test_collection_hnsw_rabitq.py:13

* check avx2/avx512

* fix refine

* fix mac ci

* add dimension check

* fix mac ci

* fix ci

* add missing lib

* fix centroids selection for Cosine/InnerProduct

* rename hnsw-rabitq to hnsw_rabitq

* address comments

* fix compile

* fix rabitqlib name

* fix mac compile

* check avx2/avx512 support

* runtime check again compile-time

* check AUTO_DETECT_ARCH

* rm debug log

* fix

* rabitq use avx2 by default

* add missing file

* fix search_bf/group_by dist

* address comments

* fix typo

* address comments

* rabitq support disable id_map

* address comments
2026-03-19 17:21:42 +08:00
ZeFeng Yin 432de52c5f [enhance] buffer storage (#83) 2026-03-19 15:24:46 +08:00