* feat(entity/search): add LinearPool/BlockHeap, split contiguous entity layout and add direct-pointer fast search path
Introduce single-heap candidate structures (LinearPool/BlockHeap) and
refactor greedy search to dispatch between pool and dual-heap paths.
Split node layout into separate vector/graph arrays for better cache
locality, add zero-copy get_vector_ptr() on the hot path, and extract
huge-page allocation into MemoryHelper. Also adds pyglass attribution.
Add a fast path that copies the first segment's index file as the merge base and only merges the tail segments into it. Limited to streaming indexes (HNSW, HNSW_RABITQ, FLAT) with matching index_type + quantize_type and no filter; IVF/VAMANA always rebuild (their Merge is dump-then-reopen and would drop the base docs).
Also, fix the incorrect concurrency of the compaction task.
* fix(vamana): medoid entry point, concurrent build crash and prune quality
- Add DiskANN-standard medoid (centroid-closest point) as the persisted
entry point, computed at dump time, replacing the fixed node-0 start.
- Fix data races on node_chunks_ / dist_chunks_ / node_chunk_bases_ during
concurrent build: add mutexes with double-checked locking and pre-reserve
capacity so the lock-free read fast path never hits reallocation.
- Fix RobustPrune for metrics with signed internal distance (quantized int8
cosine): introduce IndexMetric::build_distance_offset() to shift distances
into a non-negative range, restoring a geometrically meaningful
occlude_factor and improving graph quality / low-ef recall.
- Remove redundant `virtual` keyword from methods already marked `override`
- Replace `virtual ~Derived()` with `~Derived() override` for derived class destructors
- Add missing `override` on methods that override base class virtuals
When a RabitqReformer trained with one total_bits is reused for an entity
configured with a different total_bits, the quantized data layout (ex_code
size) silently mismatches. This causes get_full_est() to interpret binary
quantization codes as floating-point factors, producing garbage distances
that sporadically drop search results.
Add an ex_bits() accessor to RabitqReformer and validate it against the
entity's ex_bits in HnswRabitqStreamer::open(), returning IndexError_Mismatch
on inconsistency. Fix the HNSWRabitqGeneral test to re-create a converter
and reformer with matching total_bits=2 for the third invocation. Add
TestExBitsMismatch to verify the mismatch is correctly rejected.
Add modernize-use-override to .clang-tidy and apply fixes across
src/ and tests/: replace redundant virtual with override, annotate
missing override on derived methods, and drop redundant virtual on
already-overridden methods.
fixes#328
Problem:
SQ8 metadata (squared_sum, sum) was computed from pre-rounded float
values, causing mismatch with actual stored int8 values. On asymmetric
datasets (e.g. OpenAI 1536D where |x_min| >> x_max), this leads to
severe recall drop.
Solution:
Move std::round before accumulating squared_sum and sum.
Co-authored-by: rayx <rui.xing@alibaba-inc.com>
Reduces key lookup in get_id() from O(log N) to O(1) average, and index
load time from O(N log N) to O(N). Also lowers per-entry memory usage by
eliminating the three tree-node pointers (parent/left/right).
Co-authored-by: ZeFeng Yin <yinzefeng.yzf@alibaba-inc.com>
The underly core module only support a fixed-size segment due to the index file format where the section of segments meta limit the max num of data segments. This commit makes the meta section dynamically allocated and organized in a link-table manner to support much more segments.
* revert local_builder's meta cap setting
* feat: auto scalable segment meta section of mmap files
* Revert "feat: auto scalable segment meta section of mmap files"
This reverts commit 82c11e451544b912b901be33fae79e964a487b1f.
* v2: don't change to absolute offset