Commit Graph
92 Commits
Author SHA1 Message Date
egolearner f336c5c955 fix: train rabitq converter in compact-path ReduceVectorIndex (#425)
Extract the shared train+attach logic into
SegmentHelper::PrepareQuantizeField, used by both
SegmentImpl::create_vector_index and SegmentHelper::ReduceVectorIndex.
2026-05-27 12:06:21 +08:00
ZeFeng Yin 0cb2d8830e feat: small block(4K) read for 2M chunk size (#406) 2026-05-26 11:27:06 +08:00
ZeFeng Yin d9b0920ac7 fix: ivf provider sorted by local id (#422) 2026-05-26 10:33:44 +08:00
lichen2015 bdf58fc2d2 fix: prevent SIGABRT when adding nullable column to multi-segment collection (#415) (#416)
When add_column is called on a multi-segment collection with a nullable
field and no expression, segment.cc previously sliced an Arrow ChunkedArray
with an offset that exceeded the array length, triggering SIGABRT in
Arrow's chunked_array.cc:170 assertion.

Fix the slicing logic in segment.cc to materialize null values per segment.
Add comprehensive tests in collection_test.cc and segment_test.cc covering
multi-segment add_column scenarios (nullable/non-nullable, with/without
expression, with/without unflushed data, drop+re-add).
2026-05-24 14:13:57 +08:00
Qinren Zhou d351637da8 fix: optimize allocated wrong names for vector index (#421) 2026-05-23 11:15:22 +08:00
egolearner 9aae7494ad chore: enable modernize-use-override and fix existing violations (#419)
Add modernize-use-override to .clang-tidy and apply fixes across
src/ and tests/: replace redundant virtual with override, annotate
missing override on derived methods, and drop redundant virtual on
already-overridden methods.
2026-05-21 19:05:32 +08:00
ZeFeng Yin bae0e1b763 chore: fix nightly coverage run-error (#417) 2026-05-21 11:47:49 +08:00
egolearner 9bfc4b556a chore: fix more warnings (#418) 2026-05-21 11:15:41 +08:00
Jasonandrayx d580c08187 fix: correct format specifiers for uint64_t types (#374)
Signed-off-by: syaojun <libevent@yeah.net>
Co-authored-by: rayx <rui.xing@alibaba-inc.com>
2026-05-19 13:45:32 +08:00
feihongxu0824andClaude Opus 4.6 a9008b0fd2 fix: nullable scalar field filter leaks null documents without inverted index (#410)
When a nullable scalar field has no inverted index, the forward filter path
fails to handle null values from Arrow's filter evaluation:

1. get_forward_bit(): BooleanArray::operator[] returns nullopt for null entries,
   which is_filtered() treats as "no filter" (not filtered), letting null docs
   through. Fix: use value_or(false) to treat null as "not matched".

2. is_matched_by_forward_filter(): reads BooleanScalar.value without checking
   is_valid, which is UB for null scalars. Fix: check is_valid first.

Closes #409

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-18 20:25:06 +08:00
rayx 3e02031eb8 refactor: remove hamming metric (#365)
* refactor: remove binary metric

* fix: fix windows
2026-05-18 19:31:20 +08:00
feihongxu0824 273ffc2ad8 enhance: release gil lock in collection bindings (#363) 2026-05-14 10:15:49 +08:00
Qinren Zhou e0b8331116 minor: revise error messages (#400) 2026-05-13 13:18:54 +08:00
ZeFeng Yin ee40e5e0b7 fix hnsw context fetch (#386) 2026-05-11 11:17:40 +08:00
Qinren Zhou cfe9eed2f8 minor: remove unused validation in c api (#388) 2026-05-08 19:04:45 +08:00
Jalin Wang 1d4ae0b1b5 refactor: support UTF-8 file paths via std::filesystem (#359)
Rewrite file/path handling to use std::filesystem and UTF-8-safe helpers.
Switch Windows file open/create paths to wide-char APIs, replace manual
separator concatenation with PathJoin, and enable RocksDB UTF-8 filenames.

Also add UTF-8 path coverage for file IO, version manager recovery, and
collection open/flush/reopen flows.
2026-05-08 17:44:22 +08:00
Qinren Zhou 68a497efdb fix: sparse vector indices should be ordered (#382) 2026-05-07 16:40:49 +08:00
ZeFeng Yin d02ae2b74e fix: rollback hnsw ChunkSize to 2M (#383) 2026-05-07 11:00:00 +08:00
luoxiaojian efab064676 feat: refac entity and impl Vamana. (#371) 2026-04-30 10:22:25 +08:00
ZeFeng Yin 005680522b feat: merge vector arrow buffer (#320) 2026-04-29 19:24:50 +08:00
ZeFeng Yin 07e986d3d5 fix: hnsw chunk size init (#372) 2026-04-29 10:50:16 +08:00
Qinren Zhou 4e7fa0f37f fix: some minor bugs (#370) 2026-04-23 16:24:59 +08:00
egolearner 2b16ed39cd chore: rm hnsw-rabitq searcher/streamer (#367)
* chore: rm hnsw-rabitq searcher/streamer

* fix clang-tidy

* fix
2026-04-23 10:20:46 +08:00
ZeFeng Yin 4a4c02c1a5 chore: rm hnsw builder/searcher (#356) 2026-04-21 10:44:06 +08:00
feihongxu0824 bbe4ca7bae minor: fix c example cmake and build c++ dynamic lib (#347) 2026-04-20 22:08:34 +08:00
1e25294c07 feat(ci): integrate clang-tidy for changed C/C++ files (#116)
* feat(ci): add clang-tidy checks for changed cpp files

* fix(ci): correct clang-tidy workflow heredoc indentation

* test: stabilize fp16 euclidean matrix comparison

* test: fix fp16 matrix CI and clang-tidy warnings

* ci: scope clang-tidy PR and filter compile-db files

* check nullptr only

* fix nullptr

* fix: format

* chore: ignore c

* fix: tests files

* fix: workflow

* fix: specify clang-tidy version

* fix: filter regex

* fix: more files

* fix: update version

* add thirdparty cache

* add pipeline dependency

* fix clang-tidy trigger

---------

Co-authored-by: kgeg401 <kgeg401@users.noreply.github.com>
Co-authored-by: jiliang.ljl <jiliang.ljl@alibaba-inc.com>
2026-04-20 20:14:21 +08:00
Qinren Zhou c17bd8876e fix: validate query_params type (#351) 2026-04-20 10:54:06 +08:00
Qinren Zhou f602ed30ce feat: enlarge topK limit (#348) 2026-04-17 17:51:07 +08:00
Jalin Wang 1492175906 fix: add GetLastErrorString() helper for accurate OS error messages (#341) 2026-04-16 15:30:47 +08:00
Jalin Wang 95d092e9b3 fix: relax collection path restriction (#340) 2026-04-16 15:26:15 +08:00
Zihao Wangandrayx d29bffca39 fix(quantizer): use rounded int8 values for SQ8 metadata to fix recall drop (#329)
fixes #328

Problem:
SQ8 metadata (squared_sum, sum) was computed from pre-rounded float
values, causing mismatch with actual stored int8 values. On asymmetric
datasets (e.g. OpenAI 1536D where |x_min| >> x_max), this leads to
severe recall drop.

Solution:
Move std::round before accumulating squared_sum and sum.

Co-authored-by: rayx <rui.xing@alibaba-inc.com>
2026-04-16 10:07:14 +08:00
Jalin Wang 09f93c5869 fix: skip drive root in Windows MakePath (#337)
CreateDirectoryA("C:", nullptr) returns ERROR_ACCESS_DENIED on Windows
(not ERROR_ALREADY_EXISTS), causing MakePath to fail immediately for any
path on the C: drive. Skip the CreateDirectoryA call when the intermediate
path is a bare drive root (e.g. "C:").
2026-04-14 19:01:37 +08:00
feihongxu0824 8df962171f chore: clang format collection.cc (#319) 2026-04-07 14:10:33 +08:00
feihongxu0824 1c90b57e17 chore: fix some error message (#306) 2026-04-07 10:17:18 +08:00
lichen2015 6ac9bb0177 fix: refact c struct name (#302) 2026-04-02 14:54:03 +08:00
Jalin Wang ca4b893755 fix: misc fix for Windows(BUILD_SHARED_LIBS, CI) (#296) 2026-04-02 09:23:35 +08:00
lichen2015 7af6a31ae8 fix: c api ut crash on window platform (#298)
* fix c api ut

* remove c query_by_group
2026-04-01 15:46:15 +08:00
lichen2015andDonny/강동윤 ed8c010cfd feat: add c language support (#167)
* init branch

Add GitHub Actions release workflow for C API

add release linux-arm64

remove c api version

refact some code

add api reference

* feat(c-api): add nullable/doc-result APIs for agency migration (#234)


* Flattened index parameters structure

* refact c api code

* remove RAII guard

* refact use opaque pointer pattern

* refact version info

* refact use pure opaque pointers

* use typedef instead of enum

* fix set_last_error

* fix build yml

* fix doc.h

* remove wheel exclude

* fix c msvc link

* remove release yml

* fix c link libstdc++

* fix c link libgcc

---------

Co-authored-by: Donny/강동윤 <kdy.1997.dev@gmail.com>
2026-03-31 18:00:08 +08:00
Jalin Wang 8182cfff93 feat: windows support (#216)
Preliminary support for Windows with remaining TODOs
2026-03-30 20:47:34 +08:00
Qinren Zhou cbe9ece72a minor: rewrite some log messages (#265) 2026-03-27 16:35:32 +08:00
Qinren Zhou ae345ad070 fix: recovery from crash during optimization (#246) 2026-03-24 10:53:16 +08:00
rayx 31aaa2febb feat: add euclidean one2many (#188)
* add euclidean one2many implementation
2026-03-23 20:20:48 +08:00
rayx b49833bf56 Refactor/add scalar dist function and setup cmake with detailed cpu flag (#238)
* fix: add scalar

* fix: add avx512fp16

* fix: cmake config
2026-03-20 09:53:48 +08:00
egolearner e5ba11b6fe feat: add hnsw-rabitq support (#69)
* feat: add hnsw-rabitq

* undefine transform

* support config rotator type

* support sample_count

* fix ut

* refactor: update interface

* refactor: update interface

* update rabitq index params

* fix interface update

* fix searcher test

* fix streamer test

* streamer support bf and add more ut

* rm env check

* add collection ut

* add schema check

* add rabitq query param binding

* add integration test

* fix local_builder

* cleanup dist calculator

* add RaBitQ-Library submodule

* cleanup rabitq converter/reformer

* add files

* disable build on mac

* disable Feature_Optimize_HNSW_RABITQ

* disable python/tests/test_collection_hnsw_rabitq.py:13

* check avx2/avx512

* fix refine

* fix mac ci

* add dimension check

* fix mac ci

* fix ci

* add missing lib

* fix centroids selection for Cosine/InnerProduct

* rename hnsw-rabitq to hnsw_rabitq

* address comments

* fix compile

* fix rabitqlib name

* fix mac compile

* check avx2/avx512 support

* runtime check again compile-time

* check AUTO_DETECT_ARCH

* rm debug log

* fix

* rabitq use avx2 by default

* add missing file

* fix search_bf/group_by dist

* address comments

* fix typo

* address comments

* rabitq support disable id_map

* address comments
2026-03-19 17:21:42 +08:00
ZeFeng Yin 432de52c5f [enhance] buffer storage (#83) 2026-03-19 15:24:46 +08:00
rayx d4ffbf06af fix: fix metric batch (#245) 2026-03-19 13:39:10 +08:00
luoxiaojian 840bcdad26 feat: Introduce turbo. (#243) 2026-03-18 20:43:18 +08:00
Qinren Zhou 3c1241f7c6 feat: enlarge indice size limit for sparse vectors (#229) 2026-03-16 17:12:24 +08:00
rayx de1e7050d7 fix/fix mips euclidean (#226)
* fix: fix mips euclidean
2026-03-16 13:41:58 +08:00
Abdur-Rahmaan Janhangeer 65b45470d6 fix: minor typo (#225) 2026-03-16 11:31:44 +08:00