Extract the shared train+attach logic into
SegmentHelper::PrepareQuantizeField, used by both
SegmentImpl::create_vector_index and SegmentHelper::ReduceVectorIndex.
When add_column is called on a multi-segment collection with a nullable
field and no expression, segment.cc previously sliced an Arrow ChunkedArray
with an offset that exceeded the array length, triggering SIGABRT in
Arrow's chunked_array.cc:170 assertion.
Fix the slicing logic in segment.cc to materialize null values per segment.
Add comprehensive tests in collection_test.cc and segment_test.cc covering
multi-segment add_column scenarios (nullable/non-nullable, with/without
expression, with/without unflushed data, drop+re-add).
Add modernize-use-override to .clang-tidy and apply fixes across
src/ and tests/: replace redundant virtual with override, annotate
missing override on derived methods, and drop redundant virtual on
already-overridden methods.
When a nullable scalar field has no inverted index, the forward filter path
fails to handle null values from Arrow's filter evaluation:
1. get_forward_bit(): BooleanArray::operator[] returns nullopt for null entries,
which is_filtered() treats as "no filter" (not filtered), letting null docs
through. Fix: use value_or(false) to treat null as "not matched".
2. is_matched_by_forward_filter(): reads BooleanScalar.value without checking
is_valid, which is UB for null scalars. Fix: check is_valid first.
Closes#409
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite file/path handling to use std::filesystem and UTF-8-safe helpers.
Switch Windows file open/create paths to wide-char APIs, replace manual
separator concatenation with PathJoin, and enable RocksDB UTF-8 filenames.
Also add UTF-8 path coverage for file IO, version manager recovery, and
collection open/flush/reopen flows.
fixes#328
Problem:
SQ8 metadata (squared_sum, sum) was computed from pre-rounded float
values, causing mismatch with actual stored int8 values. On asymmetric
datasets (e.g. OpenAI 1536D where |x_min| >> x_max), this leads to
severe recall drop.
Solution:
Move std::round before accumulating squared_sum and sum.
Co-authored-by: rayx <rui.xing@alibaba-inc.com>
CreateDirectoryA("C:", nullptr) returns ERROR_ACCESS_DENIED on Windows
(not ERROR_ALREADY_EXISTS), causing MakePath to fail immediately for any
path on the C: drive. Skip the CreateDirectoryA call when the intermediate
path is a bare drive root (e.g. "C:").
* init branch
Add GitHub Actions release workflow for C API
add release linux-arm64
remove c api version
refact some code
add api reference
* feat(c-api): add nullable/doc-result APIs for agency migration (#234)
* Flattened index parameters structure
* refact c api code
* remove RAII guard
* refact use opaque pointer pattern
* refact version info
* refact use pure opaque pointers
* use typedef instead of enum
* fix set_last_error
* fix build yml
* fix doc.h
* remove wheel exclude
* fix c msvc link
* remove release yml
* fix c link libstdc++
* fix c link libgcc
---------
Co-authored-by: Donny/강동윤 <kdy.1997.dev@gmail.com>