* fix(cpu_features): refactor architecture detection to explicit x86 whitelist
Currently, `cpu_features.cc` assumes any non-ARM architecture is x86/x64, which leads to a fatal missing `<cpuid.h>` error on architectures like RISC-V.
This commit refactors the preprocessor macros to explicitly whitelist x86 architectures (`__x86_64__`, `__i386__`, `_M_X64`, `_M_IX86`). All other architectures (RISC-V, ARM, etc.) will now safely fall back to the default zero-initialization, allowing cross-compilation to succeed.
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: Add RISE RISC-V runner
Introduce the RISC-V CI runner provided by the RISE project.
This enables automated testing and building for the RISC-V architecture.
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: use python 3.12 for RISC-V64
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: add RISC-V numpy dependencies
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: add RISC-V wheel cache
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: use pre-built RISE numpy wheel to speed up riscv builds
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: use pre-built RISE cmake wheel to speed up riscv builds
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: split RISC-V build and test into separate jobs
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: fix
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: fix
Signed-off-by: ihb2032 <hebome@foxmail.com>
* Update hnsw_streamer_test.cc
* ci: add cache
Signed-off-by: ihb2032 <hebome@foxmail.com>
* Update hnsw_streamer_test.cc
* Update test_gil_release.py
* ci: schedule workflow to run overnight
---------
Signed-off-by: ihb2032 <hebome@foxmail.com>
Co-authored-by: ZeFeng Yin <yinzefeng.yzf@alibaba-inc.com>
Windows.h defines ERROR as a macro which collides with LogLevel::ERROR.
Rename all LogLevel enum values to kDebug/kInfo/kWarn/kError/kFatal style,
and remove the now-unnecessary #undef ERROR workaround in jieba_tokenizer.
When a RabitqReformer trained with one total_bits is reused for an entity
configured with a different total_bits, the quantized data layout (ex_code
size) silently mismatches. This causes get_full_est() to interpret binary
quantization codes as floating-point factors, producing garbage distances
that sporadically drop search results.
Add an ex_bits() accessor to RabitqReformer and validate it against the
entity's ex_bits in HnswRabitqStreamer::open(), returning IndexError_Mismatch
on inconsistency. Fix the HNSWRabitqGeneral test to re-create a converter
and reformer with matching total_bits=2 for the third invocation. Add
TestExBitsMismatch to verify the mismatch is correctly rejected.
* feat: migrate multi-vector query and reranker logic to C++
- Add Reranker base class with RrfReRanker and WeightedReRanker implementations
- Add Collection::MultiQuery interface for multi-vector queries with reranking
- Add MultiVectorQuery struct in doc.h with forward declaration for Reranker
- Add C API bindings for reranker and MultiQuery (zvec_reranker_*, zvec_multi_vector_query_*, zvec_collection_multi_query)
- Add Python binding for reranker classes with py::function bridge for callback
- Validate duplicate field names in multi-vector queries (C++ and Python consistent)
- Remove TODO comment about concurrent execution (SQLEngine is not thread-safe)
- Update collection.h MultiQuery doc comment from concurrently to sequentially
- Add C++ collection tests (6 MultiQuery test cases)
- Add C API tests (reranker functions + multi_vector_query end-to-end)
- Implement Python test cases (11 previously skipped tests now active)
- Simplify Python query_executor validation for unified duplicate field check
* style: format Python files with ruff
* fix: adapt to main branch API changes (VectorQuery->Query rename, validate_and_sanitize)
* fix: multi_vector tests now use multiple same-type vector fields (dense2, sparse2)
* fix: suppress RET501 for intentional default return None in RerankFunction._get_object
* style: ruff format test_collection.py
* refact multi vector query
* format code
* fix(multi-vector): expose SubVectorQuery in Python binding, fix tests
- Register _SubVectorQuery in pybind11 with from_vector_query() factory
- Convert _VectorQuery to _SubVectorQuery in MultiVectorQueryExecutor
- Relax RRF/Weighted score assertion tolerance from 1e-10 to 1e-6
- Fix WeightedReRanker test metric to IP (matching HnswIndexParam default)
* style: ruff format query_executor.py
* fix: define _USE_MATH_DEFINES for M_PI on Windows (MSVC)
* refactor: include reranker.h directly in query.h instead of forward declaration
* refactor(reranker): move topn from member variable to rerank() parameter
* refact code
* style(python): fix ruff UP035/UP037 in multi_vector_reranker
- import Callable from collections.abc instead of typing (UP035)
- remove redundant quotes around MetricType annotations (UP037)
* chore: trigger PR sync
* refact code
* fix(examples): restore CMakeLists.txt formatting broken by clang-format
* refactor(reranker): remove redundant metrics_ map by querying schema directly, and use insert return value to avoid duplicate set lookup
* refactor(reranker): defer schema binding to query time and remove C API callback reranker
Extract the shared train+attach logic into
SegmentHelper::PrepareQuantizeField, used by both
SegmentImpl::create_vector_index and SegmentHelper::ReduceVectorIndex.
When add_column is called on a multi-segment collection with a nullable
field and no expression, segment.cc previously sliced an Arrow ChunkedArray
with an offset that exceeded the array length, triggering SIGABRT in
Arrow's chunked_array.cc:170 assertion.
Fix the slicing logic in segment.cc to materialize null values per segment.
Add comprehensive tests in collection_test.cc and segment_test.cc covering
multi-segment add_column scenarios (nullable/non-nullable, with/without
expression, with/without unflushed data, drop+re-add).
Add modernize-use-override to .clang-tidy and apply fixes across
src/ and tests/: replace redundant virtual with override, annotate
missing override on derived methods, and drop redundant virtual on
already-overridden methods.
When a nullable scalar field has no inverted index, the forward filter path
fails to handle null values from Arrow's filter evaluation:
1. get_forward_bit(): BooleanArray::operator[] returns nullopt for null entries,
which is_filtered() treats as "no filter" (not filtered), letting null docs
through. Fix: use value_or(false) to treat null as "not matched".
2. is_matched_by_forward_filter(): reads BooleanScalar.value without checking
is_valid, which is UB for null scalars. Fix: check is_valid first.
Closes#409
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Set unique WORKING_DIRECTORY per test binary via cc_test()/cuda_test() to prevent filesystem path conflicts when running tests in parallel. Each test runs in ${CMAKE_BINARY_DIR}/test_tmp/${test_name}/.
- Enable parallel ctest execution in the unittest target with ProcessorCount-based --parallel flag (defaults to NPROC - 1).
- Set TEST_BINARY_DIR environment variable for crash recovery tests so they can locate helper binaries from isolated working directories.
- Update LocateDataGenerator() and LocateOptimizeGenerator() to search TEST_BINARY_DIR and TEST_BINARY_DIR/bin for helper executables.
Rewrite file/path handling to use std::filesystem and UTF-8-safe helpers.
Switch Windows file open/create paths to wide-char APIs, replace manual
separator concatenation with PathJoin, and enable RocksDB UTF-8 filenames.
Also add UTF-8 path coverage for file IO, version manager recovery, and
collection open/flush/reopen flows.
fixes#328
Problem:
SQ8 metadata (squared_sum, sum) was computed from pre-rounded float
values, causing mismatch with actual stored int8 values. On asymmetric
datasets (e.g. OpenAI 1536D where |x_min| >> x_max), this leads to
severe recall drop.
Solution:
Move std::round before accumulating squared_sum and sum.
Co-authored-by: rayx <rui.xing@alibaba-inc.com>
* support ios build
* fix: ci
* fix: update build_ios.sh
* fix: pr
* fix: cmake minimum version
* fix: cmake minimum version
* fix: cmake minimum version in ci
* fix: add all test in ci
* fix: ci
* init branch
Add GitHub Actions release workflow for C API
add release linux-arm64
remove c api version
refact some code
add api reference
* feat(c-api): add nullable/doc-result APIs for agency migration (#234)
* Flattened index parameters structure
* refact c api code
* remove RAII guard
* refact use opaque pointer pattern
* refact version info
* refact use pure opaque pointers
* use typedef instead of enum
* fix set_last_error
* fix build yml
* fix doc.h
* remove wheel exclude
* fix c msvc link
* remove release yml
* fix c link libstdc++
* fix c link libgcc
---------
Co-authored-by: Donny/강동윤 <kdy.1997.dev@gmail.com>
ihb2032
egolearner
ZeFeng Yin
Qinren Zhou
lichen2015
feihongxu0824
rayx
Cuiys
Jalin Wang
luoxiaojian
kgeg401
Zihao Wang