* feat: add FTS support for Collection::CreateIndex/DropIndex
Enable dynamic creation and removal of FTS indexes on existing STRING
columns through the standard CreateIndex/DropIndex API, matching the
lifecycle model already used by vector and scalar (invert) indexes.
Key changes:
- New FtsIndexer class (fts_indexer.h/cc) encapsulating per-segment FTS
RocksDB management: multi-field lifecycle, snapshot, insert, seal
- New BlockType::FTS_INDEX with block_id-based directory naming
(fts.<block_id>.rocksdb) for crash-safe snapshot-and-swap
- Segment::create_fts_index builds FTS index on a snapshot copy by
scanning forward store, then outputs new SegmentMeta + FtsIndexer
for atomic reload (same pattern as create_scalar_index)
- Segment::drop_fts_index snapshots, removes field CFs, outputs updated
meta (or nullptr when last FTS field is removed)
- Collection layer wires FTS into the existing task dispatch, version
update, and reload loops alongside vector/invert paths
- CreateIndex/DropIndex reject unsupported index types explicitly
instead of falling through to the wrong branch
* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex
fix: address review comments for FTS CreateIndex/DropIndex
- Reject CreateIndex when column already has a different index type
(e.g. FTS on an INVERT-indexed column) at both Collection and
Segment layers
- Allow same-type different-params CreateIndex to rebuild the index
(remove old + create new + replay data), aligned with INVERT behavior
- Return OK when CreateIndex is called with identical params
- Rename operator[] to get() in both FtsIndexer and InvertedIndexer
- Add test cases: create→drop→create→drop cycle, and params-change
rebuild with case-sensitivity verification
* android skip Feature_CreateOrDropFtsIndex
* fixup! feat: add FTS support for Collection::CreateIndex/DropIndex
Add a fast path that copies the first segment's index file as the merge base and only merges the tail segments into it. Limited to streaming indexes (HNSW, HNSW_RABITQ, FLAT) with matching index_type + quantize_type and no filter; IVF/VAMANA always rebuild (their Merge is dump-then-reopen and would drop the base docs).
Also, fix the incorrect concurrency of the compaction task.
- Upgrade ilammy/msvc-dev-cmd from v1 to v1.13.0 to resolve Node.js 20
deprecation warning (affects 05-windows-build and build_wheel_on_windows)
- Remove /MP from antlr4 and protobuf patches to eliminate 332
non-cacheable sccache calls caused by "multiple input files"
- Add sccache statistics steps after pip install and after C++ Tests to
capture build and test compilation cache metrics separately
Previously execute() and execute_group_by() filled the query_infos
vector with copies of the same shared_ptr, so all segments pointed to
the identical QueryInfo object. The optimizer mutates QueryInfo
in-place (set_invert_cond, set_filter_cond, set_forward, set_text,
etc.), which meant optimizations applied for segment 0 silently
corrupted the input state for subsequent segments.
Build a separate QueryInfo via build_query_info() for each segment so
they can be optimized independently. The filter-satisfiability check
is query-global and done once before the loop.
feat(ci): enable sccache on Windows and propagate compiler launcher to ExternalProject
Windows CI builds were ~6-9x slower than other platforms (35-40 min vs
4-7 min) due to missing compiler cache. This commit enables sccache for
Windows and ensures all ExternalProject_Add dependencies (Arrow, lz4)
receive the compiler launcher on every platform.
Changes:
MSVC /Zi → /Z7 compatibility:
- Replace /Zi with /Z7 in root CMakeLists.txt for DEBUG and
RELWITHDEBINFO configurations (MSVC cache variable level)
- Patch rocksdb, antlr4, and googletest source CMakeLists to replace
hardcoded /Zi with /Z7 at the source, eliminating D9025 warnings
- Clear COMPILE_PDB_NAME on googletest targets to prevent CMake from
adding /Fd<path>, which causes sccache to look for nonexistent .pdb
- Remove /FS (forced synchronous PDB writes) from Arrow ExternalProject
MSVC flags — unnecessary with /Z7
Windows CI workflow:
- Add mozilla-actions/sccache-action@v0.0.10
- Set SCCACHE_GHA_ENABLED=true to use GitHub Actions Cache backend
- Pass CMAKE_C/CXX_COMPILER_LAUNCHER=sccache via --config-settings
Compiler launcher propagation:
- Propagate CMAKE_C/CXX_COMPILER_LAUNCHER to Arrow ExternalProject on
all platforms (Windows, Linux, macOS, Android, iOS)
- Propagate to lz4 ExternalProject on Windows via CMAKE_ARGS, using
CMP0141=NEW + CMAKE_MSVC_DEBUG_INFORMATION_FORMAT=Embedded for /Z7
* fix(vamana): medoid entry point, concurrent build crash and prune quality
- Add DiskANN-standard medoid (centroid-closest point) as the persisted
entry point, computed at dump time, replacing the fixed node-0 start.
- Fix data races on node_chunks_ / dist_chunks_ / node_chunk_bases_ during
concurrent build: add mutexes with double-checked locking and pre-reserve
capacity so the lock-free read fast path never hits reallocation.
- Fix RobustPrune for metrics with signed internal distance (quantized int8
cosine): introduce IndexMetric::build_distance_offset() to shift distances
into a non-negative range, restoring a geometrically meaningful
occlude_factor and improving graph quality / low-ef recall.
- Remove redundant `virtual` keyword from methods already marked `override`
- Replace `virtual ~Derived()` with `~Derived() override` for derived class destructors
- Add missing `override` on methods that override base class virtuals
* fix(cpu_features): refactor architecture detection to explicit x86 whitelist
Currently, `cpu_features.cc` assumes any non-ARM architecture is x86/x64, which leads to a fatal missing `<cpuid.h>` error on architectures like RISC-V.
This commit refactors the preprocessor macros to explicitly whitelist x86 architectures (`__x86_64__`, `__i386__`, `_M_X64`, `_M_IX86`). All other architectures (RISC-V, ARM, etc.) will now safely fall back to the default zero-initialization, allowing cross-compilation to succeed.
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: Add RISE RISC-V runner
Introduce the RISC-V CI runner provided by the RISE project.
This enables automated testing and building for the RISC-V architecture.
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: use python 3.12 for RISC-V64
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: add RISC-V numpy dependencies
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: add RISC-V wheel cache
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: use pre-built RISE numpy wheel to speed up riscv builds
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: use pre-built RISE cmake wheel to speed up riscv builds
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: split RISC-V build and test into separate jobs
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: fix
Signed-off-by: ihb2032 <hebome@foxmail.com>
* ci: fix
Signed-off-by: ihb2032 <hebome@foxmail.com>
* Update hnsw_streamer_test.cc
* ci: add cache
Signed-off-by: ihb2032 <hebome@foxmail.com>
* Update hnsw_streamer_test.cc
* Update test_gil_release.py
* ci: schedule workflow to run overnight
---------
Signed-off-by: ihb2032 <hebome@foxmail.com>
Co-authored-by: ZeFeng Yin <yinzefeng.yzf@alibaba-inc.com>
Windows.h defines ERROR as a macro which collides with LogLevel::ERROR.
Rename all LogLevel enum values to kDebug/kInfo/kWarn/kError/kFatal style,
and remove the now-unnecessary #undef ERROR workaround in jieba_tokenizer.
When a RabitqReformer trained with one total_bits is reused for an entity
configured with a different total_bits, the quantized data layout (ex_code
size) silently mismatches. This causes get_full_est() to interpret binary
quantization codes as floating-point factors, producing garbage distances
that sporadically drop search results.
Add an ex_bits() accessor to RabitqReformer and validate it against the
entity's ex_bits in HnswRabitqStreamer::open(), returning IndexError_Mismatch
on inconsistency. Fix the HNSWRabitqGeneral test to re-create a converter
and reformer with matching total_bits=2 for the third invocation. Add
TestExBitsMismatch to verify the mismatch is correctly rejected.
* feat: migrate multi-vector query and reranker logic to C++
- Add Reranker base class with RrfReRanker and WeightedReRanker implementations
- Add Collection::MultiQuery interface for multi-vector queries with reranking
- Add MultiVectorQuery struct in doc.h with forward declaration for Reranker
- Add C API bindings for reranker and MultiQuery (zvec_reranker_*, zvec_multi_vector_query_*, zvec_collection_multi_query)
- Add Python binding for reranker classes with py::function bridge for callback
- Validate duplicate field names in multi-vector queries (C++ and Python consistent)
- Remove TODO comment about concurrent execution (SQLEngine is not thread-safe)
- Update collection.h MultiQuery doc comment from concurrently to sequentially
- Add C++ collection tests (6 MultiQuery test cases)
- Add C API tests (reranker functions + multi_vector_query end-to-end)
- Implement Python test cases (11 previously skipped tests now active)
- Simplify Python query_executor validation for unified duplicate field check
* style: format Python files with ruff
* fix: adapt to main branch API changes (VectorQuery->Query rename, validate_and_sanitize)
* fix: multi_vector tests now use multiple same-type vector fields (dense2, sparse2)
* fix: suppress RET501 for intentional default return None in RerankFunction._get_object
* style: ruff format test_collection.py
* refact multi vector query
* format code
* fix(multi-vector): expose SubVectorQuery in Python binding, fix tests
- Register _SubVectorQuery in pybind11 with from_vector_query() factory
- Convert _VectorQuery to _SubVectorQuery in MultiVectorQueryExecutor
- Relax RRF/Weighted score assertion tolerance from 1e-10 to 1e-6
- Fix WeightedReRanker test metric to IP (matching HnswIndexParam default)
* style: ruff format query_executor.py
* fix: define _USE_MATH_DEFINES for M_PI on Windows (MSVC)
* refactor: include reranker.h directly in query.h instead of forward declaration
* refactor(reranker): move topn from member variable to rerank() parameter
* refact code
* style(python): fix ruff UP035/UP037 in multi_vector_reranker
- import Callable from collections.abc instead of typing (UP035)
- remove redundant quotes around MetricType annotations (UP037)
* chore: trigger PR sync
* refact code
* fix(examples): restore CMakeLists.txt formatting broken by clang-format
* refactor(reranker): remove redundant metrics_ map by querying schema directly, and use insert return value to avoid duplicate set lookup
* refactor(reranker): defer schema binding to query time and remove C API callback reranker
Extract the shared train+attach logic into
SegmentHelper::PrepareQuantizeField, used by both
SegmentImpl::create_vector_index and SegmentHelper::ReduceVectorIndex.
When add_column is called on a multi-segment collection with a nullable
field and no expression, segment.cc previously sliced an Arrow ChunkedArray
with an offset that exceeded the array length, triggering SIGABRT in
Arrow's chunked_array.cc:170 assertion.
Fix the slicing logic in segment.cc to materialize null values per segment.
Add comprehensive tests in collection_test.cc and segment_test.cc covering
multi-segment add_column scenarios (nullable/non-nullable, with/without
expression, with/without unflushed data, drop+re-add).
Add modernize-use-override to .clang-tidy and apply fixes across
src/ and tests/: replace redundant virtual with override, annotate
missing override on derived methods, and drop redundant virtual on
already-overridden methods.
Add SYSTEM to target_include_directories in vendored CMake wrappers and
set INTERFACE_SYSTEM_INCLUDE_DIRECTORIES on imported / sub-project
targets so warnings from third-party headers no longer surface in our
build output. Introduces a mark_target_includes_system() helper in
cmake/utils.cmake (resolves aliases, skips missing targets) used by the
upstream-add_subdirectory wrappers (glog, gflags, googletest, yaml-cpp,
protobuf, antlr).
The COVERAGE build type uses -O0 and gcov instrumentation, which slows
C++ query execution ~5-10x. This causes test_gil_released_during_query
to exceed its 0.5s timing threshold and fail with an inconclusive result.
When a nullable scalar field has no inverted index, the forward filter path
fails to handle null values from Arrow's filter evaluation:
1. get_forward_bit(): BooleanArray::operator[] returns nullopt for null entries,
which is_filtered() treats as "no filter" (not filtered), letting null docs
through. Fix: use value_or(false) to treat null as "not matched".
2. is_matched_by_forward_filter(): reads BooleanScalar.value without checking
is_valid, which is UB for null scalars. Fix: check is_valid first.
Closes#409
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Set unique WORKING_DIRECTORY per test binary via cc_test()/cuda_test() to prevent filesystem path conflicts when running tests in parallel. Each test runs in ${CMAKE_BINARY_DIR}/test_tmp/${test_name}/.
- Enable parallel ctest execution in the unittest target with ProcessorCount-based --parallel flag (defaults to NPROC - 1).
- Set TEST_BINARY_DIR environment variable for crash recovery tests so they can locate helper binaries from isolated working directories.
- Update LocateDataGenerator() and LocateOptimizeGenerator() to search TEST_BINARY_DIR and TEST_BINARY_DIR/bin for helper executables.
* ci: add ccache/sccache compilation caching to speed up CI builds
- Use hendrikmuhs/ccache-action@v1.2 for Linux/macOS/iOS/Android/clang-tidy
(auto-installs ccache, manages cache, sets env vars, shows stats)
- Use mozilla-actions/sccache-action@v0.0.9 for Windows (MSVC compatible)
- Add CMAKE_C/CXX_COMPILER_LAUNCHER to all CMake build steps
- Exclude wheel build and nightly coverage workflows per decision
* ci: switch MacOS & Linux build from Unix Makefiles to Ninja generator
- Replace CMAKE_GENERATOR='Unix Makefiles' with 'Ninja' in pip build
- Replace 'make unittest -j' with 'cmake --build --target unittest --parallel'
- Add '-G Ninja' to C++ and C example cmake configure steps
- Replace 'make -j' with 'cmake --build --parallel' for examples
- Aligns with Windows and Android workflows which already use Ninja
* ci: enable parallel ctest execution with -j and --timeout
- Use CMake ProcessorCount module to detect available CPU cores
- Add -j ${NPROC} to ctest command for parallel test execution
- Add --timeout 300 to prevent individual tests from hanging CI
- Fallback to NPROC=1 when ProcessorCount returns 0
- iOS target unchanged (build-only, no test execution)
* Revert "ci: enable parallel ctest execution with -j and --timeout"
This reverts commit d196dac5f17b1f7cae443360c88bbd8c935dc145.
* fix(ci): remove sccache from Windows, fix cmake.define quote issues
Windows (05-windows-build.yml):
- Remove mozilla-actions/sccache-action: sccache incompatible with MSVC /FS flag
- Remove SCCACHE_GHA_ENABLED env var
- Remove CMAKE_C/CXX_COMPILER_LAUNCHER=sccache from build steps
- Remove 'Show sccache statistics' step
- MSVC /FS (global PDB concurrency flag) causes fatal C1041 when used with sccache
MacOS & Linux (03-macos-linux-build.yml):
- Fix cmake.define values: remove extra quotes around 'ccache' and 'ON'
- Bare values required: cmake.define.FOO=bar not cmake.define.FOO="bar"
* feat: cache key with platform and os
* ci: add compiler to ccache key to avoid cache pollution
* ci: optimize cache usage to reduce bloat
- Add max-size limits to all ccache configs (150M general, 300M Android,
100M clang-tidy) to prevent unbounded cache growth
- Remove redundant iOS full build directory cache (~1.2 GB) since ccache
already handles incremental compilation
- Fix iOS protoc cache key to use thirdparty/protobuf/** instead of
src/**, avoiding unnecessary cache misses on business code changes
* ci: trigger CI run
Rewrite file/path handling to use std::filesystem and UTF-8-safe helpers.
Switch Windows file open/create paths to wide-char APIs, replace manual
separator concatenation with PathJoin, and enable RocksDB UTF-8 filenames.
Also add UTF-8 path coverage for file IO, version manager recovery, and
collection open/flush/reopen flows.
egolearner
Qinren Zhou
Jalin Wang
luoxiaojian
Cuiys
ihb2032
ZeFeng Yin
lichen2015
Jason
feihongxu0824
rayx