Commit Graph
164 Commits
Author SHA1 Message Date
feihongxu0824 e89be98bb3 fix: optimize after crash recovery without opening wal (#600) 2026-07-17 16:56:21 +08:00
mrcs64andmrcs64 a5dfec6a65 fix(segment): propagate forward-store writer open failure instead of crashing (#579)
Co-authored-by: mrcs64 <9069178+mrcs64@users.noreply.github.com>
2026-07-17 16:13:50 +08:00
rayxandZefeng Yin ec8a78ee08 refactor(diskann): decouple from libaio via dlopen (#532)
Co-authored-by: Zefeng Yin <yinzefeng.yzf@alibaba-inc.com>
2026-07-16 17:58:25 +08:00
egolearner 23538ab876 fix(collection): preserve writing segment on DDL failure (#574) 2026-07-16 10:54:56 +08:00
rayx 8693cf9de7 refactor: turbo quantizer (#546) 2026-07-15 19:54:07 +08:00
ZeFeng Yin f9ef15a362 chore: some optimize in ivf index (#528) 2026-07-14 10:55:51 +08:00
Jalin Wangandjiliang.ljl 1afdea8dc5 feat(python): expose group-by search to Python API (#561)
Co-authored-by: jiliang.ljl <jiliang.ljl@alibaba-inc.com>
2026-07-13 20:22:07 +08:00
egolearner 4de76fdbde fix(fts): allow global doc id gaps during compaction (#588) 2026-07-13 10:54:30 +08:00
lichen2015andClaude 78ef197aaa feat(c_api): add DiskANN index type support to C API (#495)
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-10 16:11:09 +08:00
egolearner a90ec5b1d6 fix(fts): preserve zero-match filter semantics (#584) 2026-07-10 15:20:51 +08:00
egolearner 20f9be250c perf(fts): add block-max skip and score early-exit to ConjunctionIterator (#441) 2026-07-10 11:40:47 +08:00
egolearner 821a9333fc fix(index): snapshot read-only indexes without flush (#569) 2026-07-10 10:55:50 +08:00
egolearner 7449d7193b fix(fts): restore writing segment stats on reopen (#566) 2026-07-10 10:52:46 +08:00
egolearner df6af2f05d feat: Implement standard tokenizer based on UAX29 (#547) 2026-07-09 10:01:56 +08:00
egolearner 05524045f9 fix: clarify create path existence error [skip ci] (#559) 2026-07-09 08:13:53 +08:00
Jalin Wang 468c565f86 feat(core, indexer): support group_by searching and fix hnsw sparse (#527) 2026-07-07 17:15:12 +08:00
egolearner c35d24e215 feat(fts): add stemmer token filter based on Snowball 3.1.1 (#513) 2026-07-07 14:14:13 +08:00
egolearner 998a711434 feat(binding): use VectorViewClause zero-copy for Python dense vector query (#517) 2026-07-06 14:03:34 +08:00
egolearner 54340bb705 fix: IVF nprobe selected list scanning (#544) 2026-07-03 17:17:37 +08:00
egolearner 59e04ae4cc refactor: remove QueryParams::set_type() (#518) 2026-07-02 17:05:26 +08:00
egolearner b9ce030593 feat(fts): add UTF-8 support for tokenizer and token filters via utf8proc (#515) 2026-07-02 11:31:44 +08:00
lichen2015 a03ea929ed fix(collection): open LOCK file read-only when collection is read-only (#541) 2026-07-01 14:39:33 +08:00
c54f5e16e9 rotate: add an optional random rotation feature in INT8/INT4 quantization method (#483)
Co-authored-by: rayx <rui.xing@alibaba-inc.com>
Co-authored-by: Jalin Wang <wangjianning.wjn@alibaba-inc.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-06-26 19:16:19 +08:00
Jalin Wang cdef8686c7 fix(diskann, hnsw sparse builder): lost-wakeup race in builder progress loops (#530) 2026-06-26 16:35:24 +08:00
Jalin WangandCopilot Autofix powered by AI 0a008ebfbf minor(c_api): duplicated & mismatched definition (#525)
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-06-26 10:27:28 +08:00
Jalin Wang 40e91d7fc9 fix(sql engine): misused group_count/group_topk (#531) 2026-06-26 10:26:20 +08:00
ZeFeng Yin cb0d3dbdb3 fix: some bugs in ivf index (#524) 2026-06-24 21:39:27 +08:00
egolearner 22744f2c81 feat(c_api): add FTS support for sub-query (#520)
Add zvec_sub_query_set_fts and zvec_sub_query_set_fts_params to allow
setting FTS clause and FTS query parameters on sub-queries, mirroring
the existing zvec_vector_query_set_fts interface.

- zvec_sub_query_set_fts: set/clear FTS clause (copies payload)
- zvec_sub_query_set_fts_params: set FTS query params (takes ownership)
- Add unit test test_fts_wiring_on_sub_query
2026-06-24 10:12:39 +08:00
egolearner 9bd0c6eea9 fix(python): add missing FTS type annotations to stubs and wrappers (#497)
- Add FtsIndexParam/FtsQueryParam class stubs to model/param/__init__.pyi
- Add Fts/FtsIndexParam/FtsQueryParam to zvec/__init__.pyi exports
- Extend Query.param type to include FtsQueryParam
- Update Query._validate to allow fts + FtsQueryParam combination
- Extend FieldSchema.index_param type to support FtsIndexParam
- Extend Collection.create_index type to support FtsIndexParam
2026-06-23 16:14:01 +08:00
feihongxu0824 0ac294a9fd fix: clean up RocksDB files on drop_index so create_index can be repeated (#485) 2026-06-22 16:13:59 +08:00
Jalin Wang a055c7e0a2 test(index): add tests for index param builders and fix builder setters (#504) 2026-06-22 14:12:18 +08:00
egolearner aad5718112 feat(query): add VectorViewClause zero-copy path and unify validate (#478)
* feat(query): add VectorViewClause zero-copy path and unify validate

- Add VectorViewClause (string_view-based) as zero-copy counterpart to
  VectorClause; variant now holds VectorClause | VectorViewClause | FtsClause
- Add QueryTarget::get_vector_view() unified accessor via std::visit,
  returns optional<VectorViewClause> regardless of which variant is held
- Split validate_and_sanitize into QueryTarget::validate (read-only) +
  sanitize_sparse_vector (mutate); validate handles both VectorClause and
  VectorViewClause via get_vector_view()
- Collection::Query passes original request directly to sqlengine when no
  sparse sanitization is needed; only copies when sort is required
- Change build_query_info/BuildSQLInfoFromSearchQuery to take const
  SearchQuery& so VectorMatrixNode string_views point to caller's data
2026-06-22 11:32:26 +08:00
Jalin Wang 85cd08d383 feat(storage): expose copy-on-write mmap flag and fix MMAP_POPULATE flag placement
This PR exposes a copy-on-write mmap option through the public StorageOptions API and fixes a bug where the MAP_POPULATE flag was applied to the wrong mmap() argument.
2026-06-17 14:18:12 +08:00
ZeFeng Yin e12bad5e37 feat: external vector source support (#490) 2026-06-17 14:15:52 +08:00
luoxiaojian 6df823f804 fix(hnsw): restore persisted info on reopen for UniformInt8 quantizer (#496) 2026-06-17 11:28:36 +08:00
Jalin Wang b1af0e4ebf feat(index): expose is_dirty status through core Index interface (#488)
Adds is_dirty() at each layer: IndexMapping → IndexStorage (virtual) → MMapFileStorage / BufferStorage → Index::IsDirty().
2026-06-17 11:20:40 +08:00
feihongxu0824 841b1ed3f9 fix: refactor parquet buffer cache ownership (#492) 2026-06-16 10:47:07 +08:00
luoxiaojian 7a5f84e648 feat: pass prefetch config (PO and PL) as search params (#482) 2026-06-12 16:33:41 +08:00
egolearner 7c4571b8b9 fix: fix missing include in block_heap.h (#491) 2026-06-11 13:34:53 +08:00
egolearner 5aa1d7a82c fix(c_api): use IndexParams::clone() to simplify convert_c_index_params_to_cpp (#489)
convert_c_index_params_to_cpp used a whitelist switch over IndexType,
prone to miss for newly index types.

Replace the entire switch with a single clone() call (pure-virtual on
IndexParams base, implemented by all subclasses).
2026-06-11 11:04:28 +08:00
rayx 815a6783ff refactor: disable diskann index create on open (#479) 2026-06-10 21:00:22 +08:00
rayx 719e1cdb78 feat: support to pass in list size (#487) 2026-06-10 16:51:48 +08:00
feihongxu0824 4679a8f39d feat: support zvec core-only build (#481) 2026-06-09 23:45:42 +08:00
luoxiaojian da39a33feb feat(quantizer): introduce UniformInt8 quantizer with global scale/bias (#474) 2026-06-09 20:10:53 +08:00
Cuiys 0923f7c691 refactor: make Reranker stateless with std::variant value semantics (… (#471)
* refactor: make Reranker stateless with std::variant value semantics (#461)

Replace class hierarchy (Reranker/ScoreBasedReranker/RrfReranker/
WeightedReranker/CallbackReranker) with std::variant<RrfParams,
WeightedParams, CallbackParams> value type and a stateless free function
reranker::rerank().

Key changes:
- reranker.h: define RerankParams variant + reranker::rerank() API
- query.h: MultiQuery::reranker (shared_ptr) -> MultiQuery::rerank (value)
- schema.h: add CollectionSchema::get_field_ptr() returning FieldSchema::Ptr
- collection.cc: push field lookup to caller, pass vector<FieldSchema::Ptr>
- c_api: remove opaque zvec_reranker_t, add zvec_multi_query_set_rerank_*
- python binding: expose _RrfParams/_WeightedParams/_CallbackParams + setters
- python layer: WeightedReRanker(list[float]), remove Python rerank logic
- all tests updated to new interface

Benefits:
- Thread-safe by design: no mutable state, safe to share across threads
- Collection-decoupled: no bind_schema(), field info passed as parameter
- Simpler lifecycle: value semantics, no shared_ptr management

Closes #461

* chore: remove nightly_build.yml unrelated to reranker refactor

* chore: remove uv.lock unrelated to reranker refactor

* fix: raise ValueError when multi-query has no reranker

After the reranker stateless refactor the C++ MultiQuery rerank
strategy uses a std::variant with a default value, so the implicit
'reranker required' validation no longer triggered. Restore the
check in QueryExecutor._execute_multi_query so that a hybrid
(multi-query) request without a reranker raises ValueError.

* fix(reranker): use index_type FTS check for non-vector normalization

Replace dynamic_cast nullptr check with explicit IndexType::FTS check
and map FTS/BM25 positive scores to (0.0, 1.0) via 2*atan(score)/pi.

* refactor(reranker): move Params types into reranker namespace and qualify usages

Move RrfParams, WeightedParams, CallbackParams and RerankParams into the
zvec::reranker namespace, and add explicit reranker:: qualification at all
usage sites outside the reranker module (query.h, python/c bindings, tests).

* refactor(query): drop unused PendingQuery wrapper, use std::vector<SearchQuery> directly

* refactor(reranker): make _to_cpp_params non-abstract with default NotImplementedError

Remove @abstractmethod from RerankFunction._to_cpp_params and provide a
default implementation raising NotImplementedError. Drop the redundant
_to_cpp_params overrides from Qwen and Sentence rerankers since they use
the Python rerank path and don't need the C++ conversion.
2026-06-09 12:44:17 +08:00
Qinren Zhou e8b888f26b minor: fix python doc string for ivf index params (#473) 2026-06-09 11:41:03 +08:00
Qinren Zhou d96de9289e fix: validate query params types before performing a query (#476) 2026-06-08 20:02:43 +08:00
rayx 74b23ba4c6 refactor: make list size to 50 in diskann build stage (#475) 2026-06-08 17:39:42 +08:00
luoxiaojian dd4d8117df feat(entity/search): add LinearPool/BlockHeap, refac entity layout and access. (#450)
* feat(entity/search): add LinearPool/BlockHeap, split contiguous entity layout and add direct-pointer fast search path

Introduce single-heap candidate structures (LinearPool/BlockHeap) and
refactor greedy search to dispatch between pool and dual-heap paths.
Split node layout into separate vector/graph arrays for better cache
locality, add zero-copy get_vector_ptr() on the hot path, and extract
huge-page allocation into MemoryHelper. Also adds pyglass attribution.
2026-06-08 13:51:46 +08:00
feihongxu0824 faadf62bd5 fix: use project root for diskann cmake paths (#464) 2026-06-08 09:54:16 +08:00