Alibaba lightweight in-process vector database
Go to file
Cuiys c46efe1241
refactor: change rerank interface from map-based to vector-based (#458)
* refactor: change rerank interface from map-based to vector-based (#452)

- Define QueryResult = list[Doc] type alias in doc.py
- Change C++ Reranker::rerank() signature from map<string, DocPtrList> to vector<DocPtrList>
- Extend bind_schema() to accept field_names for index-based field lookup
- Update ScoreBasedReranker/WeightedReranker/CallbackReranker implementations
- Adapt collection.cc MultiQuery path to use vector<DocPtrList>
- Update Python binding to expose rerank() and use vector<double> weights
- Refactor Python RerankFunction interface to list[QueryResult] -> QueryResult
- Remove Python-layer rerank logic from RrfReRanker/WeightedReRanker (delegate to C++)
- Update query_executor to return list[list[Doc]] instead of dict
- Update all related unit tests (C++ and Python)

* refactor: replace list[Doc] with QueryResult type alias in executor and rerank functions

* refactor: replace list[list[Doc]] with list[QueryResult] in query_executor

* fix: remove unused Doc import in rerank_function.py (ruff F401)

* refactor(query_executor): merge duplicate rerank return paths

* refactor: RrfReRanker/WeightedReRanker.rerank() directly call C++ reranker

* refactor: simplify QueryExecutor into unified class, remove Factory/subclasses/validation/concurrency

* refactor: rename _VectorQuery to _SearchQuery, from_vector_query to from_search_query

* refactor(query_executor): split execute into single/multi paths, rename core_vector to search_query, drop unused core_vectors

* style: apply ruff formatter to test_reranker.py and query_executor.py

* refactor: make rescore() private in ScoreBasedReranker hierarchy

* style: apply clang-format to reranker.h

* style: apply clang-format to all modified C++ files

* refactor: rename private methods in QueryExecutor for clearer semantics

* refactor: rename mvq to multi_query for clarity

* fix: make BasicRRF test order-independent for equal scores

* fix: update collection_test to use vector-based reranker interface

* fix: update reranker tests to expect TypeError instead of NotImplementedError

* refactor: remove PendingQuery wrapper, use SearchQuery directly in MultiQuery path

* refactor: simplify MultiQuery path - remove seen_fields, merge field_names into main loop

* fix: address review comments - defensive checks and remove fields param from C API

- ScoreBasedReranker::rerank(): early return empty list when topn <= 0
- WeightedReranker::rescore(): null-check schema_ before use
- CallbackReranker::rerank(): check callback_ is not empty before invoke
- C API zvec_reranker_create_weighted(): remove unused fields parameter

* fix: remove duplicate field name test (check was intentionally removed)

* fix: address egolearner review comments

- Rename QueryResult to DocList for clarity (见名知义)
- Change docstring to #: comment for type alias
- Fix output_fields check: use 'is not None' instead of truthy check
  (None means unset, [] means explicit empty list - different semantics)
- Raise ValueError when search-by-id finds no document

* refactor: remove redundant output_fields assignment in _build_search_query

* refactor: address egolearner review comments (C++ refactoring)

- c_api.cc: simplify weighted reranker creation with inline vector ctor
- python_reranker.cc: refactor unwrap_rerank_result - take by value,
  early error return, move semantics
- Rename C API functions for consistent naming:
  zvec_reranker_create_rrf -> zvec_create_rrf_reranker
  zvec_reranker_create_weighted -> zvec_create_weighted_reranker
  zvec_reranker_destroy -> zvec_destroy_reranker
  zvec_reranker_get_rank_constant -> zvec_get_reranker_rank_constant
- reranker.h/cc: bind_schema returns Result<void>, caches
  vector<const FieldSchema*> to avoid repeated schema lookups in rescore
- python_param.cc: rename py::arg vector_query to search_query

* revert: rollback bind_schema refactoring due to thread-safety concern

The field_schemas_ caching approach introduces a data race when the same
WeightedReranker instance is shared across concurrent queries: bind_schema()
writes field_schemas_ while rerank() reads it concurrently.

Revert to storing schema_ + field_names_ and looking up fields in rescore().
Add @note thread-safety warning to WeightedReranker class documentation.

* fix: unify error message format in collection.cc

Change 'Vector field not found: X' to 'Invalid query: field X not found'
for consistent error formatting as suggested by zhourrr.

* fix: sort __all__ and remove duplicates in __init__.pyi

Fix RUF022 lint error: sort __all__ alphabetically and remove duplicate
entries (DenseEmbeddingFunction, ReRanker).

* style: format query_executor.py with ruff formatter

* fix: resolve Python test failures after FTS rebase integration

- test_query_executor.py: update method names to match refactored API
  (_do_build -> _build_queries, _do_merge_rerank_results -> _merge_and_rerank)
- test_reranker.py: fix expected exception type (TypeError from pybind11)
- test_collection_fts.py: update error message match patterns
- test_collection_fts_vector_hybrid.py: remove obsolete 'metrics' param,
  update weights from dict to positional list, adapt validation tests
  for multi-vector queries (now supported with reranker)
- test_collection_dql.py: remove 'metrics' param, update weights format
- collection.cc: distinguish FTS vs vector fields in MultiQuery path
  using get_fts_clause() to route field lookup correctly
- reranker.cc: use get_field() instead of get_vector_field() in rescore
  to support FTS+vector hybrid weighted reranking

* refactor: pass topn as rerank() parameter, move rerank_field to model rerankers

* fix: address review comments - rename test functions and restore duplicate field check

* refactor: simplify MultiQuery field lookup, let validate_and_sanitize handle type check
2026-06-04 16:03:28 +08:00
.github ci(windows): upgrade ilammy/msvc-dev-cmd, remove /MP and add sccache stats (#454) 2026-06-03 14:16:35 +08:00
cmake chore: fix more warnings (#418) 2026-05-21 11:15:41 +08:00
examples refactor: drop VectorQuery, unify single-target query on SearchQuery (#428) 2026-05-29 16:36:09 +08:00
python refactor: change rerank interface from map-based to vector-based (#458) 2026-06-04 16:03:28 +08:00
scripts ci: refact android ci (#330) 2026-04-15 15:37:51 +08:00
src refactor: change rerank interface from map-based to vector-based (#458) 2026-06-04 16:03:28 +08:00
tests refactor: change rerank interface from map-based to vector-based (#458) 2026-06-04 16:03:28 +08:00
thirdparty ci(windows): upgrade ilammy/msvc-dev-cmd, remove /MP and add sccache stats (#454) 2026-06-03 14:16:35 +08:00
tools feat: add fts support (#408) 2026-06-01 15:02:54 +08:00
.clang-format Initial commit 2025-12-30 11:02:17 +08:00
.clang-tidy chore: enable modernize-use-override and fix existing violations (#419) 2026-05-21 19:05:32 +08:00
.gitattributes feat: add fts support (#408) 2026-06-01 15:02:54 +08:00
.gitignore feat(ci): integrate clang-tidy for changed C/C++ files (#116) 2026-04-20 20:14:21 +08:00
.gitmodules feat: add fts support (#408) 2026-06-01 15:02:54 +08:00
.pre-commit-config.yaml chore: enable the conventional-pre-commit run sucess and update to latest version (#111) 2026-02-25 18:03:20 +08:00
CMakeLists.txt feat(ci): enable sccache on Windows and propagate compiler launcher to ExternalProject (#439) 2026-06-02 16:23:37 +08:00
CODE_OF_CONDUCT.md Initial commit 2025-12-30 11:02:17 +08:00
CONTRIBUTING.md doc: add v0.3.0 release note (#312) 2026-04-03 15:47:19 +08:00
LICENSE Initial commit 2025-12-30 11:02:17 +08:00
README.md minor: update readme for v0.4.0 (#389) 2026-05-09 11:40:30 +08:00
README_CN.md minor: update readme for v0.4.0 (#389) 2026-05-09 11:40:30 +08:00
pyproject.toml feat(test): enable parallel tests (#384) 2026-05-12 23:14:17 +08:00

README.md

English | 中文

zvec logo

Code Coverage Main License PyPI Release Python Versions npm Release

alibaba%2Fzvec | Trendshift

🚀 Quickstart | 🏠 Home | 📚 Docs | 📊 Benchmarks | 🔎 DeepWiki | 🎮 Discord | 🐦 X (Twitter)

Zvec is an open-source, in-process vector database — lightweight, lightning-fast, and designed to embed directly into applications. Battle-tested within Alibaba Group, it delivers production-grade, low-latency and scalable similarity search with minimal setup.

[!Important] 🚀 v0.4.0 (May 9, 2026)

  • Dart/Flutter SDK: Published the official zvec Flutter package with FFI bindings. Supports Android (arm64-v8a) and iOS (arm64) — no manual native compilation required.
  • iOS Build Support: Added support for building on iOS platforms, expanding cross-platform coverage.
  • Enlarged topK Limit: Relaxed the upper bound on topK to support larger-scale recall scenarios.
  • Bug Fixes: SQ8 quantizer recall drop; Windows path handling; sparse vector index ordering.

👉 Read the Release Notes | View Roadmap 📍

💫 Features

  • Blazing Fast: Searches billions of vectors in milliseconds.
  • Simple, Just Works: Install and start searching in seconds. Pure local, no servers, no config, no fuss.
  • Dense + Sparse Vectors: Work with both dense and sparse embeddings, with native support for multi-vector queries in a single call.
  • Hybrid Search: Combine semantic similarity with structured filters for precise results.
  • Durable Storage: Write-ahead logging (WAL) guarantees persistence — data is never lost, even on process crash or power failure.
  • Concurrent Access: Multiple processes can read the same collection simultaneously; writes are single-process exclusive.
  • Runs Anywhere: As an in-process library, Zvec runs wherever your code runs — notebooks, servers, CLI tools, or even edge devices.

📦 Installation

Python

Requirements: Python 3.10 - 3.14

pip install zvec

Node.js

npm install @zvec/zvec

Supported Platforms

  • Linux (x86_64, ARM64)
  • macOS (ARM64)
  • Windows (x86_64)

🛠️ Building from Source

If you prefer to build Zvec from source, please check the Building from Source guide.

One-Minute Example

import zvec

# Define collection schema
schema = zvec.CollectionSchema(
    name="example",
    vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4),
)

# Create collection
collection = zvec.create_and_open(path="./zvec_example", schema=schema)

# Insert documents
collection.insert([
    zvec.Doc(id="doc_1", vectors={"embedding": [0.1, 0.2, 0.3, 0.4]}),
    zvec.Doc(id="doc_2", vectors={"embedding": [0.2, 0.3, 0.4, 0.1]}),
])

# Search by vector similarity
results = collection.query(
    zvec.VectorQuery("embedding", vector=[0.4, 0.3, 0.3, 0.1]),
    topk=10
)

# Results: list of {'id': str, 'score': float, ...}, sorted by relevance
print(results)

📈 Performance at Scale

Zvec delivers exceptional speed and efficiency, making it ideal for demanding production workloads.

Zvec Performance Benchmarks

For detailed benchmark methodology, configurations, and complete results, please see our Benchmarks documentation.

🤝 Join Our Community

💬 DingTalk 📱 WeChat 🎮 Discord X (Twitter)
DingTalk QR Code WeChat QR Code Discord X (formerly Twitter) Follow
Scan to join Scan to join Click to join Click to follow

❤️ Contributing

We welcome and appreciate contributions from the community! Whether you're fixing a bug, adding a feature, or improving documentation, your help makes Zvec better for everyone.

Check out our Contributing Guide to get started!