02bfb31cf5
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
query_string and match_string syntax, phrase queries, boolean operators
(AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.
## Core
- BitPacked posting format with block-max WAND pruning
- Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
lowercase, with extensible pipeline composition
- Query parser: boolean operators, phrase queries, field scoping,
boost, MUST(+) modifier inside OR (ES query_string semantics)
- AST rewriter: dedup repeated terms with linear boost aggregation,
flatten same-type composites, canonicalize OR-with-must_not into AND
wrapper, empty-node propagation, contradiction detection
- FTS reduce/merge integrated into Optimize compaction
- Multi-segment score-descending sort
- Auto-register bundled jieba dict on SDK import
## Performance
- Block-max WAND with cached block_max_info_for (single binary search)
- AVX2/SSE bitpacked encoding with cross-arch scalar fallback
- MultiGet for batch posting retrieval and phrase position verification
- HashSkipList memtable for posting writes
- PinnableSlice zero-copy reads
- Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
- Candidate-driven (brute-force) evaluation for selective invert filters
- Precomputed BM25 IDF weights, cached SIMD dispatch pointers
- Shortest-list anchor for phrase position matching
- Single-open per-term posting iterator
## Query
- Tokenize query terms through the same pipeline as indexing
- EmptyNode for zero-token queries (all stop-words / punctuation)
- Backslash unescape after lexing in query parser
- Schema allows collections without vector fields (FTS-only use case)
- Create/Drop Index validates supported index types
- FTS fields disallowed in SQL filter expressions
## Bindings
- C API: fts query params, brute-force ratio config
- Python SDK: FTS search, jieba dict auto-registration
## Internals
- Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
- Hide tokenizer pipeline from public header (Pimpl-style FtsState)
- ListColumnFamilies to avoid double-open on segment load
- Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
|
||
|---|---|---|
| .github | ||
| cmake | ||
| examples | ||
| python | ||
| scripts | ||
| src | ||
| tests | ||
| thirdparty | ||
| tools | ||
| .clang-format | ||
| .clang-tidy | ||
| .gitattributes | ||
| .gitignore | ||
| .gitmodules | ||
| .pre-commit-config.yaml | ||
| CMakeLists.txt | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| README.md | ||
| README_CN.md | ||
| pyproject.toml | ||
README.md
English | 中文
🚀 Quickstart | 🏠 Home | 📚 Docs | 📊 Benchmarks | 🔎 DeepWiki | 🎮 Discord | 🐦 X (Twitter)
Zvec is an open-source, in-process vector database — lightweight, lightning-fast, and designed to embed directly into applications. Battle-tested within Alibaba Group, it delivers production-grade, low-latency and scalable similarity search with minimal setup.
[!Important] 🚀 v0.4.0 (May 9, 2026)
- Dart/Flutter SDK: Published the official zvec Flutter package with FFI bindings. Supports Android (arm64-v8a) and iOS (arm64) — no manual native compilation required.
- iOS Build Support: Added support for building on iOS platforms, expanding cross-platform coverage.
- Enlarged topK Limit: Relaxed the upper bound on topK to support larger-scale recall scenarios.
- Bug Fixes: SQ8 quantizer recall drop; Windows path handling; sparse vector index ordering.
💫 Features
- Blazing Fast: Searches billions of vectors in milliseconds.
- Simple, Just Works: Install and start searching in seconds. Pure local, no servers, no config, no fuss.
- Dense + Sparse Vectors: Work with both dense and sparse embeddings, with native support for multi-vector queries in a single call.
- Hybrid Search: Combine semantic similarity with structured filters for precise results.
- Durable Storage: Write-ahead logging (WAL) guarantees persistence — data is never lost, even on process crash or power failure.
- Concurrent Access: Multiple processes can read the same collection simultaneously; writes are single-process exclusive.
- Runs Anywhere: As an in-process library, Zvec runs wherever your code runs — notebooks, servers, CLI tools, or even edge devices.
📦 Installation
Python
Requirements: Python 3.10 - 3.14
pip install zvec
Node.js
npm install @zvec/zvec
✅ Supported Platforms
- Linux (x86_64, ARM64)
- macOS (ARM64)
- Windows (x86_64)
🛠️ Building from Source
If you prefer to build Zvec from source, please check the Building from Source guide.
⚡ One-Minute Example
import zvec
# Define collection schema
schema = zvec.CollectionSchema(
name="example",
vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4),
)
# Create collection
collection = zvec.create_and_open(path="./zvec_example", schema=schema)
# Insert documents
collection.insert([
zvec.Doc(id="doc_1", vectors={"embedding": [0.1, 0.2, 0.3, 0.4]}),
zvec.Doc(id="doc_2", vectors={"embedding": [0.2, 0.3, 0.4, 0.1]}),
])
# Search by vector similarity
results = collection.query(
zvec.VectorQuery("embedding", vector=[0.4, 0.3, 0.3, 0.1]),
topk=10
)
# Results: list of {'id': str, 'score': float, ...}, sorted by relevance
print(results)
📈 Performance at Scale
Zvec delivers exceptional speed and efficiency, making it ideal for demanding production workloads.
For detailed benchmark methodology, configurations, and complete results, please see our Benchmarks documentation.
🤝 Join Our Community
| 💬 DingTalk | 🎮 Discord | X (Twitter) | |
|---|---|---|---|
 |
 |
 |
 |
| Scan to join | Scan to join | Click to join | Click to follow |
❤️ Contributing
We welcome and appreciate contributions from the community! Whether you're fixing a bug, adding a feature, or improving documentation, your help makes Zvec better for everyone.
Check out our Contributing Guide to get started!