Commit Graph

30 Commits

Author SHA1 Message Date
Jalin Wang f4aa7845f2
fix(build): allow nmake for Snowball codegen on Windows (#661)
fix Snowball building when MSYS2 does not present on Windows
2026-08-07 10:13:18 +08:00
Dmitriy Shekhovtsov e906168cc1
fix: accept Xcode 26.5's renamed libtool version string in iOS Arrow build (#581)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 14:35:31 +08:00
egolearner c35d24e215
feat(fts): add stemmer token filter based on Snowball 3.1.1 (#513) 2026-07-07 14:14:13 +08:00
Jalin Wang 659951b984
fix(lz4): propagate CMAKE_C_COMPILER (#551) 2026-07-06 19:32:56 +08:00
egolearner b9ce030593
feat(fts): add UTF-8 support for tokenizer and token filters via utf8proc (#515) 2026-07-02 11:31:44 +08:00
lichen2015 48ccd44f2d
fix(arrow): propagate compiler settings to ExternalProject (#542) 2026-07-01 14:31:43 +08:00
feihongxu0824 841b1ed3f9
fix: refactor parquet buffer cache ownership (#492) 2026-06-16 10:47:07 +08:00
feihongxu0824 4679a8f39d
feat: support zvec core-only build (#481) 2026-06-09 23:45:42 +08:00
feihongxu0824 b224b17653
fix: add cmake subproject integration check (#465) 2026-06-09 16:01:21 +08:00
egolearner 9fa1b4deb6
fix(build): fix RISC-V compile FastPFOR with SIMDe via patch (#470) 2026-06-08 14:44:07 +08:00
egolearner 355523a628
ci(windows): upgrade ilammy/msvc-dev-cmd, remove /MP and add sccache stats (#454)
- Upgrade ilammy/msvc-dev-cmd from v1 to v1.13.0 to resolve Node.js 20
  deprecation warning (affects 05-windows-build and build_wheel_on_windows)
- Remove /MP from antlr4 and protobuf patches to eliminate 332
  non-cacheable sccache calls caused by "multiple input files"
- Add sccache statistics steps after pip install and after C++ Tests to
  capture build and test compilation cache metrics separately
2026-06-03 14:16:35 +08:00
egolearner e91b7802f3
feat(ci): enable sccache on Windows and propagate compiler launcher to ExternalProject (#439)
feat(ci): enable sccache on Windows and propagate compiler launcher to ExternalProject

Windows CI builds were ~6-9x slower than other platforms (35-40 min vs
4-7 min) due to missing compiler cache. This commit enables sccache for
Windows and ensures all ExternalProject_Add dependencies (Arrow, lz4)
receive the compiler launcher on every platform.

Changes:

MSVC /Zi → /Z7 compatibility:
- Replace /Zi with /Z7 in root CMakeLists.txt for DEBUG and
  RELWITHDEBINFO configurations (MSVC cache variable level)
- Patch rocksdb, antlr4, and googletest source CMakeLists to replace
  hardcoded /Zi with /Z7 at the source, eliminating D9025 warnings
- Clear COMPILE_PDB_NAME on googletest targets to prevent CMake from
  adding /Fd<path>, which causes sccache to look for nonexistent .pdb
- Remove /FS (forced synchronous PDB writes) from Arrow ExternalProject
  MSVC flags — unnecessary with /Z7

Windows CI workflow:
- Add mozilla-actions/sccache-action@v0.0.10
- Set SCCACHE_GHA_ENABLED=true to use GitHub Actions Cache backend
- Pass CMAKE_C/CXX_COMPILER_LAUNCHER=sccache via --config-settings

Compiler launcher propagation:
- Propagate CMAKE_C/CXX_COMPILER_LAUNCHER to Arrow ExternalProject on
  all platforms (Windows, Linux, macOS, Android, iOS)
- Propagate to lz4 ExternalProject on Windows via CMAKE_ARGS, using
  CMP0141=NEW + CMAKE_MSVC_DEBUG_INFORMATION_FORMAT=Embedded for /Z7
2026-06-02 16:23:37 +08:00
egolearner 02bfb31cf5
feat: add fts support (#408)
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
  query_string and match_string syntax, phrase queries, boolean operators
  (AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.

  ## Core
  - BitPacked posting format with block-max WAND pruning
  - Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
    lowercase, with extensible pipeline composition
  - Query parser: boolean operators, phrase queries, field scoping,
    boost, MUST(+) modifier inside OR (ES query_string semantics)
  - AST rewriter: dedup repeated terms with linear boost aggregation,
    flatten same-type composites, canonicalize OR-with-must_not into AND
    wrapper, empty-node propagation, contradiction detection
  - FTS reduce/merge integrated into Optimize compaction
  - Multi-segment score-descending sort
  - Auto-register bundled jieba dict on SDK import

  ## Performance
  - Block-max WAND with cached block_max_info_for (single binary search)
  - AVX2/SSE bitpacked encoding with cross-arch scalar fallback
  - MultiGet for batch posting retrieval and phrase position verification
  - HashSkipList memtable for posting writes
  - PinnableSlice zero-copy reads
  - Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
  - Candidate-driven (brute-force) evaluation for selective invert filters
  - Precomputed BM25 IDF weights, cached SIMD dispatch pointers
  - Shortest-list anchor for phrase position matching
  - Single-open per-term posting iterator

  ## Query
  - Tokenize query terms through the same pipeline as indexing
  - EmptyNode for zero-token queries (all stop-words / punctuation)
  - Backslash unescape after lexing in query parser
  - Schema allows collections without vector fields (FTS-only use case)
  - Create/Drop Index validates supported index types
  - FTS fields disallowed in SQL filter expressions

  ## Bindings
  - C API: fts query params, brute-force ratio config
  - Python SDK: FTS search, jieba dict auto-registration

  ## Internals
  - Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
  - Hide tokenizer pipeline from public header (Pimpl-style FtsState)
  - ListColumnFamilies to avoid double-open on segment load
  - Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
2026-06-01 15:02:54 +08:00
egolearner 9bfc4b556a
chore: fix more warnings (#418) 2026-05-21 11:15:41 +08:00
egolearner 51c6d9e0ec
chore(thirdparty): mark third-party include paths as SYSTEM (#413)
Add SYSTEM to target_include_directories in vendored CMake wrappers and
set INTERFACE_SYSTEM_INCLUDE_DIRECTORIES on imported / sub-project
targets so warnings from third-party headers no longer surface in our
build output. Introduces a mark_target_includes_system() helper in
cmake/utils.cmake (resolves aliases, skips missing targets) used by the
upstream-add_subdirectory wrappers (glog, gflags, googletest, yaml-cpp,
protobuf, antlr).
2026-05-19 15:03:07 +08:00
luoxiaojian 269c611f78
build(thirdparty): replace CRoaring submodule with amalgamation (~292M -> ~1MB) (#381) 2026-05-12 14:55:08 +08:00
Jalin Wang 1d4ae0b1b5
refactor: support UTF-8 file paths via std::filesystem (#359)
Rewrite file/path handling to use std::filesystem and UTF-8-safe helpers.
Switch Windows file open/create paths to wide-char APIs, replace manual
separator concatenation with PathJoin, and enable RocksDB UTF-8 filenames.

Also add UTF-8 path coverage for file IO, version manager recovery, and
collection open/flush/reopen flows.
2026-05-08 17:44:22 +08:00
feihongxu0824 30a20e14b0
feat: add iOS build support (#321)
* support ios build

* fix: ci

* fix: update build_ios.sh

* fix: pr

* fix: cmake minimum version

* fix: cmake minimum version

* fix: cmake minimum version in ci

* fix: add all test in ci

* fix: ci
2026-04-09 16:14:26 +08:00
Jalin Wang ca4b893755
fix: misc fix for Windows(BUILD_SHARED_LIBS, CI) (#296) 2026-04-02 09:23:35 +08:00
Jalin Wang 8182cfff93
feat: windows support (#216)
Preliminary support for Windows with remaining TODOs
2026-03-30 20:47:34 +08:00
Qinren Zhou c8df06ab3e
fix: build on mac (#272) 2026-03-27 20:11:22 +08:00
egolearner e5ba11b6fe
feat: add hnsw-rabitq support (#69)
* feat: add hnsw-rabitq

* undefine transform

* support config rotator type

* support sample_count

* fix ut

* refactor: update interface

* refactor: update interface

* update rabitq index params

* fix interface update

* fix searcher test

* fix streamer test

* streamer support bf and add more ut

* rm env check

* add collection ut

* add schema check

* add rabitq query param binding

* add integration test

* fix local_builder

* cleanup dist calculator

* add RaBitQ-Library submodule

* cleanup rabitq converter/reformer

* add files

* disable build on mac

* disable Feature_Optimize_HNSW_RABITQ

* disable python/tests/test_collection_hnsw_rabitq.py:13

* check avx2/avx512

* fix refine

* fix mac ci

* add dimension check

* fix mac ci

* fix ci

* add missing lib

* fix centroids selection for Cosine/InnerProduct

* rename hnsw-rabitq to hnsw_rabitq

* address comments

* fix compile

* fix rabitqlib name

* fix mac compile

* check avx2/avx512 support

* runtime check again compile-time

* check AUTO_DETECT_ARCH

* rm debug log

* fix

* rabitq use avx2 by default

* add missing file

* fix search_bf/group_by dist

* address comments

* fix typo

* address comments

* rabitq support disable id_map

* address comments
2026-03-19 17:21:42 +08:00
feihongxu0824 b10c3fc316
chore: default not use oss and use beijing-oss (#207) 2026-03-09 17:10:57 +08:00
feihongxu0824 43e3eeac40
feat: support android platform cross build (#90) 2026-03-04 16:42:23 +08:00
feihongxu0824 83c03ed0e4
chore(build): accelerate third-party downloads using OSS mirror URLs (#194) 2026-03-04 11:19:10 +08:00
egolearner 5e637bcfdf
build: disable rocksdb -arch=native (#183) 2026-02-28 11:19:44 +08:00
Aashish Thapa 0d84e30dd3
fix: update antlr4 CMAKE_POLICY from OLD to NEW for CMake 3.31+ compatibility (#145)
CMake 3.31+ has removed support for the OLD behavior of policies
CMP0042, CMP0045, CMP0054, and CMP0059. This causes a hard
configuration error when building from source with newer CMake versions.

Changing these to NEW resolves the build failure. The NEW behavior has
been the default since CMake 3.0/3.3 and does not affect functionality.

Fixes #144
2026-02-27 14:35:55 +08:00
feihongxu0824 e957442355
feat: refact cpp sdk (#27)
* refact cpp sdk

---------

Co-authored-by: zhouqinren.zqr <zhouqinren.zqr@alibaba-inc.com>
2026-01-21 20:05:42 +08:00
egolearner b3287fea3c
chore: support build arrow with debug (#11)
* chore: support build arrow with debug

* align

---------

Co-authored-by: feihongxu0824 <xufeihong.xfh@alibaba-inc.com>
2026-01-12 11:49:24 +08:00
sanyi 6524fabd80 Initial commit 2025-12-30 11:02:17 +08:00