* chore: add git commit msg and branch name in pre-commit * docs: modify org to alibaba * ci: mac build with make
3.5 KiB
3.5 KiB
Benchmarking scripts
This directory contains benchmarking scripts and reproducing steps.
COHERE experiments
Getting COHERE Data
Please download the COHERE 10M dataset to cohere_large_10m as follows:
neighbors_head_1p.parquet
neighbors_tail_1pgit.parquet
neighbors_labels_label_20p.parquet
neighbors_labels_label_50p.parquet
neighbors_labels_label_80p.parquet
neighbors_labels_label_95p.parquet
neighbors.parquet
shuffle_train-00-of-10.parquet
shuffle_train-01-of-10.parquet
shuffle_train-02-of-10.parquet
shuffle_train-03-of-10.parquet
shuffle_train-04-of-10.parquet
shuffle_train-05-of-10.parquet
shuffle_train-06-of-10.parquet
shuffle_train-07-of-10.parquet
shuffle_train-08-of-10.parquet
shuffle_train-09-of-10.parquet
scalar_labels.parquet
test.parquet
Preparing Environment
Clone code and init:
$ git clone git@github.com:alibaba/zvec.git
$ cd zvec
$ git submodule update --init
Make build docker image:
docker build -t zvec/build-image:latest -f ./.github/workflows/docker/Dockerfile.ubuntu18.10-glibc228 .
Start bulld container:
docker run -it --net=host -d -e DEBUG_MODE=true --user root --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /home/zvec/:/home/zvec/ -w /home/zvec --name=build_zvec zvec/build-image:latest bash
Turn-off complation option:
option(BUILD_PYTHON_BINDINGS "Build Python bindings using pybind11" ON)
=>
option(BUILD_PYTHON_BINDINGS "Build Python bindings using pybind11" OFF)
Build source code:
$ docker exec -it build_zvec bash
$ cd /home/zvec/workspace/zvec
$ mkdir build
$ cd build
$ cmake -DENABLE_SKYLAKE=ON -DCMAKE_BUILD_TYPE=Release ..
Converting Dataset
Export vector data using python script:
$ mkdir 10m.output
$ python3 convert_cohere_parquet.py
Convert vector data to binary formatted file.
/home/zvec/workspace/zvec/bin/txt2vecs -input=cohere_train_vector_10m.txt --output=cohere_train_vector_10m.zvec.vecs --dimension=768
Preparing Bench Config
Prepare Build Config
BuilderCommon:
BuilderClass: HnswStreamer
BuildFile: /home/zvec/bench/data/10m/cohere_train_vector_10m.zvec.vecs
NeedTrain: true
TrainFile: /home/zvec/bench/data/10m/cohere_train_vector_10m.zvec.vecs
DumpPath: /home/zvec/bench/config/cohere_train_vector_10m.index
IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.2.index
ConverterName: CosineInt8Converter
MetricName: Cosine
ThreadCount: 16
BuilderParams:
proxima.general.builder.thread_count: !!int 16
proxima.hnsw.builder.thread_count: !!int 16
Prepare Search Config
SearcherCommon:
SearcherClass: HnswStreamer
IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.2.index
TopK: 1,10,50,100
QueryFile: /home/zvec/bench/data/10m/cohere_test_vector_1000.new.txt
QueryType: float
QueryFirstSep: ";"
QuerySecondSep: " "
GroundTruthFile: /home/zvec/bench/data/10m/neighbors.txt
RecallThreadCount: 1
BenchThreadCount: 16
BenchIterCount: 1000000000
CompareById: true
SearcherParams:
proxima.hnsw.streamer.ef: !!int 250
Building Index
Conduct Build
$ /home/zvec/workspace/zvec/build/bin/local_build_original ./build.yaml
Performing Bench
Conduct Recall
$ /home/zvec/workspace/zvec/build/bin/recall_original ./search.yaml
Conduct Bench
$ /home/zvec/workspace/zvec/build/bin/bench_original ./search.yaml