141 lines
3.5 KiB
Markdown
141 lines
3.5 KiB
Markdown
|
|
# Benchmarking scripts
|
|
|
|
This directory contains benchmarking scripts and reproducing steps.
|
|
|
|
## COHERE experiments
|
|
|
|
### Getting COHERE Data
|
|
|
|
Please download the COHERE 10M dataset to cohere_large_10m as follows:
|
|
|
|
```bash
|
|
neighbors_head_1p.parquet
|
|
neighbors_tail_1pgit.parquet
|
|
neighbors_labels_label_20p.parquet
|
|
neighbors_labels_label_50p.parquet
|
|
neighbors_labels_label_80p.parquet
|
|
neighbors_labels_label_95p.parquet
|
|
neighbors.parquet
|
|
shuffle_train-00-of-10.parquet
|
|
shuffle_train-01-of-10.parquet
|
|
shuffle_train-02-of-10.parquet
|
|
shuffle_train-03-of-10.parquet
|
|
shuffle_train-04-of-10.parquet
|
|
shuffle_train-05-of-10.parquet
|
|
shuffle_train-06-of-10.parquet
|
|
shuffle_train-07-of-10.parquet
|
|
shuffle_train-08-of-10.parquet
|
|
shuffle_train-09-of-10.parquet
|
|
scalar_labels.parquet
|
|
test.parquet
|
|
```
|
|
|
|
### Preparing Environment
|
|
Clone code and init:
|
|
```bash
|
|
$ git clone git@github.com:alibaba/zvec.git
|
|
$ cd zvec
|
|
$ git submodule update --init
|
|
```
|
|
|
|
Make build docker image:
|
|
```bash
|
|
docker build -t zvec/build-image:latest -f ./.github/workflows/docker/Dockerfile.ubuntu18.10-glibc228 .
|
|
```
|
|
|
|
Start bulld container:
|
|
```bash
|
|
docker run -it --net=host -d -e DEBUG_MODE=true --user root --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /home/zvec/:/home/zvec/ -w /home/zvec --name=build_zvec zvec/build-image:latest bash
|
|
```
|
|
|
|
Turn-off complation option:
|
|
```
|
|
option(BUILD_PYTHON_BINDINGS "Build Python bindings using pybind11" ON)
|
|
=>
|
|
option(BUILD_PYTHON_BINDINGS "Build Python bindings using pybind11" OFF)
|
|
```
|
|
|
|
Build source code:
|
|
```
|
|
$ docker exec -it build_zvec bash
|
|
$ cd /home/zvec/workspace/zvec
|
|
$ mkdir build
|
|
$ cd build
|
|
$ cmake -DENABLE_SKYLAKE=ON -DCMAKE_BUILD_TYPE=Release ..
|
|
```
|
|
|
|
### Converting Dataset
|
|
Export vector data using python script:
|
|
```bash
|
|
$ mkdir 10m.output
|
|
$ python3 convert_cohere_parquet.py
|
|
```
|
|
|
|
Convert vector data to binary formatted file.
|
|
```bash
|
|
/home/zvec/workspace/zvec/bin/txt2vecs -input=cohere_train_vector_10m.txt --output=cohere_train_vector_10m.zvec.vecs --dimension=768
|
|
```
|
|
|
|
### Preparing Bench Config
|
|
Prepare Build Config
|
|
|
|
```yaml
|
|
BuilderCommon:
|
|
BuilderClass: HnswStreamer
|
|
BuildFile: /home/zvec/bench/data/10m/cohere_train_vector_10m.zvec.vecs
|
|
NeedTrain: true
|
|
TrainFile: /home/zvec/bench/data/10m/cohere_train_vector_10m.zvec.vecs
|
|
DumpPath: /home/zvec/bench/config/cohere_train_vector_10m.index
|
|
IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.2.index
|
|
|
|
ConverterName: CosineInt8Converter
|
|
MetricName: Cosine
|
|
|
|
ThreadCount: 16
|
|
|
|
BuilderParams:
|
|
proxima.general.builder.thread_count: !!int 16
|
|
proxima.hnsw.builder.thread_count: !!int 16
|
|
```
|
|
|
|
Prepare Search Config
|
|
|
|
```yaml
|
|
SearcherCommon:
|
|
SearcherClass: HnswStreamer
|
|
IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.2.index
|
|
TopK: 1,10,50,100
|
|
QueryFile: /home/zvec/bench/data/10m/cohere_test_vector_1000.new.txt
|
|
QueryType: float
|
|
QueryFirstSep: ";"
|
|
QuerySecondSep: " "
|
|
GroundTruthFile: /home/zvec/bench/data/10m/neighbors.txt
|
|
RecallThreadCount: 1
|
|
BenchThreadCount: 16
|
|
BenchIterCount: 1000000000
|
|
CompareById: true
|
|
|
|
SearcherParams:
|
|
proxima.hnsw.streamer.ef: !!int 250
|
|
```
|
|
|
|
### Building Index
|
|
Conduct Build
|
|
```bash
|
|
$ /home/zvec/workspace/zvec/build/bin/local_build_original ./build.yaml
|
|
```
|
|
|
|
### Performing Bench
|
|
Conduct Recall
|
|
```bash
|
|
$ /home/zvec/workspace/zvec/build/bin/recall_original ./search.yaml
|
|
```
|
|
|
|
Conduct Bench
|
|
```bash
|
|
$ /home/zvec/workspace/zvec/build/bin/bench_original ./search.yaml
|
|
```
|
|
|
|
|