91 lines
3.8 KiB
Markdown
91 lines
3.8 KiB
Markdown
# Evaluation Memory Framework
|
|
|
|
This repository provides tools and scripts for evaluating the `LoCoMo`, `LongMemEval`, `PrefEval`, `personaMem` dataset using various models and APIs.
|
|
|
|
## Installation
|
|
|
|
1. Set the `PYTHONPATH` environment variable:
|
|
```bash
|
|
export PYTHONPATH=../src
|
|
cd evaluation # run from the repository root
|
|
```
|
|
|
|
2. Install the required dependencies:
|
|
```bash
|
|
poetry install --extras all --with eval
|
|
```
|
|
|
|
## Configuration
|
|
Copy the `.env-example` file to `.env`, and fill in the required environment variables according to your environment and API keys.
|
|
|
|
## Setup MemOS
|
|
### local server
|
|
```bash
|
|
# modify {project_dir}/.env file and start server
|
|
uvicorn memos.api.server_api:app --host 0.0.0.0 --port 8001 --workers 8
|
|
|
|
# configure {project_dir}/evaluation/.env file
|
|
MEMOS_URL="http://127.0.0.1:8001"
|
|
```
|
|
### online service
|
|
```bash
|
|
# get your api key at https://memos-dashboard.openmem.net/cn/quickstart/
|
|
# configure {project_dir}/evaluation/.env file
|
|
MEMOS_KEY="Token mpg-xxxxx"
|
|
MEMOS_ONLINE_URL="https://memos.memtensor.cn/api/openmem/v1"
|
|
|
|
```
|
|
|
|
## Supported frameworks
|
|
We support `memos-api` and `memos-api-online` in our scripts.
|
|
And give unofficial implementations for the following memory frameworks:`zep`, `mem0`, `memobase`, `supermemory`, `memu`.
|
|
|
|
|
|
## Evaluation Scripts
|
|
|
|
### LoCoMo Evaluation
|
|
⚙️ To evaluate the **LoCoMo** dataset using one of the supported memory frameworks — run the following [script](../../../../evaluation/scripts/run_locomo_eval.sh):
|
|
|
|
```bash
|
|
# Edit the configuration in ./scripts/run_locomo_eval.sh
|
|
# Specify the model and memory backend you want to use (e.g., mem0, zep, etc.)
|
|
evaluation/scripts/run_locomo_eval.sh
|
|
```
|
|
|
|
✍️ For evaluating OpenAI's native memory feature with the LoCoMo dataset, please refer to the detailed guide: [OpenAI Memory on LoCoMo - Evaluation Guide](./openai_memory_locomo_eval_guide.md).
|
|
|
|
### LongMemEval Evaluation
|
|
First prepare the dataset `longmemeval_s` from https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned
|
|
, and save it as `data/longmemeval/longmemeval_s.json`
|
|
|
|
```bash
|
|
# Edit the configuration in evaluation/scripts/run_lme_eval.sh
|
|
# Specify the model and memory backend you want to use (e.g., mem0, zep, etc.)
|
|
evaluation/scripts/run_lme_eval.sh
|
|
```
|
|
|
|
#### Question date and `reference_time`
|
|
|
|
LongMemEval gives each question a **question date**; evaluation should use that as the reference “now”, not the time when you run the script. The LongMemEval search script passes `question_date` as **`reference_time`** where the backend supports it.
|
|
|
|
**MemOS Cloud** currently does not support supplying question date on search the same way, so LongMemEval scores there may differ from a spec-faithful run. **Prefer evaluating LongMemEval against the open-source MemOS server** when you need comparable numbers.
|
|
|
|
### PrefEval Evaluation
|
|
Downloading benchmark_dataset/filtered_inter_turns.json from https://github.com/amazon-science/PrefEval/blob/main/benchmark_dataset/filtered_inter_turns.json and save it as `./data/prefeval/filtered_inter_turns.json`.
|
|
To evaluate the **Prefeval** dataset — run the following [script](evaluation/scripts/run_prefeval_eval.sh):
|
|
|
|
```bash
|
|
# Edit the configuration in evaluation/scripts/run_prefeval_eval.sh
|
|
# Specify the model and memory backend you want to use (e.g., mem0, zep, etc.)
|
|
evaluation/scripts/run_prefeval_eval.sh
|
|
```
|
|
|
|
### PersonaMem Evaluation
|
|
get `questions_32k.csv` and `shared_contexts_32k.jsonl` from https://huggingface.co/datasets/bowen-upenn/PersonaMem and save them at `data/personamem/`
|
|
```bash
|
|
# Edit the configuration in evaluation/scripts/run_pm_eval.sh
|
|
# Specify the model and memory backend you want to use (e.g., mem0, zep, etc.)
|
|
# If you want to use MIRIX, edit the the configuration in evaluation/scripts/personamem/config.yaml
|
|
evaluation/scripts/run_pm_eval.sh
|
|
```
|