4e13f7881e
Langfuse aggregates scores by name, so one name may only carry values on
one scale. recall_top_score was emitted for every method, mixing HYBRID's
LR-sigmoid probability and AGENTIC's cross-encoder score (both comparable
in [0, 1]) with KEYWORD's unbounded BM25 and single-route VECTOR's cosine.
A chart on that name averaged the two scales, and in practice a keyword
score can read numerically higher than a calibrated one while meaning less.
Uncalibrated methods now report recall_top_score_raw, leaving
recall_top_score comparable across methods and over time. Every recall
score also carries metadata = {method, calibrated}: a structured field
Langfuse persists and can split on, which the free-text comment could not
serve. The comment stays for reading individual scores.
Breaking for anyone charting recall_top_score for keyword search; 1.2.0 is
four days old, so this is the cheapest moment to correct the naming.
Claude-Session: https://claude.ai/code/session_01UyKinsWs1MgoARPoB9R4NW
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| .gitignore | ||
| README.md | ||
| demo.py | ||
README.md
EverOS × Langfuse (native OpenTelemetry)
EverOS emits OpenTelemetry spans for its own memory operations — write, memcell boundary + episode extraction (LLM), search with recall-quality scores, and OME reflection — and exports them over OTLP to any backend, including Langfuse. There is no wrapper and no extra instrumentation code: enable it in config and the traces appear.
Enable
-
Install the optional OpenTelemetry extra:
pip install "everos[otel]" -
Add
[observability]to youreveros.toml. The Langfuse keys derive the OTLP endpoint and auth automatically:[observability] enabled = true langfuse_public_key = "pk-lf-..." langfuse_secret_key = "sk-lf-..." langfuse_host = "https://us.cloud.langfuse.com" # EU: https://cloud.langfuse.com # capture_content = true # opt-in: also record query / extracted memory textContainer/CI equivalent via env vars:
EVEROS_OBSERVABILITY__ENABLED=true,EVEROS_OBSERVABILITY__LANGFUSE_PUBLIC_KEY=..., and so on. -
Run EverOS normally:
everos server start
Off by default — with enabled = false (or the otel extra absent) there is
zero tracing overhead.
What you get
| EverOS operation | Langfuse observation |
|---|---|
POST /api/v1/memory/add · flush |
span everos.memory.add / everos.memory.flush |
| memcell boundary detection (LLM) | generation everos.memcell.boundary (model + tokens) |
| episode extraction (LLM) | generation everos.extract |
| markdown persistence | span everos.persist.markdown |
POST /api/v1/memory/search |
retriever everos.memory.search → recall / rank |
| query / recall embedding | embedding everos.embedding |
| OME reflection strategies | agent everos.ome.<strategy> (linked to the triggering request's trace) |
langfuse.session.id / langfuse.user.id group the traces. Recall quality is
pushed as Langfuse scores, split by whether the method's score is calibrated:
recall_top_score plus recall_hit for HYBRID / AGENTIC (comparable [0, 1]),
and recall_top_score_raw for KEYWORD / single-route VECTOR, whose raw BM25 or
cosine values are on a different scale and must not be averaged in with the
calibrated ones. Query and memory text are captured only when
capture_content = true.
Try it
With a server running and [observability] enabled:
python demo.py
It drives one add → flush → search (keyword / hybrid / agentic) cycle against
http://127.0.0.1:8000 using only the standard library, then tells you to open
Langfuse → Tracing filtered to session.id = langfuse_demo.
Learn more
- Langfuse OpenTelemetry: https://langfuse.com/integrations/native/opentelemetry
- Config reference: the
[observability]block insrc/everos/config/default.toml.