Files

T

cff43adb63 feat(convo): preserve authored timestamp from transcripts (#1890 )

* feat(convo): preserve authored timestamp from transcripts

Conversation drawers only carried `filed_at` (ingest time), so a bulk
re-mine collapsed every drawer to a single instant and the chronological
signal was lost — even though each Claude Code / Codex JSONL line already
carries an ISO-8601 `timestamp`. The recency-window fallback and any
date-aware consumer then saw ingest order, not when content was written.

- convo_miner: derive `authored_at` (per-file max line `timestamp`) and
  store it as drawer metadata; falls back to `filed_at` when absent
- searcher: surface `authored_at` in search results, and break exact
  hybrid-score ties toward the more recently authored drawer (ISO strings
  sort chronologically; missing dates sort oldest) — benchmark-neutral as
  it only reorders exact ties
- tests: cover `_extract_authored_at` (latest wins, skips/tolerates lines
  without timestamps, non-jsonl/missing -> None) and the tie-break

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(search): surface authored_at in CLI + backfill for existing data

Completes the authored_at work so the field is visible end-to-end and
existing palaces can adopt it without re-mining.

- layers: CLI `search` output shows an `authored:` date line per result
  (peer of the existing date; markdown drawers fall back to filed_at)
- scripts/backfill_authored_at.py: in-place migration that stamps
  authored_at on convos drawers from their source transcripts — metadata
  only (no re-embedding), idempotent, dry-run by default
- docs/authored-at.md: documents created_at (ingest) vs authored_at
  (written) and both backfill paths (in-place / drop-and-recreate)
- tests: backfill integration tests over an ephemeral ChromaDB collection

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(search): address review — non-string timestamp guard + top-level authored_at tiebreak

Two correctness fixes from the PR review:

- _extract_authored_at: only compare when the parsed `timestamp` is a str.
  A non-string timestamp on a malformed/foreign JSONL line previously raised
  TypeError outside the try and could crash the mine.
- _hybrid_rank: the tie-break read `authored_at` only from nested `metadata`,
  but the search_memories path (MCP / Claude Code) carries it at the top level
  of each hit — so the tie-break silently no-op'd there. Read both shapes.
- tests: non-string timestamp cases, and a top-level-shape tie-break test
  (which fails before this fix).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* style: apply ruff format to authored_at changes

CI ruff format --check flagged 4 files; ruff check already passed.
Formatting only — no behavior change.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>

2026-06-28 18:13:06 -03:00

2.1 KiB

Raw Blame History

Authored date (`authored_at`)

Conversation transcripts carry a per-line ISO-8601 timestamp (both Claude Code and Codex JSONL). The miner records the most recent one per file as the drawer's authored_at — when the content was actually written.

This is distinct from the ingest date:

Field	Meaning
`filed_at` / result `created_at`	When the drawer was mined (written to the palace). A bulk re-mine collapses these to a single instant.
`authored_at`	When the underlying content was written, recovered from the transcript timestamps. Survives re-mining.

authored_at is surfaced in search results (and shown in the CLI search output), and is used as a deterministic tie-break in hybrid ranking: candidates with identical scores order with the more recently authored drawer first. Drawers without per-line timestamps (e.g. markdown) fall back to filed_at.

Backfilling existing memory

New mines populate authored_at automatically. Drawers mined before this feature only have filed_at. Re-mining does not fix them — the scanner skips files already mined at the current NORMALIZE_VERSION. Two options:

In-place backfill (recommended — no re-embedding). scripts/backfill_authored_at.py reads each convos drawer's source transcript and updates only the authored_at metadata. Idempotent and safe to re-run; embeddings are untouched.
```
python scripts/backfill_authored_at.py \
    --palace ~/.mempalace/palace \
    --sessions ~/.claude --sessions ~/.codex          # dry run
python scripts/backfill_authored_at.py \
    --palace ~/.mempalace/palace \
    --sessions ~/.claude --sessions ~/.codex --apply  # write
```
For the Docker MCP image, mount the volume and session dirs read-only — see the header of scripts/backfill_authored_at.py for the exact docker run invocation.

Back up first: tar czf palace-backup.tgz -C <palace-dir> . (or snapshot the mempalace-data volume).
Drop and recreate. Delete the affected drawers and re-mine the transcripts; the fresh mine stamps authored_at. Simpler, but re-embeds everything.

2.1 KiB Raw Blame History

Authored date (authored_at)

Backfilling existing memory

2.1 KiB

Raw Blame History

Authored date (`authored_at`)