SoftDelete returns error but call sites discarded it, so a failed
soft-delete was silent — same class as the no-op implementations.
Now logged explicitly at:
- server.go: ephemeral_expired cleanup (x2)
- admin.go: auto_forget_fragment / auto_forget
- feedback.go: Deprecate endpoint
SQLiteMemClient.IncrementUseful/IncrementNotUseful were discarding the
error return of MemLanceClient.IncrementUseful/IncrementNotUseful.
Since the latter now return real errors (6e67f3e), dropping them
re-introduces the silent-failure class of bug we just fixed across
the Rust path. Also propagate persistMemory errors.
- IncrementUseful/IncrementNotUseful now return error and persist via
rc.Update() (Rust sidecar); all 4 implementations updated
- SoftDeleteBatch now delegates to SoftDelete() for real LanceDB write
- UpdateMemoryContent now persists content + version + version history
- Backup changed from return nil to explicit error (Rust sidecar unsupported)
- IncrementUseful/IncrementNotUseful signatures now include error return;
all 3 call sites (feedback.go, core.go) check the error
- Test mocks (mockLance, fakeLDB) updated to match new interface
Fixes the 'silent no-op' class of bugs discovered during dedup repair.
The <20-char exemption added earlier for human-side soulful facts also
carried a 'hasStatementSubject && !status && !question' fallback.
hasStatementSubject only checks for a subject word - it does not check
that the content is a fact - so fragments, questions and status
reports all passed:
'用户喜欢喝咖啡' (7 chars, fragment)
'用户重启了系统' (7 chars, fragment)
'用户今天完成了一次系统检查工作' (14 chars, fragment)
'用户要不要升级系统吗' (question)
'小唯a06与all桥接状态同步完成' (status report)
Five assertions in the distill suite were failing because of this
(TestIsQualityFact_ShortFragmentsRejected, _PureQuestionRejected,
_StatusReportRejected, and TestDistillOne_QualityGateBlocksFragments
which kept 2 facts where the gate should keep 1).
The exemption is now limited to daemon-written structured prefixes
(画像规则:/心迹:/牵挂:/偏好:/习惯:/目标:/行为规则:) - strong signals that
cannot be confused with noise. Natural short sentences are weak signals
and are rejected as fragments; if a writer genuinely needs one kept, it
should supply the structured prefix rather than loosen this gate.
go test ./... now passes (TEST_EXIT=0, 8 packages ok).
mockGraph declared AllEdges() []map[string]interface{} while
governance.GraphStore requires AllEdges(string) []governance.GraphEdge.
go build ./... stayed green because test files are not compiled by
build, so the routes test package could not be compiled at all and the
whole suite silently went unrun - which is why the earlier AllEdges
'compatibility fix' looked done without ever executing.
After fixing the signature the suite runs: all tests pass including
TestCommitConflictGate_E2E (end-to-end commit -> conflict -> distill
gate, 3.70s).
zz_ns_probe_test.go also gives positive evidence for the
classifyNamespace fix: human-side content now resolves to
soulful-xiaowei-main rather than a06-main.
'牧尘在招商局工作' -> soulful-xiaowei-main
'牧尘不在招商局工作' -> soulful-xiaowei-main
'画像规则: 牧尘在招商局工作' -> soulful-xiaowei-main
Two write-path fixes found while executing the dedup policy.
1. The endpoint wrote is_deleted via Update(). The system's canonical
path is SoftDelete(), which routes through SoftDeleteWriteBuffer
(5-minute window / 256-item threshold, single-transaction commit).
Update() writes one MVCC version per row, so this endpoint would
have created 411 versions in one run - the exact growth pattern the
vacuum script was written to clean up (table measured at 3.23G
before a 40M compaction).
2. The response was sent while soft deletes were still buffered. During
execution lance-vacuum (daily 03:30) stopped the service and ran
optimize(cleanup_older_than=0), discarding the unflushed writes: the
endpoint's deleted=34 never landed and is_deleted read back all
false, while the caller only saw curl exit=52 for the empty reply.
A new storage.FlushSoftDeleteBuffer() is now called before the
response, so reported counts equal what actually landed.
Demotions were unaffected in that incident because IncrementNotUseful
writes immediately (297 of 377 landed); they are now also correct
additive increments thanks to the earlier col+1 fix.
Passing namespace:'' silently scanned only hermes-main because of a
hardcoded default, and the response echoed back 'hermes-main' so the
missed scope was invisible - easy to mistake for a completed full sweep.
Empty now means all namespaces, and the response carries a by_namespace
breakdown (clusters/deleted/demoted) so a full run is auditable.
Two fixes, both silent-failure class.
1. RustLanceDBClient.IncrementNotUseful was an empty function body.
Every demotion path was a false success: the HTTP endpoint returned
200, zhiyi-dedup.py reported '34 demoted, 0 failed', and
not_useful_count stayed 0 on all 32 dropped rows. Now delegates to
Update with $inc.
2. Update emulated $inc in Go using the _local.memories cache, which is
only populated on the Insert path (Search does not fill it). On a
cache miss getIncCachedValue returned 1, so an increment wrote the
absolute value 1 - resetting an existing count of e.g. 5 back to 1.
Now the column is sent to Rust as the expression 'col + 1'; Rust's
update() detects the arithmetic, reads the row's true current value
and computes the result, so the increment is genuinely additive.
Also implements 牧尘's chosen dedup policy (option A) in admin/dedup:
keep the NEWEST entry per cluster, then for every other member
- if it normalises to the same text (punctuation/whitespace/quote
differences only) it is a true duplicate -> soft delete
- otherwise it carries different information (different numbers,
subject, timestamps, versions) -> demote only, never delete
This satisfies 'newest wins' without discarding distinct facts.
Policy set by 牧尘 on 2026-09-20:
- existing duplicates -> keep the MOST COMPLETE entry
(longest content; ties broken by earliest)
- future writes -> newest wins (handled by the write-side gate
in commit/distill)
Two behavioural fixes over the previous pairwise loop:
1. Union-find clustering instead of pairwise marking. With chained
similarity (A~B, B~C, but A!~C below threshold) the pairwise loop
counted two pairs and left the cluster with multiple survivors, so
'keep the fullest' silently kept several entries. Measured: 220
clusters via closure vs 384 via pairwise - ~12% undercount.
2. Clustering is confined to a single namespace. Identical text in two
namespaces is two agents' own copies (e.g. domain:INSIGHT and
hermes-main), which is intended, not duplication. The early string
compare also keeps the full-table O(n^2) pass cheap.
Response now reports clusters/merged/deleted/policy, and an empty
namespace means 'all namespaces' rather than defaulting to hermes-main.
RustLanceDBClient.Search decoded the Rust IPC response into a struct
that had no vector field, so every MemoryRecord returned to Go had an
empty Vector. Consequence chain:
len(results[i].Vector) > 0 && len(results[j].Vector) > 0
always false → the cosine branch was skipped in:
- commit path near_dup (core.go:634)
- distill direct-write dedup gate (server.go:179) <-- the fix I just
deployed this round was silently a no-op
- admin/dedup endpoint (core.go:634) <-- merged≈0 false
negative
Without this, no near-duplicate detection in the whole system works —
it silently fell back to exact string comparison only.
Retained: the vector is forwarded from the raw IPC JSON; if Rust stops
returning it, the degenerate behaviour is exact-string comparison
(which still works), rather than a crash.
The endpoint passed the namespace straight into LanceDB.Search with
top_k=200. Rust ranks by distance to the zero vector globally and only
then applies the namespace filter, so a namespace's rows were mostly
never returned: soulful-xiaowei has 238 rows but only 9 came back,
hermes-main has 3387 but only 200 did. The endpoint therefore reported
merged=0 on every call - a false negative that made memory
deduplication look complete when a full cosine scan finds 404
redundant rows at the 0.98 threshold.
Now it pulls the whole table (top_k=1000000, no namespace filter) and
filters by namespace plus is_deleted in Go, so 'no duplicates' is a
claim the endpoint can actually support.
Previously distill's direct-write path only skipped facts whose exact
content already existed (sim[0].Content == fact). This let semantically
equivalent facts - e.g. "牧尘在招商局工作" vs "牧尘在招商局工作。"
(trailing period) - slip through as separate memories, accounting for
most of the 404 redundant entries found by a full-vector cosine scan.
Now distill encodes each fact, searches 5 nearest neighbors in the same
namespace, and skips insertion if any neighbor has either identical
content OR cosine similarity >= 0.98. The same threshold is shared by
the commit path (routes/core.go), via a new api-level cosineSimilarity
helper.
This is preventative only: it does NOT merge existing duplicates.
Existing 404 redundant entries remain; they will be cleaned up by a
separate manual dedup run after the gate is live.
小唯