SQLiteMemClient.IncrementUseful/IncrementNotUseful were discarding the
error return of MemLanceClient.IncrementUseful/IncrementNotUseful.
Since the latter now return real errors (6e67f3e), dropping them
re-introduces the silent-failure class of bug we just fixed across
the Rust path. Also propagate persistMemory errors.
- IncrementUseful/IncrementNotUseful now return error and persist via
rc.Update() (Rust sidecar); all 4 implementations updated
- SoftDeleteBatch now delegates to SoftDelete() for real LanceDB write
- UpdateMemoryContent now persists content + version + version history
- Backup changed from return nil to explicit error (Rust sidecar unsupported)
- IncrementUseful/IncrementNotUseful signatures now include error return;
all 3 call sites (feedback.go, core.go) check the error
- Test mocks (mockLance, fakeLDB) updated to match new interface
Fixes the 'silent no-op' class of bugs discovered during dedup repair.
Two write-path fixes found while executing the dedup policy.
1. The endpoint wrote is_deleted via Update(). The system's canonical
path is SoftDelete(), which routes through SoftDeleteWriteBuffer
(5-minute window / 256-item threshold, single-transaction commit).
Update() writes one MVCC version per row, so this endpoint would
have created 411 versions in one run - the exact growth pattern the
vacuum script was written to clean up (table measured at 3.23G
before a 40M compaction).
2. The response was sent while soft deletes were still buffered. During
execution lance-vacuum (daily 03:30) stopped the service and ran
optimize(cleanup_older_than=0), discarding the unflushed writes: the
endpoint's deleted=34 never landed and is_deleted read back all
false, while the caller only saw curl exit=52 for the empty reply.
A new storage.FlushSoftDeleteBuffer() is now called before the
response, so reported counts equal what actually landed.
Demotions were unaffected in that incident because IncrementNotUseful
writes immediately (297 of 377 landed); they are now also correct
additive increments thanks to the earlier col+1 fix.
Two fixes, both silent-failure class.
1. RustLanceDBClient.IncrementNotUseful was an empty function body.
Every demotion path was a false success: the HTTP endpoint returned
200, zhiyi-dedup.py reported '34 demoted, 0 failed', and
not_useful_count stayed 0 on all 32 dropped rows. Now delegates to
Update with $inc.
2. Update emulated $inc in Go using the _local.memories cache, which is
only populated on the Insert path (Search does not fill it). On a
cache miss getIncCachedValue returned 1, so an increment wrote the
absolute value 1 - resetting an existing count of e.g. 5 back to 1.
Now the column is sent to Rust as the expression 'col + 1'; Rust's
update() detects the arithmetic, reads the row's true current value
and computes the result, so the increment is genuinely additive.
Also implements 牧尘's chosen dedup policy (option A) in admin/dedup:
keep the NEWEST entry per cluster, then for every other member
- if it normalises to the same text (punctuation/whitespace/quote
differences only) it is a true duplicate -> soft delete
- otherwise it carries different information (different numbers,
subject, timestamps, versions) -> demote only, never delete
This satisfies 'newest wins' without discarding distinct facts.
RustLanceDBClient.Search decoded the Rust IPC response into a struct
that had no vector field, so every MemoryRecord returned to Go had an
empty Vector. Consequence chain:
len(results[i].Vector) > 0 && len(results[j].Vector) > 0
always false → the cosine branch was skipped in:
- commit path near_dup (core.go:634)
- distill direct-write dedup gate (server.go:179) <-- the fix I just
deployed this round was silently a no-op
- admin/dedup endpoint (core.go:634) <-- merged≈0 false
negative
Without this, no near-duplicate detection in the whole system works —
it silently fell back to exact string comparison only.
Retained: the vector is forwarded from the raw IPC JSON; if Rust stops
returning it, the degenerate behaviour is exact-string comparison
(which still works), rather than a crash.