Previously distill's direct-write path only skipped facts whose exact content already existed (sim[0].Content == fact). This let semantically equivalent facts - e.g. "牧尘在招商局工作" vs "牧尘在招商局工作。" (trailing period) - slip through as separate memories, accounting for most of the 404 redundant entries found by a full-vector cosine scan. Now distill encodes each fact, searches 5 nearest neighbors in the same namespace, and skips insertion if any neighbor has either identical content OR cosine similarity >= 0.98. The same threshold is shared by the commit path (routes/core.go), via a new api-level cosineSimilarity helper. This is preventative only: it does NOT merge existing duplicates. Existing 404 redundant entries remain; they will be cleaned up by a separate manual dedup run after the gate is live.
24 lines
622 B
Go
24 lines
622 B
Go
package api
|
||
|
||
import "math"
|
||
|
||
// cosineSimilarity 计算两个向量的余弦相似度。
|
||
// 放在 api 包根目录,让 server(蒸馏写入路径)和 routes(commit 路径)
|
||
// 共享同一套近重复判据(>=0.98),避免两套逻辑各自漂移。
|
||
func cosineSimilarity(a, b []float32) float64 {
|
||
var dot, normA, normB float64
|
||
n := len(a)
|
||
if len(b) < n {
|
||
n = len(b)
|
||
}
|
||
for i := 0; i < n; i++ {
|
||
dot += float64(a[i]) * float64(b[i])
|
||
normA += float64(a[i]) * float64(a[i])
|
||
normB += float64(b[i]) * float64(b[i])
|
||
}
|
||
if normA == 0 || normB == 0 {
|
||
return 0
|
||
}
|
||
return dot / math.Sqrt(normA*normB)
|
||
}
|