We Published turbomem's Memory Evals

Memory products are easy to demo and hard to measure. A hiking fact comes back for “outdoor activities.” A dog’s name survives a new chat. That is a good gut check. It is not a eval.

We shipped public evals for turbomem anyway. The page is at turbomem.dev/evals. The protocol is at docs.turbomem.dev/evals. The frozen snapshot is in the repo as packages/evals/results/latest.json.

The headline number is 27.1% official LoCoMo F1 on locomo10.json (1,986 questions, 10 conversations). That is not an LLM-as-judge score. It is the category-aware token F1 from the LoCoMo evaluation script, run on the default product stack: in-memory PGlite, OpenAI text-embedding-3-small, gpt-4.1-mini for extraction and answering, search k=10, write-time dedup on.

We are not publishing a Mem0 or Zep column. Those numbers are not comparable unless the judge, dataset file, k, and ingest protocol match. Ours are written down so you can disagree with them in public.

Why we did this

turbomem is an embedded library. You npm install, call memory.add(), call memory.search(). The marketing story is architectural: no memory server, scoped facts, cosine retrieval.

Architecture claims are easy to overfit to anecdotes. The hiking example in our docs is one transcript. LoCoMo is ten conversations, hundreds of sessions, and questions that span weeks of dialogue. If we were going to blog about quality, the protocol had to exist first.

Two suites:

1. Product goldens - the claims we actually make (extraction, semantic recall, dedup, multi-tenant scoping).
2. LoCoMo - the academic benchmark people already cite, scored the way the authors scored it.

What we ran

Snapshot date 2026-09-28. Config chips on the evals page match this table.

We used the default extraction model on purpose. A stronger chat model would change the number, and not always upward: official LoCoMo F1 punishes extra words. Evaluating gpt-4o would be “turbomem plus 4o,” which is not what npm install turbomem does.

Product goldens

These are small, first-party cases. Treat them as contract tests for the pipeline, not as a substitute for LoCoMo.

The hiking / marathon / vegetarian transcript from the docs is in here. So is a noise-only chat (“Thanks!”, “Can you rephrase that?”) that should not produce durable facts, a dog named Maple, and user/session isolation.

Read the small n honestly. Five retrieval queries at 100% means the goldens are on-distribution for dense search. It does not mean LoCoMo single-hop is solved. Dedup at 2/3 means the more-specific replace path did not fire the way we wanted on that case. Scoping at zero leaks is the result I would bet production on: userId / sessionId filters are SQL, not a model.

Latency in this snapshot is OpenAI-bound (add p50 ~465 ms, search p50 ~422 ms). That is embedding plus extraction on the live path, not a PGlite SLA.

LoCoMo

LoCoMo (Maharana et al., ACL 2024) is long-term conversational QA. We download locomo10.json. We do not vendor it.

For each of the 10 conversations:

1. Map each session_N to turbomem messages. Speaker A is user, speaker B is assistant. Session timestamps and image captions stay in the text so the extractor can see them.
2. memory.add(session, { userId }) - extract, embed, dedup.
3. For each question, memory.search(question, { userId, limit: 10 }).
4. A short-answer LLM sees retrieved facts only. If nothing is there, it is told to say no information available.
5. Score with a TypeScript port of task_eval/evaluation.py.

Category IDs

The integer category field in locomo10.json does not match the numbered list in the paper. The evaluation code is the source of truth:

Token F1 lowercases, strips punctuation, drops {a, an, the, and}, Porter-stems, then computes overlap. Category 1 splits on commas. Category 3 keeps gold only before the first ;. Category 5 is a refusal check (no information available / not mentioned), not token overlap.

Many write-ups drop adversarial questions and report 1,540 items. On categories 1-4 only, our weighted F1 is about 11%. Overall 27% is real, and a large slice of it is “we abstained on adversarial prompts.” That is worth saying out loud.

Search p50 on this run was ~469 ms. Answer-side tokens averaged ~251 per question.

What the number is actually saying

Temporal is the hole. 1.8% F1 on 321 questions. turbomem stores third-person fact strings. Session timestamps are in the ingest transcript, but nothing in the default pipeline builds a timeline. “When did X happen?” is the wrong question for a bag of cosine-ranked sentences unless extraction wrote the date into the fact and retrieval found it.

Single-hop at 14% is the other tell. These questions have evidence in one session. If extract-then-search were reliably keeping the needle, this bucket would be much higher. Long LoCoMo sessions are dense. One extraction call per session can drop details. Dedup can merge them into something the later QA prompt does not match after stemming.

Adversarial at 82% is the pipeline behaving as instructed. Empty or irrelevant retrieval should produce a refusal. That is easier than recalling a buried date.

None of this is an argument that LoCoMo is a bad benchmark. It is an argument that default dense memory is not a long-context database. We would rather publish that than imply otherwise.

What we are not claiming

• We are not claiming a win over Mem0, Zep, or anyone else.
• We are not claiming 27% is a product SLA.
• We are not claiming the 100% product retrieval goldens generalize to LoCoMo.
• We are not using LLM-as-judge as the LoCoMo headline. Judge scores are too easy to move by swapping the judge.

If you see a 90% LoCoMo number in the wild, read the methodology first: dataset variant, whether category 5 is included, answer model, k, reranker, and whether F1 or a judge was used. Scores disagree for those reasons, not because one library is magically done.

Reproduce it

cd turbomem
pnpm install
pnpm --filter @turbomem/evals test          # no API key
pnpm --filter @turbomem/evals download-locomo
export OPENAI_API_KEY=sk-...
pnpm --filter @turbomem/evals exec tsx src/cli.ts --suite all --publish

A full LoCoMo pass is on the order of an hour or two and about a dollar in gpt-4.1-mini plus embeddings. --smoke is the cheap path (one conversation, 20 questions) and must not be frozen as public.

The snapshot schema pins gitSha, date, models, k, and per-category n. If those fields are missing, the percentage is incomplete.

What we will do next

The evals are a measurement loop, not a trophy.

The obvious gaps from this freeze: temporal facts, extraction recall on long sessions, and the replace/dedup case that missed. Hybrid search is out of scope until the core is not dense-only. A second snapshot with a different extractor is an ablation, not a replacement for the default.

Until then, the public number is 27.1% official F1, with the table above, on the stack we actually ship.

Live scores: turbomem.dev/evals. Protocol: docs.turbomem.dev/evals.

Citation

LoCoMo:

@article{maharana2024evaluating,
  title={Evaluating very long-term conversational memory of llm agents},
  author={Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei},
  journal={arXiv preprint arXiv:2402.17753},
  year={2024}
}