What Does AI Memory Remember? Testing Civilization on Long Conversations

A memory system is not useful just because it can store a conversation. It has to retrieve the right detail later, keep older facts from drowning out newer ones, and make its evidence inspectable.

We ran Civilization, an open-source decision service with native memory, against the complete 500-question LongMemEval set, then added focused samples from BEAM and LOCOMO. The full LongMemEval result is 69.8% at the Top-20 cutoff. That score is useful, but the breakdown matters more: performance ranges from 50.0% on single-session preferences to 87.2% on knowledge updates.

What we connected

Civilization can use an OpenAI-compatible endpoint as its decision provider while its Orion memory service stores and retrieves episodic memories. For this experiment, the model answered benchmark prompts while Civilization handled memory writes and searches through its native APIs. We connected the open-source memory-benchmarks suite to those endpoints.

The benchmark project currently documents Mem0 Cloud and Mem0 OSS backends. For this run, we added a small Civilization adapter in our local benchmark checkout. It writes each conversation chunk as an episodic memory and routes search requests to Civilization. The adapter is local evaluation glue; it is not part of the upstream benchmark repository.

Results across three benchmarks

The LongMemEval run processed all 500 questions across six question types. At the Top-20 cutoff, Civilization answered 349/500 correctly for an overall accuracy of 69.8%.

Multi-session and preference questions were the weakest categories. These scores identify where this evaluated pipeline missed; they do not by themselves reveal whether retrieval, answer generation, or the judge caused a miss. Per-question retrieval evidence should be reviewed before drawing that conclusion.

We also ran three smaller rounds on BEAM and LOCOMO:

The BEAM round used conversation 0 from its 100K bucket. We selected information extraction, multi-session reasoning, and temporal reasoning, with two questions per ability. The full history was split into 94 memory chunks before answering.

The BEAM result needs a closer read. Its judge gives partial credit, and a result counts as passing at a score of 0.5 or higher. All six questions crossed that threshold, but the mean score was 0.792; multi-session reasoning averaged 0.5. “Six out of six passed” is not the same as “the system answered every detail perfectly.”

The two LOCOMO samples returned different results: 60% for the first conversation and 100% for the second. Across these ten selected questions, the total is 8/10. We do not combine that percentage with BEAM because the datasets and scoring rules differ.

Where persistent memory can help

Coding agents need project continuity. An agent that returns to a repository should be able to recover decisions, constraints, and prior findings from earlier sessions. That can reduce repeated setup and prevent it from confidently proposing a change that was already ruled out.

Assistants need history, not just profile fields. Preferences and facts can change. A useful memory system should surface the relevant conversation, retain temporal context, and let the application inspect which memory supported its answer.

Long-running workflows need an audit trail. In support, operations, and research workflows, it matters which facts were retrieved and when. Civilization's auditable service path makes that trace available to the caller. This experiment covers provider-based text inference and native memory retrieval; it does not test the separate local Civilization Adapter path.

These are promising application areas, not product integrations validated by this benchmark. The test covers memory writes, retrieval, answer generation, and model judging on LongMemEval's full set and selected BEAM/LOCOMO samples. It does not verify a production Hydite workflow or the separate Civilization Adapter execution path.

What this result does not prove

LongMemEval used all 500 questions. BEAM and LOCOMO remain focused samples: one BEAM conversation and five questions from each of two LOCOMO conversations. Those rows are not full BEAM or LOCOMO scores. We also did not run a matched Mem0 or other-system baseline, so these results are not evidence that Civilization outperforms another memory system.

The same model generated and judged the answers, so the scores can reflect judge bias. A stronger evaluation would use more conversations, all benchmark question types, an independent judge, and a matched baseline with identical prompts and retrieval cutoffs.

For now, the practical takeaway is simple: test the retrieval path with long, multi-session histories and inspect the evidence behind each answer. Storage volume alone does not tell us whether an agent remembers well.

Open-source projects and datasets

• Civilization — auditable decision service and native memory.
• Memory Benchmarks — benchmark runners and evaluation workflow.
• BEAM dataset and BEAM paper — long-conversation evaluation, including the 100K bucket.
• LOCOMO dataset and code — multi-session conversations with annotated question-answer tasks.