Civilization 1.0.0: An Auditable Foundation for Long-Term Memory

Research release note · 29 October 2026
Civilization 1.0.0 is our first formal software release. Its latest complete LongMemEval Top-20 record is Stage311: 449/500 (89.8%). Our best complete run remains Stage299: 461/500 (92.2%). The project’s internal release gate is 485/500 (97%); the current release record is 36 answers short. Stage311 was the last complete run before the 1.0.0 tag; it was not rerun on the tag.

Abstract

Civilization 1.0.0 packages a persistent, inspectable memory layer for assistants and agent systems. The release brings together Python and TypeScript SDKs, a service baseline, multiple memory stores, session and global memory, access controls, asynchronous jobs, export, and encrypted backup workflows. The software release is complete; the separate long-term-memory accuracy gate is not.

We report the project’s accumulated LongMemEval-S Top-20 results and the experimental path behind them. The latest pre-tag complete 500-question answer-and-judge run scored 449/500. The historical best, Stage299, scored 461/500. Both remain below the project’s 485/500 target. The 67-run history is non-monotonic: focused changes can fix a specific failure while changing hundreds of answers elsewhere. We therefore distinguish full answer-generation runs from frozen-answer replays and composite scores, and do not attribute aggregate changes to a single feature without a controlled paired design.

The remaining high-impact error classes are multi-session synthesis and temporal reasoning. Retrieval coverage is necessary but insufficient: answer-marked evidence can be present in the Top-20 while the answer remains wrong. Stored-memory factual precision has not been measured at 100%, a live offsite disaster-recovery restore has not been verified, and the Stage166 token-reservation prototype has no production-generation integration or LongMemEval efficacy result.

1. Release scope and research claim

Civilization 1.0.0 is the first formal release of the Python package, TypeScript SDK, and service baseline. The release includes a dependency-free Python client and CLI, request and prediction models, runtime registry, provider and local-Transformers runtime options, audited local-adapter integration, decision traces, working/episodic/semantic/procedural memory stores, persistent session and global memory, asynchronous jobs, access controls, rate limits, export delivery, encrypted backups, and optional S3-compatible backup replication.

The software version and the benchmark target answer different questions. Version 1.0.0 marks a shippable engineering baseline; it does not certify that long-term memory has met the project’s research threshold. In particular, the 97% line is Civilization’s internal release gate, defined as at least 485 fully correct answers on the 500-question Top-20 run. It is not presented as a pass threshold set by the LongMemEval authors.

Civilization is open source under Apache License 2.0. Source code, release notes, and the versioned scorecard are available in the Civilization repository, the v1.0.0 changelog, and the LongMemEval scorecard.

2. Evaluation protocol

The LongMemEval paper introduces 500 questions over long interaction histories and studies five long-term-memory capabilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Our scorecard separates single-session questions into assistant-output, user-fact, and preference strata for diagnosis, producing six internal reporting groups: knowledge update, multi-session, single-session assistant, single-session preference, single-session user facts, and temporal reasoning. See the LongMemEval paper and official repository README for benchmark and dataset definitions.

Dataset history is not the model prompt

The runner’s default is the cleaned LongMemEval-S dataset: 500 questions. The benchmark README estimates that concatenating one S question’s full chat history is about 115K tokens (roughly 40 sessions, measured for Llama 3). Multiplying by 500 gives approximately 57.5M question-history token-instances across the evaluation. This describes the histories associated with the 500 questions; it is neither a count of unique corpus tokens nor the number of tokens sent to GPT-6 Luna.

LongMemEval-M is a separate, much larger variant: its histories contain roughly 500 sessions per question and are on the order of 1.5M tokens in this dataset-scale estimate. The Civilization results in this article use S, not M.

For Stage311, the frozen Top-200 retrieval snapshot contained 99,988 memory records across 500 questions, with 194–200 candidates per question. The answerer saw the first 20 candidates for each question, or 10,000 memory selections across the run (selections can repeat across questions). Stage311 reused Stage299’s frozen candidate rankings, so this is not a fresh live-retrieval count. The saved run did not include prompt_tokens or token-usage records; actual API input-token totals therefore cannot be reported.

For the project’s release metric, the retriever produces up to 200 candidates and the answerer receives the first 20. Every accepted full run must complete all 500 questions under the Top-20 cutoff. A question counts toward the release gate only when the strict judge marks the answer fully correct; partial judge scores do not count as a correct answer. Top-200 answer diagnostics are useful for debugging but do not satisfy the Top-20 gate.

The Stage299 and Stage311 answerer and judge used GPT-6 Luna (gpt-6-luna). The test model metadata reported:

The route’s effective context and input ceiling was 272,000 tokens; this is the tested service configuration, not a claim that each prompt used 272,000 tokens. OpenAI’s GPT-6 Luna model specification separately lists a 1,050,000-token model-level context window. The benchmark client sets a default per-request output cap of 4,096 tokens, below the route’s 128,000-token maximum. These limits describe capacity, not measured use: the run did not save per-request prompt or completion usage.

Stage311 regenerated answers and judgments for all 500 questions, but reused Stage299’s frozen Top-200 candidate rankings and did not call a live reranker. It is therefore a complete 500-question answer run, not a fresh end-to-end retrieval rerun on the v1.0.0 tag. The run metadata reports accepted answerer and judge seeds; that did not make generation or reranker selections deterministic.

We track retrieval evidence, answer accuracy, structural answer-audit validity, memory-fact precision, and backup recovery as separate quantities. A retrieved source session does not show that the exact answer-bearing statements were retrieved. An answer audit passing its schema is not proof that all facts stored in memory are correct. A frozen replay is not a new full-run score.

3. Release result

Stage311 and Stage299 by question type

Stage311 is 36 correct answers below the internal 485-answer gate. Stage299 is 24 below it. Compared with Stage299, Stage311 has 11 candidate-only correct answers and 23 Stage299-only correct answers, for a net change of −12. The two runs both answered 438 questions correctly and failed 28; answer text changed on 258 rows. Because the run configuration and prompt source differ, this is not a controlled estimate of the capacity guard’s effect.

Stage311 evaluated a narrow cross-capacity count guard. In the frozen target replay, retrieved user-authored memories described fish in a 20-gallon tank while the question asked about a 30-gallon tank; a cited abstention passed the live judge. In the full 500-question run, the guard fired zero times because the answerer independently abstained on that row. The guard is kept default-off, and the 449/500 result is not credited to it.

The main unresolved areas are multi-session and time-dependent questions. Stage311 answered 84.2% of multi-session questions and 86.5% of temporal questions, compared with 98.2% on single-session assistant questions and 100% on preference questions. The latter results show that high scores on narrow single-session strata do not remove the remaining cross-session and temporal error burden.

4. Research path: bottlenecks, interventions, and sample types

The stages below are research iterations, not a monotonic feature ladder. Retrieval snapshots, prompts, answerer settings, and reranking scope changed over time. The 67-run table later in this article preserves the measured scores; the account here explains what each phase was designed to test and what the evidence did—and did not—support.

4.1 From retrieval recall to answer construction: Baseline–Stage184

The baseline scored 349/500 (69.8%). On 470 questions with an annotated answer session, a source-session chunk appeared in 446 Top-20 lists and all 470 Top-200 lists. Stage162 raised the complete Top-20 result to 414/500; a separate Top-200 diagnostic scored 451/500, but that diagnostic is excluded from the release metric because the answerer saw 200 memories. Stage163’s speaker-aware run reached 424/500 and retrieved a source-session chunk in 458/470 annotated Top-20 cases. Those improvements showed that candidate recall and speaker attribution mattered, but did not make the answer step reliable.

Stage165 combined bounded source-session retrieval, Top-20 reranking, and structured cited answers. It had a source-session chunk in all 470 annotated Top-20 cases, yet scored 416/500 and missed 79 of those 470 questions. Stage168 added event-date intervals and before/after anchors. It reached 426/500 and 470/470 source-session coverage, while only 401 of the 470 questions were answered correctly. Stage166’s temporal-event fields and anchor checks slightly improved the temporal slice, from 93/133 to 95/133, while its full result was 414/500. Together, these runs shifted the diagnosis: retrieving a conversation containing the answer is not the same as retrieving every required fact, resolving its time and speaker, and synthesizing a complete answer.

The count-inventory experiments exposed a second failure mode. Stage169’s initial inventory over-counted cumulative snapshots and routed scalar arithmetic questions through item enumeration; its full score fell to 409/500. Stage171 narrowed the inventory path to enumerative counts and treated cumulative snapshots as updates, but scored 418/500, eight below Stage168 on the paired comparison. Stage174 added citation-backed verification for selected non-numeric multi-session, knowledge-update, and temporal answers. The verifier revised 92 answers, 74 of which were judged correct, yet paired outcomes were balanced at 24 gains and 24 losses against Stage168: net zero. Stage178 disabled count inventory and scored 424/500, close to the enabled variant.

Stage184 then skipped pre-answer inventory specifically for multi-session count questions. It scored 429/500 and improved the 95-question multi-session count subset to 74/95 from 70/95 in Stage174. This small paired improvement supported a narrower policy, not a broad claim that count inventory solves multi-session reasoning. A later 94-question frozen-evidence comparison reached the same directional decision: skip-multi-session scored 73/94 versus 70/94 when inventory was applied to all questions.

Sample types: speaker-attributed personal facts, facts that change over time, event dates and ordering, enumerative counts, cumulative snapshots, scalar quantities, and multi-session count questions.

4.2 Deterministic corrections and frozen-response composition: Stage188–Stage219

The next phase tested whether deterministic post-processing could repair specific date, count, and preference errors without rewriting the whole answer path. Stage188 applied temporal rules to saved answers and scored 434/500 on a frozen-response replay. Its temporal stratum reached 111/133, but fresh 133-question temporal replays scored 105/133 and then 106/133; they did not reproduce the five-answer frozen lift. We retain the result as evidence about fixed outputs, not as a fresh-run accuracy gain.

Other narrow experiments included a 30-question preference replay (Stage195), explicit-total deduplication for same-unit item sums (Stage196), physical-object counting (Stage203, 12 questions), user-authored evidence roles for personal counts (Stage213, one targeted row), relative time-window counts (Stage216, 19 questions), and an observation-date fallback for a just-completed event (Stage219, one question). Their 500-row totals—437, 439, 441, 442, 443, and 444—were assembled by replacing only the targeted rows and retaining other saved answers. They are useful engineering records, but they are not fresh all-500 answer-generation runs.

The pair-recall audit at Stage203 further separated retrieval and reasoning. Of 883 annotated answer-bearing message pairs, 867 appeared in Top-20; all annotated pairs were present for 456 of 470 answerable questions. Yet the answerer still missed 14 multi-session questions whose annotated pairs were all present. Remaining errors therefore included both evidence selection and reasoning over retrieved evidence.

Sample types: event-to-event intervals, two-option temporal order, preference facts, duplicate totals, physical objects, user-versus-assistant claims, relative count windows, and explicit observation dates.

4.3 Retrieval snapshots, reranking, and verification: Stage221–Stage261

Stage221’s new-provider full run scored 390/500. It became a frozen retrieval snapshot for subsequent answerer experiments; later results that reused this snapshot isolate answer-generation choices more cleanly than live retrieval, but do not measure a changed retriever. The initial reranker sequence was mixed: Stage223 scored 418, Stage224 422, Stage225 430, and Stage226 419. Stage227’s matched control scored 385/500 while its four-type reranker arm scored 433/500, showing that candidate ordering could make a large difference on that configuration. It did not establish a stable gain across reranker repeats.

Further answer checks and task-specific temporal rules also varied. Stage230 scored 427/500; Stage232’s single-session verification candidate scored 432; Stage239’s count-aware verification candidate scored 422; and Stage241’s cited temporal list-order candidate scored 433. Stage243 added a relative-week temporal candidate and scored 438/500. It retrieved a source-session chunk for all 470 annotated questions and 871/883 annotated answer pairs, but 47 of 460 questions with all annotated answer pairs in Top-20 were still wrong. That residual error is direct evidence that pair coverage alone does not guarantee correct synthesis or temporal interpretation.

A 94-question frozen A/B at Stage245 compared count policies on identical Top-20 evidence: skipping inventory for multi-session counts scored 73/94, versus 70/94 for the all-inventory route. We retained the narrower policy. Stage257 then freshly regenerated all 500 answers with six opt-in resolvers and reranking disabled; it scored 434/500, below Stage243’s 438. A two-type fact-ledger verifier was negative on its 133-question multi-session pilot (Stage258: 106/133 vs 108/133 control); a restricted non-numeric pilot was also slightly negative (Stage259: 45/66 vs 46/66 across three seeds) and was reverted. Source-backed temporal resolvers corrected specific frozen rows in Stages260–261, while both fresh full candidates scored 432/500; the new rules did not explain the aggregate score.

The matched Stage265 numeric-verification ablation was favorable on its 111-question multi-session slice: verification enabled scored 89/111 versus 84/111 with it disabled. A separate Stage266 audit found that 58 of 65 wrong answerable rows already had all annotated answer pairs in Top-20. The slice result supports keeping the verifier in that benchmark configuration; the audit shows why verification and synthesis remain necessary after retrieval. Neither result is a full-run score or a measure of factual precision across stored memories.

Sample types: multi-session and temporal questions, retrieval order, citation validation, counts with source-role constraints, and answer verification over the same frozen candidate lists.

4.4 Evidence-bound, question-specific resolvers: Stage262–Stage295

After Stage243, the work moved from broad pipeline changes to narrow rules whose preconditions could be checked against cited user-authored evidence. This phase deliberately covered heterogeneous sample types rather than assuming one general rule would solve them: event aliases and ordinal order; transport prices; contradictory currency totals; education-year spans; parenthood comparisons; property offers before/after an event; missing count state; whether two earrings represent one acquired pair; prior-career duration; weekday event matching; poster venues; loyalty-point balances; pending pickup and return tasks; yearless event dates; professional-role counts; planned consultations versus completed visits; later social-follower estimates; cumulative collection totals; sport-specific frequency; assistant-authored output; wake-time updates; elapsed days; preference extraction; and open temporal-window boundaries.

These rules were usually default-off and had explicit evidence requirements. For example, Stage283’s role-count guard tested whether an engineer count was supported by a statement about a different professional role; Stage284 distinguished a planned doctor consultation from a completed visit; Stage287 distinguished tennis evidence from table-tennis frequency; Stage288 allowed assistant-authored evidence only when the question asked what the assistant had previously said or created. Stage282’s temporal resolver used recent month/day dates only when both events shared the question-date anchor. The preference experiments separated research-topic preferences, general single-session preference facts, and hotel filters that could transfer to a new destination only when no destination-specific hotel fact conflicted.

Frozen target replays often fixed the intended row and passed citation validation. Full 500-question results nevertheless moved both up and down because hundreds of answer texts changed under the end-to-end run. Examples include Stage268 at 440/500, Stage277 at 441, Stage281 at 442, Stage283 at 446, Stage286 at 447, Stage288 at 451, and Stage291 at 452. These are complete scores, but a one-row correction cannot be credited with the aggregate change when paired comparisons show hundreds of changed responses. Stage291 became the best complete result before the reranker repeat experiments.

Sample types: exact user-source pairs, role and action states, explicit numeric units, conflicting or cumulative facts, event dates, dated preferences, and evidence-supported abstention. The engineering objective was not to add a rule for every wording pattern; it was to test whether each proposed correction had a narrow trigger, a verifiable source, a safe no-change path, and a measured effect on the relevant slice.

4.5 Reranker scope and repeatability: Stage296–Stage300

Stage296 re-evaluated a four-type reranker over multi-session, temporal, preference, and knowledge-update questions and scored 448/500. Stage297 restricted reranking to the two weakest broad strata—multi-session and temporal—and scored 456/500, then the best complete run. Stage298 tried preserving original evidence order on count-inventory rows inside reranked categories. The target subset lost three net answers, the full candidate scored 449, and the experimental patch was discarded.

Stage299 repeated Stage297’s selective reranker configuration over the same frozen retrieval snapshot with the same accepted seed. It scored 461/500, five above Stage297, and became the historical best. Repeatability remained a concern: among 266 reranked rows, only 22 preserved the exact selected order and 82 preserved the same selected set as Stage297. Only 272/500 final Top-20 orders matched the original retrieval order. The answerer and judge accepted their seeds, but those seeds did not make the reranker path deterministic. Stage300 repeated the settings and scored 452/500, nine below Stage299. We therefore report 461/500 as the best observed run, not as a stable estimate or a causal effect of reranking.

Sample types: the broad multi-session and temporal strata, plus count questions where reranking could displace relevant evidence. This phase exposed a systems-level bottleneck: a plausible reranker can improve aggregate answers, but its selection variance must be controlled before the gain can be treated as repeatable.

4.6 Final audits and release record: Stage301–Stage311

Stages301–307 examined cross-session fact ledgers, temporal event ledgers, source-pair coverage, explicit abstention, and one additional structured answer repair. Stage302’s fact-ledger verifier revised six of 22 eligible examples and corrected two failures without regressing a Stage300 pass. The multi-session category rose from 109/133 to 111/133, but the full run scored 450/500; 255 answers changed, so the overall movement cannot be assigned to the ledger. In Stage306, an extra repair pass was attempted on 24 rows and accepted on three; none changed the judge label. Structural audit errors fell from 29 to 21, while the full score was 444/500. Audit validity and factual memory precision remain different measures.

Stage308 tested reciprocal-rank fusion offline on the 266 multi-session and temporal rows. The sparse answer-pair retrieval proxy moved from 869/883 to 871/883, but the paired answer pilot fell from 231/266 to 226/266 and invalid audits rose from 17 to 20. We kept the fusion utility as an offline research tool and did not enable it in the default runtime. The two-pair proxy gain is not a measurement of memory-fact precision.

Stage310 tested a relative-mention-date fallback on four temporal questions. It corrected one answer with an exact user-authored event citation; the paired slice moved from 1/4 to 2/4. It fired once in the fresh 500-question candidate, which scored 444/500, so the aggregate score was not attributed to the fallback. Stage311 then tested the cross-capacity count guard described above. The frozen target replay passed; the guard did not trigger in the full run. The latest complete Top-20 record remained 449/500.

Sample types: fact-state reconciliation, temporal event references, explicit abstention, malformed answer audits, rank-fusion candidates, date mentions, and mismatched physical-capacity counts. The final phase emphasized auditable triggers and safe abstention, while preserving the distinction between a successful targeted replay and an aggregate benchmark result.

5. Full 500-question Top-20 run ledger

The following table contains the 67 recorded complete 500-question Top-20 results, including the initial baseline and the two Stage227 arms. It excludes Top-200 diagnostics and the frozen/composite records listed separately below. This is a chronological record, not a controlled ablation table: not every adjacent pair shares identical prompts, retrieval, or model conditions.

6. Frozen replays and composite scores

These 500-row totals were useful for examining deterministic fixes and targeted subsets. Some rows reused previous answers or judgments; they must not be read as fresh, complete 500-question answer-generation runs.

Top-200 answer diagnostics are also excluded. Stage162 and Stage163 each had a Top-200 diagnostic above their respective Top-20 scores, but those runs gave the answerer 200 memories and do not satisfy the release cutoff.

7. Interpretation and limitations

The score history establishes progress from 349/500 at baseline to a best observed complete run of 461/500, followed by 449/500 as the latest pre-tag complete run. It does not establish that every feature improved the system. Targeted replays often answer a narrow question about whether a rule can repair a known case. Full runs answer a broader but noisier question about the combined system under a particular prompt, retrieval snapshot, and model execution. We preserve both kinds of evidence and do not substitute one for the other.

The current release record has several explicit limits:

1. No 97% pass. Stage311 is 449/500 (89.8%), 36 answers below the internal 485/500 release gate. Stage299 is the historical best at 461/500 (92.2%), 24 below the gate.
2. No benchmark rerun on the tag. Stage311 is the last full run before v1.0.0; the tag itself was not re-evaluated. Stage311 also reused Stage299’s frozen Top-200 rankings and did not call a live reranker.
3. No causal attribution for the latest delta. Stage311 and Stage299 differ in prompt source; 258 answer texts changed. The aggregate −12 cannot be attributed to the capacity guard, which triggered zero times in the full run.
4. Retrieval coverage is not answer accuracy. Stage243 had all annotated answer pairs in Top-20 for 460 questions, yet 47 of those questions were still wrong. The remaining work includes evidence synthesis, temporal state resolution, and appropriate abstention.
5. Stored-memory factual precision is unmeasured. LongMemEval answer accuracy, answer-audit validity, pair recall, and factual precision of all stored memories are separate metrics. This release does not claim 100% stored-memory fact precision.
6. Live offsite recovery is unverified. The software includes encrypted backup and optional replication paths, but a live restore from an offsite copy has not been demonstrated.
7. Stage166 has no efficacy result. Its stochastic token-reservation mechanism remains a synthetic protocol. It has no production generation caller, provider-token-logit integration, durable-memory write path, or LongMemEval A/B result; it contributes no benchmark accuracy claim.
8. One answerer/judge route. Stage299 and Stage311 use the same model route for answer generation and judging. Independent-judge replication would improve confidence in the measured labels.

The next research priority is not simply to add more isolated rules. It is to improve multi-session and temporal answer construction while preserving reproducible retrieval, controlling reranker variance, and reporting paired outcomes. Future release claims should be backed by a fresh 500-question run on the tagged code, with the same frozen protocol and a separately reported model/judge configuration.

8. Intended applications

Civilization is designed for systems that need to retain information across sessions and inspect why an answer was produced. Its current product surface is suited to:

• Long-running assistants that need persistent user context, changing facts, preferences, and session continuity.
• Agent platforms that require auditable decision traces and structured memory boundaries across working, episodic, semantic, and procedural stores.
• Research prototypes that compare memory, retrieval, and answer-generation strategies under a fixed benchmark protocol.
• Developer integrations using the Python or TypeScript SDK, a service endpoint, or a supported provider/local runtime.
• Operational deployments that need access controls, rate limits, export, and encrypted backup workflows.

The LongMemEval score is evidence about one benchmark protocol. It should not be interpreted as universal reliability across domains, a guarantee that every stored fact is correct, or proof that disaster recovery has been exercised successfully.

Conclusion

Civilization 1.0.0 is our first formal, open-source engineering release for persistent and inspectable assistant memory. The project now has a versioned SDK and service baseline, a traceable benchmark history, and a clear account of where the system still fails. The latest full Top-20 result is 449/500; the best observed run is 461/500; the internal release gate remains 485/500.

We publish the gap alongside the progress. The remaining work is to make multi-session synthesis, temporal reasoning, reranking, and recovery evidence more dependable—and to rerun the complete protocol on a release tag before claiming that the software has reached its accuracy target.