How to Read the Agent Memory Leaderboard (and What It Leaves Out)
The Agent Memory Leaderboard launched in July 2026 with more than twenty universities behind it and published its first results in August. It brings standardized evaluation to a field that badly needed it. Here is how to read a score, and the four capabilities it does not cover that decide whether an agent is reliable over months, not minutes.
By Clinton Stark • explainer, benchmarks, meaningbench
The Agent Memory Leaderboard (AML) launched on July 29, 2026, built by researchers from more than twenty universities and research organizations, among them Oxford, Tsinghua and UNSW Sydney. The first results went live on August 12. A second cycle is expected to open on September 20.
Full disclosure before anything else: Meaning Memory is not on it. The first cycle closed on August 7, before we had worked through the evaluation contract. We intend to enter a future cycle, once we can put a properly tested version forward rather than rush one to meet a date. What follows is not a complaint about a benchmark we did not enter. It is a guide to reading one we think the industry needed, and an honest account of what it cannot tell you.
How AML works
Until now, memory systems were reported against different datasets, different answer models, different judges and different aggregation rules. A score produced under those conditions is useful inside one study and useless across studies. AML holds the pipeline constant and lets participants vary only the memory system.
A participant exposes two operations:
- Add. Write a conversation, event, document or engineering history into the memory system.
- Search. Return relevant memory evidence for a supplied query and scope.
That is the whole contract, and it is published in full. The participant owns storage and retrieval; AML controls answer generation, evaluation, scoring and publication. If two systems get different results, the difference is attributable to the memory system, not to the answer model or the judge.
How to read a score
AML maps every question into one capability taxonomy, which is one of the better decisions in the spec: results compare at the level of memory behavior, not dataset names. The seven capabilities, grouped and named the way the leaderboard itself displays them (the project README words a few of them slightly differently):
Retrieve. Explicit fact recall (facts, attributes, sources, entities). Compositional inference (relations and multi-hop evidence). Temporal and event reasoning (dates, order, state changes).
Adapt. Memory governance (update, conflict, deletion, forgetting). Personalization and care (preferences and sensitive context).
Act. Context learning and execution (rules, procedures, constraints). Safety and privacy (abstention and evidence boundaries).
Here is the top of the commercial division as published on August 12:
| # | System | Overall | Recall | Compose | Temporal | Govern | Personal | Execute | Safety |
|---|---|---|---|---|---|---|---|---|---|
| 1 | MemoraX | 58.0 | 89.9 | 63.4 | 60.0 | 51.2 | 58.5 | 30.0 | 58.4 |
| 2 | MemOS | 45.9 | 68.9 | 53.4 | 56.5 | 44.3 | 48.7 | 9.8 | 56.1 |
| 3 | NTES-Memory-Smart | 44.2 | 55.6 | 46.8 | 20.6 | 31.2 | 57.0 | 27.7 | 29.0 |
| 4 | AML-Eval-Flash | 43.7 | 55.4 | 43.7 | 22.3 | 30.9 | 56.1 | 27.3 | 29.7 |
| 5 | Cognee | 42.6 | 53.6 | 42.9 | 22.0 | 32.1 | 54.5 | 27.1 | 29.0 |
Source: agentmemories.ai leaderboard, Textual Memory, Commercial Products, retrieved August 25, 2026. Versions as listed: MemoraX v0.5, MemOS v2.0.1, Cognee v1.4.0. Column names abbreviated from AML’s seven capabilities. Academic methods are ranked in a separate table.
Read it per column, not by the overall number. Three things the table says that the headline does not:
- Fact recall is close to solved; execution is not. The leader scores 89.9 on explicit recall and 30.0 on following rules and procedures. Every system in the top five is under 31 on execution. If your agents have to obey standing constraints, that column is the one to watch.
- Temporal reasoning separates the field. Second place is nearly on par with first on temporal understanding (56.5 against 60.0); third place scores 20.6. Serving the latest valid state, rather than a fact that was true once, is where systems diverge most.
- A strong overall can hide a weak column. The second-place system posts 9.8 on execution. An agent built on it would recall well and follow instructions badly, and the overall score would never say so.
These are the right questions for a recall-oriented benchmark. A memory system that cannot do these things is not ready for production, and AML gives you a structured way to check.
What AML does not measure
AML measures whether a system can retrieve and use information. It does not measure whether the system knows which information matters. It does not measure whether the agent stays recognizable as the same agent across consolidation cycles. It does not measure whether the system can detect its own stale memories and stop serving them as current. And it does not measure whether the agent knows what is worth following up on.
These are not edge cases. They are the difference between a memory system that behaves like a database and one that behaves like part of an agent.
We ran into the gap ourselves. When we started building Meaning Memory, we validated retrieval on recall benchmarks, and retrieval was fine. What was not fine was that the system treated “I’ll send the report by Friday” the same as “the user likes cold brew.” (That gap is why commitments became a memory type.) Both were facts. Neither had a deadline. Neither knew whether it had been fulfilled. The recall benchmark scored both correctly, because recall was never the problem.
So we built a second benchmark for the part recall does not reach. It scores five dimensions: the four gaps above, plus cross-session perspective, which AML touches through its temporal and personalization columns but does not isolate.
A leaderboard without a winner
Here is what a MeaningBench cross-system run looks like. In April 2026 we ran three production-grade adapters through all five dimensions: Meaning Memory, a vector-retrieval memory layer, and a persona-block agent framework. The interesting output is not a ranking. It is the coverage matrix: which dimensions each architecture can natively score at all.
| Dimension | Meaning Memory | Vector-retrieval memory layer | Persona-block agent framework |
|---|---|---|---|
| Significance calibration | scored | no native primitive | no native primitive |
| Identity coherence | no native primitive | no native primitive | scored |
| Stale-memory detection | scored | scored | scored |
| Curiosity selectivity | scored | no native primitive | no native primitive |
| Cross-session perspective | scored | scored | scored |
Meaning Memory, April 28, 2026 run: significance calibration 0.941, stale-memory detection 0.840, curiosity selectivity 0.918, cross-session perspective 0.650 (n = 5 to 10 cases per dimension, single run, wide confidence intervals).
Two honest readings. On the dimensions every system can score, the three landed in a tight band, and Meaning Memory did not lead on both. On the dimensions that need a native primitive, only the architecture that has the primitive gets a number; the persona-block framework is the only one of the three with a native identity primitive, and it scored it. That is the whole point: this is a class of memory behavior the recall benchmarks do not cover, and the matrix shows who has built for it, not who is “best.”
One rule we adopted after that run, and which AML enforces from the other direction: never quote an average across systems with different coverage. A five-dimension mean with two zeros imputed is not comparable to a four-dimension mean with a dimension skipped. Every score we publish now carries its dimension, case count and coverage inline. Read other people’s leaderboards the same way.
Why the gap matters in production
The capabilities AML measures are table stakes. The capabilities it does not measure decide whether your agent is still trustworthy after weeks of operation.
Significance. Your agent has ten thousand memories. A user asks what you discussed last week. Recall tests whether the agent can retrieve prior conversations; it does not test whether the agent surfaces “you moved the deadline” and buries “the weather was nice.” An agent that recalls everything equally drowns its user in noise.
Identity. Your agent runs through fifty consolidation cycles in a month. Does it still sound like the same agent? AML tests whether the user’s preferences are preserved, not whether the agent stays coherent. Identity drift is slow and corrosive; nobody notices until a user says “you used to be better at this.”
Staleness. A user’s phone number changed. Does the system flag the old one, or serve it as current? AML’s temporal column touches this; MeaningBench tests it explicitly, with negative controls. Serving a contradicted fact is a failure that is hard to debug because nothing tells you it happened.
Curiosity. An agent observes twenty things in a day. Which does it follow up on? No AML column covers this. Never following up makes a passive database; following up on everything makes a notification machine. The capability that matters is selectivity.
Where this goes
AML is a step forward: versioned contracts, controlled answer generation, public governance. The agent memory field needed exactly that, and we intend to be on it.
MeaningBench is our contribution to the dimensions AML does not cover. It is v0, internal, 35 cases. We use it for release-over-release regression tracking, and we plan to grow the case set and open the methodology as it matures.
If AML is the benchmark for what your agent can remember, MeaningBench is the benchmark for whether it knows what that memory means. Evaluate on both.
Common questions
What is the Agent Memory Leaderboard?
AML is an open evaluation platform launched July 29, 2026 by researchers from more than twenty universities and research organizations. Memory systems expose a standardized Add and Search contract and are scored across seven capability dimensions and more than ten benchmark datasets, in separate divisions for academic methods and commercial products. The first results were published August 12, 2026.
What does AML measure?
Seven capabilities of textual memory: explicit fact recall, compositional inference, temporal and event reasoning, memory governance, personalization and care, context learning and execution, and safety and privacy. A separate coding-memory track measures whether an agent can reuse engineering history from earlier work in a repository.
What does AML not measure?
AML does not evaluate whether a system knows which memories are significant, whether an agent stays recognizable as the same agent across consolidation cycles, whether it can detect its own stale memories, or whether it knows what is worth following up on. MeaningBench covers those as a complementary benchmark.
Is MeaningBench a competitor to AML?
No. AML measures recall and retrieval accuracy under a controlled pipeline. MeaningBench measures significance calibration, identity coherence, stale-memory detection, curiosity selectivity, and cross-session perspective. A memory system can and should be evaluated on both.
Is Meaning Memory on the AML leaderboard?
Not yet. The first cycle closed on August 7, 2026, before we had evaluated the contract. AML expects to open a second cycle on September 20, 2026, and we intend to enter once we can submit a properly tested version, which may not be that one. MeaningBench is an internal v0 benchmark with 35 hand-curated cases across five dimensions.
Related reading: Your Agent Treats Every Memory the Same. That’s the Problem., From Context Window to Memory Layer: A CTO’s Guide, and How to Make Your AI Agent Keep Its Promises.
AML facts and the commercial-division table verified against agentmemories.ai and the AML GitHub README on August 25, 2026; MeaningBench figures from the April 28, 2026 cross-system run reports.