Agent Memory Reaches a Crossroads: Technical Approaches, Evaluation Methods, and AML's First Rankings

A Year in the Agent Memory Race

👦🏻 Author: GaKi

🥷 Editor: Koji

🧑‍🎨 Layout: NCon

Even if you hired the best personal assistant in the world, they couldn't remember every word you've ever spoken, every email you've written, every document you've read. But AI systems might get close to unlimited memory — remembering users, preferences, and long conversation history instead of starting from zero every time.

This comes from a podcast conversation between OpenAI CEO Sam Altman and Alex Kantrowitz, founder of the independent tech media outlet Big Technology. It was later cited repeatedly in GPT-6 planning documents and in the phrase "infinite perfect memory."

At the time, many people dismissed these remarks as marketing speak.

But a year later, with various agent harnesses launching and products that "bake memory capabilities into the model itself" emerging, people realized that agent memory is one of the core technologies that can actually deliver on that promise.

And the imagination around agent memory is already materializing. Over the past year, dozens if not hundreds of startups have entered the agent memory space. Crowds of people have piled into one赛道, none of them sure whether their chosen technical approach is the right bet.

On August 12, the Agent Memory Leaderboard (Chinese name: 记忆之巅排行榜), backed by Tsinghua University, Peking University, and other institutions, released its inaugural results. The project is currently featured on the Hugging Face Spaces Trending homepage.

This leaderboard gives us an entry point to glimpse where this赛道 actually stands.

🚥

In this article, we want to map out how far the agent memory赛道 has come, what it has achieved, and how the technical approaches have splintered.

The Agent Memory赛道: Hot on the Inside, Cold on the Outside

The agent memory赛道 has no shortage of capable players. Letta, with the most impeccable academic pedigree, grew out of the MemGPT paper from UC Berkeley's BAIR Lab and raised tens of millions in seed funding upon its founding in 2024.

Zep uses temporal knowledge graphs to achieve 63.8% accuracy on LongMemEval, specializing in scenarios where "facts expire." Mem0 has low integration barriers and is one of the most widely used vendors. SuperMemory, founded just over a year ago, has raised nearly $3 million and focuses on connecting external data sources like Gmail, Notion, and GitHub into agent memory.

But the赛道 as a whole remains "hot on the inside, cold on the outside."

A July 2024 New Market Pitch industry report showed that agent memory systems accounted for only about 0.78% of total agentic AI funding for the year. The reason: most memory system core engines are open source, making technical barriers thin.

More troubling, the model vendors themselves are moving in.

In May, Anthropic simultaneously launched three features in Claude Managed Agents — Dreaming, Outcomes, and Multi-Agent Orchestration — directly targeting orchestration tools like LangGraph and CrewAI. The "memory layer" services that startups were selling got partially absorbed by the platform.

In the battle for users, memory has become a platform bargaining chip: in March, Anthropic first enabled memory migration for Claude; three weeks later, Google followed suit, with Gemini supporting import of complete conversation history from ChatGPT and Claude.

However, platform-native memory currently remains stuck at shallow information like user preferences and can't handle structured knowledge. This leaves room for differentiated innovation by startups. Compared to a year ago, market opportunities are concentrating further toward vertical scenarios, specialized capabilities, and technical moats.

Continual Learning

Beyond helping agents remember users, memory has another purpose that's even more worth watching: making agents smarter the more they're used, achieving continual learning effects.

Earlier this year, a team from UC Berkeley and others published the FST paper, which attracted significant attention. Its core idea treats continuously updatable agent context as fast weights and model parameters as slow weights. Experiments showed it improved sample efficiency by up to roughly 3x and significantly outperformed pure RL on continuous tasks.

Media commentators judged this paper as representing "continual learning is 1000x more important than reasoning." And one of the cores of continual learning is agent memory.

The agent memory赛道 has thus entered an intensely competitive phase. On one side, funding hasn't kept pace with the hype. On the other, model vendors are consolidating territory themselves. Startups urgently need evidence they can show to investors and customers.

Leaderboards have become the simplest, most straightforward source of that evidence.

The Agent Memory Behind the Leaderboard

By now, the market isn't actually short of evaluations and leaderboards related to agent memory, and each one has fairly solid institutional and technical backing behind it.

For example, the comprehensive-route AMB and Bench'd are independent evaluation sites built by memory product创业 teams themselves, deliberately keeping some distance from commercial companies and mainly checking for score manipulation. AMA-Bench focuses on long-horizon working memory, with author lists that include Meta researchers.

The AML mentioned at the beginning of the article is a newcomer platform that released its inaugural results on August 12. It has high traction, with over 100 teams submitting registration applications, and some overseas bloggers are reposting related information.

The reason this evaluation leaderboard has garnered attention is that its exam subjects are relatively comprehensive. Text memory covers seven directions: factual recall, multi-hop reasoning, time and events, memory governance, personalization, in-context execution, and security and privacy. Code memory examines whether systems can reuse historical debugging and development experience.

Moreover, among the joint organizing units listed on its official site are nearly 30 institutions including Tsinghua University, Peking University, Fudan University, Shanghai Jiao Tong University, Shanghai Artificial Intelligence Laboratory, Institute of Automation of the Chinese Academy of Sciences, The Chinese University of Hong Kong, and The Hong Kong University of Science and Technology.

From the results, AML's inaugural release split into two tracks: commercial products and open-source methods.

On the commercial product track, MemoraX ranked first with 58.0 points, leading by a substantial margin — taking first place across all seven capability dimensions, with write and retrieval efficiency also firmly in the top tier.

On the open-source methods track, InvMem, ReFind, and ActiveMemoryIndex placed top three with 45.1, 45.0, and 44.8 points respectively. The gaps are small — no one has pulled significantly ahead — and the three have different technical orientations with respective strengths.

Among these, MemoraX, first on the commercial product track, deserves more attention.

This is a company founded only this March. After seed and seed+ rounds in April and May, it quickly secured a third large funding round. Co-lead investors include Beiyang Haitang Fund, Shangshi Capital, and Renai Capital, with Yinxingu Capital participating, and returning investors Hua Capital and Data Capital increasing their stakes.

Many people began paying attention to this company partly because it had 10 papers accepted at ICML 2026, 2 of which were selected as Spotlight papers.

For a company less than half a year old to achieve this density of paper output at a top conference is rare among agent memory startups. This is also why it's useful as a handle for understanding the technical approaches in this赛道 — to some extent, it's a microcosm of how this cohort of new agent memory teams is playing the technology game.

MemoraX's official technical description is "learnable memory policy engine, memory foundation model, self-evolving agent harness" — different from traditional vector database similarity retrieval approaches. What it emphasizes is memory that can continuously self-evolve, dynamically update, and be reused across scenarios.

This approach isn't being validated for the first time.

AML's evaluation data itself incorporates existing benchmarks like LoCoMo-Refined, and MemoraX had already separately scored 82.65 on LoCoMo-Refined earlier — more than 30% higher than second place.

This benchmark, released by Nanjing University and Shanghai Artificial Intelligence Laboratory, is significant because after tightening evaluation standards, many previously high-scoring systems saw notable score drops, as past judging criteria had been relatively lenient.

MemoraX achieved its 82.65 under these tightened standards, and their technical approach began drawing more attention from this point.

At the time, MemoraX's own framing was: model the entire memory system as a learnable sequential decision process. Each step of write, retrieve, and utilize is treated as a decision that can be evaluated and optimized, using the end-to-end task result as reward, allowing the system to continuously adjust its own strategy through multi-round use and gradually converge to better behavior.

Memory thus shifts from a static component of "write once, use once" to a dynamic system that performs better the more it's used. How much data is stored or what vector library is used isn't what determines whether a memory system gets better with use.

The real key is whether there's a self-contained continual learning loop.

Combining with technical details MemoraX has previously disclosed, this loop can be broken down into three specific stages: write, retrieve, and utilize.

This mechanism of diagnose first, optimize, iterate repeatedly — may become standard operating procedure for agent memory going forward.

Memory's Past Year and Next Year

For those following this赛道, the significance of AML's inaugural leaderboard may not lie in the rankings themselves.

From single benchmarks to unified leaderboards, one reason these results attract attention is that standardized evaluation makes memory capabilities comparable and accelerates the entire industry. The proliferation of agent memory leaderboards over the past year shows the important position the memory赛道 occupies today.

"External" memory libraries win on maturity, compliance, and plug-and-play readiness. "Endogenous" approaches win on ceiling and imagination space. One leaderboard can't prove which route wins in the endgame, but it gives every subsequent product iteration and technical decision one more reference frame to cite.

Second, memory's commercialization path is clarifying. Mem0's API call volume grew 5x in one year, showing that "Memory as a Service" payment willingness is real.

On the enterprise side, requirements for private deployment, multi-tenant isolation, and permission auditing will pull competition toward engineering depth. This赛道's revenue model will likely sit somewhere between model APIs and databases. And both of those markets have produced hundred-billion-dollar companies.

For agents to evolve from "single-conversation tools" to "long-term collaborative colleagues," memory is the puzzle piece that can't be bypassed. Now, that puzzle piece has a public report card.

---

Following this narrative thread, memory hasn't been simply "remembering information" for a while now.

It's deeply bound to agent capabilities, becoming "agent memory" as an independent technical direction, with expanded imagination space — likely to remain a hot赛道 going forward.

From the technical approaches of earlier, larger-scale players like Mem0, to the playbook demonstrated this time by MemoraX, the thinking differs considerably.

Going forward, we'll continue tracking the agent memory赛道 long-term.

---

Crossing is looking for independent contributors to write AI product and model reviews.

If you've written articles like: Hands-On with PixVerse C1, Hands-On with LibTV, please contact zeo0811@gmail.com. Your email should include: ① personal introduction, ② AI review articles you've written.

We offer competitive rates. Looking forward to observing and documenting the AI era together 🎪