Code View | AI Observations from Silicon Valley: Have Foundation Models Been Killed by a General-Purpose Formula?

Since ChatGPT's launch, global AI development has far outpaced human expectations.

Market attention has shifted from foundational model capabilities in 2023 to the practical deployment and scenario-specific applications of large models in 2024. AI capabilities need to translate into tangible products and services, with meeting user demand becoming the core priority. Behind this lies the AI industry's transition from a "training era" to an "inference era."

Technological paradigm shifts, leaps in multimodal capabilities, the reconstruction of application forms — these critical dimensions of AI development are evolving at breakneck speed. We're now halfway through 2025. What exactly has changed in the global AI landscape, and what trends and developments deserve particular attention? To explore this, we've specially invited Lex, a senior AI practitioner based in Silicon Valley and guest researcher at Delphi Intelligence, to share first-hand observations and insights on the evolution of global AI.

The following is Lex's original article. The English version has already been published on Delphi Intelligence.io, and Source Code Capital has been exclusively authorized to publish the Chinese version.**

The full article is lengthy — we recommend saving it for later reading.

"Experience isn't just the best teacher, it's the only teacher... There are no shortcuts on the path to learning." — Ted Chiang, The Lifecycle of Software Objects

"A day in heaven equals a year on earth." — This line from Journey to the West aptly captures the astonishing speed of large language model (LLM) development. This progress warps industry perceptions like a black hole distorting spacetime:

We believe that the scaling effects and emergent capabilities of large models will bring transformative changes to business models, potentially reshaping every industry that requires human creativity. AI large models will become productivity engines, driving explosive innovation across the application ecosystem.

1. AI's rapid progress is triggering an existential crisis for benchmarks — new benchmarks are established and then quickly breached, rapidly losing their evaluative value.
2. Google Search volume in Safari has declined for the first time, while ChatGPT monthly active users (MAU) have surpassed 500 million, comparable to social platform X's user base.
3. ChatGPT's DAU/MAU ratio has jumped to Reddit-level [2], signaling AI's deep integration into users' daily lives.
4. A wave of technical standardization has emerged: following ChatGPT and Gemini's adoption of the Model Context Protocol (MCP), various technical components have launched MCP service implementations, and the Agent-to-Agent (A2A) protocol was born.

Of course, the most prominent trend is the accelerating blurring of boundaries between AI models and applications. AI startups are creating growth miracles — GenSpark reached $10 million annual recurring revenue (ARR) in just 9 days; the programming agent赛道 is particularly hot, with Cursor leading a product wave covering diverse user groups...

However, tech giants' offensive posture is even more aggressive: alongside Gemini's new product launch, Google simultaneously rolled out complete agent infrastructure and an AI application matrix, fully integrating into flagship products including Search, Gmail, and Chrome; OpenAI has explicitly transformed into a product company serving 1 billion users, acquiring WindSurf and Jony Ive's AI hardware company, partnering with Anthropic to launch programming agent products, and continuously integrating memory, search, and tool-calling capabilities into ChatGPT.

This has triggered increasingly intense industry anxiety: Are startups merely exploring product-market fit (PMF) for giants like OpenAI? Will only foundation model companies ultimately have a chance? This article will analyze these questions from a dual perspective: first deconstructing the capabilities and latest progress of foundation models that serve as the bedrock of AI applications; then, based on this understanding of foundation models, exploring the opportunities and challenges facing AI applications.

01

Progress in Reinforcement Learning Has Been the Most Significant

"Everyone will eventually have their own Sedol moment." — Noam Brown

Current large language model training has developed into a relatively "standardized" process. Andrej Karpathy's excellent video tutorial [3] clearly explains this process — it's essential viewing for understanding LLM internals. Here we briefly outline the core stages of a typical training pipeline, presenting its three-phase evolution:

• Large-scale pre-training: The model learns through self-supervision on trillions of tokens (and multimodal data), forming a foundation model with broad knowledge reserves and linguistic fluency.
• Supervised fine-tuning (instruction tuning): Using curated question-answer datasets to refine the model, giving it the ability to follow human instructions (the transformation from GPT-3 to ChatGPT).
• Reinforcement learning: The model optimizes its response strategy through trial and error. For "unverifiable domains" like poetry writing and joke generation, Reinforcement Learning from Human Feedback (RLHF) is introduced — human evaluators rank model outputs by quality, training a reward model to guide policy optimization. Without altering the knowledge structure, this makes model outputs more aligned with human preferences and reduces hallucinations.

After completing these steps, the model is ready for deployment. During inference, techniques like Chain-of-Thought (CoT) prompting, self-consistency checking, and self-reflection can further improve performance. The ReAct method, for example, enables models to "think" during environmental interaction, significantly enhancing problem-solving capabilities.

However, among all these steps, the most significant recent progress has been in reinforcement learning, particularly the Reinforcement Fine-Tuning (RFT) [4] method. It is believed that OpenAI's DeepResearch was built on a special RFT version of the O3 model. This method dramatically enhances AI application capabilities in specialized domains, becoming a powerful pathway for building AI agents.

We can understand LLM learning outcomes at each training stage as follows: pre-training builds foundational knowledge, post-training standardizes behavioral patterns, and RFT transforms generalist models into domain experts.

The core difference between RFT and SFT lies in the training methodology. Supervised Fine-Tuning (SFT) is like a teacher walking students through examples step-by-step: you prepare large quantities of high-quality input-output pairs, and the model imitates the "correct answers." This approach converges quickly with controllable style, but has clear limitations:

• Data is expensive — every new domain requires tens of thousands of manually labeled examples.
• Poor generalization — models often just memorize "correct routines," and lose points as soon as question formats or scoring criteria shift slightly.

RFT, by contrast, is like a student repeatedly practicing problems while a teacher grades them. It learns more slowly but masters the material more thoroughly — or in AI terms, generalizes better. Developers only need to implement one or more graders (scoring programs) that calculate metrics — such as whether unit tests pass, facts are verified, or outputs are compliant — to score the model, then use these scores as rewards for RL updates.

It brings three core differences:

• Less labeling required: As long as you can write rules or scripts to measure quality, large-scale human examples are unnecessary.
• Bias toward functional correctness: The optimization target directly binds to "task success rate" rather than "resembling human examples" — in computer code, mathematical Q&A, long-chain tool calling, and similar scenarios, it often generalizes better than SFT.
• Gradual iteration: Developers can continuously improve graders (adding new unit tests, strengthening security reviews), and the model iterates accordingly without needing complete re-labeling.

The main difference between RFT and RLHF is that RFT uses computable rules as feedback, while RLHF uses a trained reward model to simulate human preferences. In other words, RFT's rewards come from explicit "judgment criteria," focusing the model on task performance itself rather than merely imitating human text style. This suits tasks with objective right and wrong answers (mathematics, programming, law, etc.).

Progress in reinforcement learning has greatly boosted AI researchers' confidence. Shunyu Yao, OpenAI researcher and ReAct author, noted in his latest blog "The Second Half" [1]: "We have converged on a general paradigm for AI development — with the 'right RL prior (language pre-training),' and a suitable RL environment built by 'adding reasoning to the action space,' reinforcement learning can enable language to generalize through agent reasoning." Put plainly, with the assistance of large language models and reasoning, reinforcement learning can achieve generalization and continuous improvement.

As corroborating evidence, Yao emphasizes: "This paradigm has essentially standardized and industrialized benchmark climbing, without requiring too many new ideas. Because the paradigm has good scalability and generalization, new methods for specific tasks may bring 5% improvement, while the next-generation O-series models can achieve 30% improvement without specialized optimization."

Furthermore, at the 2025 HSG AI Summit [2], OpenAI researcher Dan Roberts presented an agent capability exponential curve showing that since 2019, the duration of tasks AI agents can autonomously handle has doubled every 7 months — a trend dubbed the "New Moore's Law for AI Agents." By linearly extrapolating this trend, Roberts predicted: by 2034, agents will be able to autonomously complete tasks comparable to "Albert Einstein discovering general relativity."

We have reason to take OpenAI's claims seriously — it is reported that they have already launched subscription services at $2,000 and $20,000 per month toward this vision. Similarly, Anthropic CEO Dario Amodei stated that around 2026, we may have AI systems with intelligence surpassing most Nobel Prize winners.

But not everyone shares this vision [13]: HuggingFace Chief Science Officer Thomas Wolf criticized Amodei's prediction as "wishful thinking at best," pointing out that humanity's greatest scientific breakthroughs stem from asking unprecedented questions — not merely answering known ones. Linguist Edward Gibson explained to Lex Friedman that the brain region responsible for language processing appears independent from logical reasoning regions [5], questioning whether complete intelligence can emerge from linguistic intuition alone. Is superintelligence approaching or illusory? The truth likely lies somewhere between these extremes — let us investigate further.

2

Current LLM Chain-of-Thought May Be a Performance

"In their world, as in ours, the greatest danger is not ignorance, but the illusion of knowledge." — Isaac Asimov, The Gods Themselves

When discussing AGI, a frequently asked question is whether the Scaling Law still holds. This question is largely meaningless, because since its introduction in 2022, the meaning of Scaling Law has shifted multiple times — from pre-training to post-training, to Test Time Computing. Someone might well bundle tool use, memory, and other agent characteristics into a fourth Scaling Law.

A more meaningful question might be: using human intelligence as a benchmark, what capabilities has this emerging intelligence already acquired, and where does it still fall short? Examining internal mechanisms, actual performance, and direct comparisons with the human brain, this intelligence appears "serrated" — highly uneven development across different dimensions.

Anthropic's recent Attribution Graph research delves into models' internal reasoning processes, revealing intermediate concepts they consider. In one case, when asked "What is the capital of the state containing Dallas?", the attribution graph revealed that the model internally activated a feature representing "Texas" (because Dallas is in Texas), then arrived at "Austin" as the final answer. In other words, it executed explicit two-step reasoning entirely internally (Dallas → Texas, Texas → Austin), very similar to human reasoning — and the attribution graph made this reasoning visible.

Another striking case: Claude was asked to write a multi-line rhyming poem. Attribution analysis showed that before composing the first line, Claude had already planned key rhyming words for subsequent lines. For example, it pre-selected candidate rhymes ("moon" and "June") and embedded this plan into its generation process. In short, Claude "planned ahead" — a clear sign of latent reasoning. These observations suggest models can form latent conceptual plans, not merely output text word by word. Their behavior resembles a chain of thought, even when we don't explicitly see it.

But the research also found that Claude's generated chain-of-thought (CoT) does not always faithfully reflect its true reasoning process. In other words, the thinking steps the model outputs are sometimes cobbled together to make the answer look plausible, while the final answer wasn't actually derived step-by-step from them. That is, current LLM chain-of-thought may be a performance — unfaithful reasoning. This means in scenarios requiring rigorous reasoning (complex mathematical proofs, causal inference), LLMs may cut corners through pattern matching without truly understanding the problem, like someone who first guesses an answer by intuition, then fabricates reasons to justify it.

More seriously, reinforcement learning fine-tuning has not overcome this limitation. A team from Tsinghua University and Shanghai Jiao Tong University, in their paper "Does Reinforcement Learning Really Incentivize Reasoning Beyond the Base Model?", compared multiple models and found that the "reasoning paths" of RL-trained models already existed in the base model's random sampling distribution. RL didn't teach models entirely new categories of reasoning; it merely selectively picked out correct lines of thought that the base model could already potentially produce, improving efficiency at answering questions correctly.

For example, RL fine-tuned models significantly outperform base models on Pass@1 (single-attempt accuracy), but if the base model is allowed multiple samples, its ability to eventually find the correct answer (at high Pass@k) is comparable. This shows RL mainly re-ranks the output distribution, making the model emit correct answers more frequently, but the "ceiling" remains constrained by the base model's original capability range.

The paper further notes: RL improves performance by biasing model outputs toward paths more likely to score highly, essentially a strategy of narrowing diversity in exchange for higher accuracy. The result is that RL fine-tuned models actually have narrower reasoning boundaries than base models, because they no longer explore rare but potentially correct alternative solutions. This research sounds an alarm about the limits of a "universal recipe": reinforcement learning has not enabled models to break through their own limitations and learn truly new reasoning skills; performance gains come more from "selection" and "bias" than from "epiphanies" of logic no human taught.

Another paper by researchers at Stanford University and Tsinghua University, "The Four Habits of Highly Effective STaRs" [9], indirectly supports this view. They compared two models on RL-based self-improvement tasks:

The Qwen-2.5-3B model showed significant improvement in math games through RL, while LLaMA-3.2-3B improved little under the same training. Deeper analysis found that Qwen naturally possesses some effective "cognitive habits" — autonomously verifying answers when solving problems, backtracking and revising when encountering contradictions, skillfully breaking down subtasks, and working backward from goals. These habits resemble expert human problem-solving strategies, dubbed the "four habits of highly effective thinking" in the paper. LLaMA, by contrast, initially lacked these behavioral patterns, so RL training struggled to take hold.

But surprisingly, when researchers prompted/fine-tuned LLaMA with examples containing these reasoning habits — even when the examples' conclusions were wrong, as long as the thinking framework was correct — LLaMA subsequently made substantial progress in RL. That is, the reasoning behavior itself (even without arriving at the correct answer) matters more than the correct answer alone.

This explains why some models (Qwen) thrive in RL self-improvement while others (LLaMA) stall — if a base model lacks fundamental thinking habits, no amount of RL reward and punishment can get it going. Therefore, to truly expand a base model's reasoning boundaries, perhaps these habits need to be instilled during pre-training.

To use a human analogy: if you don't have this talent innately, you can't train it in later.

3

LLMs Lack Real-World Concepts of Time and Space

A hallmark of human intelligence is surviving and accumulating experience in temporal and spatial environments, giving us intuitive physical understanding and continuous memory. Yet current LLMs lack real-world concepts of time and space. Though they've read countless texts, these are discrete fragments; models have not personally experienced the continuity of events.

A vivid example: Anthropic's Claude 3.7 model livestreamed playing Pokémon Red on Twitch. Claude 3.7 was enhanced with so-called "hybrid reasoning" capabilities, able to think and act simultaneously, and was indeed smarter than the older 3.5. It successfully won three gym badges, while Claude 3.5 couldn't even get out of the starting town.

Seemingly the model demonstrated planning and exploration capabilities in a large RPG, but problems exposed during the stream were comical — Claude would often get stuck in front of a rock wall for hours, repeatedly trying to walk through it, only belatedly realizing it needed to go around. One viewer quipped: thousands of hours of AI training versus one wall — who wins?

Claude eventually struggled past the obstacle, but its sluggishness throughout resembled Slowpoke, the slowest Pokémon of all. This anecdote highlights LLM agents' limitations: lacking spatial topology understanding and environmental memory. What seems obvious to humans — "if you can't get through, find another way" — the model had to learn through repeated trial and error, because it doesn't truly "see" the map's overall layout, only exploring step by step like the blind men and the elephant.

Why so painful? Unlike humans, Claude has no internal map of the virtual world, nor persistent memory of early game stages. It can only "see" the current screen and decide. It doesn't remember navigating a maze yesterday, nor infer that an unseen wall is impassable. Fundamentally, current large language models have no experiential common sense about the physical world or spatial topology. And they're not very sensitive to time either.

In contrast to Claude's struggles, Google's Gemini 2.5 Pro model recently made headlines: in a livestream organized by developer Joel, it successfully completed Pokémon Blue, becoming the first large AI to finish the game — even Google CEO Sundar Pichai celebrated online. On the surface, this is a striking example of a foundation model "swallowing the application" — Gemini defeated 1990s human gamers.

But deeper analysis reveals some "humblebrag" truths: Gemini's success heavily depended on human-provided assistive tools. First, researchers didn't have the model play blind like a human; instead they provided key information from game screenshots as text prompts — current location, walkable paths, inventory items — then had the model choose the next action, translated into button presses by an "agent harness." That is, Gemini received more high-level environmental knowledge than human players, sidestepping visual understanding challenges.

Second, Joel admitted to occasional human intervention during the livestream, prompting Gemini to adjust its thinking when it got stuck for long periods, though these interventions were "carefully limited to improving Gemini's reasoning quality." Gemini essentially completed the game with training wheels on. Of course, this remains an impressive achievement — Gemini demonstrated long-horizon strategic planning and adaptation. But as the article concludes, we shouldn't overstate this milestone's significance. "We're still a long way from the kind of envisioned future where an Artificial General Intelligence can figure out a way to beat Pokémon just because you asked it to."

To use another human analogy: this child read too many books in their education, and did too little sports and hands-on practice.

4

The "Intelligence Gap" of Foundation Models: Humans vs. Base Agents

As mentioned, current foundation models have approached or even surpassed human levels in language and logic dimensions — passing bar and medical exams, generating code competent for junior programmer work. But in overall intelligence structure, foundation models still differ significantly from humans. This imbalanced capability matrix can be called an "intelligence gap." Researchers from the MetaGPT project propose analyzing intelligent agents through a "brain-like modular" architecture, mapping them onto human brain functional regions. For example, the human brain has the hippocampus dedicated to memory, the occipital lobe responsible for visual-spatial processing, the limbic system related to emotion and motivation, and so on.

So how do LLM-centered agents perform across these modules? The good news is that LLMs have developed very well in language, vision, and other areas; the bad news is that some aspects remain insufficient or entirely untouched.

• Memory: LLMs have limited context windows and cannot form long-term memories the way the human brain does, let alone continuously updated autobiographical memory. Each response is essentially a "reset" unless external databases are attached. While research is exploring long-term memory modules (retrieval-based memory, knowledge bases), these remain immature. This makes it difficult for models to accumulate knowledge from their own experiences — a new fact learned today simply ceases to exist for them tomorrow unless provided again. Similarly, models lack any internal representation of time's passage, unlike humans who possess a sense of time and causal sequencing. This leaves LLMs struggling with tasks involving scheduling, historical evolution, and real-time learning.

• Perception and Spatial Understanding: Although multimodal models are beginning to incorporate visual input, LLMs are congenitally deficient in spatial cognition. Without embodied experience of interacting with environments, their understanding of the physical world can only be learned indirectly from text and images. Asking GPT to draw a room layout or imagine rotating a 3D object is profoundly challenging, whereas human children grasp basic spatial concepts through crawling and grasping. Initiatives like MetaGPT advocate introducing world models and perception modules into agents to give AI some capacity for internal environmental simulation, but this remains in its infancy.

• Contextual Understanding and Goal Orientation: Current foundation agents are also weak at sustained, focused tracking of long-term goals. The human brain continuously integrates external information with internal objectives, driving behavior toward long-term purposes. Most LLM agents, by contrast, execute round-by-round conversational instructions, easily becoming trapped in local dialogue context without a global task perspective. For example, an automated agent writing code might, without additional constraints, endlessly refine details while forgetting the overall requirements. This is the so-called defect of "instrumental rationality," requiring planners or human feedback to remedy. MetaGPT's multi-agent framework attempts to have different role-based agents cross-check each other to maintain correct direction, but this still falls short of human flexibility in goal management.

• Creativity and Problem Discovery: As Thomas Wolf's view noted earlier, truly Einstein-like creativity lies in discovering new problems. LLMs excel at combining existing knowledge (writing poetry, imitating paintings, assembling creative ideas), but lack the years-long process by which researchers formulate novel hypotheses and test them. This may relate to the absence of autonomous motivation and genuine interaction. Without autonomous experience accumulation, AI can only compete within existing corpora — it can rapidly output "standard answers" already present in a field, but rarely asks as humans do: Are we even asking the right questions? Is there a more fundamental problem?

To bridge these intelligence gaps, the MetaGPT paper advocates drawing on the brain's modular design, including introducing independent memory storage modules, explicit world models, hierarchical planning and execution units, even simulating simple emotional and motivational mechanisms. By enabling these modules to work in concert, AI systems can evolve toward architectures closer to biological intelligence.

To some extent, this means foundation models need to evolve from "models" to "agents" — possessing proactivity, persistence, and environmental adaptability — to truly support more complex applications. In other words, these problems can be addressed through multi-agent approaches, particularly when problems can be clearly defined. This is why Shunyu Yao emphasizes the importance of environment and evaluation.

Through the above analysis, we can draw several conclusions:

1. Under the current paradigm, general-purpose models cannot possibly swallow all application scenarios in terms of capability. They still require adaptation through RFT, inference-time techniques, multi-agent frameworks, or additional auxiliary tools to function in practical use cases — just as Gemini needed assistance to complete the Pokémon game.
2. Beyond effectiveness, the "intelligence" that emerges from RL on top of LLMs may be inefficient in certain respects. This is like using soldier formations as computers in The Three-Body Problem — it might work, but the cost and latency may not be practical, so application logic remains necessary.
3. Following the current general recipe, producing an Einstein in nine years may not be realistic. Of course, a new recipe may already be in the lab.

5

AI Surpassing Humanity May Require Entering the Era of Experience

"No algorithm can compress experience." — Andy Jassy

In the film Ex Machina, the genius programmer Nathan tests AI Ava's intelligence by telling Caleb the famous "black and white room" thought experiment: scientist Mary, who has lived her entire life in a room containing only black and white, has mastered all the physical knowledge about color. Yet when she steps outside and sees blue sky for the first time, she truly "experiences" what color feels like.

So perhaps the direction to fill these gaps lies in entering what Richard Sutton calls the "Era of Experience."

DeepMind's David Silver and RL pioneer Rich Sutton co-authored Welcome to the Era of Experience in 2025, proposing that for AI to acquire capabilities beyond existing human knowledge, agents must autonomously gain experience through environmental interaction.

They divide AI development into three acts — the Simulation Era (AlphaGo and others learning in closed simulations), the Human Data Era (GPT series learning from internet human output), and the Experience Era (AI autonomously learning from real or open environments). The first two eras both rely on "stones from other hills": Act I uses human-built simulators, Act II uses massive human-provided data. Now returns are diminishing on existing human data: in mathematics, programming, science, and other fields, knowledge extractable from human data is approaching its ceiling. LLMs have already mastered 99% of patterns on the internet, but cannot expect to invent insights far beyond human boundaries from them. Just as students must eventually leave books and enter laboratories, AI too must step out of the comfort zone of human-gifted data to learn directly from the world.

Silver and Sutton argue this means AI agents require several key characteristics:

1. Sustained, autonomous long-horizon goals: Experiential learning requires agents to set and track long-term goals themselves, rather than completing only short instructions each time. Future AI should strive toward grand visions over years, like researchers or engineers. For example, a medical AI might autonomously establish the long-term goal of "discovering new treatments for a disease" and persistently learn and experiment.
2. Environment-driven feedback: Agents should learn not merely from human rewards but from the environment itself. In the Experience Era, AI's driving force comes from the results of its environmental interactions, not just humans telling it what is good or bad. As animals in nature evolve through success and failure, AI should adjust strategies by trying actions and seeing actual consequences. For instance, a robot trying different gaits adjusts based on whether it falls and how fast it walks.
3. Open, continuously evolving data: Experiential learning requires data that is not a fixed training set but new data constantly generated by the agent. Silver emphasizes avoiding "static synthetic data generation programs," as fixed patterns are quickly learned and lead to stagnation. Instead, environments and tasks themselves can dynamically increase difficulty, co-evolving with agent capabilities. This resembles game level progression — AI always faces challenges slightly beyond its current level, ensuring there is always something to learn.
4. Autonomous reasoning and planning: Agents must transform experiences into knowledge, further planning actions through self-reflection. This returns to the earlier topic of "cognitive habits": AI must acquire human-like ability to summarize lessons from experience and revise strategies. Silver/Sutton emphasize that AI must "plan or reason about the things they experience," rather than blindly trial-and-error. The Experience Era does not reject reasoning; it precisely demands stronger reasoning to navigate real-world complexity.

The vision painted by the Experience Era is captivating, but the path to realization is fraught with challenges.

First is the training efficiency problem: training agents in real or high-fidelity simulated environments is vastly less data-efficient than training LLMs on static data. Teaching a physical robot to walk may require tens of thousands of attempts, equivalent to months or even years of human infant practice.

Classical RL algorithms' poor data efficiency was once mockingly described as "requiring a power plant to train a small walking robot." While researchers explore various self-playback and model-assisted techniques to improve efficiency, the data cost of real-world interaction remains prohibitive.

Moreover, experience acquisition is relatively slow: LLMs can read Wikipedia in hours of GPU computation, but an AI driver learning through actual road driving would need to authentically drive millions of kilometers, taking months or even years.

To mitigate these issues, simulated environments become a critical pathway. NVIDIA researcher Jim Fan and others advocate acquiring experience through virtual worlds: building high-fidelity simulators where AI agents train at high speed, then transferring skills to reality. This is known as the "sim2real" (simulation to reality) strategy. Jim Fan's Embodied AI team proposes a "three-pronged" data strategy: combining internet data (rich but static), simulation data (infinitely generable but biased), and real robot data (most authentic but expensive) to train "foundation agents."

Simulation is a crucial component: leveraging NVIDIA's graphics technology, they can generate virtual environments approaching reality, producing large amounts of synthetic interaction data to accelerate experience accumulation. For example, they have LLM agents conduct autonomous exploration and tool use in open worlds like Minecraft, continuously improving skills through self-play. In this process, LLMs can even be used to dynamically write reward functions, guiding agents to learn more effectively. These attempts (such as the MineDojo and Voyager projects) demonstrate simulation's enormous benefit for experiential learning: AI agents achieved sustained autonomous learning previously unattainable in virtual environments.

Many other ideas aimed at helping agents achieve continuous learning and reason like humans are constantly emerging. A few examples:

Geoffrey Hinton's "Forward-Forward Algorithm" was proposed as a learning rule without backpropagation. It replaces conventional backpropagation training by running two forward passes — one on real data, one on generated negative data — and adjusting weights so each layer shows high "goodness" on positive samples and low "goodness" on negative samples. If efficiently implemented, this could allow agents to learn online from streaming sensor data without heavy gradient descent at every step.

Meta has explored "continuous latent reasoning" as a complement to textual chain-of-thought.

In their COCONUT paradigm ("Chain of Continuous Thought"), models reason in continuous latent space rather than outputting all reasoning in language. Special tokens can switch between language mode and latent mode, enabling breadth-first or backtracking search in "thought space." Practically, Meta introduced Large Concept Models (LCMs) that operate on high-level concepts rather than tokens — these methods aim to improve reasoning efficiency.

Regarding temporal cognition, Llion Jones, one of the Transformer authors, recently proposed the "Continuous Thought Machine" — an AI model that uniquely utilizes synchronization of neuronal activity as its core reasoning mechanism... using temporal information at the neuronal level to achieve more complex neural behaviors and decision processes, making its reasoning more human-like.

Conclusion

In summary, we are indeed at another inflection point in AI development:

On one hand, the current paradigm — large-scale pre-training plus fine-tuning plus alignment — still has considerable room to grow. In the foreseeable future, we should expect to see: foundation models scaling up from their current 20–30 step reasoning with a dozen or so tools to hundreds of steps and 100+ tools, enabled by greater scale, smarter prompting, and better fine-tuning. Foundation models will absorb more agentic capabilities, yet a vast array of vertical scenarios will still require applications to adapt to them, and RFT should prove to be a solid technical option for building those applications.

On the other hand, multiple studies and experiments have converged on the limitations of the existing paradigm: just as humanity in The Three-Body Problem could build sub-light-speed ships but never reach the stars without breakthroughs in fundamental science, AI needs to move into its "second half" — shifting from imitating humans to experiencing autonomously. The focus of reinforcement learning is pivoting from algorithmic innovation toward creating environments conducive to reinforcement learning, allowing agents to learn continuously from real-world feedback.

We are seeing new explorations emerge across training data, model architectures, and methodologies — from sim-to-real, concept models, to forward-forward algorithms and the Continuous Thought Machine — all aimed at accelerating this transition. Though existing general-purpose methods are remarkable, it is time to step into the era of experience.

(English version of this article: https://www.delphiintelligence.io/research/shift-toward-the-experience-era)

References

1. Yao Shunyu — The Second Half
2. Joyce Birkins — Deep Dive into OpenAI's Reinforcement Fine-Tuning (RFT)
3. Amanda Silberling — Anthropic's Claude AI is playing Pokémon on Twitch — slowly
4. Eric Hal Schwartz — Google's Gemini AI Is now a Pokémon Master
5. Why Google Gemini's Pokémon success isn't all it's cracked up to be
6. Kanishk Gandhi et al. — Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
7. Yang Yue et al. — Does RL Really Incentivize Reasoning Beyond the Base Model?
8. Jack Lindsey et al. — On the Biology of a Large Language Model (Attribution Graphs)
9. Ashish Vaswani et al. — Rethinking Reflection in Pre-Training
10. Maxwell Zeff — The AI leaders bringing the AGI debate down to Earth (TechCrunch)
11. Bang Liu et al. — Advances and Challenges in Foundation Agents (MetaGPT Survey)
12. Sequoia Capital — The Physical Turing Test: Jim Fan on Nvidia's Roadmap for Embodied AI
13. Silver, David & Sutton, Richard — Welcome to the Era of Experience
14. TechRepublic — Is 'The Era of Experience' Upon Us?
15. The Forward-Forward Algorithm: Some Preliminary Investigations
16. Meta's COCONUT: Better alternate than Chain Of Thoughts for LLM reasoning
17. Large Concept Models: A Guide With Examples

---