Dr. Mu Li: Every Era Is a Model, Every Model Is a Species | Oasis Capital Sharing Session

In April, Oasis Capital invited deep learning expert and former Amazon Chief Scientist Mu Li to walk the founders of its portfolio companies through the evolution of AI and large language models, helping them better prepare for the changes ahead.

In the midst of this AI revolution, we hope everyone can embrace it, understand it, and put it to work for themselves. Below are excerpts from that session. Enjoy.

Waves of Machine Learning

Machine learning started in the 1960s — about sixty years ago. Looking at the big picture, each era can be understood as a single class of models. Beginning with linear models, every decade or so we've taken one model to its limit and essentially "killed" it. By "killed," I mean pushing performance improvements until one day you simply can't move forward anymore. That's how most technology revolutions work: each time you mature a technology by taking it to its peak.

For machine learning, what does "can't move forward" look like? When no matter how much you scale up the model or the data, you only get marginal gains — that's saturation, the end of the road.

From an academic perspective, linear models peaked in the 1970s and 80s. Next came neural networks — the CNNs and multi-layer RNNs we use today were already around in the 1990s. But even with specialized hardware attempts back then, compute couldn't keep up, so that approach hit a ceiling too.

Then in the 2000s, SVMs and kernel machines performed about as well as neural networks, but with better algorithms, stronger theory, and easier hyperparameter tuning — they weren't as finicky as neural networks. So academia went all in on those.

In 2010, neural networks came roaring back: deep learning began. In 2007, Princeton's Kai Li and Stanford's Fei-Fei Li built ImageNet, which was eye-opening. At the time it was the largest dataset available — one million images, which was massive for that era.

GPUs entered the picture in 2007, with CUDA enabling general-purpose matrix computation. By 2010–2012, programming them had become straightforward. At this point, data had scaled up 100x from the previous era of tens of thousands of samples, and GPU compute had jumped by several orders of magnitude. People found that 1980s CNNs, with slight modifications, worked dramatically better.

By 2018–2019, though, CNNs had stalled too. CNNs are built on the assumption that pixels are locally correlated with their neighbors — add more images and it doesn't help. RNNs assume sequential dependency, which is a text-specific prior.

Google was using TPUs (Tensor Processing Units) at the time. TPUs excel at fully-connected layers — they're essentially giant matrix multipliers with large memory. So Google explored attention mechanisms, which are essentially structured fully-connected layers, scaled them up, and created the Transformer.

Because Transformers have both an encoder and decoder, they split into two paths:

The first, more practical path was BERT. Before this, deep learning had been dominated by computer vision; NLP lagged behind. BERT brought the entire NLP field into the deep learning era — finally there was a model that could extract features from all language and then apply them. That's why all of NLP pivoted to BERT starting in 2018.

Around the same time, OpenAI jumped in: "We also believe in the Transformer and its 'brute-force miracles'." But BERT had already claimed the encoder side, so OpenAI took the harder half — the decoder — and built GPT. I suspect they hadn't fully figured out what GPT would be useful for; they just needed something harder than BERT to publish. GPT and BERT launched around the same time, but through GPT-2, GPT was consistently outperformed by BERT.

Everyone had been scaling models up incrementally, but OpenAI's GPT series kept getting ignored. So they pulled a big move: scaled GPT-2 up by 100x and launched GPT-3. Combined with more data, GPT-3 delivered performance that stunned the world. After GPT-3, OpenAI didn't rush to GPT-4 — they built a consumer product, ChatGPT, which became a breakout hit. GPT-4 came later with even bigger upgrades, and that's where the ceiling sits today.

Essentially, over the past 70 years, each generation of models has operated within a major architectural family. Each time you switch architectures, you scale data and model size until you hit the ceiling, then move to the next architecture. That's the full lifecycle of models.

Can the GPT Path Lead to AGI?

Based on this history, people started asking: can the Transformer architecture, specifically the GPT path, lead to AGI?

Two main camps have emerged:

The conservative camp believes that if this wave tops out at GPT-4 scale — roughly 500B parameters — then we're waiting for the next architecture. The current approach may not work within ten years, but the next architecture will likely emerge on a similar timescale, without the long wait we had for deep learning.

The optimistic camp believes Transformers can push much further. Even if AGI isn't achieved at the current ceiling, if the next architecture transition is as smooth as CNN to Transformer, we could see AGI in 3–5 years, even by 2025.

In short, each model is a species. Just as evolution moved from plankton to animals to humans, there's a good chance we'll see species that surpass humans.

The Core of Large Language Models

If you want to understand LLMs, you only need to know three things:

First, the assumption about the world. What you ask the model to do is simple: read books, then read a sentence and predict the next word — statistical modeling, repeated endlessly. "Read a hundred times, and the meaning will emerge." As it keeps reading and predicting, the model compresses data into its internals, where it forms a good semantic space representation. It seems that regardless of the specific model, they all map data into a useful language space — and quite possibly, the human brain works the same way. Just keep reading, and understanding emerges; intelligence manifests from this process.

Second, Scaling Laws. This holds for all of machine learning: exponentially increase the model, and intelligence increases linearly. Model IQ is essentially the log of model size. Every time you double the model, you add a linear increment of intelligence. So the gap between desired model quality and current quality, measured in these increments, tells you how many doublings you need.

Compute follows something like Moore's Law — also doubling — so in practice, intelligence grows linearly. Training cost equals model size squared: double the model, double the data too. Compute equals model size times data volume; model quality and cost have a straightforward conversion. To estimate what a model will cost, it's this simple formula.

This is where it gets expensive. A 50B parameter entry-level model runs about $1 million. Below 50B is basically research territory; practical utility starts at 50B. GPT-3 was 175B — seemingly only 3.5x larger than 50B, but data has to scale 3.5x too, so it's actually more than 10x. Training GPT-3 once jumped to $12 million. GPT-4 is likely around 500B, costing $100 million per training run, not even counting hyperparameter tuning.

Because the barrier is so high, most open-source models you see are under 50B — a few hundred thousand to a million dollars, painful but doable. Open-sourcing 50B or 100B+ models is rare because it's genuinely expensive, so nobody releases bigger ones.

Third, emergent capabilities. This is the most exciting aspect of current models: as models scale, new capabilities appear, and new capabilities open new application possibilities. As your model gets larger, you don't just see intelligence improvement — you see more emergent capabilities.

Human intelligence far exceeds that of chimpanzees, yet the IQ difference is merely linear — chimps around 30, humans around 100. Using human cognition as a framework, emergent intelligence from lower to higher consciousness manifests across four domains: language, reasoning, knowledge, and planning. While we don't know if LLMs evolve along exactly these lines, these four domains represent what's most distinctive about large language models.

Language ability covers conversation, text comprehension, and understanding semantic variation. BERT already had strong language capabilities.

Reasoning ability requires not just understanding text but logic — mathematical reasoning, for example.

Knowledge ability is somewhat fuzzier: how much information the model can retain and reasonably apply internally, not externally. Search engines fetch data from outside each time; this refers to what's inside the model. Once large enough, a model can memorize everything in a search engine.

The planning capability discovered recently is the most intriguing. Planning is the highest form of consciousness known to humans, and now models seem to be acquiring this skill. Of course, above planning for humans is "consciousness" itself — there's no consensus that models have shown consciousness yet, but given current trends, perhaps soon models will genuinely be aware of what they are.

Model Applications and Capability Evolution

Let's look at some applications of model planning capabilities.

The classic example is "chat for fun," plus foundational NLP tasks like text classification (determining if someone is happy or sad), machine translation, and text summarization. What's remarkable is that models weren't explicitly trained on these tasks, yet can perform them after general training.

GPT-3 essentially replaced all NLP tasks upon release — previous algorithms couldn't match it. One interface solved everything text-related. Over the past year or two, writing applications have actually made money: business writing, academic papers, etc. GPT's English writing ability is genuinely top-tier.

Another application was Bing's attempt to challenge Google by integrating ChatGPT for complex question answering — returning direct answers rather than summarizing search results. When Bing launched this, people thought Google was finished. But in reality, Google traffic didn't drop much, and Bing traffic only grew 30–40%, not doubling. The feature is useful but still niche.

Then there's code generation (coding by providing comments or function signatures). A classic language model application: give the model preceding code in a file, and it completes what follows. After ChatGPT emerged, the application landscape evolved incredibly fast over just a few months. Language models increasingly resemble people — specifically, assistants.

Here are some more complex examples:

1. Blender with GPT-4: Generate a graphic, and the model writes the code directly. You can see code generating continuously in Blender, creating a falling effect. This is using relatively simple commands for complex content.

2. Web development: Using Tailwind to build a login page — the user chats requirements with ChatGPT, and the entire page generates in 10 minutes, completely unlike traditional coding mindset. This evolution means even someone who knows zero code can act as the "client" making demands, with the model as the developer completing tasks, not just filling in blanks.

3. iOS app: Someone with zero iOS programming experience spent 2 hours building an app with payment functionality, enabling user interaction with GPT-4 — using GPT-4 as a developer, completely outside an editor.

4. Ordering pizza via browser: Input "order pizza online" to GPT-4, and the model demonstrates planning: go online, find a pizza place, fill out the order — becoming a pizza-ordering assistant.

5. AR personal assistant: This example is crude but fascinating. A private assistant integrated into VR glasses that follows you constantly. Previously you'd only consult the model for specific tasks; now the model can help you in real-time for any need, because it knows your context intimately — like the omniscient mentor from web novels.

6. Most exciting: AutoGPT. In this demo, input "go online and learn what AutoGPT is," and the model uses Google search, eventually analyzing AutoGPT's GitHub source code, summarizing and writing it up. Throw any task at AutoGPT — say, a research project — and it automatically searches, summarizes, and digs deeper, without step-by-step instructions.

Someone built a project on this that blew up — less than a month old, already more GitHub stars than PyTorch. They made a UI with the goal: "How to make the most money in New York in one day, without a job." In short, the model becomes your intern, constantly proposing ideas, doing homework based on your choices, and executing the next step. While perhaps less intelligent than the previous example, it's more practical — not burning compute and time, but optimizing the search space based on your feedback.

Next Stop

Of course, these examples are viral but I still consider them "toys" — not yet practical. The impressive examples you see online are the successes; most of the time the model is still pretty "dumb." Still, the emergence of "planning" capability has genuinely ratcheted up the pressure.

These are just simple application introductions — there are many more unexpected possibilities. Going forward, models will keep improving on existing capabilities, and undiscovered capabilities will be unearthed, opening possibilities for new applications.

As for how things develop: first, I think the GPT series remains somewhat toy-like. Useful in many scenarios, but guaranteeing reliability even 90% of the time is hard, let alone 99%. The journey from toy to tool still has distance to cover.

Second, multimodality. What I've discussed is mostly text-based; GPT-4 has some image capability but it's still weak. For a person, for an intelligent agent, understanding shouldn't be limited to text — it should include images, video, and surrounding environment; generating images, video, and other outputs. While text-to-image applications exist, they haven't truly integrated with large language models yet. LLM image understanding remains relatively weak.

Third, costs need to come down. Training and inference are currently extremely expensive. GPT-4 at a few cents per 1,000 tokens sounds cheap, but actual usage adds up fast because GPT-4's deployment uses many GPUs, and GPU efficiency in deployment is quite low.

Today, most attention focuses on models advancing continuously, perhaps like chips with new releases every year or two. For model builders, with constant updates, the question is how to manage model iteration and improve iteration speed. For 80–90% of startups, the thinking should be: what products to build? How can models enhance existing product capabilities? What new modalities do models enable? How to leverage this wave to improve products?

How Can Large Models Be Trustworthy and Controllable?

For any technology, effectiveness comes first, safety second — that's the iteration pattern. OpenAI only started seriously addressing safety after ChatGPT blew up.

But because models are pure statistical objective functions, there aren't great methods for safety within the model itself. Current mainstream approaches:

1. Heavy filtering of training data — like raising a child, keeping them away from corrupting influences.
2. During fine-tuning, telling the model what's wrong and what's off-limits.
3. Post-hoc processing: blacklists/whitelists for specific keywords — simple, fastest to implement. But the problems are: (1) mechanical filtering can be circumvented, and models not understanding underlying rules is problematic; (2) it makes models schizophrenic — either no results at all, or suddenly very different outputs.

From a long-term research perspective, the goal is to bake data filtering and post-hoc adjustments into model parameters. But for 100% safety, a single model is difficult; multiple models together is a possible approach: say, three models with one as the "class monitor" enforcing discipline. If other models produce bad outputs, the monitor tells them what's wrong and to fix it — so the model doesn't become schizophrenic because it understands what you want. Why hasn't this been done? Mainly because it's expensive.

The simplest method is reject sampling: generate 10 outputs, pick the least harmful one. Not yet done particularly well. Anthropic works on this; OpenAI is starting too. The advantage is this isn't limited to a few companies — academia can contribute too, so expect many papers.

Vitality

What do you think is technological vitality?

Curiosity drives humanity's continuous exploration of the unknown. A small, surprising discovery draws more people to investigate, leading to technological breakthroughs and world-changing transformation.

— Dr. Mu Li, Deep Learning Expert

Oasis Capital is a new-generation Chinese venture capital firm dedicated to discovering the most vital entrepreneurs of the next decade in China, growing alongside them to create long-term value. "Vitality" is both the direction of structural transformation in our era and the resilience and evolutionary force of entrepreneurs.

Oasis Capital focuses on early and growth-stage investments, with check sizes from $3 million to $30 million, concentrating on robotics, artificial intelligence, and technology services — empowering China's technology-driven new service upgrade.