20 Questions to Understand DeepSeek and the "Second Half of AI" It Ushered In

This week, Crossing launched its inaugural "20 Questions" column: a carefully curated set of 20 heuristic questions that delve layer by layer into the story behind DeepSeek — from technical breakthroughs and commercial impact to future trends.

Why "20 Questions"? Because when facing a major technological breakthrough like DeepSeek, we need a structured thinking framework. These 20 questions aren't simple knowledge primers, but a meticulously arranged path of inquiry: from technical principles to industry impact, from phenomenon interpretation to future prediction. We hope this Q&A format helps listeners build a clear, systematic understanding of DeepSeek amid today's information deluge. Especially now, as the initial DeepSeek fervor has cooled, revisiting the topic allows for more rational, comprehensive discussion — unswayed by short-term emotions.

This week, we're joined by Leding Li, who spent ten years as a principal senior architect at Baidu and has previously appeared on Crossing. His last episode, "I Want to Shatter Your Unrealistic Fantasies About AI, and Rebuild a Correct Understanding"[1], became our most popular episode out of 45 last year. We're honored to have him back, as the DeepSeek hype gradually subsides, to explore this topic again.

Our curated 20 questions about DeepSeek cover multiple dimensions: algorithms, compute, data, applications, and business.

In a person's lifetime, major technological breakthroughs are rare, and DeepSeek's release is undoubtedly one of them. I believe everyone owes it to themselves to invest time in understanding: what exactly DeepSeek is, what it means, and how it will shape our future.

Listen on WeChat:

Listen on Xiaoyuzhou:

The Pre-training Bottleneck and AI's Second Half

🚥 Koji

Teacher Li, let's dive straight into the first question. There's a popular notion online that AI has entered its "second half," with DeepSeek's release as the marker. Do you agree with this? Can you explain what the first half and second half each represent?

👦🏻 Leding Li

Hello everyone. Last time we focused more on pre-training. After two years of development, people have noticed that the ceiling of AI improvement through pre-training has been gradually flattening. From GPT-3.5 to GPT-4, we saw a major leap, but from GPT-4 to now — nearly two years — the upward evolution has been limited. Months ago, the United States began debating whether pre-training had hit its peak, including Ilya weighing in with his assessment. So a basic consensus has formed:

"Pre-training has reached a阶段性顶点 [phase peak/staging point]; we're less inclined to train larger models."

One reason is that publicly available, high-quality data with sufficient diversity has largely been exhausted. AI needs multi-directional knowledge — it's not that increasing knowledge in one area tenfold makes it stronger; we need new directions. From this angle, diverse, high-quality public data is already scarce, so from a data perspective, it's hard to scale further.

There's also much discussion about synthetic data. The current consensus is that if you arbitrarily generate synthetic data and feed it directly into pre-training models, it causes model collapse — and many universities have published papers arguing this point.

Beyond data, there's another discussion that hasn't fully reached consensus but that many are beginning to recognize: what Professor Yann LeCun previously raised — that due to limitations in the Transformer architecture, generative AI's data learning efficiency is too low.

Training on trillions of tokens, models still struggle to achieve intelligence comparable to a house cat. So people feel new methods may be needed. From these two angles, pre-training may have run its course, and we can explore new directions. Especially after OpenAI's o1 release, more attention shifted to post-training.

Post-training can add more vertical skills to models — for example, o1 enhanced various STEM capabilities like math and coding. You can do domain-specific fine-tuning to make it a finance, legal, or medical model. So post-training is now where most effort is focused.

🚥 Koji

So was o1's release a signal of the "second half"?

It's just that o1 was closed-source, so it didn't reveal more technical details. Plus it was paid, and the chain of thought wasn't公开 [made public], so although capable, it wasn't strongly perceived by the industry or ordinary users — which is why DeepSeek R1's release garnered such attention?

👦🏻 Leding Li

Yes, DeepSeek's release was certainly stunning. o1's release was like our first time seeing ChatGPT — solving word problems without human assistance, generating very high-quality code, this was extremely impressive. But OpenAI has become "Close AI." Researchers on the front lines had some understanding of the methods, but the specifics weren't公开 [made public], and they even blocked o1's generated tokens during its thinking process — no one knew exactly how it thought.

So R1's release was essentially revealing to the world how the most advanced technology is accomplished. From this angle, R1 is like the Llama of the second-half era.

Deconstructing DeepSeek R1: From Zero to the Full Model

🚥 Koji

Then let's move to our second question. Teacher Li, please give everyone a primer on DeepSeek's R1, and V3 which was released a few months before R1 — what are their working principles and training methods respectively, so we can have a basic understanding.

👦🏻 Leding Li

DeepSeek published papers directly this time, including for R1 and V3. Beyond that, for those who want deeper understanding, you can also look at two earlier papers to get the full picture. The first two are DeepSeek-V2 (the previous MoE version) and DeepSeek-Math (which introduced how mathematical capabilities were incorporated).

Let's start with R1. R1 brought the model's mathematical and multi-step reasoning capabilities to o1's level. How it achieved this is what everyone cares about most. This paper is the most brilliant I've read in the past two years. Actually, DeepSeek trained three models in this R1 paper, not one. I think the most valuable is the initial R1-Zero.

R1-Zero has a very elegant idea: "Can we purely rely on reinforcement learning, letting the model explore on its own to achieve multi-step thinking, thereby improving logical reasoning in math, coding, and similar areas?" The answer is yes.

DeepSeek introduced a reinforcement learning approach different from what was used before. Previously, especially in the United States, when people used reinforcement learning, they tended to try rewarding intermediate steps. For example, with math word problems, the intuitive idea is that the problem is solved through step-by-step reasoning, and the reinforcement learning system should incentivize each step for correctness and quality, optimizing through continuous feedback. But DeepSeek brilliantly analyzed this problem.

They discovered you should incentivize the model's results, not intervene in the intermediate process.

R1-Zero's reward model is actually very simple — so simple it only judges two things.

1. It only judges whether the model's final output is correct or not, not caring about the intermediate thinking process, with no incentives for intermediate thinking whatsoever.
2. It introduced a format. The format system's purpose is to let the reward model focus on judging the output result without being contaminated by the intermediate process. This format clearly distinguishes between intermediate thinking and final result, only looking at the result without caring about the process. With just this one method, the model's performance approached OpenAI's o1. This is a very elegant method.

Why does only incentivizing results without intervening in the process work? DeepSeek gives a brilliant explanation in the paper. It argues that intermediate process content is difficult to quantify and score correctly. For example, in humanities content, human speech has contextual coherence between sentences — you can't "quote out of context"; any single sentence extracted can't represent the full picture, so it's hard for us to precisely score any particular sentence someone says.

For more STEM-oriented content, even with a math problem, there are multiple solution methods (for instance, many high school problems can be solved algebraically, geometrically, or analytically; and even restricting to algebra, you could combine like terms first, multiply both sides by a square, or do the reverse). The reasoning processes are infinitely many, hard to enumerate. So rather than intervening in the intermediate process, better to just look at the result.

That is, machines think differently from humans — you can't force machines to think like people. It pre-trained on trillions of tokens; let it train in whatever way is most natural for machines, and it can achieve good results.

This is R1-Zero's greatest revelation: proving that reinforcement learning can work when used directly.

🚥 Ronghui

This innovative idea you mentioned — had other model vendors tried it before?

I saw in reports that OpenAI researchers mentioned relatively early in sharing sessions that models need incentives rather than intervention.

👦🏻 Leding Li

Yes. OpenAI was the earliest to do this, along with DeepMind — the grandmasters of reinforcement learning. This idea showed up in AlphaZero and AlphaFold, and OpenAI had related discussions too.

But from an open-source perspective, R1 is the first to directly show us that this can be done. I believe OpenAI and DeepMind have extensive internal research on this, and it's very likely that o1 went through the same journey described in the R1 paper to develop its own model. So many frontier researchers are working in this direction. But in terms of open source, R1 is the first to do this. R1-Zero is a brilliant innovation, but the paper also clearly points out two shortcomings that make it unsuitable as a model for everyday use.

The first shortcoming: the reasoning steps that the model figures out on its own are hard for humans to understand. Many people online have vividly described R1-Zero's thinking process as not speaking "human."

The second shortcoming: the intermediate thinking process mixes languages, combining Chinese, English, and others. Because the model has learned all languages, to it there's no difference between languages — all languages are tokens, all computation is probability selection, all calculation is attention. Whichever token better expresses the direction of reasoning, it picks that one. For it, there are no barriers between different languages, no boundaries. So whichever word better expresses the current reasoning direction, that's the one it chooses. This creates the effect we see in R1-Zero.

But such results aren't suitable for daily use. We want to understand the intermediate thinking process, and we want language consistency. To solve these two problems, DeepSeek added more methods, evolving from R1-Zero to R1.

The methodology is described in detail in the paper. Step one: perform SFT fine-tuning on the DeepSeek V3 base model, using a batch of COT data. This is equivalent to giving the model a basic response framework — for example, when encountering a certain type of problem, first do this, second do that, and so on.

For instance, if we're doing a summary statement, first you should look at various materials, second compare them, third write the summary. It would write out a framework like this. After fine-tuning with these frameworks, the model's responses become closer to how humans speak, ensuring consistent language use and overcoming the language consistency problem. Then on this basis, reinforcement learning is applied — essentially giving only the basic framework and letting the model figure out all multi-step problem-solving methods on its own, yielding an intermediate model. This intermediate model still has some issues. Although the language may be consistent, after extensive capability expansion through reinforcement learning, many capabilities in terms of response style still don't fully sound like "human speech."

So how to solve this? DeepSeek used distillation. The intermediate model is distilled to produce a batch of Q&A pairs (these pairs contain large numbers of questions and response styles). These Q&A pairs are then filtered using a "rejection sampling mechanism." This algorithm is quite common. The idea is that humans calculate, through examples, the difference between human response frameworks and the distilled data's response frameworks. For cases with particularly large differences that don't sound like "human speech," they're rejected — it's essentially a filtering algorithm.

After this processing, DeepSeek obtained 600,000 very high-quality multi-step reasoning Q&A pairs. Using this data as a foundation, plus 200,000 traditional fine-tuning data points to give the model conventional LLM capabilities like conversational language, text creation, and reading comprehension — all best practices using traditional fine-tuning. Combined, that's 800,000 data points total. With this data, they re-fine-tuned the DeepSeek V3 base model. What you get at this point is R1. It has both traditional large language model capabilities and multi-step thinking and reasoning capabilities. The reasoning results conform to human thinking habits, and the language is consistent. It achieves o1-level results. The entire process is truly brilliant.

Distillation and Innovation: Can R1 Surpass OpenAI?

🚥 Koji

Sounds like it all flows together seamlessly. Teacher Li's explanation is brilliant too — like a documentary, very engaging.

Right on to the third question. Teacher Li just mentioned distillation. There's been controversy online: did DeepSeek actually distill from OpenAI? Setting aside questions of originality or infringement — another question is, if you only use distillation, is it possible to surpass OpenAI, to surpass SOTA?

👦🏻 Leding Li

First, I believe DeepSeek didn't use OpenAI's data, and there was no need to.

Suppose it did use distilled data — what does distillation do? From the paper's discussion we can see that the only place external data is used is in the first step of SFT from the DeepSeek V3 base model, using a batch of COT data to let the model learn basic response frameworks.

So what if it used OpenAI's results? The final capabilities come from the reinforcement learning that follows. If it didn't use them, no problem — DeepSeek could absolutely find experts to do the annotations themselves. That's not a particularly major difficulty. Without predecessors' data for reference, there might be some detours — for example, discovering that the R1 path works better for math problems, perhaps after doing large amounts of humanities COT then distilling and experimenting, finding the results poor, then reworking on math problems. Just wastes some time, but eventually they'd figure out suitable seed COT data. So there was no need to distill OpenAI's data. Maybe there was some reference at the start — that's normal, everyone references others' work.

Can you surpass SOTA with distillation alone? Absolutely not. This "no" doesn't mean you produce a final product and look at its performance metrics/benchmarks and find it's not as good as the current leader — that situation can happen.

The impossibility lies here: if the model you produce is distilled, where did its distillation data come from in the first place? This distillation data must necessarily come from extensive expansion through reinforcement learning, not from distilling others. If you distill from others, your capabilities will certainly be less than theirs — this is established experience.

Traditional distillation is always from a large model to a small model. The small model will then possess 80%, 90% of the large model's capabilities, depending on your distillation technique, but it absolutely cannot reverse-surpass the large model — that would be stepping on your own left foot to reach the sky, which is wrong. So in the middle, if you want your model to perform well in reasoning, you must introduce reinforcement learning. At that point, it surpasses distillation.

🚥 Koji

The industry consensus still says that o1 brought a paradigm shift, inspiring many people — whether DeepSeek's R1 or Moonshot AI's reasoning model Kimi 1.5. This knowledge relay, innovating while standing on others' shoulders, gives a feeling of technological idealism driving human progress.

Recently, Anthropic's CEO Dario has been getting quite a lot of flak. He used to be quite respected in the industry. After DeepSeek R1's release, he wrote an article representing certain American viewpoints, calling for further chip restrictions on China to protect American absolute leadership in large models and AI.

Does this mean — and this is our fourth question — that DeepSeek R1's RL in the post-training route has clearly won out compared to Anthropic's consistently used RLHF route, so Anthropic feels threatened, leading to Dario's series of reactions?

👦🏻 Leding Li

That article went viral across the entire internet. We need to look at it objectively, analyzing from three levels.

First, looking at the whole thing, the article is entirely China conspiracy theory, trying to amplify the China threat and cover up the fact that Anthropic and OpenAI's capabilities are being caught up by China. This part is pure conspiracy theory. But the article's discussion of purely technical matters is actually quite fair — the extensive technical discussion of DeepSeek's internals is worth taking on board.

He stated a fact: DeepSeek's capabilities still have a six-month gap with theirs. There's no need for us to avoid this. Looking at the metrics, we indeed haven't surpassed OpenAI and Anthropic. For example, R1's performance only reaches the lower bound of o1, and OpenAI already has o3 — so there is still a gap in capabilities. But what he didn't say is that Anthropic hasn't made much progress in these six months either. The gap isn't that large. As leaders, their pace has already slowed, while we're catching up.

Speaking of technology, I think we need to maintain an open mindset. You can't preset that a certain technology is definitely right or wrong. If you do that, DeepSeek couldn't have produced R1, because the mainstream approach before was Llama, which has huge architectural differences from DeepSeek.

I agree with Liang Wenfeng's view in his interview: "Large model research right now is more in the research phase." The typical approach in the research phase is to propose hypotheses and confirm them through experiments. Before experiments yield conclusions, you can't say a hypothesis is definitely right or wrong. So I can't answer whether pure reinforcement learning or Anthropic's method is correct.

Anthropic believes model capabilities are a continuous spectrum, that the base model should possess both humanities response capabilities and STEM multi-step thinking capabilities. This is also a very powerful idea — perhaps they have better methods to achieve it later.

We follow this with a scientific mindset, watching who can prove their ideas. But in terms of spirit, we don't agree with America's approach. America is incapable of preventing China from developing large models. We already fully possess all generative AI techniques. We only have difficulties with hardware manufacturing, but domestic AI training chips exist too. No matter what, they can't stop us from making the world's best models.

🚥 Koji

So our inference cards this time have already been proven usable by R1 (referencing SiliconFlow). The API R1 released this time was done on Huawei Cloud inference cards. So what level are our training cards at now?

👦🏻 Leding Li

Training cards are currently at the usable stage, but somewhat difficult to use.

If you want to reproduce DeepSeek in three months, the optimal solution is still to use the full NVIDIA stack — GPUs, NVLink, ideally InfiniBand for networking — because the entire system is very mature. If you want to explore completely new models, CUDA provides the most complete support. There's no need to avoid this. But domestic cards can absolutely be used for training. They require some time for model adaptation.

My hardware friends say that for relatively fixed, established model training, adaptation takes about three months. If you want to do custom operator exploration, it takes longer.

Open Source Beats Closed Source: The Key Factor Behind DeepSeek's Success

🚥 Koji

The first four questions were pretty technical, so to give everyone a breather, the fifth one will be lighter — and it's something everyone cares about.

DeepSeek fought this battle beautifully. The factors behind this sweeping victory include open source, strong technical chops, low costs, and decent product experience. Teacher Li, in your view, which of these factors matters most, and what percentage would you assign to each? Could you share your thoughts?

👦🏻 Li Leding

In my view, open source is the most important. Professor Yann LeCun put it well: "Rather than saying China beat the United States, it's more accurate to say open source beat closed source."

Looking back at DeepSeek's papers, starting from R1 and tracing back to V3, V3 had MoE and MLA, and going further back to V2 and DeepSeek-Math's exploration of reinforcement learning — we can very clearly sense that frontline researchers in the industry had a feel for where things were headed, but what held them back was likely the lack of resources to run innovative experiments. I think experiments often come at a cost. DeepSeek, for instance, ran massive numbers of experiments to validate which methods worked and which didn't. For R1, they tried process reward modeling, and it didn't pan out. In V3, they compared how much their new MTP multi-token prediction actually improved model performance. This all means that building a great model requires enormous experimentation, because nobody truly knows the right path forward.

You could say that generative AI — AI as a whole, really — is theoretically incomplete. Unlike computer science, which has Turing completeness that tells you everything, AI doesn't. Often, the right method can only be found through experimentation.

So what's more important is giving researchers enough opportunities to experiment. You can't punish someone because the model they trained this time performed 2% worse than the last version, and then deny them resources. If you do that, you'll never build the best model.

So when we look at how o1 came about, I think there were two key factors behind it. First, DeepSeek-V3 dramatically reduced training costs. V3's entire training run cost only $5.5 million, using just over 2,000 GPUs. And building on top of it for inference and further reinforcement learning was vastly cheaper than using the kind of oversized models we traditionally see. This meant that with equivalent resource investment, DeepSeek could run far more experiments.

Second, I think DeepSeek itself maintained a very good original intention. As Liang Wenfeng said, "We're not doing commercialization, pure research." In that context, I can accept model failures, I can accept that this run didn't work well, I can even accept underperformance for a stretch of time.

For example, if we look carefully at the paper, we see they ran reinforcement learning for 8,000 steps — which is virtually unheard of.

No previously published paper had pushed reinforcement learning this far. It's not that researchers couldn't think of this, whether in China or the United States — we have excellent researchers everywhere. Everyone could think of it. But the problem is that improvement from step 0 to step 8,000 isn't linear — there's decline in the middle, there's back-and-forth. So how do you guarantee sufficient resources to persist when, say, at step 2,000, your reinforcement learning slope stops rising quickly, or even dips temporarily?

I think this requires better management support. So a research-oriented, open-source approach is what can nurture such advanced technology.

🚥 Ronghui

I've seen analyses saying that what everyone sees is DeepSeek's massive success this time. But behind that success were numerous failures — we just probably don't know about them.

Teacher Li, you just mentioned these 8,000 steps. Could you explain more concretely and accessibly what those 8,000 steps actually mean?

👦🏻 Li Leding

Sure. In R1-Zero's application, we can understand it this way: first, you have the model answer math problems knowing absolutely nothing. Where do these math problems come from? We refer to the DeepSeek-Math paper — it used 8,000 seed math problems from a specialized evaluation set, with problems and standard answers, and had the machine generate answers on its own.

Obviously, most answers should be wrong, with only a tiny fraction correct. At this point, the reinforcement learning system scores them, marking correct and incorrect answers. Of course, there's also a very important technique here — the GRPO reward model, which we'll get to shortly.

In this process, I can score the model's responses, select the good ones, discard the bad ones, and have the model fine-tune my base model on these better data. The model's capability improves by one step. For instance, out of 8,000 problems initially, it might only get 10 right — so the accuracy is 10/8,000. Then I take these 10 correct answers to fine-tune the model, and begin step two of reinforcement learning: still having it answer those same 8,000 problems. Likely, the number it gets right will go from 10 to 20, 30 (these numbers are illustrative). Again using the reward model to select good answers, discard bad ones, and feed more good answers back to the model.

Then comes round three, and so on — fourth, fifth, sixth round. At a glance, we see that with each round, the number of tokens in the model's generated reasoning chain, the intermediate reasoning process, keeps growing longer. The paper describes this very well — it's basically linear growth. The model says more and more, which means it's thinking more and more.

Meanwhile, we see that as steps increase, overall accuracy improves. But what's interesting is that this improvement isn't linear. In the first 4,000 steps, the accuracy improvement slope is basically a straight line shooting upward — performance gets much better. But from 4,000 to 6,000 steps, the slope decays, shifting from very rapid growth to relatively slow growth.

And after 6,000 steps, model accuracy even dips a bit, still declining around step 7,000. Then after 7,500 steps, model accuracy starts rising again, forming this kind of curve.

Of course, DeepSeek may have done so much preliminary work and exploration that their curve looks very clean.

Many reinforcement learning studies might see their improvement slope decay by step 1,000. By step 2,000 it plateaus, or even turns downward. At that point, people might not be willing to spend resources on the next four or five thousand steps.

🚥 Ronghui

So could we say that they were really trying to test and prove this direction, this choice, wanting to see what results it could yield — so they were very persistent?

👦🏻 Li Leding

Yes, exactly. First, I think we'll insert something about GRPO's reward model in a moment — because when you truly think through this paper, you'll find the entire R1 training approach so natural that you're bound to gain confidence, to believe this thing can definitely succeed.

Then combined with DeepSeek's excellent low-cost training and a great boss, I think everyone formed a shared decision. Let's do it — train it to a point no one else imagined, and see what happens. That's the reason for success. Here I want to add that most of R1's techniques come from simple ideas, things you can roughly judge as feasible through intuition, and GRPO is the most complex idea among them.

Returning to the reinforcement learning process as a whole, we can simplify: first have the system try to answer, say, 100 problems, then take the 10 correct ones back to fine-tune the model. Next training round, maybe 20 out of 100 are correct.

At this point, do we feed all 20 correct answers to the model for fine-tuning? There's a problem here: among these 20 correct answers, a considerable portion may overlap with the previous 10. Going back to pre-training large models two years ago, we found that while data matters, it's best not to reuse data for training — it easily causes overfitting. To put it intuitively, as the model continues training through reinforcement learning, we'd rather it focus more on newly discovered tricks than repeatedly rehearsing skills it already mastered.

So GRPO's job is to try to quantify which newly trained tricks are newer and better.

R1's approach works like this: first, for each problem, the system generates multiple answers — here they chose 8 — then scores these 8 answers sequentially, giving 1 point for correct, 0 for incorrect. Then these 8 scores are converted into a statistical measure called a Z-score. This score is commonly used in statistics. What's its purpose? Its purpose is to create better differentiation.

For example, if most of the eight answers are correct, we consider that the model has basically mastered this problem type well, so its Z-score will be relatively low. Conversely, if only a few of the eight answers are correct, we consider that the model has learned a new trick, and those few correct answers will have very high Z-scores. Coincidentally, from a mathematical standpoint, Z-scores happen to fall between 0 and 1 as decimals, so they can be directly treated as sampling probabilities.

With such a score and sampling probability, we can have the model fine-tune on newly learned tricks with higher probability. And for already-mastered, already-known old knowledge, feed it to the model for review with lower probability. Focus more on the new, less on the old. Repeating this cycle, the model continuously selects the best of the best, achieving the entire process.

Of course, beyond its conceptual advantages, GRPO also has significant benefits over traditional reinforcement learning algorithms like PPO and DPO in terms of computational cost and memory consumption — because it only rewards the final outcome without attending to intermediate steps. Taken together, these factors make it the excellent algorithm that R1 employs.

🚥 Ronghui

I saw that Lex Fridman blog post where he explained in detail some of the low-level optimizations DeepSeek made. He even listed this as key to their low-cost training success.

He mentioned three points. The first is the MoE and MLA you mentioned earlier — because of chip constraints, they had to improve efficiency through low-level optimizations. Without access to NVIDIA's NCCL library, they had to innovate quite a bit. Do you know much about this? Like, what specific innovations did they make?

👦🏻 Li Leding

What exactly they did is written quite clearly in the paper. Basically, it's through some low-level CUDA optimizations, plus orchestration optimizations between different computation and communication during training, to ensure their system had sufficient bandwidth.

With enough bandwidth, communication doesn't get bottlenecked. But reading through the paper, I think DeepSeek only had H800s — neither H100s nor H20s. Why? Because the main difference between H800 and H100 is that NVLink bandwidth was cut in half. So they were forced to do this optimization; they had no choice. On the other hand, the H20's communication capability and compute power are quite far from the H800's. With their current optimization approach, running on H20s probably wouldn't yield the same results. So I think they used H800s. But they haven't said so, and nobody really knows.

🚥 Ronghui

I find it quite inspiring — they turned their constraint into innovation.

👦🏻 Li Leding

Yes, exactly. But if they had H100s, the results would certainly be even better. The H800's limitations were indeed quite severe.

🚥 Ronghui

Lex listed these three points as the most important. Do you agree?

👦🏻 Li Leding

I basically agree. I think there's another point — their FP8 low-precision training. I think that's very important; these are the main contributing factors. Americans are generally reliable when analyzing technology — what they say is correct. But their interpretations, well, take those with a grain of salt.

Low-Cost AI: The Technical Breakthrough Behind $5.5 Million

🚥 Koji

Our sixth question follows directly: how did DeepSeek's V3 manage to cost only $5.5 million?

And not long ago there was another piece of news — Fei-Fei Li's team reportedly trained an AI reasoning model for just $50 that could supposedly match DeepSeek R1 and OpenAI o1. What's going on with that? Could Teacher Li share some thoughts?

👦🏻 Li Leding

These are two separate questions — one about V3's $5.5 million, and one about Fei-Fei Li. I think let's start with Fei-Fei Li. The "$50 to train R1" is indeed a bit clickbaity, but this paper is nonetheless very important. To understand its significance, we need to go back to the R1 paper itself, which is so excellent that it turned what should have been dessert into the main course.

At the beginning I mentioned that R1 actually trained three models in the end. We've covered two — R1-Zero and R1. What's the third? DeepSeek was actually trying to answer this question: both R1 and R1-Zero were trained on top of DeepSeek V3, this massive base model — 671B parameters (though MoE, still a very large model).

Training on this very large model, we confirmed that both reinforcement learning and distillation approaches can give it excellent multi-step reasoning capabilities.

So the natural question becomes: on a smaller model without this scale, can reinforcement learning and distillation also give it good multi-step thinking ability? DeepSeek ran experiments. Their method was to take those 800,000 precious distilled data samples from the intermediate model we discussed earlier, and use this data to fine-tune Qwen and Llama, these relatively smaller models (if I'm remembering the paper correctly, 7B and 30B). This is essentially distillation — using data to distill.

The answer is: yes. Using precious distilled data on small models can also give them excellent math capabilities — the paper has corresponding benchmarks, and the results are very good. But the paper also tells you: if you directly use reinforcement learning on small models, using R1-Zero's approach — does that work? Not really. The paper suggests this is because small models simply don't have enough knowledge.

It's like if you want to cultivate yourself through practice, it's better to have more knowledge first. Without any foundation, you're prone to going astray. This is absolutely not dessert — this is the main course. Because this means taking a 30B small model, or even a 7B small model, can achieve step-by-step reasoning capabilities similar to R1.

Following this line of thought, let's think further: fine-tuning small models with 800,000 data samples — what about 80,000? If fine-tuning with less data is beneficial for applications to acquire their own capabilities, does 80,000 work? What about 8,000?

This brings us to Fei-Fei Li's S1 paper.

Here, the final answer it gives us is: if you want good math performance, 1,000 samples is enough.

How did they get there? They first distilled 59,000 high-quality question-answer pairs from Google's Gemini 2.0 Flash Thinking (also an excellent reasoning model — objectively speaking, its performance is no worse than R1). Then Fei-Fei Li curated these questions, because her goal was to see how small this distilled dataset could be compressed while still letting the model learn the capability. She used three principles in between: the questions need to be sufficiently difficult, not repetitive, and cover a broad enough range. This ultimately yielded 1,000 samples.

The paper very clearly tells you what these roughly 1,000 samples are: all kinds of math, from relatively simple linear algebra and calculus to more complex differential equations, plus lots of quantum physics and so on — basically all STEM problems.

Using just these 1,000 samples to fine-tune Qwen, you can achieve math performance comparable to R1.

So Fei-Fei Li is telling you here: if you just want to quickly get R1-level capabilities on an ordinary model using distillation, you only need 1,000 data samples.

And the paper states that fine-tuning with these 1,000 samples only requires 16 H100s, training for 26 minutes, which works out to about $50. Of course, $50 is an extremely极限的测试, so I say it's rather clickbaity — in practice it would definitely cost more. But this also means that if you simply want to acquire vertical capabilities, post-training costs can be very low.

Let's connect this back to V3. In V3's paper, they not only tell you that pre-training cost $5.5 million — 2.6 million GPU hours (let's use GPU hours because unit prices vary by region) — these 2.6 million GPU hours can pre-train V3. They also tell you that the compute used for post-training doesn't even amount to a fraction of pre-training.

So sometimes I get a bit annoyed when people online say pre-training scaling is over, and now it's all about post-training scaling, with compute continuing to expand infinitely. Post-training does have scaling, but its scale is not comparable to pre-training — it's several orders of magnitude smaller.

As for V3's $5.5 million, how was this 2.6 million GPU hours achieved? We need to go back to V2 — this was the result of accumulated efforts over a long period. The most important techniques for saving model parameters were completed in V2, namely MoE, which was further optimized in V3. How to reduce the compute consumed by attention calculations in each forward pass — this was also completed in V2, with their MLA attention, which V3 inherited and optimized.

What V3 did was use FP8 training. Actually, I think Americans should have done this long ago, because FP8 is one of the H100's most important selling points. Americans have so many H100s in hand, yet they didn't try this — I do think there's a bit of a resource curse at play. Doing it isn't some magical thing; it's just running lots of experiments, figuring out which parts of the forward and backward propagation can use FP8. This contributed another portion. These three factors together account for the vast majority of the gains. Put them all together, and 2.6 million GPU hours can basically complete a training run.

🚥 Koji

I think Teacher Li explained this wonderfully — to make things so clear without video or slides is really not easy.

👦🏻 Li Leding

I spent a long time on these two papers. They're indeed very well written.

🚥 Ronghui

I actually really want to ask this question. You mentioned this several times, and in hidecloud's presentation he also repeated it several times — that this paper is just too excellently written.

I even asked ChatGPT today: how does the industry generally measure whether a paper is particularly well written? It gave me some dimensions. But I want to ask your subjective judgment — from what angles do you feel this paper is so excellently written?

👦🏻 Li Leding

In simple terms: it's packed with substance. This paper has just so much substance. To be honest, although I've been very focused on AI, over the past two years I've become increasingly reluctant to read papers, because most papers simply don't contain much useful content.

You could say the hard content in this paper is equivalent to ten or more typical papers. Take R1, for instance — training three models could easily have been three separate papers. And then in DeepSeek V3, there are so many different optimization techniques. To be honest, many of them DeepSeek hasn't fully refined yet — the effects are modest, and later versions like V4 or V5 will likely improve on them. But with so many experiments, so much work, if published normally, you could easily get five or six papers out of it. So the content density really is extraordinarily high.

🚥 Ronghui

And I think this approach actually fits the communication style of the open-source world very well. They publish one paper, and the community interprets it through that paper — it's essentially doing massive secondary distribution for them.

👦🏻 Li Leding

Yes, this is the community spirit.

DeepSeek's Profound Impact on China's Tech World

🚥 Koji

Let's move into our second major section — how to properly assess the series of impacts brought by DeepSeek. This is our seventh question.

DeepSeek has clearly stirred strong national sentiment, especially considering that every person on the team was trained within China's engineering university system. Just a couple days ago, I heard an investment veteran say at a private gathering, very emotionally, that he considers this China's national destiny. Without it, we likely would have slid into deflation. DeepSeek's release, he said, may even have prevented China from entering something like Japan's Lost Three Decades.

Teacher Li, how do you view DeepSeek's impact on China's tech industry and the broader Chinese business world?

👦🏻 Li Leding

That's too grand a question — I really can't answer it.

From my perspective, one thing is very clear: DeepSeek has proven that Chinese researchers and AI software practitioners possess knowledge and capabilities no less than those in the United States.

We have all the knowledge needed to build the most advanced things in near-complete independence — that's excellent. As for its impact on larger levels, I don't know, but I believe people will certainly feel excited by it. Of course, I think some of the online discourse recently hasn't been great — we shouldn't suddenly feel like we've completely surpassed the United States.

In Liang Wenfeng's interviews, there are two viewpoints I particularly liked. I mentioned the first earlier; the second is his view on where exactly the gap between China and the US lies.

In AI, we may only be a year or even half a year behind. But rather than a difference in time, it's more a difference between originality and following.

We must and should acknowledge that almost all original ideas in AI still come from the United States.

Who first truly applied reinforcement learning in industry? DeepMind. Where did the GPT scaling approach — increasing model parameters to boost capability — originate? OpenAI. All from the United States. So if at this stage we can't transform from followers into originators, our gap with the US may forever remain at one or two years. We can follow closely, we can learn, but we can't surpass it.

I have a feeling that DeepSeek today is more like OpenAI in the Ilya era, not the Sam era.

Current OpenAI is too path-dependent, completely closed off. One, it's too obsessed with scaling — adding compute. Two, productization has consumed too much of its energy. Actually, what people most expect from OpenAI is to build GPT-5, to surpass what came before, to do these things — that's what people want to see.

🚥 Koji

That brings us perfectly to our eighth question: why was Liang Wenfeng able to lead DeepSeek to these results, while other domestic giants — we won't name names, but everyone knows who they are — why couldn't they do it?

👦🏻 Li Leding

Subjectively, I think often it's excessive KPI pressure, with KPI targets set unreasonably. Liang Wenfeng's team, from what I understand, doesn't have KPIs. People maintain more of a pure researcher attitude. We don't assume from the start that a certain path is definitely correct — everyone discusses together, sees which path works, invests resources there. If it doesn't work, it doesn't work; if results are poor, maybe persist a bit longer. If it's clearly not working, maybe switch approaches.

This is the normal path of scientific exploration. From childhood we learn about Edison inventing the lightbulb, trying so many times — without failure, where does success come from? But working at a major company may not be like this. Big companies have far more resources, talent, and data, but also far more pressure. Often if a model version can't improve results by 3% or 5%, you might not keep your job. This really affects how people work — they may be less willing to try things that currently look unpromising, more inclined to follow established patterns: predecessors did it this way, so build on that with some scale expansion, that's how you survive. This is just my personal opinion.

🚥 Koji

This actually leads to our ninth question: DeepSeek's victory appears to be a victory for engineering students, for technology. So the question becomes: what should product managers do in the AI era?

👦🏻 Li Leding

I think product managers are absolutely crucial for AI — they're the soul. Let's not just say good things; let's also splash some cold water.

Honestly, current AI model development and real AI application needs are misaligned.

R1 is truly excellent, o1 and o3 are excellent. But applications don't need this. Applications need stable, reproducible, and horizontally scalable capabilities across domains — not doing well in one pure math domain but still not being 100% correct. That's different from what applications need.

For example, besides the consensus on pre-training hitting walls and post-training showing promise, there's a third consensus: applications don't choose the most powerful model.

The common practice now is using small models, even very small ones, no longer pursuing models with strong reasoning capabilities. Leave all these issues to other things — to prompt engineering written by product managers, to RAG systems, to external programs for judgment. Leave logic to traditional approaches; only have AI handle work similar to traditional NLP. This is how applications actually use AI today. Because from the application perspective, current models simply don't achieve the quality people want, so we settle for less and use what we can. Having said all this, what is the product manager's role? It's incredibly important.

Product managers must define boundaries in products, clearly identifying what AI cannot realistically accomplish and how to handle it.

Like the example I gave last time we talked. Say I ask AI to book a flight — you can't imagine that two years later, such a product still hasn't been made. Why? Even with o1, Americans have had it for six months and still haven't built it, it hasn't gotten better. Why? Because in practice, booking a flight isn't a one-shot deal.

You can't tell AI upfront exactly what ticket you want. It's impossible to say: book me a flight tomorrow from Beijing to Shanghai, what time, which airline, economy or business, upgrade or not — I can't say all that in advance. It must involve constantly looking, constantly thinking, comparing with my itinerary, with very complex processes in between. This process can't be accomplished in one AI step. How to improve overall efficiency in the middle — this is definitely the product manager's job. So product managers are truly the soul of AI moving toward applications; they determine the work here, and at this level, they absolutely cannot be replaced by anyone.

Consider this example: RAG, which has massive application in enterprise intranet search. Anyone who's worked with RAG knows the best practice: RAG retrieval. The content RAG needs should come only from your internal search system, your networked retrieval system. You want to use knowledge from RAG, not the model's own knowledge, because the model's own knowledge produces hallucinations — but you want the model's language capabilities for LLM understanding, summarization, and so on. This should be a good combination: RAG provides knowledge, LLM provides capability.

But in practice, you find their boundaries can't be precisely controlled. What often happens is the model's output couples RAG knowledge with the model's own knowledge, with the model's own knowledge used even more — causing hallucinations.

Conversely, sometimes even when fully applying RAG knowledge, the RAG data actually suppresses the LLM's inherent capabilities. These situations happen constantly, but they're hard to change because you can't retrain a model — you can only add additional methods outside your RAG system. So from this perspective, large models have this problem in use too. And you can't modify large models; fine-tuning costs are even higher, so you might as well use a small model and leave more logical work, stability work to RAG.

DeepSeek's Impact on ToC, ToB, and ToD

🚥 Koji

Very clear. Okay, let's move to part three: after DeepSeek's release, its various impacts on ToC, ToB, and ToD (To Developer).

Tenth question: DeepSeek's release, especially R1 — Teacher Li, what impacts do you think it has brought to ToC products?

👦🏻 Li Leding

I think for ToC, actually for the entire industry, it has massive impact. Because costs have genuinely been reduced by an order of magnitude.

Right now, our AI applications are often bottlenecked not by cost, but by capability. If this portion of costs comes down, I can reserve more budget to bring in humans, allowing product managers to do more planning and design on the human-intensive side. That could make ToC products better — that's one angle. The second is that for places already using AI, but held back by cost — basic NLP understanding, basic planning, and so on — all of that can now be deployed. That might trigger a wave where everyone starts experimenting.

🚥 Koji

What about the impact on ToC products? Some people feel like the model is the product now, and that it'll drown out a lot of actual products. Do you share that feeling?

👦🏻 Li Leding

I don't think so. In my view, whether it's OpenAI's app, or others like Gemini, or domestic apps — if you look at the data, none of them have particularly long user session times. Including our own usage, we don't actually use them that many times per day.

This largely reflects that we haven't accurately identified people's pain points yet.

It's probably because the technology is evolving too fast, and there aren't enough strong product managers to define what problems AI solves best. So when people's needs aren't properly defined, positioned, and met, they can only try things out, see whose tech is better — that's what creates this effect. I think once the market enters a more mature operating phase, with more product managers coming in to define things, it won't be like this anymore. Unless AGI happens.

🚥 Koji

I agree. Okay, our eleventh question: DeepSeek's release's impact on ToB — and ToB splits into To Enterprise and To Government. What impacts do you see?

👦🏻 Li Leding

For ToB and ToG, I think the impact could be quite far-reaching. You observed that very shortly after DeepSeek launched, Azure started supporting it, then AWS quickly followed, and today virtually all tier-one clouds have followed suit, allowing you to use it as an inference model.

This actually raises a question. From what we've observed in open source before: when a technology's ceiling stops advancing rapidly, the existential crisis for commercial companies is removed. Because you no longer face the scenario where, if you don't immediately跟进 the most advanced model, someone releases a dramatically improved model six months later that kills your business entirely. In this situation, companies may shift their research focus from an arms race of who can build a GPT-4-level model to something more rational, directing research teams toward approaches like DeepSeek's — exploring next-generation models beyond Transformer. There's no rush on when that gets done.

Meanwhile, on the product side, everyone directly uses open-source DeepSeek, giving everyone the same product. Nearly all Chinese internet companies, including American ones, heavily use open-source software — everyone uses Linux, everyone uses MySQL, Redis. Open-source usage is massive. It's just that when open source is genuinely the best in the industry, and there's no existential crisis, there's no reason not to use it. So the entire ToB space may be affected, especially in China — I think there'll be deep impact.

But for the US, I think America may get its own DeepSeek. Because of players like Anthropic, and due to the current US political climate, there's likely hostility toward China. They probably don't want to see a Chinese product build a massive ecosystem in America, so they'll likely build something similar themselves. But if frontier model improvement stalls, in the future the ToB direction may see everyone using open-source models.

🚥 Koji

Okay, our twelfth question is about ToD, since ToD is also a very important direction for large language models — AI coding, To Developer.

Teacher Li, what impact do you think DeepSeek has had? Because I saw that Cursor officially came out and said that based on both their own evaluations and user feedback, they still believe Claude 3.5 Sonnet is in a league of its own for coding.

👦🏻 Li Leding

Yes, I think for To Developer the impact is probably the smallest. Because Developer happens to be the direction that Claude, OpenAI o3, and R1 all excel at most, and this direction is still rapidly evolving, so who's ahead is still uncertain — there may be major shifts ahead.

And the quality of these models directly determines what models To Developer software like Cursor uses. But I think it's fine — this kind of competition is what births new technology. Maybe we've fallen behind America again at this stage, so what? After some time, with good research, we'll surpass them again.

The ultimate beneficiaries may be developers themselves — everyone will have a great AI chatbot in hand to help with writing code, tests, documentation, and so on. That's a good thing.

🚥 Koji

Truly fascinating. I was just suddenly reflecting — in my first thirty-some years of life, when else have I felt technological progress happening right before my eyes so intensely? I don't think I've ever felt it this strongly before.

🚥 Ronghui

But it's like the first time you used an iPhone.

🚥 Koji

Right, but after the first iPhone, there wasn't this catch-up dynamic — like using an iPhone today, then a new iPhone three months later. No, so this feeling is quite peculiar. I think it may be once-in-a-lifetime.

Our thirteenth question: looking at DeepSeek's impact on AI applications. Actually, before DeepSeek's release — I don't know if everyone remembers, there's been so much news — but Crossing did two pieces where we were quite excited. Davin's launch event marked 2025 as potentially the first year of AI agent deployment.

Now with DeepSeek's release, what additional impact does it have on the application space as a whole? Teacher Li mentioned some earlier; any further thoughts here?

👦🏻 Li Leding

For applications, I think it's essentially an extension of the ToB question. One is what tech stack applications use — the ToB issue of everyone potentially shifting to open-source models, which seems quite likely. The other is whether applications themselves advance dramatically because of R1's release — I'm fairly skeptical about that.

Because from the application perspective, it's still the same fundamental problem: the capabilities applications need are ones current LLMs don't possess.

We've seen o1 out for months now, and no app has solved a previously unsolvable problem because of it. And we're seeing more and more apps, after trying all the models, deciding they don't need GPT-4-level models, or even Claude-level powerful models.

I saw something particularly interesting in America: when you ask people about Google's Gemini app, most say it's not great — feels a bit dumb, no real advantage. But if you ask them about Gemini's API, everyone says it's good. Because Gemini Flash is genuinely cheap, fast, and good.

From a cost-performance perspective, Gemini 2.0 Flash is arguably even better than DeepSeek. Looking at actual pricing, especially with Google's TPU support, its costs are extremely low. So for applications, we'd already gone down a path somewhat different from today's LLM research direction. Because everyone was already using small models, playing with RAG, playing with prompt engineering — quite different from this. So from the application frontend perspective, I don't think there'll be too much change.

🚥 Ronghui

This Gemini Flash is actually also a reasoning model, but it seems pretty unknown.

👦🏻 Li Leding

Right, yes — Google probably didn't do great on PR this time.

Its Gemini 2.0 Flash performance is SOTA-level. It's comparable to previous models like GPT-4o, but its cost is extremely low. There's also Gemini 2.0 Flash Thinking, which is a reasoning model, and it's quite good too.

Fei-Fei Li's paper distilled from it, and its data capabilities are strong too — it just hasn't quite caught up on benchmarks yet. And Google recently released other models too, like its Flash Lite model, pushing costs down further.

I think Google sees this very clearly. Because Google is actually the world's largest application player — the biggest AI application is search. It saw early on that reducing AI costs is absolutely critical to pushing existing applications toward larger-scale AI adoption, so it's been emphasizing this. Though Google probably also suffers from big-company disease — it really hasn't done well on PR lately.

🚥 Koji

I also think their product execution is poor. This Gemini 2.0 Flash Thinking is buried inside Google AI Studio, and that Google AI Studio interface really isn't built for ordinary users. I used it several times when it first launched and was genuinely impressed. But the entry point is so deep, every time I wanted to use it again I had to stop and remember how to navigate there — after a few times I just couldn't be bothered. It's a real shame.

Okay, to our fourteenth question: after DeepSeek's launch, it very quickly hit number one on App Store charts in dozens of countries. It's also the fastest app in history to go from zero to 30 million to 40 million DAU. But on the other hand, we also saw that DeepSeek never had enough GPUs internally. The app's massive popularity reportedly affected their training, so they apparently never prioritized stable app maintenance as their top priority. It seems they still wanted to allocate more GPUs to training rather than serving end users.

So Mr. Li, what's your take? Is a chatbot actually important to DeepSeek? Do you predict they'll keep investing in their chatbot, or will they gradually decide it's not worth it and move on?

👦🏻 Li Leding

I can't answer that — that's probably a decision only Liang Wenfeng can make.

But speaking from my own emotional perspective, I don't want them to build a chatbot. Because once you build a chatbot, you get locked into a massive amount of productization work. In real-world usage, most users aren't going to ask complex applied math problems every day, or questions that require multi-step reasoning.

That's not what we're optimizing for. Productization forces you to focus on everyday utility questions, and those require a lot of extra work. And once you launch a product, you start caring more about quality, maintainability, and so on. All of that drains enormous energy — just look at the problems OpenAI is dealing with today.

So emotionally speaking, I don't think China needs "Another APP." China needs an OpenAI.

So I hope they keep it simple, let people experience it, and don't pour energy into this. Keep focusing on original technology — that's what gets people most excited.

🚥 Koji

Actually, back in December, we had an episode where we reviewed the year in large language models with Manqi from LatePost.

We went through all the domestic model companies and mentioned DeepSeek. One story that really stuck with me: Liang Wenfeng once stated publicly that DeepSeek doesn't care about commercialization and won't pursue it. Because he believed any attempt at commercialization would disrupt, dilute, and distract from their research focus and investment.

So what you just said, Mr. Li, aligns with that — no commercialization, not even consumer products, just pure research. That might actually be where the highest ROI lies.

👦🏻 Li Leding

Exactly.

Future Outlook: Compute, Data, and the Next Steps in AI R&D

🚥 Koji

Alright, onto part four — the future. First question of this section, and our 15th overall: will compute still matter in the future?

This is on the minds of countless retail investors. A lot of people hold NVIDIA stock. If compute doesn't matter, that means the tech giants' continued investment in chips and compute would decline. What's your view, Mr. Li?

👦🏻 Li Leding

I'll stick with my long-standing "hot take": I think people should stop investing in compute. Investing in compute ultimately means doubling down on pre-training scaling laws, and I can clearly see where the ceiling is.

There are many ways to reduce costs that people simply haven't used. Now that DeepSeek has emerged, everyone will focus on cost reduction and inference. In real-world inference applications, people are already using more small models. So I think, at least for this phase, demand for compute will definitely decline. Especially if my earlier prediction holds — when model capabilities plateau and enterprise customers start choosing open-source models, and the tech giants stop the arms race — then there's no reason to keep burning training compute.

Inference might also decline temporarily, because everyone will use small models, until AI applications cross the productization threshold and then ramp back up. So for this intermediate phase, I don't think there's any need to keep chasing compute. People should redirect that energy toward algorithms, toward research that goes beyond Transformer architectures. We've fed models a trillion tokens, several trillion tokens, and they still haven't figured out why 3 plus 5 equals 8. That's not right.

🚥 Koji

Interesting. If your "hot take" is that compute doesn't matter anymore, don't chase chips, don't invest in buying chips.

Then question 16: do you think data will still matter in the future? Can proprietary data still serve as a moat for companies that own it?

👦🏻 Li Leding

I think data is more grounded than compute, and still very important — especially in vertical domains.

All our previous applications, whether search, recommendation, or short video, have proven how much value data distribution brings to quality improvement. Because at the end of the day, whether it's AI or big data, these are all statistical models. More data means more accurate statistics. So data still matters.

But don't cross an excessive threshold. If you've already accumulated several trillion tokens for training and keep piling on more data, that's unnecessary. But foundational data remains a very important moat. And vertical application data — vertical knowledge and industry expertise — that's always a moat.

🚥 Koji

Question 17 — let's make a bold prediction about DeepSeek's next move. Right now, as we're recording, Liang Wenfeng and the DeepSeek team are probably thinking about what's next, what's on their to-do list.

Do you have any bold predictions, Mr. Li?

👦🏻 Li Leding

If it were me, I'd continue the unfinished work from V3 and R1. One thing is to keep exploring the limits of the R1, R-Zero path. I think it's worth exploring. We've gone 8,000 steps — what happens if we keep going? I suspect at 10,000 steps, 20,000 steps, we'll still see improvements. But at what point does marginal returns diminish? That's worth finding out.

And another important question: can pure reinforcement learning directly solve the problems of models not speaking naturally and language mixing? After all, distilling it one round isn't a particularly elegant approach. In technology, we should pursue elegance. I think these are all worth pursuing.

And V3 actually opened up a lot of new explorations. For example, its MTP multi-token prediction, and its new routing algorithms — these didn't show clear benefits in V3, and they ran explicit experiments, the data is all in the paper. But these directions are still very promising. If they have resources, continuing these experiments in V4, V5 — I think they'll do that.

And I think there's another point they actually mentioned in their paper: exploring new architectures beyond Transformer. This is actually what I most hope they'll do. Just the other day they were saying that if you truly want to achieve AGI in the future, it definitely won't be an autoregressive model — you have to try approaches through concepts, through world knowledge, and so on.

I still hope to see our research teams really try to surpass OpenAI and DeepMind at the level of big ideas, to think about what the path to full AGI should look like.

🚥 Koji

I can tell Mr. Li still has high hopes for DeepSeek — just like everyone else. If we're really expecting more miracles, DeepSeek does seem like the most likely source to keep giving us that sense of belief and momentum.

Alright, now question 18. Beyond DeepSeek, we have many other impressive companies and major players. I'd like to ask Mr. Li to make some bold predictions about what ByteDance, Tencent, and Alibaba will do next.

👦🏻 Li Leding

For commercial companies, if they don't face an immediate existential threat — meaning they don't see any technology developing rapidly enough to directly disrupt them in the short term — and at the same time they observe that this technology's costs can be dramatically reduced and open-source alternatives exist, their strategy will definitely involve some reassessment and adjustment.

They probably don't need to keep burning massive money on the original path. Better to build on open-source, make improvements, do more refinement, and conduct some research.

I think using open-source as a foundation for improvement, plus some first-party R&D specific to themselves, plus forward-looking exploration — that's actually the normal state of big tech work.

That's how big tech has operated for the past decade. It's not normal to pour crazy investment into something that hasn't been proven out. I think they'll all adjust after some time.

🚥 Koji

Question 19 — let's talk about the United States.

Whether it's Meta, Google, Microsoft, Amazon, or OpenAI and Anthropic — what will they do next? Do you have any information or predictions, Mr. Li?

👦🏻 Li Leding

On the US side, I think Americans were definitely shaken — that's certain. But different people were shaken by different things.

For researchers — objectively speaking, top-tier researchers — everyone roughly knows what directions to pursue. In this regard, I think the US will still be the originator of breakthroughs. So the next small breakthrough, or big breakthrough, still has a higher probability of emerging in the US. They'll pursue other directions. We need to be objective about this — we still need time to catch up.

The big US companies are similar. I understand Meta is also evaluating whether its massive investment in Llama was the right approach. I imagine they'll make adjustments similar to what Chinese commercial companies are doing. And I'll boldly guess that the US will likely produce an open-source product to rival DeepSeek, to satisfy Americans' own sense of security.

But at the same time, for many people outside the industry, this is terrifying. They thought they had unique world-leading technology, and now not only has China cracked it technically, but China has also dramatically undercut them on cost. They might react in many ways, but what can they really do?

Today we have to acknowledge that China and the US are the only two countries globally that possess the most advanced technology. Only these two sides have experimental tech, the internet, cloud computing, AI, new energy, automated robotics — China already has all of this. You can't stop it. So instead of spending time on conspiracy theories, better to focus on working harder yourself.

🚥 Ronghui

I've seen some analyses suggesting that Anthropic and Google might be among the most negatively affected.

👦🏻 Li Leding

Anthropic and OpenAI are probably under pressure too, but the biggest impact might be on Llama — it used to be the open-source leader, and now that position may be at risk.

🚥 Ronghui

I've seen a lot of people discussing model commoditization, and how the commoditization of models combined with falling inference costs will have significant implications for various companies going forward.

👦🏻 Li Leding

I think the idea of model commoditization is correct, but I don't necessarily agree that DeepSeek is the sole cause.

It's more that the general public is finally feeling it. Even without DeepSeek, the low-cost commoditization of models has been ongoing — it's been happening throughout the past year.

What we'll see is that the models people use daily have rapidly gone from hundred-billion-parameter models to ten-billion-parameter models, and even down to a few billion or tens of billions. This downward trend is continuing. To push applications forward, costs must come down — this is unavoidable.

🚥 Ronghui

Okay, question 20. When reading analyses of DeepSeek, I was reminded of something Peter Thiel once said. He believed that true innovation requires many different layers of innovation to happen simultaneously, and to be combined in a highly synergistic way.

If I could ask you to summarize, Teacher Li — DeepSeek's tremendous success in low-cost inference, viewed through Peter Thiel's framework, rests on which factors?

👦🏻 Li Leding

DeepSeek actually has quite a few innovations. But frankly, many of them aren't original to DeepSeek.

If you look at the paper citations, you'll see that many ideas were first proposed by other researchers. DeepSeek was the first to apply them at scale in an open-source setting. It's hard to say that one single company encompassed all the original innovations.

But I think the most important thing is adhering to a fundamental principle of science: whether in science or innovation, there is no fixed direction. It's unpredictable, with many different paths. Among these paths, you need to seriously experiment, try things out, and actually do the work — I think doing the work is what matters most.

🚥 Koji

Let's thank Teacher Li first. This is also Crossing's first attempt at using a 20-question format to discuss a topic. We hope this approach lets us more comprehensively understand a hot issue, a trending technology, or a popular product from various angles. Thank you again for your time, Teacher Li, and we welcome you back to Crossing in the future.

And DeepSeek is a product that I believe will remain hot for a long time. If any listeners have thoughts to share or questions to ask, feel free to leave us a comment — we'll respond to all of them. Alright, thank you all, and thank you again to Teacher Li.

If you think friends would enjoy this episode of Crossing, please forward it to them on WeChat. Finally, we welcome you to join Crossing's membership group, where we encourage everyone to chat, interact, make friends, and find future kindred spirits.

---

Subscribe to the "Crossing" Podcast

🚦 We track the industry shifts and entrepreneurial opportunities brought by the new wave of AI technology. Crossing is a metaphor Steve Jobs used for Apple — standing at the intersection of technology and liberal arts, where great products are born. AI is transforming industries across the board; we seek out, interview, and bring together "active doers" of the AI era, exploring and embracing new changes and new possibilities alongside them.

👦🏻 Host Koji: Co-founder of The Fair and Tangdao. I believe technology, especially AI, will fundamentally transform society and empower humanity in the future. Feel free to reach out to chat, exchange ideas, and connect on what's next. Koji on Jike[2], Koji's website[3]

👧🏻 Host Ronghui: Works at a tech VC, former Silicon Valley correspondent for CBNweekly. Ronghui on Jike[4]

Join the "Crossing" Membership Group

☀️ First-hand AI news and insights

👫🏻 We encourage dating / making friends / finding future kindred spirits

🦀 Add our assistant on WeChat to join: Rwkfbcianvd, or scan the QR code below

References [1] "I Want to Shatter Your Unrealistic Fantasies About AI and Rebuild a Correct Understanding": https://www.xiaoyuzhoufm.com/episode/6692529b37236c546edd4b2f

[2] Koji on Jike: https://okjk.co/0JSUes

[3] Koji's website: https://koji.super.site/

[4] Ronghui on Jike: https://okjk.co/0cbnYV