AIGC 'Peak Series' | Dr. Li Wei: Humanity's Language 'Tower of Babel' Built, ChatGPT's Glory and Challenges

"This time, communication between machine and human is the machine accommodating the human, not the other way around." Looking back at the last decade of contemporary AI history, from ImageNet to NLP language models to ChatGPT's meteoric rise, the technology has not only brought a fresh reshuffling opportunity to the AI ecosystem but also turned the red-hot AIGC赛道 into the most sought-after keyword for investment and entrepreneurship. In late February, Code Brain launched its "Together Toward the Future" AIGC-themed lecture series, inviting technical experts and industry entrepreneurs from China and abroad to share and discuss hot topics including "AIGC Concepts and Business Opportunities," "AIGC's Evolution and Development in the United States," "AIGC Implementation: Pain Points and Challenges," and "The Future of AIGC: Focusing on Commercial Applications and Opportunities." On March 1, we invited Dr. Wei Li, senior NLP architect, Chief Scientist at Trend, and former Vice President of iFlytek's AI Research Institute, to deliver an online presentation titled After the Tower of Babel: The Ecosystem and Applications of LLMs for nearly 200 MaHui members, bridging a 16-hour time difference from Silicon Valley. The talk covered topics including "ChatGPT Marks the Completion of Humanity's Tower of Babel" and "ChatGPT Is Not a Foolproof Knowledge Base," delivering a knowledge feast that was both substantive and refreshingly original.

We have transcribed and edited Dr. Li's presentation into the following article. This is the first installment, focusing on ChatGPT's impact and how the LLM behind it was built.

Selected excerpts from the presentation:

01 ChatGPT: The "Tower of Babel" for Human Language

Natural Language Processing (NLP) is the crown jewel of AI. AI is broadly divided into perceptual intelligence and cognitive intelligence, and the leap from the former to the latter is marked primarily by progress on NLP tasks. Human language is the vessel of human knowledge; mastering it is the gateway to cognitive intelligence. For millennia, eliminating language barriers has been humanity's dream. The Tower of Babel in the Bible represents this aspiration — long considered a fantasy, destined to remain unbuilt. We NLP practitioners have pursued this dream, and it felt harder than reaching the heavens.

But November 30, 2022 — remember this date. With the official release of the ChatGPT model by the American AI company OpenAI, the Tower of Babel was formally completed! It not only successfully eliminated barriers between human languages but also built a bridge for human-machine interaction. This historic moment initially drew little reaction in China; unfortunately, the country was at the peak of its COVID outbreak. Two months later, as we emerged from the pandemic, we discovered this monumental transformation. A ChatGPT tsunami began sweeping across China and the world.

Why call ChatGPT the Tower of Babel for human language? Because its linguistic ability surpasses even that of native speakers: native speakers inevitably have slips of the tongue and ungrammatical expressions, yet ChatGPT, powered by a large foundation model, generates output that is consistently idiomatic and natural. On the input side, it can listen — there is no language it cannot understand, with exceptional comprehension. On the output side, it can speak — often at great length and with fluency. Most stunningly, its linguistic performance reveals an underlying "chain of thought" and certain logical reasoning abilities, giving the impression of clarity and coherence. Behind this input-output capability lies what we call the LLM (Large Language Model). To users, it appears as an unfathomable black box containing many layers of neural networks, with internal representations as multi-dimensional vectors — colloquially dubbed the "toiler," tirelessly analyzing, understanding, and composing sentences. This "toiler's" work, expressed through ChatGPT, perfectly realizes a natural language interface between human and machine.

Let's examine how the LLM behind ChatGPT was built. Technical introductions to this topic already abound; I'll briefly summarize the underlying principles. Its foundation is GPT-3, specifically the latest version known as DaVinci, or GPT-3.5. First, the model's scale is enormous — brute force produces miracles. Hundreds of billions of tokens serve as training data, forming a model with hundreds of billions of parameters. Research shows that general-purpose large models, once they reach a certain scale, exhibit a phenomenon of capability "emergence." These emergent skills, with minimal prompting, can perform excellently across diverse tasks. Previously, this was vaguely attributed to quantitative change leading to qualitative change, essentially treating the miracle as a mystery — as if it were divine favor, a modern version of the Foolish Old Man who moved mountains: the modern fool's relentless effort moved God. Now it appears far less mysterious: the emergence of multi-task capabilities requires an ultra-large-scale LLM as foundation, because without an LLM, there is no room to align the model with human preferences. The generative large model, learned from language sequences, is characterized above all by its productivity — given a prompt, there are many possible ways to continue the sequence, yet only a tiny fraction of these are desirable and beneficial. Many generations remain shallow, hollow, or even toxic.

ChatGPT's breakthrough came in this final tuning step, using reinforcement learning as its core to find an effective method for aligning with human preferences. This is like having a giant basin with 1,000 children bathing in it, and wanting to drain the bathwater without losing the children. Nearly impossible. But with coarse-grained control, you drain the water and lose 900 children. What people know is that it effectively preserved a batch of qualified children. The prerequisite for doing this is: the basin must be large. Being able to do this, daring to do this — only an ultra-large-scale model can. For example, in normal corpora, what proportion of data is directly relevant to translation or question-answering skills? A tiny fraction. When data scale is small, these skills are hard to learn from sequence-learning patterns. But with ultra-large data, a small proportion multiplied by a large number creates the conditions and soil for learning. If the model is large enough, these skills will be learned latently.

In a foundation model with nearly infinite generative possibilities, without sufficient subsequent refinement, the output would likely still be mostly dross. Thus "meeting human expectations" became the paramount goal of later-stage tuning (fine-tuning). In this process, many children are also discarded — in the literature, this is called the alignment tax (the "tax" that must be paid to build a natural language interface model aligned with humans). No matter — people cannot see the discarded treasures; they only need to see the good children. Large models have sufficient redundancy to survive layer after layer of filtering and pruning. In truth, it is not the large model itself that works miracles, but the large model that prepares the breeding ground for miracles to occur.

What distinguishes ChatGPT from previous large models is its carefully orchestrated reinforcement learning from human feedback.

For a general-purpose open system, humans cannot clearly articulate what is good or bad, but they can at least say whether a particular round of dialogue was good or bad. Using this feedback to reinforce training and fine-tune the large model, ChatGPT suddenly appears remarkably understanding. Human-machine interaction has shifted from humans accommodating machines, forced to write code, to machines accommodating humans, understanding human speech. This is a monumental transformation.

Among learning algorithms, reinforcement learning is notoriously difficult to manage because of its long chains and because the definition of the ultimate goal is not explicit and direct, but indirect — judged by results. Tuning means suppressing the high-probability dross of the original foundation model and bringing the low-probability treasures hidden within to the surface: the children are the reinforcement target aligned with human expectations, yet no specific child serves as the optimization target. After all, there is no unique answer form in this world; generation typically has no gold standard. What we have is vague human feedback based on preference: this answer is good, that one is nonsense; this is on the right track, that is discriminatory. The typical method that can effectively utilize this endgame feedback is precisely reinforcement learning. Once this feedback loop is established, the model can continuously strengthen and iterate, naturally performing better and better. Thus, reinforcement learning advanced to November 30, 2022 — the curtain rose, and humanity witnessed the miracle.

To be honest, having spent my entire career in NLP, I never imagined I would see such a miracle in my lifetime. The ancients said, "Hear the Way in the morning, die content in the evening." Witnessing the completion of the Tower of Babel surpasses even that for me; every day since feels like borrowed time. Three months have passed since ChatGPT's debut, and it still feels like a dream. Sometimes I stare at the ChatGPT icon and ask myself: is this truly the gateway to a new ecosystem, the door to a starry sea of possibilities? All signs indicate that infinite possibilities lie behind ChatGPT.

02 ChatGPT: The Pinnacle of AI's Golden Decade

Let's step back and review the contemporary history of AI's golden decade.

Ten years ago, in the ImageNet competition, deep learning overwhelmingly outperformed all other machine learning approaches for the first time, sparking an iconic neural network revolution in computer vision. Deep neural networks relied on supervised learning with big data. From then on, we knew: given sufficient labeled data, deep learning could solve the problem. After sweeping through image recognition, speech, and machine translation, it encountered a stubborn obstacle in NLP, because many NLP tasks lack large-scale labeled language data.

Five years ago, the NLP field saw the emergence of large language models — LLMs (represented by BERT and GPT) took the stage. LLMs learn from language itself, "consuming" it directly without labels — what the field calls self-supervised learning. The LLM marked the arrival of a second revolution, thrusting NLP to the center of AI and making it the core engine of cognitive intelligence. AI freed itself from dependence on labeled data, overcame the knowledge bottleneck, and leaped from perception to cognition.

Three months ago, ChatGPT was born, building an almost perfect natural language interface between human and machine. From now on, machines accommodate humans, interacting in natural language, rather than humans accommodating machines with computer languages. This is an epoch-making change.

Only with the arrival of ChatGPT did the LLM truly externalize its linguistic talent and knowledge potential, making it accessible to ordinary people. Looking back, human-machine interaction and its applications have been developed for many years, but it was never truly cracked before ChatGPT. When GPT-3 was launched two years ago, those of us in the industry already knew how capable it was. Give it a few examples, and it could follow them to produce similar output — the so-called few-shot approach, requiring no modification to the large model and no large-scale labeled data. A handful of examples could unlock GPT-3's potential to accomplish various NLP tasks. This was already stunning; it overcame the knowledge bottleneck of supervised learning. But awareness of these remarkable capabilities and their fundamental limitations remained confined to a small circle of practitioners. Its explosion still needed a linguistic bridge. ChatGPT rose to the occasion. Its defining characteristic is the so-called zero-shot capability — no examples needed at all; you can directly tell it what to do. Five years of supervised learning, five years of self-supervised learning, ultimately culminating in the completion of the ChatGPT-branded Tower of Babel — this is the pinnacle moment of AI's golden decade. ChatGPT burst onto the scene like a tsunami, "stirring the heavens to bitter cold." Borrowing the Chairman's poetic spirit:

Now I say to Chat, no need for such height, no need for such snow. How to draw the heaven-relying sword, cut you into three? One for symbols, one for domains, one for science returned. A world at peace, the globe from now on frenzied.

Incidentally, the "three cuts" mentioned here all relate to current shortcomings or limitations of LLMs: first, how to interface with symbolic resources (such as databases) and the symbolic approach. Second, how to break down domain barriers, land in scenarios to create value, and align with specific domains. The final one is the scientific theoretical demand: LLM breakthroughs so far remain empirical, with far insufficient theoretical mechanism research — for many of its behaviors, we know the what but not the why.

From a longer historical perspective on AI, 30 years ago the field primarily used symbolic logic to tackle NLP tasks. The symbolic approach and machine learning are the two routes that have alternated in dominance throughout AI history — like a pendulum swinging between east and west of the river — but over the past 30 years, machine learning has risen steadily as the mainstream, reaching new heights with deep learning in the last decade, with no sign of the pendulum swinging back. I came up through symbolic logic and rule-based systems; though not mainstream, it was not without innovation and had its own differentiated advantages. It is worth noting that symbolic rule-based models also embraced empirical data-driven approaches, relying on multi-layer module pipelines to ultimately crack linguistic structure — what we call deep parsing. Similar to LLMs, deep parsing first digests language, but converts sequential symbols into internal graph-based symbolic structures rather than the vector representations of LLMs. Different paths, same destination: deep parsing uses structure to empower downstream NLP, greatly improving efficiency. But it pales in comparison to LLMs, because LLMs crack not just syntactic structure but also context, discourse, and stylistic computation — directions involving long distances and subtle variations in linguistic expression that pose major challenges to parsers focused on sentence-level structure.

I recount this history to tell you that AI has always had two main threads. Beyond machine learning, there is traditional symbolic logic; raised to the philosophical level, the two routes belong to empiricism and rationalism respectively. They have waxed and waned over time. For 30 years, machine learning has risen steadily, from traditional statistical models to the neural network revolution of deep learning, while the other route has disappeared from the mainstream stage, though industry has never abandoned its use. The symbolic side has its own advantages in transparency and interpretability — transparency directly translates to engineering convenience for targeted debugging, in contrast to LLMs: the internal vectors of neural networks remain largely a black box, where we can only employ retraining for macro-level control, or use fine-tuning or few-shot prompting for guided induction. LLMs cannot perform surgical precision fixes like a scalpel. Perhaps one day, at the confluence of the two routes, there will be a new AI revolution or breakthrough.

From the perspective of AI generality, we see that pre-LLM models were almost all specialized — the narrower the task, the better the performance. One exception was the parser, which was essentially the "symbolic foundation model" of the pre-LLM era, using structure to empower downstream NLP tasks — similar in spirit to LLMs. Fifteen years ago, I used it to crack multi-language social media sentiment mining, building Netbase into the leading American NLP SaaS player and securing market position — to this day, it remains the absolute leader in that赛道 in the United States. But parser empowerment had high barriers to entry, achievable at best as low-code, similar to large model fine-tuning; it could not achieve the on-the-spot few-shot or zero-shot task completion of LLMs, and thus naturally lagged behind in scalable普及 and penetration.

The ultimate goal of LLMs has been designated as AGI (Artificial General Intelligence) — general-purpose artificial intelligence. AGI has long been a controversial goal; like many other scholars, I once doubted and even mocked AGI as unrealistic. Looking back now, the emergence of LLMs five years ago brought AGI from utopia into the realm of science. OpenAI, flying the AGI banner high, became the brightest star, with a dazzling transcript of influential general-purpose LLM models: beyond the NLP-oriented GPT series, there was Codex for code writing and debugging (ultimately powering Microsoft's Co-pilot service), and DALL-E for image generation.

With ChatGPT as the pinnacle, using natural language directly to convey task instructions, large models have taken on all NLP tasks — not just the various tasks and benchmarks defined by the NLP community, but also many user-defined tasks. Its NLP tasks are completely open; for any language- or knowledge-related task, you can try giving it instructions in any language, and it often responds immediately and effectively — that magical. Someone listed 49 scenario tasks it can complete, but the actual number far exceeds this; every moment, someone discovers new scenarios it can handle. This is an unprecedented spectacle in NLP history, what the industry calls skill "emergence."

We can examine why it appears so versatile and erudite. Overall, humanity's systematic knowledge is basically expressed through language. Human knowledge is carried in linguistic form; artificial mathematical formulas can be seen as an extension of human written language. From a linguistic perspective, human knowledge can be divided into intra-linguistic knowledge and extra-linguistic knowledge. Intra-linguistic knowledge includes dictionaries, syntax, morphology, discourse, style, and other knowledge, forming the foundation of human knowledge. Extra-linguistic human knowledge is a much larger circle with far broader boundaries. For extra-linguistic knowledge, current language large models are far from having cracked it; they have only captured some knowledge floating on the surface. For ChatGPT specifically, it has essentially mastered all intra-linguistic knowledge, but extra-linguistic knowledge is roughly estimated at only about 20% mastered — including common sense, basic logic, and encyclopedic knowledge. More rigorous research is needed for quantification, but the general feeling is around 20%, with 80% remaining unlearned.

Then why does mastering 20% of human knowledge make it seem like an omniscient, erudite professor? This is the law of large numbers — the so-called 80-20 rule: mastering the 20% of knowledge floating on the surface effectively covers 80% of scenarios. Precisely because 80% of knowledge remains absent, it also frequently makes fools of itself, bluffing with ignorance. Nevertheless, knowing its strengths and weaknesses and using it wisely, LLMs can still transform ecosystems and the world.

How to judge whether it has learned something, and how well it completes tasks? In AI work, every task requires a test set, and test sets need samples. Now ChatGPT uses zero-shot, with tasks randomly assigned, so there isn't even a test set — theoretically, its performance cannot be judged. With internal testing absent, we can only indirectly employ external methods: customer surveys, or using my former company Netbase's services (social listening) to collect online customer feedback. All indications suggest customer satisfaction exceeds 80%; in various task attempts, customer needs are mostly met, with many feeling surprised and impressed. Another relatively objective external indicator is user stickiness and growth — and ChatGPT has set unprecedented records in this regard, reaching hundreds of millions of users in just a few months, not even counting the countless domestic friends who want to use it but are currently unsupported by OpenAI. ChatGPT's customer growth has exceeded everyone's imagination. The LLM tsunami triggered by ChatGPT shows no signs of abating.

In China, this LLM storm arrived one to two months late; after people emerged from the pandemic, ChatGPT began fermenting, and now it is a topic on everyone's lips. Months of information bombardment, continuous sharing sessions, roundtables, and lectures have shown no signs of fatigue — this is truly a phenomenon-level spectacle. Recently, Qi Lu came to Silicon Valley to give a speech, adding fuel to the fire. He compared the ChatGPT revolution to Web 1.0. As we all know, during the first internet explosion, the iconic product was the Netscape browser; though Netscape itself did not become huge, the internet revolution it triggered created giants like Yahoo, Google, and Amazon. Slightly later, the same revolution occurred in China, giving rise to world-class companies like Baidu, Tencent, Alibaba, and Toutiao. Qi Lu believes we are now in such an era. The roadmap is so clear, the trend so obvious, he said, that he has not a shred of doubt in his mind. His tone carried the confidence and certainty of a prophet. Qi Lu is also an extraordinary person — this man with the appearance of a typical Asian programmer radiates a certain presence when he speaks, that high-level, panoramic, future-foreseeing demeanor rarely seen among compatriots. Overall, I agree with his view on technology trends.

ChatGPT marks the emergence of a new era. Some call it the "iPhone moment" or "Android moment" in contemporary information technology history, destined to lead a brand-new ecosystem. I feel Qi Lu's comparison has more grand vision — ChatGPT is equivalent to the "Netscape browser" that triggered the first internet revolution. Whatever the comparison, it will be a world-changing presence.

03

ChatGPT's Shortcomings and Challenges

While ChatGPT marks the emergence of a new era, we must also note its existing shortcomings and challenges. What everyone senses is the so-called "一本正经胡说八道" (speaking nonsense with a straight face), fabricating details, mixing things up. This is because although it has mastered the form of any human language, it has merely pried open a tiny corner of the iceberg of cognitive intelligence. Is there hope of completely solving this problem? Following the LLM route, my personal view is that it is impossible.

As mentioned earlier, ChatGPT has currently mastered only about 20% of human knowledge; scaling LLMs up several more orders of magnitude might reach 40-50% at most. The remaining 50% is the deep-sea zone, beyond reach. Because the long tail of this knowledge represents an absolute combinatorial explosion — an explosion of this magnitude might not even be reachable at the edges of the universe, so how could it possibly be mastered through mere sequential language learning? The nature of LLMs is that they do not hide their ignorance; for any knowledge they cannot handle, they will unhesitatingly fabricate details, looking completely genuine — this is deadly. This precision defect in long-tail knowledge is an unavoidable硬伤 for AI ecosystem building and application services built on LLMs.

Beyond this, many other problems remain to be overcome. For example, when large models empower downstream applications in every domain scenario, how can customer data privacy and security be protected during large model invocation? This problem remains unsolved, but we can believe someone will find a good isolation method; for large model providers seeking to develop their ecosystems, this is a problem that must be given special attention and resolved.

Another issue is complex reasoning capability. From ChatGPT's dialogue, we can see it already possesses preliminary reasoning ability. The source of this ability is quite interesting: it benefits primarily from massive self-supervised learning of computer code sequences. As we know, the GPT-3.5 that ChatGPT relies on was not only trained on human natural language but also on program code in various computer languages from GitHub open-source repositories — and most of this code comes with natural language comments. Because computer language code is more logical than natural language, this helped train ChatGPT's coherence in expression. Reportedly, this was an unexpected bonus that even the developers did not anticipate — computer languages in LLM language skill training compensated for the deficiencies of natural language. Nevertheless, it still falls short in complex logical reasoning. Fortunately, complex reasoning capability differs from boundless knowledge networks; it is relatively a logically closed set, and there is hope of cracking it in the near future (perhaps the much-anticipated GPT-4 can more or less handle it?).

04

Progress in Multimodality

Finally, let me introduce progress in multimodality. The power of LLMs as foundation models was first validated and brilliantly demonstrated on NLP multi-tasks. After the NLP breakthrough, the framework of foundation models empowering downstream tasks began radiating toward other modalities — an extremely active direction in multimodal academia and industry, continuously bringing us surprises. Everything is still in progress; multimodality is currently one beat behind language LLMs, now at the prompt engineering stage, lacking precisely a natural language interface. Those playing with prompts ("incantations") in image and music generation large models already know the potential and effects of foundation models — exactly the same situation as when people played with GPT-3 playground before ChatGPT. It is foreseeable that in the near future, a smooth natural language interface will be built, allowing users to describe whatever artworks they desire, whether paintings or music, to their heart's content. Work aligning with human taste is also in progress; by then, the hit rate and satisfaction of natural-language-to-image (NL2img) for a ChatGPT-like "ChatDALL-E" graphic art large model will continuously improve. The same trend is destined to occur in natural-language-to-music (NL2music). We are in an exhilarating new era of AI-generated content (AIGC). Another foreseeable vision is that, from the trend of multimodal LLMs, there will ultimately emerge a Unified large model that unifies various modalities and their associated knowledge. The打通 and unified construction of multimodal large models will open broader paths for multimodal new-ecosystem applications.

As entrepreneurs, how can we use LLMs to empower downstream applications in different scenarios? Here there is a domain barrier. As we all know, whether finance, law, or medicine, each possesses a thick, long-accumulated knowledge system. Large models, as general-purpose sequence-learning models, learn whatever they are fed; the depth of domain knowledge they acquire is severely insufficient, while the domains themselves have already accumulated rich structured symbolic knowledge, typically manifested in domain ontology knowledge bases or databases. But large models' knowledge representation is vector-based; how to interface with domains' symbolic databases faces precisely this domain barrier. But my current view is that this barrier will certainly be broken within the next two to three years.