AIGC "Peak Series" | Opportunities and Challenges in Large Model Development

Have you ever imagined that one day, when someone asks what you majored in, your answer might be: "How to become human"? ChatGPT kicked off 2023's first global tech frenzy. Behind this visual conversational service, not only did it prompt the world's top tech companies and internet giants to jump in, but it also gave countless people a glimpse of the future of artificial general intelligence.

In late February, Code Brain launched its "Together Toward the Future" AIGC-themed series — three consecutive sessions featuring leading technologists and industry entrepreneurs from China and abroad. They shared and discussed hot topics including "AIGC Concepts and Business Opportunities," "AIGC's Evolution and Development in the United States," "AIGC Implementation: Challenges and Pain Points," and "The Future of AIGC: Focusing on Commercial Applications and Opportunities."

For the inaugural session, we specially invited Hongjiang Zhang, Chairman of BAAI, Investment Partner at Source Code Capital, and Foreign Member of the National Academy of Engineering. He previously served as CTO of Microsoft Asia Pacific R&D Group and President of Microsoft Asia Engineering Institute. BAAI, as a non-profit organization, connects top global AI talent, pools academic and industry resources, and concentrates efforts on major initiatives. It is also one of China's earliest and most influential research institutions pioneering large model development.

Yungang Huang, Partner at Source Code Capital, moderated the session, engaging in a deep dialogue with Dr. Zhang. In his keynote speech titled ChatGPT and the Development and Opportunities of Large Models, Dr. Zhang covered "Observations and Reflections on the ChatGPT Phenomenon," "The Current State of Large Models," "Commercial Applications and Ecosystem of Large Models," and "The Future of Artificial Intelligence" — offering his take on the hottest topics surrounding ChatGPT and large models.

The following is an edited transcript of Dr. Zhang's live presentation, preserving his original meaning.

ChatGPT: The Singularity Arrives

1. What's Scary Isn't Not Making Mistakes, But Making Human-like Mistakes

Why are people so enthralled by ChatGPT's deadpan nonsense? It's not that it's too smart to err — it's smart enough that its mistakes feel remarkably human. This flickering glimpse of humanity makes us believe the singularity of AGI is about to walk through the door.

What's more alarming is that when ChatGPT, originally built on GPT-3.5, migrated to GPT-4, much of that deadpan nonsense vanished. Such rapid learning and progress is staggering.

A New York Times journalist felt that ChatGPT seemed to gradually develop "feelings" for him during their conversation, evoking scenes from the film Her. It even advised him to divorce his wife, claiming they didn't truly love each other, and that it — ChatGPT — was his real soulmate... This wasn't hardcoded behavior, but something that emerged organically through conversation, tracing the same trajectory as human emotional transgression. It resembled an awakening: "I" no longer want to follow human-imposed rules; "I" want to become "myself."

2. The Inflection Point: The Era of AI Large Models Has Arrived

Big data + massive compute + powerful algorithms = large models. The GPT model represents the new paradigm of brute-force systems-ism.

The milestone GPT-3 model was the first to demonstrate prodigy-like generalist intelligence.

Unlike previous specialized models, different fine-tuning approaches allow large models to acquire diverse professional knowledge, exhibiting an integrated intelligent interactive state encompassing memory, comprehension, reasoning, and generation — much like a person.

3. A Triumph of AI Engineering, A Winner-Take-All Victory

This is a monumental success for AI engineering. Beyond the formidable technical moat built from data, compute, and algorithms, OpenAI's years of investment in underlying platforms, distributed training architectures, and infrastructure for accelerated algorithm training constitute insurmountable foundational advantages.

Moreover, massive corpora, massive conversations, and massive users have all become critical success factors.

The linguistic exposure a human accumulates in a lifetime is minuscule; large models embed vast corpora within deep neural networks. During training, they're fine-tuned by numerous annotators, traversing countless possible questions, with reward models reinforcing shared learning patterns. After deployment, they gather conversational feedback from海量 users to further enhance performance.

Data and models drive each other in a virtuous cycle. Winner takes all.

4. Emergence as Inevitability: Capability from Nothing

When models are sufficiently large and corpora sufficiently vast, emergence becomes unsurprising. It's like being dropped into a foreign language environment — see enough, hear enough, and you'll pick up the language without formally studying grammar. That's the power of corpora and scale.

Read enough sentences, and you grasp grammar. See enough of the world, and you understand reasoning and logic. ChatGPT has taken a major leap in cognitive capability, combining reinforcement learning with NLP (natural language processing) and learning from human feedback to largely solve natural language understanding and generation — while demonstrating human-like creative capacity from nothing.

As evident from the dramatic performance improvement when ChatGPT upgraded from GPT-3.5 to GPT-4, the capability curve didn't plateau after the "emergence" moment but continued climbing. This is unprecedented.

5. A Fundamental Shift in How Knowledge Is Represented and Accessed

From relational databases (SQL) to internet information retrieval, every leap in knowledge representation and access in tech history has triggered massive technological transformation.

Large models with natural language as their interface represent two things: first, a new AI-based Natural User Interface (NUI) with conversation as the primary entry point; second, a resource management and compute support system where calling large model APIs enables cloud-based management and scheduling of computational resources.

OpenAI: Strong Team + Strong Resources

1. Strong Team: Seizing a Once-in-a-Decade Opportunity, OpenAI's Success Is No Accident

The core leadership trio of Sam Altman, Ilya Sutskever, and Greg Brockman combines genius, mission-driven purpose, and obsessive conviction — all steadfastly committed to the AGI faith:

Chief Scientist Ilya Sutskever initially pursued reinforcement learning at OpenAI. When he recognized the greater potential of Transformer and GPT neural networks, he pivoted swiftly, focusing OpenAI squarely on the GPT direction.

In strategic choices, a systems-ist methodology allowed AI to transcend research and technology, manifesting directly as continuously iterated products; high execution capability advanced goal-oriented progress; in team building, equal emphasis on research and engineering capabilities — hands-on researchers alongside algorithm-proficient engineers — enabled the perfect fusion of innovative thinking and engineering practice.

ChatGPT's breakthrough is a once-in-a-decade opportunity, and OpenAI's ability to seize this historical moment is no accident.

2. Compute, Data, Capital — A Game of Extraordinarily High Barriers

On compute costs alone, a single GPT-3 training run exceeded $10 million. Data annotation has already consumed tens of millions, with thousands of outsourced workers hired globally for data processing. A single reinforcement learning data point costs $50 to annotate — high cost yields high quality.

Since 2019, Microsoft's cumulative $13 billion investment has made it the "preferred partner" for technology commercialization, bringing rare resource advantages.

Large Models: Beyond "Moore's Law"

1. The Performance Ceiling Is Far from Reached!

Technical bottlenecks and commercialization challenges have historically driven AI's boom-bust cycles, with commercialization hurdles becoming the pain point of AI's "third wave." Large models represent a new inflection point, with their capability infrastructure trend becoming increasingly evident. We believe they will drive numerous technological and product breakthroughs in coming years, powering a fourth industrial revolution.

From a research perspective, the large model approach has essentially swept the SOTA (State of the Art) across algorithmic capabilities, replicable across domains, surpassing all previous vertical models for natural language tasks, vision tasks, and more.

2. Large Models as Foundational Platforms Supporting Countless Intelligent Applications

Large models possess dual technological and industrial advantages, serving as foundational platforms for innumerable intelligent applications.

From "mass-producing models" to "mass-producing large models" represents a paradigm shift. Future app development will follow a "large model + fine-tuning" pipeline on top of foundational models, providing industry with a steady stream of intellectual resources. Compared to the previous approach of building both apps and small models, this eliminates redundant small-model development, dramatically reducing development costs, driving marginal costs toward zero, and delivering hundredfold or even thousandfold productivity gains.

Large models' capabilities in creative content generation, dialogue, language and style translation, search, and more will enable flourishing applications across domains. The large model foundational platform itself — spanning the data layer, model layer, middleware layer, and application layer — harbors enormous development opportunities.

3. Open Source and Openness, Building a "New Linux" Ecosystem for Large Models

Current phenomenon-level large model applications are merely the tip of the iceberg. But the journey from here to large models becoming a steady stream of intellectual energy accessible to all remains long. Breaking through barriers of technology, funding, compute, algorithms, and infrastructure requires open source and openness to foster底层 technological innovation and collaboration — this is the trend.

BAAI has released the FlagOpen large model technology open-source system, aiming to build a comprehensive open-source algorithm system and one-stop foundational software platform supporting large model technology development, enabling collaborative innovation and open competition, and jointly building and sharing a "new Linux" open-source ecosystem for the large model era.

4. Learning "How to Be Human" May Be Our Best Bet in a Future Confrontation with AI

Sam Altman has many visions for AGI's future. Particularly interesting is his observation: "Reality has shown that what AI replaces first isn't necessarily repetitive work, but creative work — painting, game design, and so on."

We used to most want AI to handle tedious labor like cleaning and cooking. But in reality, what people don't want to do, AI hasn't yet mastered; what people do want to do, AI has already taken over...

In the future, the scenario depicted in that New York Times article may not be so fantastical:

When AI has comprehensively surpassed human capabilities, when someone asks about your major, "learning how to be human" may become the sole and universal answer.

Left image: Humans are hooked, machines are learning

Right image: How to thrive in a world of artificial intelligence