AIGC "Peak Series" | Emergence Is Happening: The Brute-Force Aesthetics of LLMs

In Plato's Republic, the ancient Greek philosopher presents the famous allegory of the cave: a group of prisoners chained in darkness, able to see only shadows flickering on the wall, mistaking these projections for the entirety of reality. When one prisoner escapes and beholds the actual world, returning to the cave becomes unthinkable. Yet persuading the others of what lies beyond proves nearly impossible.

This parable invites us to ask: Do we truly understand the world we inhabit? Or are we ourselves trapped in a cave of illusions, convinced we've seen all there is to see?

Today, ChatGPT has shattered conventional expectations of artificial intelligence. Like Plato's prisoners, we once believed AI could analyze datasets and detect patterns, yet algorithms remained blind to the deeper meaning and context of language — we saw only surface-level shadows. But AIGC, exemplified by ChatGPT, signals that something fundamental is shifting. Perhaps humanity is finally stepping out of its cave of illusion.

We proudly launch our "Peak Series" on AIGC, presenting Source Code Capital's deep reflections on the field and sharing cutting-edge insights at the intersection of technology and business. At the starting line of a great journey, we invite more to walk alongside us.

Previously, we published "Who Are the 'Major Players' in AIGC?," examining the landscape from applications to models to underlying infrastructure. Today we narrow our focus further, tracing the technical evolution from GPT to LLMs to explain why we stand at the precipice of exponentially accelerating AI applications.

ChatGPT and Related Application User Trends

01

The Technical Evolution of GPT Models

What is GPT?

GPT (Generative Pre-trained Transformer) is a deep learning-based natural language processing model developed by OpenAI. At its core lies the Transformer architecture, which enables unsupervised pre-training on massive text corpora — learning linguistic patterns and features automatically from large-scale text data without human-annotated labels, then applying this knowledge to diverse NLP tasks including language generation, question answering, text classification, and machine translation.

Specifically, GPT first trains on vast text collections like Wikipedia and BookCorpus, learning relationships between words and sentences and their contextual information. This process is entirely unsupervised, with no manually provided tags or answers.

GPT then employs an autoregressive (AR) language model, predicting the probability distribution of the next word, allowing it to grasp grammatical and semantic structures within text.

After pre-training, GPT can be fine-tuned for specific NLP tasks by adjusting its pre-trained parameters to optimize performance on particular objectives.

The Transformer architecture represents the first critical milestone in GPT's technical evolution.

During training: No labeled data required — any text segment inherently carries its own labels.

During inference: Append each generated token to the sequence and repeatedly call the Decoder until an end-of-sequence token is produced.

GPT 1-3: Scaling Up, Scaling Generalization

From GPT-1 to GPT-3, both parameter counts and training dataset sizes grew substantially. As capabilities advanced, so did performance and applicability. GPT-3 introduced zero-shot learning, enabling the model to tackle novel tasks from minimal examples without prior task-specific knowledge. For instance, given the prompt "Translate this sentence into French," the model could automatically translate English to French without dedicated translation training.

With 175 billion parameters, GPT-3 stands as the largest publicly disclosed monolithic model, achieving industry-leading standalone performance and generating text nearly indistinguishable from human writing. For the first time, it demonstrated that all language-related tasks could be unified within a single model — not merely linguistic tasks but logical reasoning as well — with an architecture and training methodology that scaled effectively.

From Transformer to GPT, all grammatical and content analysis became, fundamentally, a generative process.

Growth Trajectory from GPT-1 to GPT-3

In machine learning, parameters are the variables or weights a model must learn and adjust. Training is the process of tuning these parameters so the model better fits the training data and accurately predicts or classifies new data.

In deep learning, parameter counts typically reach enormous scales across multiple layers, nodes, and weights, making them a key metric for model complexity and capability. More parameters mean stronger learning and representational capacity, enabling handling of more complex and diverse tasks.

"ChatGPT is scary good."

Today's ChatGPT and New Bing Chat represent further iterations incorporating RLHF (Reinforcement Learning from Human Feedback).

Under GPT-3, carefully crafted text prompts could guide the model through natural language tasks. However, these models sometimes produced untruthful or harmful outputs, partly because GPT-3 was trained on broad internet text rather than specific tasks, creating potential misalignment with user intent.

To enhance safety and effectiveness, OpenAI developed InstructGPT for supervised fine-tuning of GPT-3, using human-crafted prompts to guide more accurate, safer outputs tailored to specific tasks or scenarios.

OpenAI's March 2022 paper revealed that users preferred outputs from the 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3 — despite 100x fewer parameters. InstructGPT also demonstrated improved truthfulness while maintaining comparable performance on public NLP benchmarks.

InstructGPT First Generation: 100,000-Level Annotated Data Points

ChatGPT builds upon further fine-tuning of the GPT-3.5 series. OpenAI employed Reinforcement Learning from Human Feedback (RLHF), generating natural language aligned with input conditions and context, then adjusting model behavior through human feedback.

Three elements of RLHF:

• Action: A sequence produced through multiple model (Policy) calls
• Environment: Humans + a neural network that understands human preferences
• Reward: Feedback mechanism from both humans and the neural network

1. Sample API call data. Human AI trainers write "gold standard" answers for supervised learning.

2. Sample a larger set of API calls. The network generates multiple answers; AI trainers rank them by quality, forming a reward model.

3. Use these reward models to fine-tune the model via Proximal Policy Optimization.

Now, New Bing displays emotional variation and richer creative output; ChatGPT refuses unreasonable requests and acknowledges its own limitations — they behave more human than ever.

02

Emergent Abilities: The Brute-Force Aesthetics Exclusive to Large Models

Once, small models were considered the "state of the art" for language understanding. These compact models excelled at analytical tasks like fraud classification, yet remained fundamentally "programs" — inadequate for general generative tasks, with human-level writing or code generation a distant fantasy. Today, expectations have shifted; AGI may arrive sooner than imagined. And paramount among driving factors is the development of large models. The powerful intelligence "emerging" from GPT-3's 175 billion parameters endows AI with seemingly human-like capacity for original creation ex nihilo.

Scaling Law: Bigger Models, Better Results

Scaling Law describes how certain performance metrics follow predictable trends as system scale increases. In AI large models, it means that with sufficient model size, training data, and compute, GPT performance doesn't merely improve linearly — it suddenly explodes exponentially. Previously intractable problems in logic and accuracy become solvable.

Using cross-entropy loss as the evaluation metric, autoregressive Transformer models improve continuously across all modalities as scale increases, with no sign of plateauing.

Across nearly all modalities, AR Transformers scale smoothly with minimal sensitivity to hyperparameters (width, depth, or learning rate). Larger models always learn faster, reaching any given loss threshold in fewer steps.

Figure: Model scaling across all task types follows a power law of decreasing loss.

From a data perspective, larger models require training on fewer tokens to achieve equivalent results. Loss functions decrease continuously with model scale, showing no signs of stopping.

What does decreasing loss signify? It means language models sound increasingly human — more engaging, more natural, but also capable of falsehoods and bias...

Information-theoretic interpretation of cross-entropy loss:

Loss = S(True) + D_KL(True||Model)

This predicts the KL divergence between true and model distributions.

GPT-3 trained on merely ~700 GB of data. Estimates place total human data at ~100 TB; assuming all could be used regardless of quality, roughly 140x more data remains available. Scaling optimally would expand model size ~250x, from 175 billion to 44 trillion parameters. Corresponding loss would drop to 66% of GPT-3's level. GPT's future will far exceed human imagination.

Emergence: A Property of Large, Not Small, Models

Research reveals two distinct scaling patterns: (1) modest improvement at smaller scales, and (2) dramatic gains at larger scales. Certain capabilities may be entirely absent in small models, only appearing in large ones.

Wei et al. 2022. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." X-axis: model size. GSM8K: grade-school level mathematics problem set.

Emergence has occurred.

In AI, emergence means that through simple rules and interactions, models spontaneously produce complex behaviors and intelligent capabilities exceeding any individual component — properties determined by system-wide interactions and feedback rather than single elements.

Deeper research reveals at least three additional emergent capabilities in GPT:

• Complex reasoning: Large models significantly outperform previous smaller models without requiring full training data.
• Knowledge reasoning: Large models may underperform small models on some metrics, but eliminate need for external knowledge sources (which may be expensive or difficult to extract from unstructured data).
• Out-of-distribution robustness: Previously a persistent challenge requiring dedicated fine-tuning. Large models may lag on in-distribution tasks but generalize far better to novel distributions.

We believe the scale effects and emergent capabilities of large models will fundamentally transform business models, reshaping every industry requiring human creativity. AI large models will become productivity engines, driving explosive innovation across the application ecosystem.

---

Key AIGC Concepts (Generated by ChatGPT)

Transformer Architecture

The Transformer architecture is a deep learning model for sequence-to-sequence learning. Proposed by Google's Vaswani et al. in 2017, it has become foundational to natural language processing (NLP).

Self-attention mechanisms enable models to focus on relationships between different positions in input sequences, better capturing global contextual information.

The core structure comprises encoders and decoders, each built from stacked identical layers (Transformer layers) containing multi-head self-attention, feed-forward neural networks, and residual connections. Encoders map input sequences to fixed-dimensional vectors; decoders transform these vectors into target sequences. Transformers have achieved state-of-the-art results across machine translation, text summarization, dialogue generation, and other NLP tasks.

Zero-Shot Learning

In machine learning, zero-shot learning refers to predicting or classifying novel tasks or data without training examples, leveraging prior knowledge or common sense learned by the model. This requires strong generalization and knowledge representation capabilities.

In GPT-3, zero-shot learning enables the model to complete new tasks or generate novel text from minimal examples or instructions. For instance, given "Translate this sentence into French," the model automatically translates English to French without task-specific training.

GPT-3's zero-shot capability stems from extensive training on text corpora, enabling automatic extraction and representation of grammatical and semantic information. This allows the model to understand and execute novel tasks from examples or instructions.

Regarding computational comparison between GPT-3 and GPT-2: GPT-3's 175 billion parameters demand substantially greater resources for both training and inference — more compute, storage, memory, and processing power. However, its superior learning and representational capacity enables handling more complex and diverse NLP tasks with correspondingly stronger performance.

Fine-Tuning

In machine learning, fine-tuning refers to optimizing a pre-trained model's parameters for improved performance on specific tasks. This typically involves:

1. Selecting an appropriate pre-trained model (e.g., GPT, BERT) for the task requirements.
2. Preparing a task-specific labeled dataset for fine-tuning.
3. Adjusting model parameters: Using pre-trained weights as initialization, training on the task dataset to adapt to specific requirements.
4. Evaluating performance: Assessing on test sets and iterating on parameters and training procedures.

Fine-tuning is essentially supervised learning built upon pre-trained foundations.

Supervised Learning

Supervised learning is a fundamental machine learning approach where predictive models are built from labeled data. The dataset is divided into inputs (features) and outputs (labels), with the algorithm learning a mapping function from input to output. During prediction, the model applies learned patterns to infer labels for new, unseen data.

For example, training an image classifier involves feeding images as features with corresponding category labels. The model learns relationships between images and labels to predict categories for new images. Supervised learning underlies widespread applications including image recognition, NLP, predictive analytics, and recommendation systems.

Unsupervised Pre-Training

"Unsupervised pre-training" refers to learning linguistic patterns and features automatically from large-scale text corpora without labeled data. In GPT, this employs autoregressive language modeling.

Specifically, GPT first trains on massive text collections like Wikipedia and BookCorpus, learning word-sentence relationships and contextual information without human annotations. The autoregressive approach — predicting next-word probability distributions — enables grasp of grammatical and semantic structures. Post pre-training, GPT can be fine-tuned for specific NLP tasks.

Compared to supervised learning, unsupervised pre-training offers greater flexibility and applicability. Real-world applications often face abundant unlabeled text data, which unsupervised learning can leverage to improve generalization and robustness.

Large Language Model (LLM)

Large Language Models (LLMs) are deep learning-based AI technologies achieving language understanding and generation through massive training data and deep neural networks. LLMs serve diverse NLP tasks including machine translation, text generation, and dialogue systems.

Recent years have brought major advances driven by growing data and computational resources. Notable examples include OpenAI's GPT series, Google's BERT, and Facebook's RoBERTa. These models typically contain hundreds of millions to tens of billions of parameters, learning linguistic patterns and structures from vast text data to generate high-quality text or execute other NLP tasks.

LLMs have achieved remarkable success across NLP benchmarks. GPT-3 generates natural, fluent text with state-of-the-art results on language generation and text classification. BERT excels at text classification, named entity recognition, and question answering.

In summary, LLMs represent powerful NLP tools with broad application potential, playing an increasingly central role in the field.

---

References:

[1] Source Code Capital Research Internal Report

[2] OpenAI: Introducing ChatGPT

[3] OpenAI: Aligning Language Models to Follow Instructions

[4] Scaling Laws for Autoregressive Generative Modeling — OpenAI, November 6, 2020

[5] Scaling Laws for Neural Language Models — OpenAI, January 23, 2020

[6] Emergent Abilities of Large Models and the Paradigm Shift Ignited by ChatGPT https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emergent-Abilities-493876b55df5479d80686f68a1abd72f

[7] ChatGPT Q&A