The Parametric Reflection of the World: Why Next Token Prediction Produces Intelligence | 5Y View

5Y Capital Partner Kai Liu

For outsiders, the best way to understand disruptive technological innovation is through metaphor and analogy — essentially a dimensionality-reduction method for compressing information. For practitioners on the front lines, there's only one best approach: find the most credible information sources, digest them, and deeply understand them. You could also think of this as lossless compression.

In the LLM field, where iterations happen on a weekly basis, we face an information explosion every single day. We need the best practitioners to filter and curate first-hand information for us. The article I'm sharing today is, without question, the best collection of paper commentaries I've read in recent memory. The author has deeply studied numerous key papers and summarized each important one. He didn't stop at simply removing the chaff from individual pieces — he went further, connecting these essential articles together to form higher-dimensional thinking and hypotheses. That's incredibly rare and valuable. Hope you all enjoy.

Reposted from Zhang Junlin's original article on Zhihu Author: Zhang Junlin

---

"Two English-speaking castaways are stranded on adjacent islands, separated by dangerous waters. Fortunately, they discover telegraph machines left by previous inhabitants, connected by an underwater cable that allows them to send messages to each other. What they don't know is that in the nearby waters lives a superintelligent octopus, which has hijacked the underwater cable and is intercepting the messages they send to each other. Although the octopus doesn't understand English, its superintelligence enables it to detect statistical patterns in the telegraph messages and accurately represent the statistical relationships between various telegraph signals. Once the octopus feels it has learned these statistical patterns well enough, it cuts the underwater cable and positions its two long tentacles at the two ends of the cable. Based on the statistical patterns it has identified, it receives and responds to the telegraph signals of the two castaways itself. Whether or not the two survivors notice that their communication partner has changed, the information sent by the octopus seems to have no intrinsic meaning whatsoever. After all, the octopus is merely following the statistical patterns it learned from the previous human-to-human communication, without ever having seen any human interpretation of the signals — the real meaning of 'coconut' or 'seawater,' for instance. Moreover, the octopus might not even understand that these signals are meaningful or serve a communicative function." — "The Octopus Test" — Bender & Koller

If we replaced the octopus in "The Octopus Test" with ChatGPT or GPT-4, what would you think? In other words, which of the following two viewpoints do you support? One view, similar to that of "The Octopus Test," holds that LLM models like GPT-4 have merely learned shallow, surface-level statistical relationships such as word co-occurrence in language, and don't actually possess intelligence — they're essentially stitched-together fragments of language, parroting like a parrot. The other view argues that GPT-4 has not only learned the surface statistical relationships between linguistic elements, but has also grasped the intrinsic operating laws of human language and even the physical world — that text is produced by underlying intelligence, and therefore LLMs possess human-like intelligence.

These two perspectives are diametrically opposed, and I'm not sure which camp you fall into. Currently, in both academia and society at large, there are substantial numbers on both sides, and the debates between them are quite fierce. On the anti-LLM-intelligence side, prominent figures include AI heavyweight Yann LeCun and linguist Noam Chomsky, both of whom deny that large language models trained through Next Token Prediction can possess intelligence. The pro side also has many representatives — OpenAI, needless to say, is unquestionably the most influential proponent. Based on public statements to date, Geoffrey Hinton clearly holds the pro position as well, and quite strongly at that. He not only believes GPT-4 possesses human-like intelligence, but also thinks that human carbon-based intelligence will likely serve as a bootstrap program (booster) for silicon-based LLM intelligence in the future. On this point, Hinton and Elon Musk (full name: Elon · King of Electric Vehicles · Rocket Pioneer · Twitter Rebuilder · Environmental Vanguard · Mars Colonizer · OpenAI Renouncer · Musk) share similar views.

Currently, LLMs of sufficient scale all employ "Next Token Prediction, NTP" (sometimes abbreviated as NTP for brevity in what follows) as their training task for base models. Next Token Prediction is such a simple operation — using preceding words in language to generate the next word. Clearly, wouldn't this only learn surface-level statistical relationships between words? For those holding the pro position, this question is actually difficult to refute, because at first glance it certainly appears to be the case. I believe the vast majority of pro-side advocates would struggle to provide a well-reasoned explanation that could convince those on the other side.

As for myself, if you've read my early-year article "Zhang Junlin: The Road to AGI: Essentials of Large Language Model (LLM) Technology," it's easy to see that I hold the pro position. In fact, in the original version of that article, there was a section discussing why NTP produces intelligence. Based on my understanding of LLMs as of January 2023, I summarized NTP's production of intelligence as follows: "Through the NTP task, the LLM learns an implicit knowledge graph in its model parameters. When a prompt is input, the concepts contained in the prompt activate relevant nodes in the knowledge graph, which then triggers activation spreading and information transfer between pieces of knowledge on the knowledge graph according to <activation - diffusion> theory, thereby causing the LLM to produce intelligence." That's how I understood the answer to this question in that earlier version. Looking back at this view now, I can't say it's wrong, but it's clearly somewhat shallow or rough around the edges. At the time, because that article was already too long, and because the evidence supporting the above view was insufficient, I deleted that section when publishing.

This article is dedicated to exploring this topic. I attempt to organize and summarize some currently available fragmentary evidence to provide a relatively well-grounded answer to the above question. In fact, the pro side hasn't produced dedicated research to explain this problem. However, if we connect together various known research conclusions that were originally used to study other problems — that is, if we treat finding the answer to this question as a jigsaw puzzle — I believe the pro side can roughly provide some at least seemingly reasonable explanations, based on known research puzzle pieces plus some reasonable inferences and hypotheses. Structurally, this article will first introduce OpenAI's view on this problem in some detail — a fresh angle for most people — then collect and summarize existing research conclusions, and finally present what I consider a reasonably plausible explanation.

The Two Ends of the Scale: Compression as Intelligence

Imagine a scale, with the left pan measuring a large language model's data compression capability, and the right pan measuring its intelligence level. The question is: Is this scale's measurement reliable? In other words, if a large language model has stronger data compression capability, does that mean it possesses stronger AGI intelligence?

OpenAI certainly believes in an equivalence between the two. Currently, this appears to be a core philosophy driving OpenAI's large model development direction. OpenAI Chief Scientist Ilya Sutskever hinted at this thinking in some early-year public interviews. Subsequently, Jack Rae from OpenAI's large model team gave a presentation at the Stanford MLSys Symposium on the theme of "Compression for AGI," which conceptually论证了这一理念 at the theoretical level.

This section primarily follows Jack Rae's presentation, recounting OpenAI's firmly held "compression as intelligence" argument. We'll start with a hypothetical experiment on data compression and transmission.

Using LLMs for Data Compression

Let's assume Xiao Shuai and Xiao Mei live on Earth and Mars respectively. Xiao Shuai has obtained a batch of confidential data that he needs to transmit to Xiao Mei on Mars with minimum transmission cost. Xiao Shuai plans to compress the data using an LLM model like GPT, then send the compressed data to Xiao Mei to reduce the amount of data transmitted. At the same time, he wants the compression to be lossless — that is, Xiao Mei must be able to fully recover the original data from the compressed data she receives, without any discrepancy. This seems rather difficult to achieve. What can be done?

First, Xiao Shuai sends Xiao Mei the GPT model code, including the code itself, initialization methods, and random seed information. Based on the GPT model information sent by Xiao Shuai, Xiao Mei uses the same code, initialization methods, and random seed to replicate and initialize her own GPT, ensuring that her GPT model matches Xiao Shuai's in its initial state.

Next, Xiao Shuai launches the GPT training process using Next Token Prediction as the task and as the training data. The training process itself is essentially a data compression process. Let's assume Xiao Shuai has already compressed the data through GPT, with the corresponding compressed data being , and has been sending this batch of compressed data to Xiao Mei in chunks. Now he's preparing to transmit the data . Let's hit the "slow motion" button here and take a close look at how GPT compresses, encodes, and decodes the data .

Encoding Stage:

Our goal is to use GPT to compress the data . Xiao Shuai feeds into GPT as input, and uses the current version of the GPT model to perform a Next Token prediction. Assuming the token vocabulary is , the GPT model produces a generation probability for every word in the vocabulary through Next Token prediction. Some words in have higher generation probabilities, others lower, and all probabilities sum to 1, forming a probability distribution over . Let the original data be . At this point, we can use a data compression algorithm — for example, Arithmetic Coding (AC) — to compress into data based on and (how Arithmetic Coding works will be explained shortly), that is, . Thus, Xiao Shuai can send the resulting compressed encoding to Xiao Mei.

Additionally, if GPT's Next Token prediction based on the preceding context yields a highest-probability word that isn't the correct answer , that means the model hasn't been trained well enough. So Xiao Shuai has GPT perform one backpropagation pass to correct the model's parameters, hoping that in the future, GPT will make more accurate predictions when encountering similar contexts. After backpropagation, the model parameters change, and the GPT model is updated from version to version .

As you can see, the process described above is essentially a standard GPT training step for a particular token — it's just that during normal GPT training, we don't use the distribution probabilities obtained from Next Token Prediction together with arithmetic coding to get a compressed encoding for , and record it. If you wanted to, you could absolutely generate and record the corresponding for each step by step during training. Doing so would give you a losslessly compressed version of the data .

Decoding Phase:

After receiving the compressed encoding sent by Xiao Shuai, Xiao Mei wants to use her own GPT model to recover the original data . She could also use arithmetic coding to reverse-decode , but to decode , the information is insufficient. Besides itself, she also needs to know the probability distribution of words in the vocabulary corresponding to . But Xiao Shuai didn't send over, because that would be too much information and not worth transmitting. What to do?

As you can see, the decoding process is essentially Xiaomei running a synchronous GPT training step, using the probability distribution of vocabulary words obtained from Next Token Prediction to help decode the compressed data back into the original data .

In this way, Xiaoshuai and Xiaomei complete the compression and decompression of data by synchronously running the GPT model training process on . By repeating this process, Xiaoshuai can transmit all data in to Xiaomei losslessly, achieving lossless compression and decompression of data through LLM. So we can say that the GPT model training process is essentially a lossless compression process for training data — it's just that we normally skip this step during ordinary training.

Arithmetic Coding Mechanism

The arithmetic coding mechanism wasn't explained above, so I'll briefly illustrate it with a simple example here. As shown in the figure above, suppose the vocabulary contains four words, and the raw data we want to compress and encode is . After GPT runs Next Token Prediction, the probability distribution for words in the vocabulary is listed on the left side of the figure. That is, at this moment, GPT's predicted Next Token has the highest generation probability for the word "too," rather than the ground truth "MaskNet."

Now, given and its corresponding , we use arithmetic coding to compress the data. First, we can divide the interval from 0 to 1 according to each word's probability score in the vocabulary — the higher a word's generation probability, the longer the interval it occupies. This gives us the lower and upper bounds of each word's coverage interval. For example, for the word "MaskNet" to be encoded, its lower bound is 0.4, and since its own generation probability is 0.2, its upper bound is 0.6. To make the binary encoding as short as possible, arithmetic coding looks for the decimal fraction with the shortest binary representation within the coverage interval of "MaskNet" (0.4 to 0.6). Clearly, in this interval, the decimal number 0.5 has the shortest binary representation, so we choose 0.5 as the encoding number. After base conversion, we get the binary 0.1, which is the binary arithmetic code corresponding to the word "MaskNet." Xiaoshuai only needs to send Xiaomei the binary digit 1 after the decimal point.

Next, let's walk through Xiaomei's decoding process after receiving the binary digit 1. As described earlier, Xiaomei uses her own GPT model to generate the same probability distribution over words:

Following the principles of arithmetic coding, she partitions the interval from 0 to 1 using this probability distribution, producing exactly the same segmentation diagram that Xiaoshuai obtained. Xiaomei then converts the binary 0.1 to its decimal equivalent, 0.5, and checks which word's upper and lower bounds contain this value in the segmentation diagram. This points her to the word "MaskNet," and she successfully decodes the word that 0.1 represents:

The elegance of arithmetic coding lies in its dynamic encoding of input sequences — it can encode an entire input in binary using fractional numbers, with coding efficiency approaching the entropy limit proposed by Shannon. However, in the scenario we've described, because each $x_i$'s corresponding $P(x_i)$ constantly shifts during GPT training, a given distribution

only needs to compress or decode a single token. The concept seems straightforward in this case. For arithmetic encoding of longer input sequences, you can refer to: "What is Arithmetic Coding."

Compression Is Intelligence

From the above explanation, we can see that if a GPT model assigns higher generation probability to the ground truth

then that outcome occupies a longer segment in the arithmetic coding interval, making it easier to find a shorter arithmetic code — which means higher model compression rate. In other words, the more intelligent the GPT model, the more accurate its NTP predictions, and the higher its compression efficiency. Therefore, we can evaluate a model's intelligence by its compression efficiency: higher compression efficiency implies greater intelligence. This is a core philosophy that OpenAI currently follows in advancing large model development.

Consider two extreme cases. In the first, the model possesses superhuman intelligence: for every ground truth

in Next Token Prediction, the generation probability is always 1. Suppose that after Xiaoshuai transmits some data

to Xiaomei, the model's intelligence accumulates to this level. This means that for the remaining unsent data

Xiaoshuai doesn't need to transmit anything further. Xiaomei's GPT can now perfectly predict every subsequent token on its own. At this point, the GPT model achieves ultimate data compression capability through superhuman intelligence — given any input context, it knows exactly what comes next.

The other extreme: a GPT that learns no intelligence whatsoever during training, making Next Token Prediction by pure guessing. Assume the vocabulary

has size

Then every ground truth

has generation probability

permanently. Here the GPT has zero data compression capability — the amount of data that must be transmitted equals the original information content

These are the two extremes. In most cases, an LLM's intelligence — or compression capability — falls somewhere between them, and we can evaluate the model's intelligence based on its compression ability. With some mathematical derivation, we can show that in this scenario, for data

the corresponding

the number of bits required for arithmetic coding — that is, the code length — is:

Take a moment to look at this formula. Does it remind you of anything? It is, in fact, the cross-entropy loss for this token

during GPT training. In other words, from the perspective of data compression, each token's encoding length is equivalent to its cross-entropy loss during LLM pre-training. The two are identical. Quite interesting, isn't it? So lossless data compression offers a refreshingly novel lens through which to view LLM training.

We can take this a step further. For a dataset D, after LLM-based compression and transmission, what's the total amount of data that Xiaoshuai sends to Xiaomei? The formula is in the figure above. As shown, the total transmitted information consists of two parts: one is the LLM model code description, including the code itself, initialization methods, and random seed information; the other part — if we expand the summation formula — each xᵢ corresponds to a compressed encoding bit count equal to that token's cross-entropy loss. So this part is simply the sum of losses for all tokens when GPT pre-trains on dataset D. Add both parts together and you get the total data transferred.

So, do different LLM models have different data compression capabilities? The answer is obviously yes. The figure above shows the compression capabilities of different-sized LLaMA models (from the smallest 7B to the largest 65B): for the same total training data (say, the 1000B tokens point on the x-axis), the total area covered by each model's loss curve represents that model's data compression capability — the smaller the area under the loss curve, the stronger the model's compression ability. With the groundwork we've laid, I trust this is intuitive. We can make an extreme assumption: if during training each batch contained only one token, the required encoding bits would equal that token's loss value. Integrating over the model's loss area gives us the total loss across all tokens, which is equivalent to the total compressed encoding length needed to compress those tokens.

As shown in the figure, larger LLaMA models have smaller loss areas, meaning higher compression rates and stronger compression capabilities — which in turn indicates higher intelligence. Roughly estimated, current LLaMA models achieve about 14x data compression, surpassing the best result in the dedicated data compression competition Hutter Prize, which currently stands at 8.7x. What does this tell us? If we assume that mainstream text compression algorithms rely primarily on surface-level factors like word frequency and repeating patterns, then this additional compression likely represents LLMs' deep understanding of text — an intelligent encoding born from AGI.

Further Reflections

The above is the "compression is intelligence" argument from Jack Rae's presentation. When I watched the talk around March, I found it deeply inspiring — OpenAI had opened up a fascinating line of thinking. I had genuinely never considered LLMs from a data compression perspective before, and I believe this is a novel angle for the vast majority of people. However, after reviewing related literature, I discovered this "compression is intelligence" line of thinking wasn't pioneered by OpenAI — it actually has a longer history. For instance, the Hutter Prize mentioned above, established in 2006 to encourage better data compression algorithms, was founded by Marcus Hutter precisely because he believed data compression capability and AI intelligence are equivalent problems. That was his original motivation for funding the prize. And using AI models for data compression has already become a small research direction, with numerous related papers.

We can dig deeper into two related questions. The first: The above discussion views LLM intelligence through the lens of data compression — but why does stronger compression capability indicate higher intelligence?

The Minimum Description Length (MDL) principle can explain this. It's an important concept in machine learning and the formalized expression of Occam's Razor ("entities should not be multiplied without necessity"). MDL's core idea: given many models that can explain our data, the best explanation is the one that describes the data as accurately as possible with the shortest description. The shorter the model description, the better its generalization — that is, the more intelligent it is. Why is shorter more intelligent? Because this short description abstracts the underlying patterns from the data. Compared to massive amounts of data, descriptions of underlying patterns are naturally much shorter. And if a model can produce shorter descriptions, it has learned more underlying patterns — hence it's smarter. That's the logic. Let's use an example. Suppose the sequence to transmit is ten thousand consecutive prime numbers:

There's no pattern to the gaps between prime numbers, so Xiaoshuai would have to honestly encode and transmit all ten thousand digits. But actually, Xiaoshuai could describe these numbers with a single sentence like "output 10,000 consecutive primes starting from 2," compress this sentence and send it to Xiaomei, and if Xiaomei's LLM is smart enough, it could reconstruct the 10,000-prime sequence from this. By now, I trust you understand what MDL means.

Of course, this only works if the LLM understands the abstract concept of prime numbers. So, can large models really grasp such abstract concepts? And is it true that only large models can understand abstractions like "prime numbers" while small models cannot? I ran a test comparing large and small model capabilities. To prevent models from simply memorizing prime sequences seen during training, I transformed the description to ensure the phrasing wouldn't appear in large language model training data. The test prompts and outputs from both large and small models are shown below.

As shown, GPT-3.5 has learned the abstract concept of prime numbers — otherwise this question would be nearly impossible to answer well. Without understanding the concept, you get the incoherent response from the small model on the right. This demonstrates both that large models can indeed learn abstract concepts, and that large models outperform small models in this regard.

Another point: Jack Rae emphasized in his presentation that LLMs' data compression capability is lossless, directly contradicting the influential argument by renowned sci-fi writer Ted Chiang earlier this year that "ChatGPT is a lossy compression of internet data." But if you think carefully, the claim that LLMs perform "lossless compression" is somewhat questionable. If we examine this more rigorously, although LLM training can be viewed as lossless data compression, achieving "lossless" results doesn't rely on the LLM alone — it's accomplished by "LLM + arithmetic coding" together.

If an LLM learns to become sufficiently intelligent, guaranteeing that NTP (next token prediction) loss for subsequent text sequences reaches zero — meaning it can perfectly predict the next token given the context — then arithmetic coding becomes unnecessary, and the LLM alone could fully compress and decompress the data. In this scenario, saying that LLM training, or a trained LLM, achieves "lossless compression" would be perfectly valid. But this is the ideal case. Can current LLMs achieve this? Definitely not. So the LLM's next token predictions will inevitably contain errors, and these mispredicted tokens represent information loss in the LLM's compression. This loss is compensated by arithmetic coding's additional encoding, which is what achieves the "lossless compression" effect. So a more precise formulation would be:

Lossless data compression = LLM's lossy data compression capability + arithmetic coding's compensation capability

In other words, current LLMs are still lossy in their data encoding and cannot achieve lossless compression through their own capabilities alone. Whether future LLMs could become powerful enough to achieve lossless compression independently remains unknown.

Data compression is merely the means; the goal is for GPT to acquire intelligence through this means. The question now is: OpenAI has identified the means and the end from a theoretical foundation, but hasn't addressed a more fundamental question — what kind of AGI intelligence does GPT acquire through next token prediction via data compression? The remainder of this article attempts to answer this.

The Jigsaw Puzzle: Known Fragments

If we compare LLMs acquiring AGI intelligence to a jigsaw puzzle, at present we only hold scattered fragments — we haven't yet glimpsed the full picture of this machine intelligence. This section gathers and introduces research findings from several different angles.

How GPT Models Extract Knowledge

Let's first examine what happens after an LLM is trained and receives a prompt: how does GPT extract knowledge? The paper "Dissecting Recall of Factual Associations in Auto-Regressive Language Models" studied this in detail. As shown in the figure, given the prompt "Beat music is owned by," GPT can return the correct answer through NTP: Apple. In this example, "Beat music" is an entity, and "owned by Apple" is an attribute of that entity.

The research found that GPT goes through a clear three-stage process to extract this knowledge. First, the word "music" — the final and most critical descriptor of this entity — has its information flow upward through Transformer blocks, where Attention first integrates the modifier "beats" into the position corresponding to "music." Then, as Transformer layers grow higher, each block's FFN layer continually adds information to the embedding at the "music" position. So as information flows upward, the layer-by-layer embedding for "music" can trigger more and more attribute words related to "Beat music." This first step occurs primarily in the lower layers of the Transformer.

Second, at the position of the word "by" — the last position where NTP will generate the output token — GPT integrates the information from "own" to this final position through Attention. Note that this last word's Transformer position is quite critical, because its top layer will produce the Next Token output. During inference, GPT gradually integrates important information from the input context to this position through Attention. This operation also occurs in the lower layers of the Transformer.

Third, at the "by" position — the last position — in the higher layers of the Transformer, the information from "own" already integrated in the lower layers extracts the attribute "apple" corresponding to "Beat music" through Attention. The specific extraction is accomplished by a certain Attention Head, and this paper proved that Attention Heads encode information. For concrete examples see the figure below — this should be new knowledge for us (Attention was previously generally understood to mainly perform information comparison and搬运; this proves it also stores certain knowledge).

Through these three steps, GPT completes the extraction of a piece of knowledge.

Another work, "Understanding Transformer Memorization Recall Through Idioms," explored how LLMs extract memorized information, including idioms/proverbs that require precise reproduction from memory, as well as factual knowledge. The research concluded that LLM extraction of memorized information divides into two stages: in the first stage, lower-layer Transformer Blocks gradually raise the ranking of the correct answer word until it reaches first place at the middle layers; in the second stage, higher-layer Transformers increase the confidence in the correct answer, i.e., continually raising the probability distribution score for this correct answer.

Beyond these two works, there are other similar studies on knowledge extraction. Synthesizing existing research conclusions, I think we can roughly outline GPT knowledge extraction as follows: when a trained GPT model receives a prompt, for a given input word at some Transformer position, as the Transformer progresses upward, GPT integrates information from the context relevant to itself into its own embedding through Attention, while each layer's FFN transforms and adds information to the current word's embedding — thereby continually triggering knowledge stored in the FFN and refining the word's embedding layer by layer (similar to the "music" example above).

The Transformer's last token position works similarly, with a special characteristic: from bottom to top, it first copies the most critical information from the entire input context to its own position through Attention, then gradually filters out relatively important information from the context through this key information. At the lower layers of this position, there should be many candidate answers available for output, with the correct answer not ranking highly. As the Transformer progresses upward, the correct answer's ranking improves, and competing candidate answers diminish — manifested as increasingly higher probability distribution scores assigned to the correct answer, until at the highest layer of the last token, GPT can output the correct answer (similar to the "by" example above).

Distribution of Knowledge Points in the Transformer

This section covers how knowledge points are distributed across Transformer structure — meaning how different types or specific knowledge points are distributed across high, middle, and low layers of the Transformer. Understanding this is very helpful for grasping GPT's internal operating mechanisms.

Before introducing research conclusions, to facilitate understanding, we first explain three basic concepts (referencing Toy Models of Superposition): monosemantic neurons, polysemantic neurons, and superposition.

Current research has found many individual neurons in LLMs that each respond only to some specific knowledge point in the input — activated only by particular input patterns, remaining silent for irrelevant inputs, with one neuron encoding one piece of knowledge in perfect one-to-one correspondence. These neurons in Transformers are called "monosemantic neurons" (this resembles neuron mechanisms in the human brain). In contrast, there are also numerous polysemantically encoded neurons, meaning many knowledge points with different linguistic meanings can activate a given neuron; these are called "polysemantic neurons." The figure above gives examples: some neurons respond only when the input prompt is written in French — a typical "monosemantic neuron"; while others respond to multiple semantically disparate 2-gram language fragments — a typical "polysemantic neuron."

The concept of superposition means: assuming the number of features n to be encoded far exceeds network parameters d, we can find ways to use d-dimensional neurons to encode far more than d features. This encoding mechanism is called superposition — it is an information compression encoding mechanism discovered to exist in Transformer structures.

Superposition is closely related to "polysemantic neurons." Current research finds that LLMs operate internally as follows (referencing Finding Neurons in a Haystack: Case Studies with Sparse Probing): as shown above, the LLM's superposition mechanism is jointly constructed by multiple "polysemantic neurons," each responding to several different knowledge points in the input. Thus, through a single "polysemantic neuron" alone, we cannot detect what it is currently responding to; but if there are multiple "polysemantic neurons" that all respond to some knowledge point, a linear combination atop their responses can detect the specific knowledge point we want to recognize (the blue portion in the figure above). That is, LLMs encode a specific feature or knowledge point by combining multiple "polysemantic neurons." Therefore, the relationship between "polysemantic neurons" and knowledge points is many-to-many: one knowledge point activates many "polysemantic neurons" that encode it, while one "polysemantic neuron" responds to multiple input knowledge points.

Now that we've covered these three foundational concepts, here's what current research has found: in a trained GPT model, the lower layers of the Transformer encode a large number of concrete features or knowledge points — n-gram features, syntactic features, and so on — using the superposition pattern described above, composed of multiple "polysemantic neurons." As the Transformer layers go deeper, concrete knowledge points gradually decrease while abstract ones increase (such as "French" or "prime numbers"). These abstract knowledge points are generally encoded independently by "monosemantic neurons," and the higher the layer, the more abstract the encoded features become. In other words, the encoding of features or knowledge points in Transformers follows a bottom-up abstraction process, with knowledge becoming progressively more abstract. This phenomenon was also noted in OpenAI's recent article "Language models can explain neurons in language models."

Additionally, the paper "Polysemanticity and Capacity in Neural Networks" points out that during model learning, to increase parameter efficiency, "monosemantic neurons" are allocated to important features, "polysemantic neurons" to less important ones, and for even less important features, the model doesn't encode them at all. Here, "importance" refers to impact on training loss — meaning "monosemantic neurons" have a relatively large effect on reducing loss during NTP training. This suggests that abstracting features and knowledge points is an intrinsic driver for NTP to rapidly reduce loss, and this is likely one of the keys to how GPT models develop intelligence through the Next Token Prediction task.

Evidence of Knowledge Circuits in GPT

Here we introduce research on knowledge circuits in LLMs — the idea that completing a specific task involves a corresponding circuit. A "circuit" refers to the path that information takes after a task prompt is fed into the Transformer: propagating upward from the bottom layers until the last token at the highest layer outputs the answer via Next Token Prediction. Within the network, there exist critical paths for completing this task, along which information primarily travels upward, undergoing continuous information transfer and knowledge processing to accomplish the task through NTP. If you read the following content, you'll find that the workings of LLM knowledge circuits are actually quite similar to certain information processing circuits in the human brain. And the various knowledge circuits formed during GPT's NTP pre-training process may be another key to unlocking the mystery of AGI.

The paper "How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model" explores why GPT models can acquire mathematical capabilities through pre-training. Specifically, using prompts like "The war lasted from the year 17YY to the year 17," GPT can output a year XX where the next token's year is greater than YY — indicating it learned numerical comparison relationships during pre-training. Through investigation, researchers found that the model developed a knowledge circuit to solve this problem during pre-training, as shown in the right figure above. There are two key components: first, certain attention heads in the middle layers — for example, a5.h5 represents the 5th attention head in the 5th Transformer layer — which primarily focus on the YY year and propagate it upward; second, the MLP layers from layers 8 to 11, which perform the "greater-than" operation, enabling GPT to output the correct result. Moreover, there are corresponding transmission relationships between the middle-layer attention heads and the upper MLP layers — for instance, the 9th layer MLP mainly receives information from a9.h1, while the 8th layer MLP draws from more diverse sources. As you can see, information forms a specific propagation path from bottom to top.

Digging deeper, researchers found that key neurons within the MLP perform the mathematical operations. As shown in the right figure above, they identified the 10 most influential neurons in the 10th layer MLP — this layer can roughly complete the "greater-than" operation using just these 10 neurons. The left figure shows that the a7.h10 attention head primarily focuses on the key information "YY." Additionally, the study found that even when prompt formats were varied, as long as they expressed numerical comparison relationships, this same circuit was activated — suggesting this circuit may be specifically dedicated to comparing numerical relationships.

Most knowledge circuits are composed of both attention and MLP components, but some attention-dominant circuits have also been discovered. A typical example is the "Induction Head" circuit, whose existence has been verified by multiple studies. Its main function is that when GPT predicts the next token, it tends to find similar output patterns from earlier in the context and copy them to subsequent token outputs. In the sentence shown above, the second "so" is the last token. As GPT is about to generate the next token via NTP, the "Induction Head" circuit tends to find the same "so" from earlier in the context and treat the word "bad" that followed it as the next token output. The study "Localizing Model Behavior with Path Patching" probed the internal mechanism of the Induction Head: when predicting the next token based on the second "so," the content of "so" itself is copied into the Query of the corresponding attention mechanism, while the "bad" that appeared earlier in the context has its preceding semantic content integrated into the Key corresponding to "bad" through the PTH (Previous Token Head to key) attention head. When "so" performs attention, the two achieve high similarity, allowing "bad" to be copied to the position of "so" through attention — making it easy for the next token to output "bad," thus achieving the goal of copying "so…bad" from earlier context.

Beyond "Induction Head," there are attention circuits with more complex functions. For example, the paper "Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small" discovered a knowledge circuit in Transformers that is primarily attention-based and used for "Indirect Object Identification." This concept, illustrated in the example above, refers to having two entities in the input — one repeated, one non-repeated — and finding the correct answer from them. From the example above, we can see that GPT can output the correct answer "Mary" because the model learned a complex recognition circuit composed mainly of attention heads.

As shown in the diagram above, the "Indirect Object Identification" knowledge circuit that identifies the correct answer consists of three main steps: first, Duplicate Token Heads are used to identify tokens that appear multiple times in a sentence, with Induction Heads serving a similar function; second, S-Inhibition Heads act at the position where the next token is output, removing or suppressing repeated names from the attention of Name Mover Heads; finally, Name Mover Heads output the remaining name token. From this, we can see that during pre-training, LLM models learn highly complex attention knowledge circuits to better perform next-token prediction — circuits that execute copying of certain input tokens and output them in the next-token prediction result.

OpenAI Chief Scientist Ilya Sutskever once said in an interview: "When we trained LSTMs to predict the next character (NTP) in Amazon reviews, we found that if you predict the next character well enough, the LSTM develops a neuron that corresponds to sentiment. This nicely demonstrates the effect of unsupervised learning and validates the idea of next-character prediction. This discovery had a big impact on us." I understand that what he's describing here — the emergence of sentiment-corresponding neurons in the network — is essentially a sentiment-judgment knowledge circuit formed inside the model through the NTP training task. This discovery (refer to: Learning to Generate Reviews and Discovering Sentiment) was indeed an important factor that later motivated OpenAI to replace LSTMs with larger-scale Transformers and adopt NTP for pre-training on more data.

Currently, there is relatively little work exploring knowledge circuits in GPT models, but I personally believe this is particularly important. For instance, I suspect there likely exists a complex logical circuit that can explain the Chain of Thought (COT) phenomenon, and the formation of this circuit was probably driven by the introduction of program code or STEM paper data into the pre-training corpus. Because the logical relationships in such data are relatively tight, in order to rapidly reduce loss and accurately predict subsequent tokens during NTP tasks, GPT may be compelled to generate numerous abstract conceptual knowledge points internally and, on this basis, form complex logical circuits. I feel this line of work is highly valuable and worth further strengthening.

Differences in Knowledge Learning Across LLM Scales

This section summarizes research findings on how LLM models of different sizes differ in their learning of knowledge points.

The paper "Finding Neurons in a Haystack: Case Studies with Sparse Probing" mentions an interesting phenomenon: for the same abstract feature encoded by a "monosemantic neuron" — "whether French" (used to judge whether input content is in French) — if we ablate it, we can observe the impact on GPT's next-token prediction loss. The more the loss increases after ablation, the more important that feature is to the model. Interestingly, after ablation, small models show significant loss increases, while for large models, the impact is minimal. This indicates that this feature is important for small models but less so for large ones.

This phenomenon is strange, but the paper offers an explanation: as model scale increases, a feature splitting phenomenon occurs. That is, while a small model represents a certain knowledge point with only one coarse-grained neuron responding independently, a large model will refine this knowledge point and, depending on different context inputs, split into multiple neurons representing that knowledge point under different preceding contexts — corresponding neurons are activated only when specific contexts appear. In other words, for the same knowledge point, large models represent it more finely than small models.

For example, a small model might have just one neuron responding to "return" in the input, but a large model might differentiate into neurons responding to "return" in different programming languages — one neuron responding to "return" in Python, another responding to "return" in C++, and so on. Therefore, when a small model has a feature ablated, the impact is substantial because if this knowledge point appears in the input, it cannot be captured at all, which heavily affects loss. But for a large model, ablating this feature has less impact because it has split out neurons responding to different contexts; while this particular neuron may no longer function, other neurons represent various other cases. I consider this research conclusion very important, as it demonstrates a significant difference in knowledge representation capability between large and small models.

Additionally, other research findings indicate that as models become increasingly large, a higher proportion of "monosemantic neurons" can be detected. I think this suggests a possibility: the larger the LLM, the more abstract knowledge it encodes with independent neurons.

Another paper, "The Quantization Model of Neural Scaling," proposes that according to the degree of impact on NTP loss, we can rank knowledge units (referred to in the paper as "quanta") from most to least important, forming a Q queue. LLM models prioritize learning quanta at the front of the Q queue, while large models can learn more quanta from the back of the queue — those of lesser importance — than small models can. To summarize the core idea: large models can learn more less-important features than small models.

The above points are the conclusions available from current literature regarding differences in representational capacity across model scales.

Below the Iceberg: The Circuit Competition Conjecture (CCC)

If we piece together the fragmentary evidence from the known puzzle pieces, I feel that the hidden principles beneath the iceberg become faintly visible before us. This section offers some inferences based on known research conclusions, presenting the "Circuit Competition Conjecture (CC Conjecture)" as an explanation of the internal mechanism by which GPT constructs intelligence through next-token prediction. I require myself to find reference support for key points and to provide inference processes where deductions are made, so that this conjecture is built upon existing research conclusions. But overall, it remains an unverified conjecture, so please treat it with caution.

Circuit Competition: The Breakthrough of Task Circuits

First, let's summarize the known research conclusions to form a holistic impression. In this article, I uniformly refer to certain features or knowledge as knowledge points, because the traditional term "feature" alone struggles to encompass certain contents. Specific knowledge points include linguistic knowledge points (n-gram, morphology, syntax, semantics, etc.), context knowledge points (such as "input is in French"), world-knowledge-related knowledge points (entity-attribute, commonsense, events, etc.), and simple functional circuit knowledge points — these are fine-grained, and we collectively call them knowledge points.

Synthesizing the above, we can see that GPT models learn knowledge from data through NTP tasks, establishing two types of knowledge systems internally: hierarchical knowledge structures and various task circuits (refer to the diagram above). Task circuits are built upon hierarchical knowledge structures — they are fixed pathways formed by mutually exciting knowledge points, used to solve specific tasks.

Assuming a trained GPT model, we can clearly detect their existence. First, these knowledge points exist at different levels of abstraction. The lower they are stored in the Transformer, the more concrete, reusable, and universal they are, the greater their quantity, and the more likely they are encoded through dense encoding methods like superposition and polysemanticity. The higher they are stored in the Transformer, the more abstract, less reusable, and more task-specialized they become, tending toward separate encoding via "monosemantic neurons" (in the Transformer diagram above, white nodes represent concrete knowledge points, red nodes represent abstract knowledge points).

Second, certain knowledge points form bottom-up excitation relationships, where excitation paths progressively activate increasingly abstract knowledge points from lower to upper layers. For example, a knowledge point encoded at layer L of the Transformer can be activated by other excited knowledge points from layers 1 to L-1. Activated neurons, besides collecting, synthesizing, and abstracting information passed upward, may also add new knowledge through their own FFN structures (such as extracting world knowledge) or perform mathematical logical calculations (such as comparing numerical magnitudes). A trained GPT model contains massive amounts of these "local" knowledge point-based "micro-excitation structures," which should be the fundamental units forming GPT intelligence. Thus, the entire GPT structure constructs a world knowledge structure with hierarchical encoding of world knowledge. And training the model according to the NTP objective is essentially a process of gradually establishing increasingly complex hierarchical knowledge structures — from simple to complex, from universal to specialized, from concrete to abstract, from lower to upper layers — including knowledge points and the micro-structures produced by excitation relationships between knowledge points. The reason these emerge is that their existence helps with accurate prediction of subsequent tokens in NTP, that is, they help reduce training loss for the GPT model in NTP.

On this basis, we can re-examine the formation of task circuits. Task circuits should be structures that GPT forms, starting from the input layer of the Transformer and progressively associating relevant "excitation micro-structures" layer by layer, in order to more accurately predict the next token of certain special data types — ultimately forming a complete pathway structure that excites from low to high layers and finally associates to the output position to determine output token probabilities (refer to the red line in the diagram above outlining a certain task pathway). Having learned such a task circuit, if GPT subsequently encounters this type of data again, next-token prediction accuracy increases, manifested as reduced loss in the NTP task. For example, if the training data contains large amounts of arithmetic examples like "13+24=37," GPT will likely learn a task circuit for simple mathematical calculation, thereby increasing next-token prediction accuracy for the numbers after the equals sign.

Additionally, the Transformer blocks at each layer corresponding to the final input token may carry special significance. Through the Attention mechanism, this final position likely performs an information aggregation function over all preceding input content. If the input prompt is designed to accomplish a specific task, the Transformer blocks at the final token position progressively consolidate task-circuit information layer by layer, enabling accurate next-token prediction at the highest layer of the final token. In effect, the final token sketches out a prompt-specific sub-world from the vast knowledge system embedded in the Transformer.

The above synthesizes current research findings at a macro level, showing where our understanding of GPT's operational mechanisms currently stands. What follows incorporates some of my own inferences.

First question: during GPT training, with so many knowledge points to learn, there must be some sequential priority to how it acquires them. What order does it follow? While some research suggests that important knowledge points are learned first, this "importance" typically refers to contribution toward reducing the model's NTP loss — the more a knowledge point reduces loss, the more important it is considered. This is certainly correct from a loss-reduction perspective, but remains too abstract.

My personal view is that during training, GPT prioritizes knowledge points with the following characteristics: high-frequency knowledge points, general knowledge points (where higher probability of reuse indicates greater generality), and concrete rather than abstract knowledge points. These three principles should govern learning priority. Why? Because according to the principle of Next Token Prediction, the more frequently a knowledge point appears, the more opportunities it has for backpropagation to correct model parameters when GPT mispredicts, ensuring correct prediction next time. High-frequency knowledge points, due to their repeated occurrence, receive more parameter correction through backpropagation and thus more readily establish both the knowledge point itself and its connection pathways to other knowledge points. Once learned, high-frequency knowledge points are easily encountered in subsequent training data, contributing significantly to NTP loss reduction. The same logic applies to the other two categories: general knowledge points, due to their broad applicability, are used more often in subsequent predictions, receiving more backpropagation corrections and thus being more readily learned; concrete rather than abstract knowledge points also appear more frequently in training data, making them easier to establish. And so on. Conversely, low-frequency, domain-specific or task-specific, and abstract knowledge points are learned later by GPT. Or put another way, learning such knowledge points requires exposing the model to substantially more data, increasing the necessary backpropagation correction opportunities during learning.

Now we formally begin discussing the "circuit competition" hypothesis. Before introducing it, I make one assumption:

Assumption: To improve parameter utilization efficiency in GPT models, the NTP task encourages sub-circuit reuse.

By "sub-circuits," I mean circuits that perform simple computations, involving relatively few knowledge points with straightforward activation structures between them. GPT models likely first develop numerous sub-circuits for simple tasks or calculations, with complex circuits formed by further connecting many sub-circuits. To increase parameter efficiency, GPT models should encourage these sub-circuits to be reused across different complex circuits as much as possible, enabling more task types to be completed with the same parameter budget. For example, the "Induction Head" circuit discussed earlier is a typical sub-circuit — as we saw, it forms one component within the more complex "Indirect Object Identification" knowledge circuit. The relationship between sub-circuits and complex circuits is roughly illustrated by this example.

For two complex circuits solving different tasks, sub-circuit reuse means they share some identical sub-circuits — we can call these shared components "overlap circuits." It's easy to infer that the more similar two tasks are, the more overlap circuits they share. Moreover, overlap circuits likely occur more frequently at lower Transformer layers, since lower-layer circuits involve more concrete, numerous, and reusable knowledge points. The diagram above illustrates "sub-circuit reuse and overlap circuits," where the red line (red task) and blue line (blue task) represent two different complex task circuits, with some sub-circuits at the bottom layer being reused by both.

The "circuit competition" hypothesis can be explained using the above example. Suppose we input a prompt intended to complete the red task. When information propagates upward from lower to higher layers, activating the correct pathway, the lower-layer knowledge points and sub-circuits — being more reusable — tend to produce an "excess activation phenomenon": besides activating our desired red task, they also activate many knowledge points and sub-circuits leading to other task circuits. This phenomenon is more pronounced at lower layers. As information progressively moves upward, the red task circuit gradually receives further strengthening, while fewer incorrect upper-layer knowledge points and sub-circuits are activated, ultimately sketching out the correct red task pathway. This is the core thinking behind the "circuit competition" hypothesis.

If during this bottom-up activation process the correct circuit we desire is activated, we can consider this a victory in circuit competition, and the model outputs the correct answer. If an incorrect task circuit is activated instead, this constitutes circuit competition failure, and the model outputs an incorrect answer. We can infer that more complex tasks, involving more knowledge points and sub-circuits with more intricate interrelationships, are more likely to overlap with other similar task circuits and thus more prone to circuit competition failure.

We can use this "circuit competition" framework to think through many LLM phenomena and provide explanations; later sections will apply this hypothesis to explain some currently unexplained LLM behaviors.

Differences in Model Scale: Larger Models, Clearer Worlds

Based on existing research conclusions, when considering differences between large and small LLMs, we can roughly infer the following: small LLMs construct a coarse-grained, fuzzy world image, while as model scale increases, large LLMs build increasingly clear world images capable of representing more detailed information.

As described above, LLM representational capacity manifests in two main aspects: hierarchical knowledge structures from concrete to abstract, and task circuits capable of solving many problems. Let's examine differences between large and small models from both perspectives.

Differences in Hierarchical Knowledge Structure

Much research has shown that as model scale increases, model sparsity increases. Polysemantic neurons encode features densely, representing large numbers of relatively concrete features, while monosemantic neurons exhibit sparse single-neuron representation. This indicates that as models grow larger, the proportion of monosemantic neurons increases. Since monosemantic neurons encode important and abstract knowledge, their increasing numbers mean the model has learned more knowledge points. New knowledge points come from two possible sources: first, knowledge that smaller models failed to learn but larger models acquire — entirely new knowledge. This category subdivides further: world knowledge (common sense and events) that small models cannot encode due to low frequency in data, which large models encode through monosemantic neurons (large models can learn more low-frequency knowledge from data compared to small models — this is well-validated, and currently world knowledge appears to be single-neuron encoded); and more abstract knowledge newly induced from data (such as "prime numbers"), representing increasingly complex abstract knowledge or capabilities learned by large models.

The other source of new knowledge points comes from feature splitting of abstract features as discussed earlier. That is, where small models had only one coarse-grained abstract knowledge point, larger models develop new fine-grained representations of such knowledge, with different knowledge points learned for different context. For example, monosemantic neurons have been found in LLMs that respond to consecutive uppercase characters — if the input contains "ABCD," this neuron activates. Small LLMs might have only a single neuron responding to this; deactivating it causes NTP loss to spike when GPT predicts subsequent tokens, indicating this feature's absence causes errors for all consecutive uppercase character predictions. Large LLMs, however, besides this general neuron, develop fine-grained representation neurons: one specifically responding to corporate name abbreviations like "IBM," another to pharmaceutical abbreviations like "GS (glucose injection)," and so forth. This abstract feature splitting in large models demonstrates that even for abstract knowledge, large models possess more nuanced abstract feature expression capabilities compared to small models.

In summary, relative to small models, large models encode more low-frequency world knowledge — learning more detailed information about the world — while new abstract knowledge and abstract feature splitting demonstrate that large LLMs possess more difficult and more fine-grained abstract knowledge expression capabilities.

Differences in Task Circuits

Task circuits are pathways formed by bottom-up activation and connection between knowledge points arranged in hierarchical structures. Based on the above analysis of structural knowledge differences between large and small models, we can reasonably infer that large LLMs can likely construct circuits involving more fine-grained abstract knowledge points along their pathways, with more complex paths. This is likely the primary reason large models can solve complex problems.

Synthesizing the two perspectives, we can think of small models as coarse-grained world models, while large models are fine-grained, high-resolution world models. And the scaling law tells us that as we add more data and increase model size, LLMs can render the world with ever-greater clarity. From this angle, describing LLM parameters as lossy compression of the world isn't really off the mark.

The Endless Frontier: Explaining Unexplained Phenomena Through "Circuit Competition"

In this section, we use the "circuit competition" framework to explain several observed phenomena in current LLMs.

Emergent Capabilities Through the Lens of Circuit Competition

Emergent capabilities refer to tasks (mostly in-context learning or CoT-related) where small models show virtually no ability to solve them, but once model scale crosses a certain threshold, they can handle the task well. Although some research (see Are Emergent Abilities of Large Language Models a Mirage?) suggests that so-called "emergence" is merely an artifact of poorly chosen evaluation metrics — that there is no real emergence, just imprecise measurement — I personally find this explanation plausible for some tasks that appear to show emergent behavior. But I suspect it may not be the whole story; some tasks may be difficult to explain away so easily. So the question of why large language models exhibit emergent capabilities deserves further investigation.

Viewed through the circuit competition framework, there are two possible reasons why small models fail at certain tasks: either the activating circuit for that task hasn't been established in small models but has been in large ones, or the circuit exists in small models but consistently loses in circuit competition, making it appear as if they can't do the task.

I lean toward the first explanation for the emergent capabilities we currently observe. As discussed earlier, small models build a blurry, low-resolution mirror of the world, while large models construct a high-resolution, high-clarity mirror. Small models likely struggle to build complete activating circuits for certain tasks — perhaps because some abstract conceptual knowledge point critical to circuit formation is beyond their weaker abstraction capabilities (like the "prime number" example at the beginning of this article), or because emergent tasks tend to be complex and small models lack the capacity to build complex pathways. When model scale increases, capabilities in abstract concepts and complex circuit construction strengthen. Once a complete activation pathway for solving a task is established, the model suddenly appears capable of solving it — emergent capability. However, such complex circuits may still be relatively weak in competitive activation, which is why adding a few task-relevant examples via few-shot prompting helps the task circuit win in pathway competition, yielding better results.

In-Context Learning and Chain-of-Thought (CoT) Through the Lens of Circuit Competition

From the circuit competition perspective, ICL likely involves two types of circuits: task circuits and attention circuits, which compete or cooperate to determine ICL performance. CoT is a special form of ICL, and the mechanism should be similar.

The role of task circuits is straightforward to understand. In-context learning first provides the LLM with several task-relevant examples , then inputs , expecting the model to output the correct result corresponding to The role of the examples provided in the input is to activate the corresponding task circuit that the LLM learned during pre-training. Then when is input, it tends to follow this activated pathway, producing the correct output . CoT likely works similarly: without CoT, the LLM may activate a simple-structured task circuit, but with CoT examples, it tends to activate a complex reasoning circuit with detailed representations, causing subsequent inputs to follow this sub-pathway and produce detailed reasoning steps. Thus, in ICL scenarios, task circuits consistently play a positive role in generating correct answers for .

Now let's look at the Attention circuit — though here too we're in speculative territory. (The paper "In-context Learning and Induction Heads" aims to explain ICL through induction heads, but I find the induction head mechanism overly simplistic and in need of some strengthening.) Suppose there exists an enhanced version of the induction head circuit, which we might call the "Enhanced Induction Head" or EIH. Its operating mechanism would likely work as follows (as shown in the figure above): the EIH circuit would, based on the semantic similarity between the current input and each of the ICL examples' , copy the corresponding . The higher the similarity between and , the greater the probability of copying that corresponding . This process somewhat resembles a KNN model built from EIH circuits — it can arrive at the correct answer simply by voting based on the similarity between input examples and their corresponding labels, without the model needing to learn the mapping function from to through parameter updates. It amounts to a conditional induction head copy operation, where the triggering condition is the attention-based similarity between the input examples . We can see that what determines which label outputs should primarily depend on three types of examples in the ICL context: examples more similar to exert greater influence; that appear more frequently in the ICL context have greater influence; and examples closer in position to have greater influence (the positional information encoded by position embeddings and the pervasive local correlations in NLP likely produce this effect).

If the EIH circuit truly exists, we can infer from its operating mechanism the influence of the Attention circuit on correctly predicting the output in the following three scenarios:

Scenario 1: If the labels $y_1$ to $y_k$ in the ICL examples are ground-truth labels, the EIH circuit clearly exerts a positive influence — analogous to the KNN mechanism described above, which judges based on the examples from $x_1$ to $x_k$.

Scenario 2: If the labels in the ICL examples are not ground-truth labels but are instead randomly assigned from the label space, the EIH circuit should negatively impact the model's ability to arrive at the correct answer for $x_{test}$. This is because $x_{test}$ will search among the preceding examples from $x_1$ to $x_k$ for similar content to copy the corresponding label, but since that label was randomly assigned, it will most likely be wrong. In this case, the EIH circuit produces a harmful effect.

Scenario 3: If the labels in the ICL examples come from a different label space altogether, but one that maintains a corresponding mapping relationship with $y_{test}$, the EIH circuit should again have a positive effect. The logic mirrors Scenario 1: the KNN mechanism can learn this mapping relationship, thereby yielding the correct $y_{test}$ — the only difference being that it uses $y'_i$ rather than $y_i$. Of course, if you were still evaluating performance under the $y_i$ label space, ICL would necessarily be harmful.

When we consider both the LLM's intrinsic task circuit and the pure Attention-based EIH circuit together, they sometimes reinforce each other in the same direction, and sometimes compete in opposite directions. In the three scenarios above, the first represents synergistic alignment — both circuits promote the correct answer — while the second and third represent competition: the task circuit facilitates the correct answer, while the EIH circuit exerts a negative influence.

Following this logic, we can roughly explain many seemingly inexplicable phenomena observed in ICL research. Here's an example: current research shows that if an ICL label space contains two labels, A and B, and we reverse the labels in the ICL examples — swapping A for B and B for A — the ICL task performance degrades (see: Overthinking the Truth: Understanding how Language Models process False Demonstrations). Suppose the correct label for input X is A. From the perspective of the task circuit and the EIH circuit, the task circuit will tend to output label A. In this case, the EIH circuit corresponds to scenario three described above: label reversal is a special case of label substitution, because the correspondence between X and Y still exists. So the EIH circuit appears to learn the mapping from X to Y, and will tend to output label B. At this point, one circuit pushes in the positive direction while the other pushes in the negative direction — they compete, reducing overall model performance.

Many other phenomena can largely be explained within this framework, but I won't elaborate further for reasons of length. Interested readers can work through the derivations themselves using this analytical framework.

Fine-Tuning for Domain Tasks Through the "Circuit Competition" Lens

We can re-examine the effects of fine-tuning a general-purpose model on domain-specific data through the "circuit competition" perspective. We already know that fine-tuning on domain data causes "catastrophic forgetting" in the base model — that is, subsequent fine-tuning adjustments to model parameters cause the model to forget certain previously learned knowledge. My assessment is that, currently, any form of tuning performed on top of a base model will result in some loss of capabilities, and this includes the instruct tuning that ChatGPT underwent to understand commands and align with human values. This too should degrade certain base model capabilities, though we can't yet specify exactly which ones. This is the unavoidable price of tuning models under present technical conditions.

But why does fine-tuning a base model cause capability degradation? What's the underlying mechanism? Through the "circuit competition" lens, we can analyze the effects of fine-tuning. I suspect there are roughly two types of effects, which may operate independently or in combination. The first effect: fine-tuning uses large amounts of domain data to strengthen the response circuit that the LLM uses to solve this particular task. This probably doesn't affect the lower-level knowledge points much, since the bottom layers contain more general-purpose features that this task also requires. The adjustments likely target more of the upper-level abstract knowledge nodes and the activation pathways connecting lower-level knowledge points to upper-level abstract nodes. Another possible effect: fine-tuning may establish shortcut pathways inside the model, causing information to bypass many pathways it should have traversed. Take text classification, for instance — the internal logic of this task is relatively simple, likely just establishing activation pathways from lower-level domain-specific vocabulary knowledge points to upper-level abstract category concept nodes. So it's quite possible that fine-tuning creates a very short shortcut directly from the bottom-layer knowledge points to the high-level category concept nodes, with other complex circuits getting bypassed by this shortcut. It's not necessarily that the upper-level abstract nodes get rewritten; rather, the other circuits get routed around via the shortcut.

Regardless of which mechanism is at play, the consequence is the same: for new inputs, even when the task at hand is different, this specially strengthened circuit is easily triggered. That is, this reinforced circuit tends to win the competition even when it shouldn't, degrading performance on other tasks.

Instruct Tuning Through the "Circuit Competition" Lens

Instruct tuning is essentially a specialized form of fine-tuning designed to align model behavior with human expectations. GPT-4's technical report notes that instruct tuning does not enhance the base model's knowledge or capabilities; in fact, it may even damage certain abilities. High-quality instruct tuning is certainly important, but it only makes large language models appear more effective — a subjective perception from users, not an actual improvement in the model's fundamental capabilities.

So how might we understand what instruct tuning actually does through the "circuit competition" lens? I think it works like this: instruct tuning establishes a special activation circuit, creating a connection between the activation circuit formed by the input instruction itself and the corresponding task circuit. After training with instruct data, when an instruction is input, this facilitates activation of the relevant task circuit — making it seem as though the large language model understands the instruction's meaning. This resembles the mechanism of "conditioned reflex" in Pavlov's biological experiments, essentially building a reflex pathway between user commands and their corresponding task circuits.

The "circuit competition" hypothesis can not only provide plausible explanations for phenomena whose internal mechanisms remain unknown, but also account for other observed behaviors. Take the frequent problem of large models "confidently hallucinating nonsense" — this can be understood as a failure of the correct circuit in the competition process, or as a situation where the correct circuit and some incorrect circuit are activated with similar intensity, producing a mixed result that appears reasonable but is factually wrong. And so on.

The Parametric Reflection of the World: From the Real World to Possible Worlds

The physical world operates according to its own Hidden Rules. Conceptually, we can understand that a parsimonious Hidden World exists, from which the colorful phenomenal world emerges. If we categorize the phenomena of the world, they broadly fall into natural phenomena, social phenomena, and psychological phenomena. Humans are constituents of the physical world, observing its appearances and attempting to comprehend its operating principles in order to better sustain the survival of both the species and the individual within it.

From the species perspective, the evolutionary process of natural selection over millions of years constitutes humanity's pre-training, with the optimization objective of "Next Person's Survival Prediction." The smaller the loss, the more individuals survive in the population. Genetic encoding serves as the model parameters; individuals expressed by genetic encoding survive if they fit the environment, and are eliminated if they do not. Survivors persist because certain features expressed by their genetic encoding match the survival environment, so these environment-matching encodings are strengthened in the population — completing one parameter update for humanity's pre-trained model. The continuous changes in the external physical environment drive changes in the population's genetic encoding, thereby promoting survival under shifting conditions. The genetically pre-trained model we are born with records survival strategies learned over millions of years, forming System 1 — the fast, unconscious reactive system in our brains that represents the collective memory of the species.

From the individual perspective, beyond the innate survival strategies obtained through genetic pre-training, individuals engage in "continual pre-training" throughout their lives to maintain survival in specific environments. The optimization objective here is "Next Action Prediction," pursuing correct behavioral outputs to sustain survival. This employs a parameter update strategy similar to LoRA: for the individual, innate genetic encoding is an immutable base model that determines many behavioral patterns, but a portion of the brain remains modifiable. By adjusting the connection patterns between neurons in this region, new knowledge and skills can be learned. If a behavioral output negatively impacts continued survival, model parameters are adjusted to better cope with the environment in the future. This brain region's function forms System 2 — the slow, deliberate decision-making system that represents personalized survival experience. "Innate genetic encoding + individual survival fine-tuning" shapes the diverse behaviors of different individuals, with commonalities deriving from collective species memory and individuality stemming from unique survival experiences.

Language initially served as a communication and collaboration tool between human individuals, facilitating species survival. As technology developed, it was gradually recorded on tortoise shells, bamboo slips, paper, and electronic signals as written text. Each person can be viewed as an independent "encoder-decoder": individuals observe and experience the physical world, encoding knowledge and thought in the brain, with decoding outputs forming written text that records personal perspectives on worldly experience and reflection — both subjective feeling and objective record. Populations form distributed "encoder-decoders," whose decoding outputs produce vast textual records containing objective facts about world operation alongside conflicting subjective viewpoints. Thus, text is merely surface appearance; what it internally records is human-formed cognition of the physical world and subjective experience of it (physical knowledge, social knowledge, event records, individual feelings, individual imagination, and various other types), behind which lies a world model from the human perspective. GPT, through its Next Token Prediction task attempting to correctly reproduce human-generated text, is essentially decoding and reconstructing the world model hidden behind textual appearances, storing it in GPT's model parameters — forming a parametric reflection of the physical world.

If we think more deeply, we may discover that GPT, from vast quantities of text, has learned not only how to generate content consistent with real-world facts, but has potentially learned to become a "possible world" generator. It begins by simulating our real world through text, then generalizes and abstracts. Though it follows our world's physical laws, it can produce not only knowledge and content true to our perceived world, but also other possible worlds consistent with physical laws and human-understandable logic. Perhaps you cannot call it wrong simply because its output doesn't match the real world — you can only say it has the capacity to show you all logically consistent possible worlds. Inevitably, many cases will not correspond to reality, since the existing world is merely one realized choice among possible worlds, and it has the ability to present you with various reasonable possibilities.

The End of the World and the Hard-Boiled Wonderland: The "Digital Brain in a Vat" Thought Experiment

"A mad scientist performs an operation, removing a person's brain and placing it in a container filled with nutrient solution. The nutrients in the solution are sufficient to maintain normal brain function, while the brain's nerve endings are connected to wires, with the other ends connected to a computer. The computer simulates real-world parameters and transmits information to the brain through the wires, making the brain feel that everything is completely normal — as if familiar people and routines continue as always, with nothing amiss.

One day, the brain in the nutrient solution has a sudden inspiration for an interesting thought experiment. In its perceived reality, it is currently on the subway to work or at its office desk, with faint footsteps audible nearby. It pulls out its phone and writes the idea in its notes app:

'OpenAI has released a new LLM model called GPT-4 with very powerful capabilities, likely heralding the arrival of AGI. Everyone around me is discussing it enthusiastically. Today I read an article analyzing its possible working mechanisms, titled The Parametric Reflection of the World: Why GPT Can Produce Intelligence Through Next Token Prediction. It was very inspiring and got me thinking. We can imagine: if AGI becomes sufficiently powerful in the future, it could read everything I've written, my photos and videos, even scan and replicate my brain's response patterns, reconstructing a digital brain identical to my physical one. Then another version of myself would live in digital space, with AGI接管我的数字大脑的各种感知信号，模拟我的工作和生活场景，让大脑感到一切都完全正常，好像周围认识的人、熟悉的事情还照常进行，没有任何异样。那么，这个数字大脑里的我，或者现实生活里的我，能区分现在是生活在数字空间，还是物理空间吗？我把这个思想实验称为：数字缸中之脑。这个思想实验，是不是很有意思？'

I call this thought experiment: the Digital Brain in a Vat. Isn't this thought experiment rather interesting?"

5Y Capital seeks out, supports, and inspires founders who are going it alone, giving them everything from moral backing to operational support. We believe that if the world starts believing in the vision that others call crazy, things will never be the same.

BEIJING · SHANGHAI · SHENZHEN · HONG KONG

WWW.5YCAP.COM