A 10,000-Word Chronicle: The LLM Infrastructure Years, From the Dark Ages to the Golden Age to the Silver Age

十字路口Crossing·July 17, 2024·0·0

The low-hanging fruit has been almost entirely picked. What now?

📝 Editor's Note:

This long-form chronicle comes from Fang Jiarui, a seasoned systems engineer who has spent years working on large model infrastructure. The article distills the history of large models into what reads like a biography of the technology itself. For engineers, it serves as an invaluable guide. For non-engineer readers, it offers a window into the world of large models. We hope it helps you understand where large models have been, where they stand now, and where they're headed — and that it inspires you to seize the new opportunities that AI is creating.

Author: Fang Jiarui, PhD in Computer Science from Tsinghua University, has worked on large model systems software at major tech companies and startups alike. Zhihu profile[1], GitHub profile[2].

More and more people are paying attention to large models, and many engineers have asked me how to get started with large model training and inference systems software (colloquially known as "LLM Infra").

As someone who once worked in backend development, I deeply relate to where they're coming from. In recent years, China's internet industry has been broadly stagnant, yet large models have been rising against the tide — the daily news cycle is a riot of color, a cauldron boiling over, forming a stark contrast with every other subfield.

The same dynamic plays out globally. In 2023, Silicon Valley laid off hundreds of thousands of software engineers, yet American tech giants were frantically buying GPUs and hiring at premium salaries to build large models. This half-year's US stock market highs were driven by AI expectations. The contrast between hot and cold could not be clearer. As computer professionals, none of us want to miss this train.

But I must caution that LLM Infra is transitioning from its golden age into its silver age.

I started writing large model training systems in 2021, making me a full-cycle witness to this era. Let me share the phases I've observed:

2019–2021: The Dark Ages

Overall, before 2022, large models were simply too far ahead for China, and consensus was extremely weak. Hence, I call this the Dark Ages of LLM Infra.

The seminal 2017 paper "Attention Is All You Need[3]" was published, and Transformer-based architectures began proliferating: decoder-only GPT-1 (2018), encoder-only BERT (2018), and encoder-decoder T5 (2019), each conquering new territory in NLP. Translation, dialogue, recommendation — one application after another fell to Transformers within internet companies.

Riding this wave, some researchers began exploring scaled-up Transformers. Enlarging models ran counter to CNN-era intuition, when the community was desperately trying to shrink models through NAS and AutoML to fit them into cars, cameras, and phones. The true believers in Scaling Laws were mainly OpenAI and Google. In 2018, while ELMo[4] used LSTMs for next-token prediction, OpenAI swapped in Transformers to create GPT-1, applying the philosophy accumulated from Dota AI and beginning to scale. GPT-2 and GPT-3 followed. Google, as Transformers' birthplace, had proposed a MoE architecture[5] for Transformers in 2017, and by 2020 could train a 600B parameter model called GShard[6] on 2K TPUs, developing the TensorFlow-based distributed training framework MeshTensor[7] for this purpose. Unlike OpenAI, Google didn't bet solely on decoder-only architectures, instead vigorously pursuing encoder-decoder structures (the original Transformer formulation). DeepMind, a British research organization under Google but relatively independent, also produced decoder-style models similar to GPT: Gopher[8] and Chinchilla[9].

Though GPT-3's 175B parameters and few-shot learning capabilities in 2020 sent some shockwaves through the industry, domestic skepticism toward the large model technical path dominated; true believers were few, and most algorithm engineers remained unenthusiastic. I attribute this to two factors:

First, most people lacked any concept of pretraining. The NLP development paradigm then was private data plus BERT fine-tuning. Drawing from small-model experience, the focus should be on data quality, not a larger base model. Even doubling model size would require substantial budget for training, with questionable ROI.
Second, the infrastructure wasn't ready. Training a large model wasn't something just anyone could attempt. Today, training requires algorithm and engineering collaboration; back then, no such organizational pairing existed. Business teams had only algorithm engineers, for whom GPUs were a headache — especially deploying a model requiring two GPUs, which was practically a disaster. Meanwhile, infrastructure teams in platform departments were unaware of large model trends; information silos between departments were formidable.

The most motivated to scale models domestically were leaderboard-chasing algorithm teams, yet most of their models inevitably followed the PR-then-shelved fate. Still, this process trained many teams, who later became the backbone of domestic large model training after ChatGPT exploded.

One man's poison is another man's meat. Model scaling, conversely, represented new opportunity for AI infrastructure people. The dominant AI Infra application then was still recommendation systems; NVIDIA was betting on the metaverse as its next growth engine. Large models were as novel to Infra engineers as to anyone else. With OpenAI and Google blazing trails, some American institutions began exploratory work in model scaling, mainly along the path of scaling up encoder-architecture BERT — though for training purposes, the difference between encoder and decoder architectures was minor.

It was through this wave that large model training frameworks Megatron-LM and DeepSpeed first took shape. In 2020, Microsoft trained a 17B BERT called Turing-NLG, whose training code became DeepSpeed's prototype. In 2019, NVIDIA trained an 8.3B model called Megatron-LM — yes, Megatron-LM was originally a large BERT's name. The Megatron-LM repository contained training scripts implementing tensor parallelism, which gradually evolved into the most popular training framework. Both were built atop PyTorch and both were open-source.

During this period, American tech giants trained models mainly to test the waters and flex their muscles, so DeepSpeed and Megatron-LM were open-source from the start — a fortunate development. Imagine if large models had debuted as nuclear weapons, with everyone hoarding secrets like OpenAI post-2022; Infra technology diffusion would never have been so rapid.

Training models at GPU scale rendered small-model-era data parallelism inadequate. Fundamental training optimization concepts began crystallizing: ZeRO, tensor parallelism, pipeline parallelism, offloading, hybrid parallelism, and more. These techniques all had predecessors — FlexFlow[10] (2018) for hybrid parallelism, GPipe[11] (2018) for training AmoebaNet from NAS, ZeRO as a special form of Parameter Server[12], and so on. But these techniques could be more purposefully optimized for the Transformer architecture and high-bandwidth interconnect networks.

Domestic understanding of large models as a novelty was limited then. Jinhui Yuan's OneFlow[13] was among the earliest attempts at large model distributed training; its SBP-based approach to automated model parallelism was conceptually ahead of its time, but unfortunately China lacked business demand to support their product-market fit. Huawei's MindSpore also began automatic parallelism experiments. Earlier movers included Alibaba's M6[14], BAAI's GLM[15], and Huawei's Pangu[16]-alpha.

Beyond a handful of large model believers, domestic tech giants remained unenthusiastic. Each major cloud/platform team had just completed a round of X-minutes ImageNet training arms races and was still pondering what business value could come from using GPU clusters at such scale to train a single model. Some NLP technology startups still pursued custom BERT fine-tuning for different businesses. Some institutional investors would call technical consultants to ask whether large models were a scam.

Overall, before 2022, large models were simply too far ahead for China, and consensus was extremely weak. Hence, I call this the Dark Ages of LLM Infra.

I personally began researching large model training systems in early 2021, at the twilight of the Dark Ages. At Tencent's WeChat AI, the research atmosphere was relaxed and free, and I was fortunate that both my direct and skip-level leaders possessed genuine technical foresight — they supported this work despite no clear path to production. The greatest obstacle to large model training was compute resources; the GPU hardware I could access was mainly single-machine 8-card V100s. So I chose to design better CPU-offloading strategies to run large models on low-end machines — this became PatrickStar[17]. At the time, several teams within the company attempted to jointly apply for hundreds of cards to train models, but were denied due to unclear ROI.

2022–2023: The Golden Age

Following the Dark Ages' accumulation period, the Golden Age of LLM Infra began in 2022, marked by Meta's open-sourcing of the OPT-175B model.

On one hand, with NVIDIA's chip roadmap iterations, a single DGX SuperPOD (2021) could already comfortably handle hundred-billion-parameter model training. To promote these supercomputers, NVIDIA positioned large models as a killer app — besides large models, what application could possibly saturate them? In November 2021, Tencent acquired a SuperPOD prototype and pulled together infrastructure teams company-wide for two months of testing. They'd drawn up an intensive testing schedule, only to discover that only our PatrickStar tuning was actually using the machine; we eventually had exclusive access to the entire system.

On the other hand, some people were beginning to see large models' power. Silicon Valley felt the spring water's warmth before the duck knew to swim.

In May 2022, Meta open-sourced OPT-175B's weights, trained to reproduce GPT-3 using PyTorch FairScale. The model quality left much to be desired, but it was an enormous boon to large model researchers — AI Infra people finally had a real model to experiment with. In June 2022, HuggingFace organized a multinational coalition across thirty-plus countries and institutions to open-source the Bloom-176B model, using the DeepSpeed framework.

Among Silicon Valley startups, Character.ai[18] and Anthropic.ai[19] had already been established for some time. From the start of 2022, while the country united against COVID, several teams abroad were already intensively training GPT-3-level models. What particularly stuck with me was April 2022: two authors of the Transformers paper had left Google to found Adept.ai[20], a large model company using large models to help people complete complex tasks. Their Twitter demo showed a large model generating Python code to draw a bar chart from text input — I found it absolutely incredible. That was probably my ChatGPT moment; today, such things barely raise an eyebrow.

Then came November 2022. ChatGPT burst into mainstream consciousness, detonating the large model phenomenon. This event dramatically accelerated consensus-formation around the value of scaling models. Large model teams rapidly amassed human, material, and financial resources. LLM Infra leaped forward.

Large model development: training first. In training systems, Megatron-LM and DeepSpeed iterated rapidly during the Golden Age. To be honest, pre-ChatGPT, both were architecturally questionable — I sensed no architect in charge, just frantic stitching together of research ideas, riddled with bugs and unfriendly user interfaces. Yet for large model training frameworks, first-mover advantage matters enormously. Selection criteria weren't usability, but which framework had previously trained which model. After all, a single training run represents investment ranging from hundreds of thousands to tens of millions of dollars — stability trumps everything. Post-ChatGPT, Megatron-LM, backed by NVIDIA's brand and technology, began snowballing in advantage; today it's basically unified under its commercial version, Nemo.

Technically, it was a land of opportunity everywhere. Parallel strategies and algorithm optimization abounded with openings — simple yet effective works like Sequence Parallelism and Flash Attention gained enormous influence. Beyond pretraining came SFT and RLHF demands, spawning training-inference hybrid scheduling and works like S-LoRA.

LLM Infra also profoundly shaped upper-layer algorithm trajectories. For a long time, people dared not increase context length because the intermediate QK^T matrix in Attention computation incurs memory costs quadratic in sequence length. This led to a period where Linear Attention and Approximate Attention research flourished, trading accuracy for length. Once Memory Efficient Attention engineering optimizations appeared — most famously Flash Attention, which eliminated the quadratic memory term entirely — everyone returned to straightforward standard Attention.

In inference systems, major development came much later than training, mainly after 2023. For one, models hadn't been trained yet, so there was no inference demand. For another, before decoder architectures achieved dominance, inference acceleration research hadn't found its proper target. Attention had focused on optimizing encoder Transformer inference — operator fusion, eliminating padding from variable-length inputs. In the BERT era, FasterTransformers[21] (FT) was most widely used; my 2019 project TurboTransformers[22] was directly benchmarked against FT. FT was developed by NVIDIA's China team (made in China); I witnessed its growth from a small corner of NVIDIA's examples into a standalone product line.

BERT-era system optimizations could be reused for GPT's prefill phase, but solutions for the decoding phase's critical problems were still lacking. The paradigm shift from encoder to decoder changed training minimally, but impacted inference enormously — from a compute-bound problem to a hybrid where prefill is compute-bound and decoding is memory-bound. BERT-era optimizations couldn't transfer to decoding. Decoding's uncertain output length created two stubborn problems: one, how to dynamically batch to reduce wasted padding computation when output token lengths are unknown; two, how to dynamically allocate GPU memory for KV Cache without fragmentation.

Though inference started late, its development velocity far outpaced training's. Inference requires fewer resources and has lower barriers to entry, so broader participation is possible — pooling collective wisdom, problems surface quickly and get solved quickly. In 2022, the OSDI paper ORCA[23] proposed Continuous Batching, solving problem one. Exactly one year ago from today — June 2023 — the vast majority of domestic large model practitioners didn't even know about Continuous Batching. The 2023 SOSP paper Paged Attention[24] solved problem two.

Open-source community development was explosive. Excluding Accelerate, DeepSpeed-Inference, and other libraries that handle computation without scheduling, the earliest open-source inference framework was HuggingFace's TGI (text-generation-inference[25]), originally created to host models on HuggingFace's platform. But first-mover advantage failed in inference frameworks — partly because this framework wrote its scheduling in Rust, making it inaccessible to most and thus niche. By then, some decent open-source inference frameworks had already emerged domestically, such as Shanghai AI Lab's LMDeploy[26].

The true game-changer was UC Berkeley's vLLM[27], open-sourced in June 2023, which shot to fame with its original Paged Attention technology. This coincided with various large models releasing their first versions; vLLM immediately met this wave of concentrated deployment demand. In September 2023, NVIDIA launched TensorRT-LLM, initially with targeted open-sourcing for enterprise beta testing before broader open-sourcing, also capturing a substantial slice of inference. NVIDIA had only formally organized forces to develop decoder model inference frameworks in early 2023; TensorRT-LLM stitched together TensorRT, Triton Server, and FT — three flagship products — showing that inference demand convergence had really only formed in the past year.

With inference frameworks in place, many experimental works could unfold: GPTQ, AWQ and other quantization methods; speculative sampling, multi-head Medusa and other techniques to increase decoding phase compute-to-memory ratios; FastGen, ChunkPrefill and other batching scheduling strategies; DistServe, SplitWise and other disaggregated scheduling strategies; broader NPU support.

Training and inference demand surged, attracting much talent to LLM Infra. The field experienced a boom. Ordinary people with strong learning ability had a chance to get on board — hence, I call this the Golden Age.

2024–: The Silver Age

In 2024, despite large models blooming everywhere, production resources concentrate toward the top, practitioner class stratification intensifies, and LLM Infra enters the Silver Age.

After the FOMO-driven frenzy of 2023, people are calming down. Some are exiting; some are expanding.

In pretraining, GPU resources concentrate toward the top. Startups have winnowed to perhaps six or seven, some aligning with cloud providers. Within large companies, only one anointed team controls all GPUs for pretraining. This differs markedly from the small-model era, when every business team could train its own model and manage some GPU compute. It's as if, previously, every province had its own army; now the nation has only central forces. Thus, talent demand is lower than traditional AI business, yet aspiring entrants have surged enormously, raising barriers precipitously. Unless you join one of those dozen-or-so domestic pretraining teams, you'll likely have no access to pretraining.

In fine-tuning and inference, opportunities are also contracting. The causes differ between open-source and closed-source models.

For closed-source models, fine-tuning and inference remain monopolized by pretraining teams — models worth hundreds of millions to train cannot leak; customers must bring data on-site for private deployment.

For open-source models, one might expect that with open models, everyone can do fine-tuning and deployment downstream. Counterintuitively, despite growing open-source large model quantity and capability, fine-tuning and training demand is decreasing.

First, fine-tuning is actually extremely difficult. Without model training experience, you won't achieve expected results. Hence RAG has become dominant — it merely requires calling large model MaaS APIs.
Second, inference is intensely competitive. Integrating quantization, scheduling, and speculative sampling at best-practice level is non-trivial. And now trends like disaggregation and co-location increasingly demand engineering sophistication. A small team doing inference deployment often underperforms free open-source MaaS APIs, which have professional optimization behind them.

Overall, large models are a technology very decoupled from business — more like cloud providers or chips. Traditional backend online and offline systems, being business-dependent and non-standardized, retain value even without best-practice implementation. For LLM Infra, with open-source frameworks as a baseline, it's the best or nothing — if you can't be the best, there's no reason for your existence. Thus, mirroring the chip industry, resources concentrate in a few giants, while most can only participate in downstream supporting roles like RAG and agents.

Overall, in the Silver Age, LLM Infra aggregate demand is increasing, but with Matthew effects: "For whoever has will be given more, and they will have an abundance. Whoever does not have, even what they have will be taken from them." Fair grounds don't last long; grand feasts are hard to repeat. Those already aboard can feel the thrilling push of acceleration; those not aboard can only stare helplessly.

But the Silver Age is still a precious metal — opportunities remain. The LLM Infra pie is still growing, with more chips and more algorithm innovations on the way. Getting in now, the physically strong can still squeeze onto the train.

Advice for the Silver Age

Entering the Silver Age, I deeply feel the low-hanging fruit has been almost entirely picked; papers today are less exciting than two years ago.

People often say seven-year cycles: 2016's AlphaGo launched one cycle with deep learning; 2022's ChatGPT launched a new cycle with large models.

Many now enter LLM Infra expecting excess returns — in the Silver Age, this expectation must be tempered. Those who catch cycles are destined to be few, because divergence creates risk, and risk creates excess returns. Consensus on large models has long since solidified; this field will inevitably reach supply-demand equilibrium, becoming governed by market principles. It's like when your auntie starts buying a certain stock — that stock won't make money anymore.

Large models will inevitably transform our world profoundly, but the trend of resource and information concentration at the top is deeply concerning. On one hand, core large model technology is held by a tiny handful — OpenAI. On the other, facing enormous information and compute asymmetries, ordinary people cannot participate in large model development. Just as robotaxis are replacing ride-hail drivers, AI will someday replace programmers. How to find one's place in a world after productivity advances, and not become second-class citizens of an AGI world — this is the root of everyone's anxiety.

For society to not fear AI, for society to rationally plan a future of harmonious coexistence with AI, we need both regulation of giants and more equal opportunities for people to understand large model technology. Many are working toward the latter, diffusing large model technology through open-source code and public papers. As aspiring entrants, you can leverage open-source forces to keep yourself synchronized with industry. There remain numerous unsolved technical challenges awaiting your contribution. Additionally, directions like agents, multimodality, and embodied intelligence are still nascent — you can position yourself ahead of the next wave.

As LLM Infra practitioners, the Silver Age demands rigorous fundamentals. In 2023, many people derived value from information asymmetry — I know something you don't, and even experimenting would cost you time. Many, amid intense competition, were willing to pay for this information gap. Such opportunities will diminish greatly this year. Now we compete on genuine capability: can you rapidly follow new technologies? Can you independently handle a complex large system? Do you have the broader technical vision to dialogue with other stakeholders — requiring not just Infra knowledge but some algorithm and cloud computing understanding? Traditional engineering competence has become paramount.

I've always hoped to contribute modestly to democratizing AI; my GitHub tagline[28] is "Democratizing LLM." My Zhihu sharing over the past year has focused on interpreting cutting-edge LLM Infra papers.

Entering the Silver Age, I deeply feel the low-hanging fruit has been almost entirely picked; papers today are less exciting than two years ago. Earlier papers often contained insights defining critical LLM problems — ZeRO, ORCA, PagedAttention — while today's papers rarely produce such eye-opening content.

I plan to systematically share LLM Infra fundamentals going forward, aiming to reveal underlying universal principles and make them accessible to backend developers without AI backgrounds or to undergraduates. This idea remains immature; feedback is welcome.

References

[1] Zhihu profile: https://www.zhihu.com/people/feifeibear

[2] GitHub profile: https://link.zhihu.com/?target=https%3A//github.com/feifeibear

[3] Attention Is All You Need: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

[4] ELMo: https://arxiv.org/abs/1802.05365

[5] MoE architecture: https://arxiv.org/abs/1701.06538

[6] GShard: https://arxiv.org/abs/2006.16668

[7] MeshTensor: https://proceedings.neurips.cc/paper/2018/hash/3a37abdeefe1dab1b30f7c5c7e581b93-Abstract.html

[8] Gopher: https://arxiv.org/abs/2112.11446

[9] Chinchilla: https://arxiv.org/abs/2203.15556

[10] FlexFlow: https://proceedings.mlsys.org/paper_files/paper/2019/hash/b422680f3db0986ddd7f8f126baaf0fa-Abstract.html

[11] GPipe: https://proceedings.neurips.cc/paper_files/paper/2019/hash/093f65e080a295f8076b1c5722a46aa2-Abstract.html