Introduction to AI Data Centers | Bolt's Reading Pick

线性资本·October 30, 2024

The History, Construction Technology, and Corporate Landscape of AI Data Centers

Eric Flaningam is an investor and researcher at Felicis, a venture capital firm. He recently published an article on Substack about AI data centers, covering their history, technology, and industry landscape in detail — a solid primer for anyone looking to understand the space.

We've organized and translated the piece, hoping it offers some food for thought. You can read the original via the link at the end.

📝 Article

We are in the middle of the largest compute infrastructure buildout in history. The parallels to the electrical grid a century ago are striking. Back then, we saw power plants scaling up, astronomical capital expenditure, and a steep drop in electricity costs. Today, we're seeing data centers scaling up, massive capex from hyperscalers, and a sharp decline in the cost of AI compute.

This article serves as an introduction to AI data centers. We'll break the topic into several parts: what they are, the upstream and downstream value chain, and potential investment opportunities.

01 Overview of AI Data Centers

The term "data center" doesn't quite capture the sheer scale of these AI factories. Research shows that building a hyperscale facility costs billions of dollars, covering land, power and cooling equipment, construction, GPUs, and other compute infrastructure.

That doesn't even include energy costs. Factor those in, and a single hyperscale data center can consume up to 1 gigawatt of power. For reference, New York City uses about 5.5 gigawatts. So for every five of these facilities, we're adding roughly another NYC-sized load to the grid.

We can roughly divide the data center value chain into several categories: initial development and construction, industrial equipment that supports operations, the compute infrastructure inside, and the energy required to power everything. Additionally, some companies own or lease data centers to provide end-user services.

Here's how we can visualize that value chain:

Note: This diagram doesn't cover every company involved. Financiers, real estate developers, construction firms, and many others all contribute to data center builds.

Before diving deeper, we should look at the history of data centers — particularly relevant given the energy constraints we're seeing today, especially in Northern Virginia.

02 A Brief History of Data Centers

Data centers have largely evolved alongside computers and the internet. I'll briefly trace these trends and how we got here.

2.1 Early History

Early computers actually resembled what we'd call data centers today: a centralized machine built for computationally intensive, complex tasks.

Two early examples:

  1. Colossus: Built by Alan Turing to crack the Enigma code.

  2. ENIAC: Designed by the U.S. military during WWII but completed in 1946. Colossus was built earlier, but due to its classified nature, ENIAC is often credited as the first computer.

Both could be considered the "earliest data centers."

In the 1950s, IBM dominated computing with its mainframes and maintained that dominance for decades. AT&T was another tech giant of the era.

ARPANET, launched in 1969, was developed to connect the growing number of computers across the U.S. and is considered a precursor to the internet. As a government project, its densest connections concentrated around Washington, D.C.

This is the root cause of Northern Virginia's computing dominance. Even today, as new data centers are built, operators want to leverage existing infrastructure — so more facilities keep springing up in the region.

2.2 The Rise of the Internet and Cloud

In the 1990s, the internet's growth demanded more physical infrastructure to handle increasing data flows. Data centers emerged as interconnection hubs in this context. Telecom providers like AT&T already had communications infrastructure, so building data centers was a natural extension.

However, the competitive dynamics among these telecoms resembled the vertical integration we see in today's cloud providers. AT&T owned both the data transmitted through its infrastructure and the infrastructure itself. So when capacity was constrained, AT&T prioritized its own traffic. Companies grew wary of this dynamic, which led to the rise of data center companies like Digital Realty and Equinix.

Data centers saw heavy investment throughout the dot-com bubble, but funding slowed dramatically after it burst.

In 2006, the launch of Amazon Web Services began to reverse the data center downturn. U.S. data center capacity has grown steadily since. That growth continues today, with estimates suggesting capacity will double by 2030.

2.3 Enter AI Data Centers

The AI frenzy of 2023 brought renewed focus on data center scale, driven by the compute demands of model training. Research shows that the closer compute infrastructure is physically packed together, the better the performance. Moreover, when data centers are designed as compute units rather than just server rooms, companies gain additional integration advantages. And since model training doesn't need to be near end users, data centers can be built anywhere.

To summarize the defining traits of today's AI data centers: they emphasize scale, performance, and cost — and they can be built anywhere.

03 How to Build an AI Data Center

3.1 Construction

There are two approaches: compute providers can build themselves, or partner with data center developers like Vantage, QTS, or Equinix.

First, they need to find a suitable location. Then they hire a general contractor to manage the entire construction process, who in turn brings in specialized subcontractors for electrical, plumbing, HVAC, and so on. Workers relocate to the area for the project's duration. Once the building shell is up, equipment installation begins.

Industrial equipment for data centers broadly falls into two categories: electrical and cooling.

  1. Electrical equipment: This includes main switchgear connecting to external power sources, power distribution units, uninterruptible power supplies (UPS), and server power cabling. Most data centers also have diesel generators as backup for outages.

  2. Cooling equipment: This covers chillers, cooling towers, HVAC systems, and liquid or air cooling devices connected to the servers themselves.

3.2 Compute

The compute infrastructure needed for data centers consists of equipment running model training and inference workloads. The main components are GPUs or accelerators. Beyond Nvidia, AMD, and supercomputers, numerous startups are also competing for a piece of the AI accelerator market.

While less central than before, CPUs still play an important role in complex computations and "task orchestration." For data storage, storage devices hold data off-chip, while memory is dedicated to frequently accessed data. Networking connects all components, enabling communication within servers and between servers and the outside world.

Finally, everything gets packaged into servers installed in the data center. Below, you can see a visual of one such server.

3.3 Powering the Facility

The energy supply chain breaks down roughly as follows:

  1. Power sources: Fossil fuels, renewables, and nuclear that can generate electricity.

  2. Generation: Power plants convert fossil fuels to electricity; renewable generation is closer to the source.

  3. Transmission: Transformers and substations convert high-voltage electricity to manageable levels and transmit it via high-voltage lines to destinations.

  4. Distribution: Utilities manage the last-mile delivery and power scheduling through power purchase agreements (PPAs).

Transmission and distribution together constitute what's commonly called the grid, managed locally. Depending on location, either can become a bottleneck in energy delivery.

As it turns out, energy is the critical bottleneck in AI data center construction.

Unfortunately, rapidly expanding energy capacity isn't easy. Data centers have two options: grid-connected and off-grid energy. 1) Grid-connected energy flows through the grid and is distributed by utilities. 2) Off-grid energy bypasses the grid, such as solar, wind, and batteries. 3) Alternatively, you could build a gigawatt data center next to a 2.5-gigawatt nuclear plant — that would work quite well.

The problem with grid energy is the time required to expand grid capacity. The chart below shows how long companies wait for commercial power hookups.

04 Characteristics of AI Data Centers

The new generation of data centers is larger, denser, faster, and more power-hungry. Building at hyperscale isn't new — from a few megawatts in 2001, to 50 megawatts in the 2010s, to 120 megawatt facilities in 2020, and now gigawatt-scale data centers. Every few years, articles appear about the next leap in data center scale.

These gigawatt facilities are designed from a systems perspective and pack in more density. The core problem here is the slowing of Moore's Law — the idea that semiconductor performance improves as transistor density increases. Yet transistor improvements are becoming increasingly difficult. The solution is to pack servers, even entire data centers, more tightly together.

In practice, this means data centers are being designed as integrated systems rather than rooms full of individual servers. The servers themselves are designed as integrated systems, bringing all components closer together.

This is why Nvidia sells servers and PODs (the smallest deployable compute units), why hyperscalers build system-level data centers, and probably why AMD acquired ZT Systems.

Below, we can see the Nvidia DGX H100 visualized — it can function as a standalone server, connect to other GPUs via POD, or link through SuperPOD for even greater connectivity.

Nvidia also helped pioneer accelerated computing — offloading tasks from CPUs, which raises the importance of all other components including GPUs, networking, and software.

Beyond this, AI's unique demands require processing massive amounts of data. This makes storing more data (memory/storage) and moving data faster increasingly critical. It's analogous to a heart pumping blood: the GPU is the heart, data is the blood.

These components come together to form the most powerful computers on the planet. Yet as compute power increases, so does energy consumption, heat generation, and the cooling required per server — and this energy intensity will only grow.

05 Opportunities and Challenges in AI Data Center Construction

5.1 Grid-Connected and Off-Grid

Clearly, our energy infrastructure needs to evolve to support this buildout. Nearly every tech company prefers grid power because it's more reliable and easier to manage. Unfortunately, when the grid can't deliver, hyperscalers must solve the power problem themselves. For example, AWS is investing $11 billion in an Indiana data center campus and building four solar farms plus one wind farm (600 megawatts) to power it.

Over the medium to long term, I'm most bullish on two areas solving the energy bottleneck: nuclear and advanced energy storage batteries. Both can provide more sustainable power for data centers.

The benefits of nuclear are well-known: clean and reliable. The challenge is building it economically. In my view, some of the most exciting startups in the world are tackling this.

Advanced battery storage will be a major step forward for renewables. The problem with solar and wind is their intermittency — they only generate when the sun shines or wind blows. Long-duration batteries can store excess energy and dispatch it when supply is scarce, helping solve this.

5.2 Permitting Automation and Liquid Cooling

On the industrial side, I'm excited about two trends: permitting automation and liquid cooling.

  1. Permitting automation: In conversations with researchers, one topic consistently comes up as a bottleneck for data center construction: permits. For data centers and energy expansion, developers need permits for construction, environment, zoning, noise, and more. They may need approvals from local, state, and national agencies. Plus, right-of-way laws vary by region.

  2. Liquid cooling: One obvious difference in new AI data centers is the dramatically increased heat from servers. This generation will adopt liquid cooling; the next may move to immersion cooling.

5.3 Credit Where It's Due

We have to acknowledge: (1) the incredible work Nvidia has done in building its ecosystem; (2) AMD's efforts to solidify its position as a strong follower.

From applications to software infrastructure to cloud computing, systems, and chips, Nvidia's position in AI is extraordinarily high. If you wanted to write the perfect playbook for preparing for a technology wave, Nvidia has done it. Additionally, Crusoe is another excellent compute infrastructure company, offering both AI computing resources and energy services.

Overall, tech companies involved in data center construction should continue performing well, as revenue flows through the value chain. From networking to storage to servers, if a company delivers top-tier performance, their results will stand out.

5.4 Final Thoughts

My closing thought on data center construction: this is indeed a major trend of our era, but it's probably just one part of computing's broader history. I see AI, data centers, and compute as an integrated whole that can't be discussed separately.

As Sam Altman described it:

"We can look at human history from a narrow lens: after thousands of years of scientific discovery and technological progress, we've figured out how to melt sand, add some impurities, arrange it at astonishing precision and incredibly small scale into computer chips, run energy through it, and ultimately create systems capable of producing increasingly powerful AI."

Throughout the last 100 years, creating intelligence has been a constant trend, and data centers are at the heart of that trend today.

Linear Bolt

Bolt is an investment initiative by Linear Capital focused on early-stage, globally-oriented AI applications. It upholds Linear's investment philosophy, targeting projects where technology drives transformative change, and aims to help founders find the shortest path to their goals — whether in speed of execution or investment approach. Bolt's commitment is to be lighter, faster, and more flexible. In the first half of 2024, Bolt invested in seven AI application projects including Final Round, Xin Guang, Cathoven, Xbuddy, and Midreal.