7 Judgments to Rethink AI Infrastructure

Cheaper Tokens, Pricier Race

👩 Author: Shirley

🥷 Editor: Koji

🧑‍🎨 Layout: NCon

Recently, SemiAnalysis founder Dylan Patel appeared on Sequoia Capital's Training Data[1] and WisdomTree's The Next Big Thing[2].

The two interviews range from models and chips to memory, energy, data centers, and AI business models, converging on one shift: the AI race is moving beyond pure model capability comparisons toward who can deliver model performance stably, at lower cost, and at greater scale.

Following this thread, Crossing has systematically organized Dylan's views into seven judgments. Together they form an analytical framework extending from efficiency, demand, and physical bottlenecks to competitive dynamics, organizational capacity, and commercial闭环.

The central causal chain running through this framework can be summarized as:

Cross-layer coordination drives down per-token costs → cost declines amplify total demand → demand collides with physical constraints of memory, networking, and power → supply landscape is redrawn → high-margin model companies gain stronger compute bidding power

---

Judgment 1: 100x Gains Don't Come From Single Breakthroughs, But Cross-Layer Coordination

The industry typically breaks AI efficiency gains into three layers: chip hardware, systems software, and model algorithms.

Dylan believes looking at any single layer in isolation easily misses what actually matters:

A chip can get 2x faster, a kernel can get 2x faster, model architecture can improve efficiency 2x. Multiply those three numbers and you only get 8x. But when model structure, communication patterns, memory hierarchy, and compute units are redesigned simultaneously, the final gain can reach 100x.

DeepSeek offers a more intuitive example. DeepSeek V3's expert architecture was primarily optimized for NVIDIA's Hopper architecture; the subsequently released 1.6-trillion-parameter V4 was further adapted for Blackwell and Huawei Ascend architectures.

However, hardware adaptation doesn't mean performance is fixed from day one.

InferenceX[3]'s ongoing testing of DeepSeek V4 shows that at launch, GB300 NVL72 led significantly in some test configurations, while AMD MI355X's early performance lagged noticeably. Over roughly six weeks of testing, AMD's team improved MI355X throughput by over 100x through precision switching, kernel completion, scheduler optimization, and inference framework improvements — exceeding H200 in some ranges. (Note: this 100x improvement came from software stack maturation from a low early baseline under specific test configurations, not a 100x general hardware performance gain for MI355X.)

This catch-up process better illustrates Dylan's point: final performance isn't determined by silicon specs alone, but by the combined effect of model, chip, memory, network, and software maturity.

This cross-layer coordination is also redefining NVIDIA's moat.

Open-source models like DeepSeek, Moonshot AI, GLM, and Qwen extensively optimize around GPU memory structures, communication patterns, and compute units. Even if Google's TPUs offer better energy efficiency for specific workloads, migrating these models there may not yield equivalent performance.

Thus, NVIDIA's barrier lies not only in the CUDA development ecosystem, but in the deep adaptation that downstream models, inference frameworks, and open-source tools have formed around GPU system characteristics.

---

Judgment 2: The Cheaper Each Token Gets, The Stronger Total Compute Demand Becomes

Dylan makes a radical claim about the token market: in the future, model inference could become a market larger than oil, accounting for several percentage points of global GDP.

Whether this prediction materializes remains uncertain, but the cost curve behind it is already steep.

By Dylan's estimates, at comparable capability levels, model usage costs once fell at roughly 60x annually. DeepSeek V3's price gap from GPT-4 two years prior reached about 600x on some comparable tasks.

By conventional economic logic, sharp price declines should mean reduced spending, but AI is showing the opposite.

The cheaper and more capable models become, the more tasks that previously weren't worth automating become viable; when models can handle longer, more complex tasks, users further increase call volume. This is AI's Jevons paradox: falling unit costs don't reduce total consumption, but expand overall demand.

SemiAnalysis itself is an extreme example. Per Dylan's disclosure in the interview, the company's annualized AI spending was under $100,000 in November 2025. With Claude Code's internal adoption, this rose to roughly $4 million by late January 2026, and has now reached about $11 million, with peak weekly spending annualizing to nearly $14 million.

But two distinct workload types need separation here:

Type one: fixed workflows. For example, a company needs to check whether a document meets specific rules. Once a model hits the quality threshold, the company can simply wait for a cheaper small model and use cost declines to improve margins.

Type two: AI assistants. Employees use models daily for research, coding, analysis, or content generation. In this scenario, the cheapest model doesn't necessarily have the lowest total cost.

A stronger new model might complete in one interaction with 1/4 the tokens what an old model needed multiple rounds to finish. Though each token is more expensive, total token count is lower, and less employee time is consumed.

Thus, Dylan's cost optimization approach isn't to always switch to cheaper models, but rather: fixed workflows pursue unit cost; knowledge work pursues total cost to complete a task.

And when new models raise productivity, people typically don't stop to save budget — they immediately reinvest saved time and tokens into more tasks. So total spending briefly dips, then quickly rebounds.

---

Judgment 3: Memory Is Structural Scarcity, CPU Is Cyclical Recovery, CPO Still Waits

In The Next Big Thing, Dylan offers a framework for evaluating AI supply chains.

To judge whether a segment merits attention, at minimum three questions must be answered:

How much real AI demand flows through this segment? Is it a monopoly, oligopoly, or commodity market? When supply is tight, can vendors convert scarcity into pricing power?

Applying this framework to memory, CPU, and optical interconnect yields three distinct conclusions.

Memory: From Inventory Cycle to Structural Scarcity

The traditional memory industry typically runs on 18- to 24-month cycles: price rises stimulate capacity expansion, new supply builds inventory and drives prices back down.

Dylan believes this cycle may last longer, because AI is simultaneously changing memory's demand scale and product mix.

On one hand, model parameters, inference traffic, and context lengths continue growing. Conversational Q&A is evolving toward complex reasoning and agent tasks, requiring models to store and repeatedly access more intermediate states — pushing demand for HBM, server memory capacity, and bandwidth.

On the other hand, supply expansion hasn't kept pace.

Global leading-edge DRAM capacity remains concentrated in Samsung, SK Hynix, and Micron. Faced with higher-margin HBM3E and HBM4, vendors prioritize allocating advanced capacity to HBM.

But HBM is not only more complex to manufacture and package; it also consumes more wafer area. As more capacity shifts to HBM, traditional server DRAM, PC memory, and mobile memory receive squeezed allocations.

This resource competition is already transmitting to consumer electronics. Counterpoint Research[4] projects that叠加 memory crisis and geopolitical shocks, global smartphone shipments will fall 13.9% year-over-year to 1.08 billion units in 2026 — the lowest since 2013. Rapidly rising storage costs are forcing phone makers to restructure product mixes, pricing, and shipment plans.

Dylan's rough assessment in the interview: in coming years, memory capacity may grow roughly 20% to 30% annually, while related demand could maintain near-doubling growth rates.

If this growth differential persists, this cycle isn't merely inventory restocking, but structural reallocation between AI demand and limited capacity.

Of course, "structural scarcity" doesn't mean prices rise forever — memory retains commodity attributes. When new capacity comes online, or price-sensitive demand in phones and PCs gets squeezed out by high costs, the market will still seek new equilibrium.

CPU: Real Demand Recovery, Plus Historical Catch-Up

Reinforcement learning and agents are increasing CPU usage.

During pretraining, most compute is concentrated on GPUs or other AI accelerators. Once reinforcement learning begins, systems also need to run code tests, sandbox environments, task validators, and simulators. Agents frequently call databases, search engines, code interpreters, and external tools — work typically handled by CPUs.

TrendForce[5] judges that as Agentic AI becomes widespread, the workload ratio between CPUs and GPUs inside data centers may gradually evolve from roughly 1:4 or 1:8 on traditional AI servers toward 1:1.

What this describes is a shift in workload and system requirements, not that CPU and GPU physical chip counts or market values will mechanically become 1:1. It reflects that as AI moves from single model calls to executing complete tasks, more and more computation happens outside the model itself.

However, Dylan cautions that part of the current CPU market's high growth rate comes from historical catch-up.

In past years, AI accelerator shipments grew rapidly while supporting CPU deployment lagged behind. As reinforcement learning and agent workloads increase, cloud providers and labs are now集中补齐 this gap.

New demand and inventory restocking are happening simultaneously, amplifying current growth.

But inventory restocking doesn't continue forever.

Using Dylan's example from the podcast: if one CPU pairs with two GPUs, each GPU costs roughly $50,000 and each CPU about $5,000, then every $100,000 in GPU spending corresponds to only about $5,000 in CPU spending.

CPU demand elasticity can be high, but its absolute value remains difficult to compare with AI accelerators.

Therefore, the more accurate judgment is that the CPU is in a phase of recovery driven jointly by "new demand" and "historical catch-up," not another supercycle on par with GPUs.

Optical Interconnect: CPO May Still Take Longer Than Market Expects

As AI clusters scale, network equipment's importance and value share in overall systems will continue rising.

But growing network demand doesn't mean all optical interconnect technologies will immediately enter large-scale deployment. Dylan is cautious about near-term co-packaged optics (CPO) adoption.

CPO integrates optical components more deeply with switch chips or compute chips, theoretically reducing power consumption, shortening signal transmission distance, and improving bandwidth density. But this integration also introduces new engineering challenges, including manufacturing yield, thermal management, repair difficulty, and coordination between optical and semiconductor supply chains.

Dylan's judgment: CPO will be difficult to deploy at scale by 2027; true volume ramp may not come until late 2028 or even 2029.

Until then, copper cables, backplane connectors, and traditional pluggable optical modules may enjoy a longer commercial window than the market expects.

This also shows that AI demand growth doesn't evenly convert to profits across every supply chain link. Scarcity itself doesn't automatically create returns — only vendors with technical barriers, stable delivery capability, and pricing power can translate supply tightness into sustained gains.

Judgment 4: Energy Isn't Unsolvable; What's Truly Scarce Is Power Available On Schedule

A data center's endpoint is electricity.

According to SemiAnalysis estimates, global data center capacity additions may reach roughly 20 GW in 2026, exceed 30 GW in 2027, and approach 50 GW by 2028.

Such massive load growth makes energy one of the most watched constraints on AI infrastructure.

Dylan breaks the energy problem into three links:

• Generation: where the electricity comes from;
• Transmission: how to deliver power to the data center;
• Conversion: how to transform high-voltage AC into stable power usable by chips.

Of these, transmission is the hardest to solve.

Transmission lines and substation construction have long cycles, involving regulatory approval, cost allocation, local utility company coordination, and community resistance. Even if a power plant gets built, the electricity may not reach the data center on time.

Generation has more workarounds.

SemiAnalysis expects that in coming years, close to half of new data center load may come from behind-the-meter generation built by enterprises themselves. Rather than waiting for public grid expansion, companies deploy gas turbines, gas generators, or other on-site power directly adjacent to data centers.

Some operators are even experimenting with repurposing mass-produced truck, marine, or locomotive engines into on-site generation sets — engines driving generators. Hundreds of units deployed around a data center, maintained by rotating mechanical repair crews.

This approach isn't elegant, and efficiency and reliability may not be optimal, but it can come online faster.

Solar plus storage represents another path. Dylan judges that under certain reliability requirements, the combined cost of solar plus batteries may fall below gas within two years. But if the system must handle extended cloudy periods and higher-tier reliability requirements, required storage capacity and costs rise significantly.

The most radical proposal is sending compute equipment into space to use solar power directly.

Dylan's judgment here carries strong speculative weight: by 2030, space-based compute may still account for less than 1%. Only when ground grids, behind-the-meter generation, and industrial supply chains all approach limits would space-based data centers become economically viable.

Thus, energy isn't a completely unbreakable physical ceiling, but every workaround implies higher capital investment, construction difficulty, environmental cost, and regulatory contention.

Judgment 5: NVIDIA Needs a Multipolar World

In his Sequoia Capital interview, Dylan offered an intriguing judgment: Jensen Huang doesn't want an AI world that ends up with just two or three closed labs and a handful of hyperscalers.

If model demand concentrates increasingly in the hands of leading labs like OpenAI, Anthropic, and Google, while Google, Amazon, and Microsoft continue raising their custom silicon ratios, NVIDIA would face fewer and fewer customers, with a handful of large buyers gaining ever more bargaining power.

Therefore, for NVIDIA, a more ideal customer structure would feature enough competing model companies, cloud providers, and compute operators. It needs Chinese open-source models, independent AI labs, NeoClouds, and enterprise customers to continue existing, and more companies able to deploy compute outside the hyperscalers.

Custom silicon is intensifying this urgency.

Anthropic[6] was reported to have held preliminary discussions with Samsung on 2nm custom chips, with project purpose and specifications not yet determined; Reuters cited sources saying DeepSeek[7] is developing inference chips; earlier, OpenAI[8] and Broadcom formally announced the Jalapeño inference accelerator. The three differ in progress, but together they show large model companies increasing vertical integration, seeking greater control over compute supply, costs, and technology roadmaps.

However, custom silicon doesn't mean general-purpose GPUs will be fully replaced.

Specialized chips suit optimization around relatively stable model architectures and mature workloads, lowering inference costs at sufficient scale; but from design and tape-out to large-scale deployment takes a long cycle, while model architectures and training methods may change faster.

Therefore, the more likely pattern is: leading companies use custom silicon for some mature workloads while continuing to purchase general-purpose GPUs for frontier training, architecture exploration, and tasks that are hard to predict in advance.

In other words, general-purpose GPU value lies not only in current performance, but also in providing insurance against technology roadmap uncertainty.

In this context, NVIDIA's support for independent labs and NeoClouds carries deeper strategic significance.

Sharon AI[9] is one representative case. In June this year, the company disclosed a six-year strategic compute partnership with NVIDIA, planning to add 72 MW of data center capacity in Australia and deploy up to 40,000 GB300s in batches. The agreement's maximum contract value is $4.88 billion, using revenue-sharing and credit support mechanisms that let Sharon AI expand with lighter capital, while NVIDIA gains hardware revenue and a cut of related cloud services revenue.

But this isn't without cost. When NVIDIA uses credit support to push customers to deploy GPUs, it not only bears manufacturing and sales risk, but also becomes indirectly exposed to customer credit, GPU utilization rates, and equipment residual value changes.

So "NVIDIA needs a multipolar world" doesn't mean it's unconditionally supporting the industrial ecosystem. More precisely, maintaining enough customers capable of buying and operating GPUs is itself part of NVIDIA's strategy to protect market scale and bargaining power.

Judgment 6: Catching the Frontier Requires Three Resources — Data, Talent, and Compute

SemiAnalysis proposed a relatively clean framework in The Future of Meta Superintelligence[10]:

Building frontier models requires simultaneously possessing data, talent, and compute. These three are not independently additive elements — any obvious shortfall in one, or inability to coordinate all three, ultimately becomes the ceiling on model capability. By this standard, Meta is currently one of the few companies that may be able to complete all three pieces of the puzzle.

But "having resources" and "catching the frontier" remain two different things. The real test is whether a company can, through its organization, engineering, and research systems, continuously convert these three resources into model capability.

Data: Reinforcement Learning Is Creating a New Data Supply Chain

As the focus of model training shifts from large-scale pre-training to post-training and reinforcement learning, the definition of data is changing too.

In the past, data mainly meant internet text. Now, what's scarcer is high-quality tasks, interactive environments, toolchains, and verifiers capable of judging whether an outcome is correct.

A coding agent needs code repositories, testing environments, and unit tests. A finance agent needs spreadsheets, real workflows, and clear outcome standards. A general-purpose office agent needs operation traces that approximate real white-collar work.

Around these demands, a new supply chain of human data and reinforcement learning environments is taking shape.

According to SemiAnalysis estimates, data service providers like Mercor, Surge, and Handshake have all surpassed $1 billion in annualized revenue. Fleet, Mechanize, and Afterquery — all relatively young companies — are also growing rapidly.

What's special about Meta is that it already has massive engineering, advertising, legal, finance, and operations teams. In theory, it can extract a large volume of tasks approximating real economic activity from internal workflows, without having to rely entirely on external data vendors.

In April this year, Meta launched an internal tool that attempted to record employee screen, keyboard, and mouse activity for training general-purpose agents aimed at white-collar work. But just two months after launch, it was suspended due to internal pushback and concerns about sensitive information leaks.

At the same time, Meta has been making organizational adjustments, transferring roughly 3,000 engineers to work full-time on building RL tasks and environments. For comparison, data company Mercor logged 2.517 million expert hours across its entire platform in Q2 — equivalent to 4,800 people working 40 hours a week. If we roughly calculate Meta's 3,000 engineers at full quarter, 40 hours per week, that corresponds to about 1.56 million theoretical hours, or roughly 62% of Mercor's Q2 expert hours.

The two datasets aren't directly comparable in terms of task types, work efficiency, or quality standards. But the order of magnitude alone makes it clear: Meta is trying to turn its internal engineering organization into a massive RL task-and-environment factory.

Talent: Frontier Capabilities Remain Concentrated in a Handful of Teams

Model papers and engineering methods are diffusing rapidly. But the people who can actually organize large-scale training, design reinforcement learning systems, and solve complex infrastructure problems remain concentrated in a small number of labs.

Over the past year, OpenAI, Anthropic, and Google DeepMind have waged fierce competition for top researchers and engineers. High compensation is only the surface. The deeper competition is over how much compute researchers can access, what quality of data they get, and how much experimental autonomy they have.

Meta has ample financial resources and is willing to use organizational restructuring and heavy investment to close its talent gap. But talent migration doesn't immediately translate into model capability. New teams still need time to develop stable research cultures and engineering collaboration patterns.

Compute: Meta Is Turning Data Centers Into Part of Model Capability

Compared to model companies that rely on external cloud providers, Meta has a stronger balance sheet and doesn't need to allocate compute between internal model training and external cloud customers.

SemiAnalysis projects that by end of 2026, Meta's internal AI compute could exceed that of OpenAI and Anthropic combined.

To support this goal, Meta is simultaneously building five "Titan-class" clusters of over 1GW each, including Prometheus in Ohio, Hyperion in Louisiana, and additional campuses in Texas, Iowa, and Indiana.

Among these, Prometheus isn't a single data center but comprises six campuses and 27 data centers. Five of the campuses are within six kilometers of each other; the sixth is roughly 75 to 80 kilometers from the main group.

Meta plans to connect these sites through an AI-Backbone network, pushing past single-point power and cooling limits while also handling the bandwidth and latency challenges of cross-campus communication.

At that point, frontier model training will shift from being contained within a single data center to becoming a systems engineering project spanning multiple campuses, energy systems, and fiber networks.

But compute leadership doesn't equal model leadership. SemiAnalysis also notes that Meta's current public models still trail some open-source models on multiple metrics.

Meta has the resources needed to catch up to the frontier. But it still needs to prove that it can convert internal data into effective training environments, organize newly hired talent into stable teams, and turn massive compute into sustained model progress.

Judgment 7: Compute Keeps Growing, but Monetizable Demand Is Growing Faster

If the world is adding massive amounts of data center capacity every year, why is compute still scarce?

The problem is that the rate at which expanding model capabilities can take on economic tasks is currently outpacing the rate at which new compute comes online.

Anthropic is one of the clearest examples of this mechanism. (Note: the following figures are primarily SemiAnalysis estimates of Anthropic's unaudited finances, not company-disclosed audited numbers.)

According to SemiAnalysis, Claude Code's growth drove Anthropic's annual recurring revenue from roughly $9 billion at end-2025 to over $60 billion by mid-2026. Roughly 80% of revenue comes from API usage billed by token; consumer subscriptions make up a relatively small share.

SemiAnalysis also estimates that portions of Anthropic's API business carry gross margins above 80%. The company achieved profitability in Q2 after excluding stock-based compensation, and may post over $1 billion in GAAP EBIT in Q3.

While these numbers aren't enough to prove that all AI companies have found a viable business model, they do show that a handful of leading labs already have the ability to pay a premium for compute.

Dylan offered an intuitive illustration on the podcast: if an API business originally has roughly 75% gross margins, even a doubling of compute costs could still leave margins at around 50%.

For such a company, adding another GPU isn't just adding a capital expenditure — it's acquiring a productive asset that can be rapidly converted into paid-for tokens. As long as additional compute can be immediately deployed for inference and sold at relatively high margins, leading labs have incentive to keep bidding up compute prices.

SemiAnalysis forecasts that by 2030, the unconstrained compute demand from Anthropic and OpenAI combined could exceed 100GW, while the two companies' currently available compute totals just over 6GW.

Without considering supply and capital constraints, the scale of compute that leading labs want to deploy remains far above what they currently actually have.

Beyond Models, Competition Is Only Beginning

Returning to that main causal chain from the beginning, model capability is only the starting point.

Whether it can translate into genuine competitive advantage still depends on whether cost declines can unlock enough paying demand, whether scarce resources can be delivered on time, and whether additional compute can generate high enough returns.

This is why the question of "whose model is stronger," discussed repeatedly over the past two years, is expanding into a more complex and harder-to-replicate set of capabilities: who can continuously convert electricity, chips, memory, networking, data, and engineering capacity into cheaper, more stable, and more economically valuable tokens.

References

[1] Training Data: https://youtu.be/f6D_aiy8qyU?si=XFxMq8LJafiu_ZY5

[2] The Next Big Thing: https://youtu.be/lHnxU9f-rwc?si=Q1-mTRu2R-l_3uOd

[3] InferenceX: https://inferencex.semianalysis.com/

[4] Counterpoint Research: https://counterpointresearch.com/en/insights/2026-smartphone-shipments-to-post-worst-annual-decline-on-record-as-memory-crisis-and-geopolitical-shocks-converge

[5] TrendForce: https://www.trendforce.com/insights/ai-inference-drives-memory-demand

[6] Anthropic: https://techcrunch.com/2026/07/02/anthropic-is-discussing-a-new-custom-chip-with-samsung/

[7] DeepSeek: https://www.reuters.com/world/china/chinas-deepseek-developing-its-own-ai-chip-sources-say-2026-07-07/

[8] OpenAI: https://openai.com/zh-Hans-CN/index/openai-broadcom-jalapeno-inference-chip/

[9] Sharon AI: https://sharonai.com/press-releases/sharon-ai-announces-six-year-strategic-compute-collaboration-with-nvidia/

[10] The Future of Meta Superintelligence: https://newsletter.semianalysis.com/p/the-future-of-meta-superintelligence