How to Produce More Tokens for Less Money?
Anything that gets 10x cheaper becomes a new category.
👩 Compiled by: Shirley
🧑🎨 Layout: NCon
Crossing Editors' Note:
Everyone senses a massive opportunity: when Agents start running continuously in the background for days on end, the entire AI infrastructure stack needs to be redesigned — how GPUs divide labor, how Context gets stored, how different chips mix and match, whether data centers must never go offline, even whether power can come from intermittent wind and solar.
This podcast episode strings together concepts like Tensor Cores, NVLink, KV cache, SRAM/HBM, Cerebras, and heterogeneous computing, but it all converges on one question: how to produce more Tokens for less money?
The guest's approach is unconventional, almost like "scavenging" — not competing with NVIDIA for the most expensive chips and the most stable power, but picking up undervalued, idle compute and energy from the market, then cobbling them together into a cheaper "Token factory."
Guest bio: Neil Movva | B.S. and M.S. in Electrical Engineering from Stanford University. During his time at NVIDIA, he witnessed Tensor Cores debut with the Volta architecture. He then built Together AI's kernel team from scratch. In late 2025, he founded Sail Research with the goal of making open-source LLMs the cheapest per-Token in the industry; six months later, the company raised $80 million across seed and Series A at a $450 million valuation.
Anything that gets 10x cheaper becomes a new category
🎙️ Host (Host of podcast Invest Like the Best)
Neil, tell us about your company.
👱♂️ Neil Movva
Sail Research is a Token factory. Anyone can send requests through our API and use open-source LLMs for whatever task they want. We also let customers build Agents on top of this, hosting a cloud VM called Sailbox specifically for Agents that need to run continuously for hours, days, or even weeks.
Our company's keyword is "abundance." We want to turn AI into a new commodity that everyone can afford. Anything that gets 10x cheaper can spawn a new category. We want to make Tokens 10x cheaper. The fact that machines can think is remarkable in itself. Our job is to get as many machines thinking as possible.
🎙️ Host
Does Token cost matter today?
👱♂️ Neil
Our goal is to make the per-Token cost the lowest in the industry, and meaningfully lower than everyone else.
But Tokens aren't the ultimate measure of work or AI capability — they're just the most tangible one right now. Going forward, people will care more about "outcomes." Today when you ask an Agent to do something, you can't control how long it reasons or how many tools it calls. In the future, you just hand an Agent a task and let it keep trying. How many Tokens it consumes is determined by the task itself.
The best latency is no latency at all
🎙️ Host
So rather than capping how many Tokens engineers can use per month, you let the Agent decide what it needs to complete the task. But the whole world is trying to make Tokens better, faster, cheaper — why do you still have an opportunity? Where is the market still inefficient?
👱♂️ Neil
Two main reasons.
First, open source is rising. More and more customers want control over the technology they depend on. Whether using a fine-tuned model or the original open-source weights, they own them, can deploy them freely, and no one can take them away.
Over the past few years, hosting these models at scale has become a fairly mature business. The problem is that companies like Baseten, Fireworks, and Together have all bet on low-latency inference, and they were pulled in that direction by one very important customer: Cursor.
A year ago, that was the right call. But starting about six months ago, what people want from Agents isn't just low latency — they want them to work for long stretches. Now I'm convinced that the future of Agent inference is long-horizon tasks. When a machine runs continuously for hours or even days, generating 100 Tokens per second doesn't matter — 10 might be enough. And that actually improves overall efficiency.
🎙️ Host
Why are you so sure? I want everything to be as fast as possible.
👱♂️ Neil
If you're waiting for an answer, of course faster is better. But the ideal scenario is that you don't wait at all — the Agent works proactively in the background.
The best latency is no latency at all. You wake up, and the work was done overnight without you even having to ask. We're not fully there yet. But more importantly, if you're constantly sending prompts and waiting for responses, the human becomes the bottleneck, limiting what the Agent can accomplish.
We want Agents to run at human pace. You don't check on a colleague's progress every five minutes — you give them an objective, and check back in a day, or even a week. That's how humans and Agents will collaborate in the future.
🎙️ Host
Can you say more about what early signs suggest this is actually happening, and therefore worth building a company around?
👱♂️ Neil
The most important signal is test-time compute scaling: give an Agent more time, and it produces better answers. This idea emerged about two years ago, but it wasn't really worth betting on until Opus 4.5 shipped late last year.
Opus 4.5 was the first Agent that could barely handle long-horizon tasks, and it was pretty mediocre at launch. But look at the more recent models, and what we're doing in open source — Agents can now run for an hour at a stretch. Days, not yet. But an hour is absolutely doable today. Average task duration keeps lengthening; it doesn't take many data points before you can draw that exponential curve.
🎙️ Host
Three years from now, what share of the market do you think long-running Agents will capture?
👱♂️ Neil
I love this market because it has almost no ceiling. As long as it doesn't require constant human involvement, the background can keep consuming Tokens.
But human attention is finite. I couldn't use 10x more Tokens on Codex or Claude Code, because most of my time at the computer is already spent writing code. So in the long run, I think by year-end it'll be split 50/50 between background and real-time tasks; after that it becomes 9:1, with background at 90%.
🎙️ Host
What kinds of things get done much better as background tasks than with a human in the loop?
👱♂️ Neil
Most deep research. The kind where you want a definitive answer, and you're not checking 100 or 1,000 sources but 10,000 or more. One of our customers, Parallel Web Systems, is indexing the entire internet and monitoring its changes in real time. That's an exabyte-scale insane task, requiring a completely different magnitude of AI. Deep research is one of our most important directions.
Cybersecurity is moving this way too. Yes, you can generate a lot of code, but the number of ways to break the same code is exponentially greater than the number of ways to write it. We have some great customers working to build Agents that can break any software and then proactively patch vulnerabilities.
When Fable or Mythos first came out, the security world went through a phase where people took every line of code they'd ever written and had Fable try to find bugs in 20 different ways — memory errors, business logic errors, network-layer vulnerabilities. These are really cases where you want different Agents written for different objectives. You don't just have Fable statically read the source; you actually set up an environment and do penetration testing on the application.
Later people joked that security had become a kind of "proof of work." Want to know how secure your software is? Look at how much money you spent attacking yourself with Anthropic's API. It's a direct measure, because that's currently the best tool.
And the capability frontier here is jagged. Some bugs Haiku — a small model — can find, while Fable can't, and vice versa. So it creates a very diverse sampling approach: use different models, different Agents to autonomously attack software, then patch it.
🎙️ Host
We've talked about concrete applications like deep research and cybersecurity. If we think even bigger, what entirely new product categories would emerge from very cheap, long-running Agents? What happens if you achieve maximum success?
👱♂️ Neil
For consumers, what excites me most is proactive Agents. Imagine a Siri that's always running in the background, having read every email and text you received today, knowing your life inside out, knowing how to help. Current solutions are fragmented — you have to keep writing prompts, Siri isn't proactive at all, and abundant inference can fill that gap. If you trust it enough, it can even learn how you interact with it, so every time you unlock your phone, it presents exactly what you need to do next.
🎙️ Host
Can you build a good enough model to predict what you're going to do next?
👱♂️ Neil
I think absolutely.
🎙️ Host
And the key is extremely cheap AI.
👱♂️ Neil
The key is, are you willing to burn tokens with no return.
Zooming out, we already have an AI that can handle any problem with a verifiable answer — most software tasks, many formal mathematical proofs, even scientific discovery. Today, solving these costs money, and the cost depends on how many tokens you can throw at it. We've already brought the cost of long-horizon tasks down from millions of dollars to thousands. Soon, maybe it'll be hundreds or even tens of dollars to have AI take a shot at a scientific question.
🎙️ Host
If you follow that future forward, is the only thing ultimately limiting us "what questions humans can ask"?
👱♂️ Neil
Basically, yes. The model is nearly at the point where it can take a high-level question and just chase down every possible follow-up itself.
🎙️ Host
So the question becomes: what's your token budget?
👱♂️ Neil
And the token budget problem — we'll solve that.
🎙️ Host
What about unverifiable tasks?
👱♂️ Neil
I lump all of "human taste" into this bucket. Human taste we haven't solved, and I don't even know if it's fundamentally solvable. I'd love to be proven wrong. Right now we're focused on quantifiable problems. The quality of writing, the beauty of art — leave that to humans.
GPUs can optimize for throughput or latency. We choose throughput.
🎙️ Host
Let's talk about the full stack you're planning to build: ultimately this massive "token factory" to become a supplier of extremely low-cost intelligence. My understanding is you're thinking in three layers — software, hardware, and power. What's your overall plan, and how does it differ from how others are thinking?
👱♂️ Neil
You have to start with software, and first see how much efficiency can still be squeezed out of existing chips and data centers.
The first thing we're doing is rebuilding all large-model software around "peak GPU efficiency." We're still using NVIDIA GPUs, but we want the same chip to produce more tokens. To do that, you have to optimize from the very bottom, from the kernel level.
I've spent my entire career on GPUs and kernels. NVIDIA was my first job out of college. I watched Tensor Core earn its place on the die in 2017.
🎙️ Host
For people unfamiliar with chips, what exactly is a Tensor Core?
👱♂️ Neil
A Tensor Core is just the hardware unit on a GPU dedicated to accelerating matrix multiplication. That's it. We can talk later about how it's evolved over the years.
🎙️ Host
Why is matrix multiplication so important?
👱♂️ Neil
I don't have any grand theory, but I heard a very clean explanation: matrix multiplication is the simplest way to make two sets of numbers interact with each other. And linear algebra happens to represent all kinds of data relationships in very compact forms.
NVIDIA dominated gaming graphics for a long time. Starting around the mid-2010s, they launched some internal skunkworks projects to make GPUs more suitable for machine learning. When I was at NVIDIA, I read the lab notebooks of a few directors who went to ICML and NeurIPS back when they were still small conferences. The notes basically said, "Deep learning seems to be taking off. Grad students are training big models on our gaming cards. We should look into this." By 2015 or 2016, Jensen Huang at least was convinced this usage would only grow, and started allocating more and more precious die area to it.
It was still early. The moment you asked for area, you were fighting the graphics team. In any chip company, die area is fought over inch by inch. Bet wrong on a technology, and the opportunity cost of that area is wasted. So we fought tooth and nail to secure maybe 5-10% of the area on the first-generation chip, just to add a little acceleration for the convolution operations used in computer vision at the time.
What I learned most from being on the software team responsible for squeezing out every drop of performance was a certain quality NVIDIA has. They have a term called "speed of light." Any hardware they build, they pursue that limit: if we believe this chip can run at this frequency, can produce this many multiplies per cycle, we have to get there, breaking every bottleneck. To this day, I tell my engineers the same thing: we pursue 100% "speed of light." I don't care about relative numbers against competitors. I only care about absolute numbers.
🎙️ Host
Before we leave your time at NVIDIA, besides this cultural point you just made, what else changed how you think, or left the deepest impression on you about how NVIDIA operated and its culture?
👱♂️ Neil
I have a million stories about NVIDIA. Let me share a few. My favorite one is tenure: many people I worked with in 2015 and 2016 are still there today. The retention rate is shockingly high. They're the best engineers I've ever worked with. They've always believed in parallel computing as a direction — it's their life's work.
NVIDIA is also an extremely frugal company. After 2008, like many Silicon Valley companies, they cut perks, but they cut harder. There wasn't even free milk in the fridge. If you wanted milk for your coffee, you had to pay $1 a month to the "milk club," and the club would go buy milk at Costco and put it in the fridge. I remember this very clearly — but we don't do this at Sail, haha!
🎙️ Host
Taking that experience with you, what did you learn about using software to more efficiently utilize underlying hardware? What does that look like connected to today's environment?
👱♂️ Neil
GPUs are fundamentally better at throughput: give them a lot of tasks at once, keep the compute units fully loaded, and that's where efficiency peaks.
But for the past few years, AI has mainly been used as chatbots. To get answers to users as fast as possible, everyone has prioritized generation speed, which often doesn't keep GPUs fully loaded. GPUs struggle to optimize for both throughput and latency simultaneously. Because the dominant use case has been chat, everyone has prioritized low latency. I believe the most important shift in the coming year is AI moving from chatbots to proactive background agents. When that happens, designing the entire technology stack around throughput makes much more sense.
🎙️ Host
Can you explain technically why this throughput-latency tradeoff can't be broken? Why can't the same hardware have both?
👱♂️ Neil
This is a choice that almost every system faces: either process less at a time but faster, or process more at a time but slower. All of computer science deals with this.
On GPUs, the key is batching. We group many users' requests into a batch and run them on the GPU all at once. But a large batch means more total work, and more work per forward step, so any single user's request in that batch stays on the GPU longer because it has to travel with everyone else's.
To use an analogy: if you're going to downtown San Francisco, you can take the bus or call a private car. The private car takes its own straight line, from point A to point B directly. The bus serves more people, so its route has to make sense for everyone.
Diagram: prefill, decode, and batching
🎙️ Host
So the bus takes a slower route and has to stop for people to get on and off — that bus versus private car analogy is very apt. So your first optimization step is building the best possible bus on NVIDIA GPUs?
👱♂️ Neil
That means trying different parallelism schemes.
One important NVIDIA innovation is connecting GPUs with NVLink. It can split a large matrix multiplication across up to 8 GPUs to compute in parallel, then merge the results. This can significantly reduce latency for a single operation, but speed doesn't scale proportionally with hardware. With 8x the hardware, you might only get 4-5x the speed, because GPUs need to communicate with each other, and each GPU gets a smaller slice of work, so efficiency drops.
For the lowest possible latency, this is the only way to speed up, but I wouldn't choose it. I'd rather use expert parallelism or pipeline parallelism, and try to overlap communication latency with computation to hide the communication cost — which gives you much less room to maneuver on a low-latency server.
🎙️ Host
So, can NVLink be understood as a technology that only improves latency performance?
👱♂️ Neil
Yes. I'd say if you want to do the best possible low-latency inference, NVLink is basically essential. NVIDIA is excellent at low-latency inference. But I'm telling you, we don't actually care that much about low latency. So what does that mean?
I don't expect other vendors to replicate NVLink anytime soon. It's a technology that's hard to build, hard to scale, and hard to productize. But let's say I get chips from another vendor. Their base compute units are good, their matrix multiplication is strong, they just can't exchange results with other chips at high speed like NVIDIA can. They could still be a very strong "compute per dollar" option in my stack.
What I really care about is usually compute per dollar: how many FLOPS a chip delivers, and what it costs to own and run per hour. Some chips have significantly better FLOPS per dollar than NVIDIA, just weaker interconnect. My job is to find the right parallelism approaches to make these chips work well for inference. Tensor parallelism might not fit, since that's NVIDIA's strength, but other methods can be tried.
The Original Sin of Transformer
🎙️ Host
Before we leave the latency thread, what do you think of companies like Cerebras that can achieve extremely fast computation? Where does hardware focused on ultra-low latency go?
👱♂️ Neil
Cerebras, Groq, and a few companies just emerging from stealth have made an interesting bet. Instead of building another GPU, they've built accelerators focused on a different memory hierarchy — maximizing on-chip SRAM (static random-access memory) to serve as ultra-fast storage for weights and the KV cache (key-value cache, the dynamic memory a model retains for the current conversation).
There are two paths to adding memory to a chip:
The first is to put memory directly on the logic die — you tell TSMC you want this many megabytes of storage on the chip. The problem with SRAM is that it's area-hungry. Even if you turned an entire 800-square-millimeter NVIDIA Blackwell die into SRAM, you'd only get single-digit gigabytes.
The second path uses a different process. Instead of TSMC, you go to Micron, SK Hynix, or Samsung, who manufacture DRAM (dynamic random-access memory), where the focus is on capacitors rather than transistors. Standard SRAM uses a 6T cell — you write a bit and it stays, more or less indefinitely, which is why it's called "static." DRAM writes charge into a capacitor, and that charge starts leaking the moment it's written. You have to refresh every bit roughly every 50 milliseconds, which is like juggling billions of balls simultaneously. The upside is vastly higher density.
Stack DRAM into many layers and solder it around NVIDIA's logic die, and you get hundreds of gigabytes. Blackwell has 288 GB of HBM (high-bandwidth memory) surrounding it, while the SRAM on the logic die itself is probably around 500 MB. So DRAM and SRAM differ in density by several orders of magnitude — roughly three orders of magnitude.
What Cerebras saw is that SRAM density has no clear path to improvement, but it's incredibly close to the logic gates actually doing computation — compute units can fetch data at unbelievable speeds. Cerebras rates its third-generation Wafer Scale Engine at 21 PB/s, while Blackwell's HBM is in the ballpark of 10 TB/s. HBM has more capacity, but proportionally far less bandwidth.
So Cerebras simply ignored TSMC's 800-square-millimeter reticle limit and took the whole wafer, connecting each die to neighboring dies through the scribe lines, packing in as much SRAM as possible. Each wafer gets you roughly 50 GB, and stack many wafers together and you have nearly 1 TB of extremely fast memory that can move all the parameters of a large model like Moonshot AI's into and out of the logic cores in about a millisecond — hence the extremely high tokens per second.
🎙️ Host
So there's a path to 1,000 tokens per second. How do you see the future of this segment?
👱♂️ Neil
I think their endgame is some kind of hybrid — pairing Cerebras's extreme fast memory access with something else that has larger capacity. You can indeed fit a 1-trillion-parameter model like Moonshot AI's onto a stack of Cerebras wafers, but the KV cache is a different story. It grows as more people use it, it's always dynamic, and you don't even know how much you'll need — that depends on how many users you have and how many you want to serve.
🎙️ Host
Can you explain KV cache in the most basic terms?
👱♂️ Neil
Every token you send into the model stays in the context window as long as the conversation continues. If we've exchanged 100,000 tokens, the 100,001st token still has our entire conversation behind it, and the model references all that history to better predict what to say next.
So the KV cache is essentially a block of memory. For every token you input, you save a representation inside it, and the space it occupies is often larger than the model weights themselves. Think of it this way: model weights store fixed knowledge; the KV cache stores information that keeps changing during this conversation.
🎙️ Host
Is this why people sometimes feel the model degrading after a conversation gets very long — there's actually a technical reason behind it?
👱♂️ Neil
The KV cache is an exact representation of everything that came before, but models were primarily trained on conversations at the 8,000 or 16,000 token scale. They've seen some 200,000-token sequences, but that's not their strength. Making a model as smart at 200,000 tokens as it is at 10,000 tokens has been an open problem for frontier labs — it'll be a long war. The million-token context window has been talked about for years now; I think Anthropic was the first to get there. But when I use Claude Code myself, I hit /compact long before reaching a million — I don't think using the full length is necessarily a good thing.
🎙️ Host
So these ultra-fast, ultra-low-latency approaches are all ultimately constrained by this KV cache factor?
👱♂️ Neil
Yes. You can do whatever you want with the model weights — achieving unmatched performance there is entirely possible — but the KV cache will be a very large thorn in your side.
🎙️ Host
Looking three to five years out, what role will these ultra-low-latency chips play in the heterogeneous chip market? What share will they capture?
👱♂️ Neil
You should think of Cerebras and Groq as accelerators that are genuinely good at working alongside more traditional GPU-class devices whose key feature is having their own off-chip memory. Off-chip memory gives you capacity; on-chip memory gives you speed — we want to mix the two.
Pushing Transformer to the limit, say a million-token context, you have a compute-bound phase: the matrix multiplications in the MLP (multilayer perceptron) layers, where most of the model's world knowledge is encoded; and the attention layers, where the model dynamically adapts to the current conversation. At the limit, attention is typically memory-bound, while MLP is compute-bound when the batch size is large enough.
I'd say the "original sin" of Transformer is placing an extremely memory-bound layer right next to a compute-bound layer. It's hard for one chip to be great at both computation and memory operations. GPUs are fairly balanced in this regard, but you ultimately have to pick a side. Cerebras has extremely fast memory access, making it well-suited to hosting the MLP — the weights; GPUs can scale to very long contexts. So you might want to put attention on the GPU and the MLP on Cerebras.
🎙️ Host
You just explained a lot of core concepts very clearly. Can we spend a few more minutes on Transformer? For those unfamiliar, what problem did this 2017 innovation solve, what are its strengths and weaknesses, and will it remain AI's dominant architecture going forward?
👱♂️ Neil
Transformer enables models to learn efficiently from unsupervised data. Simply put, it finds patterns in sequential data. The crucial attention mechanism dynamically judges which parts of a sequence are most relevant. After every Transformer layer, the model re-evaluates which previous inputs matter most for the next prediction. This makes it well-suited to learning from all kinds of sequential data. And one of the most important kinds of sequential data we produce is language. That's why Transformer dominates language models.
More importantly, Transformer scales easily. It relies little on human-preset rules, instead trusting that if there's a pattern in the sequence, adding parameters will find it.
Transformer also benefited from experience accumulated in computer vision. One historical difficulty was scaling beyond thousands to hundreds of thousands of parameters — the range where linear models like support vector machines and traditional machine learning models topped out. Then came deep learning: computer vision models reached tens of millions of parameters, with the largest around 150 million, which was already considered enormous for vision models at the time. Now we casually talk about trillions of parameters. Transformer is the path that took parameter scale from millions to trillions.
🎙️ Host
If we treat data and compute as the two key variables for scaling, and we're especially good at getting more of both, does that mean Transformer stays around forever?
👱♂️ Neil
Data still has many unanswered questions. But on the compute side, Transformer is very good at consuming it: 10x more compute usually means continued performance gains. So far, the Scaling Laws have held. Transformer adapts to almost any dataset — it's a very strong general learning method.
It has another particularly useful property: it can represent any pairwise relationship — any token in a sequence can attend to any other token. You don't necessarily need every token to look at every token, but when you need to, the option is there. Until a better way to prune the search space is found, attention is a very good operation.
This also reminds me of another trick from the computer vision era. Karpathy used to have an old saying:
When you get a new dataset, your first goal should be to make your model overparameterized — overfit that data, prove that there are modelable, even memorizable relationships in it, prove that your learning algorithm works and that knowledge can indeed fit inside the model. Once you can overfit, you can compress, and compression is how you get generalization. You don't actually want to memorize the data in front of you; you want generalization. So after overfitting, you can work backwards to find the general pattern that fits the data with the fewest parameters.
🎙️ Host
How do you predict the future of data? How important is data in the whole story — unpack that.
👱♂️ Neil
I often say that the internet gave AI a one-time data subsidy, and it was almost free. High-quality text is roughly 30 trillion tokens; if you relax standards, you can get to 300 trillion. Models have already read the entire internet multiple times over. There isn't much usable human data left.
The next stage in my mind is getting models to improve themselves through reinforcement learning environments. In fact, we may not even benefit much more from random users interacting with AI. We used to value the thumbs-up and thumbs-down clicks from people using ChatGPT, but our median-served model is already far more advanced than a random human rater. Unconditional human preference isn't worth much anymore — you want expert-level preference.
So the future of data is giving models a difficult, verifiable task, letting them run in an isolated environment with problems they can make progress on and metrics to measure whether they're actually making progress. Programming and math both fall into this category, and increasingly we're just handing an Agent a computer and letting it work like a human, then giving feedback based on whether it's moving toward the goal. That environment itself becomes the data. I don't think this is a particularly unique insight, but from what I've seen so far, it's very effective.
🎙️ Host
Do you think this path will last a long time? If we think of internet data as one big chunk, will there be another big chunk that has its moment in the sun, and once we've consumed it, we move on to the next source?
👱♂️ Neil
I think this runs deeper. One viable path to AGI is continuously expanding the specific tasks models are good at, until there are no obvious weak spots. The prerequisite is that task outcomes can be verified, and that models can score themselves. With this mechanism in place, models can keep improving across different tasks. The investment patterns at frontier labs bear this out: they used to spend heavily on data, now they're pouring more money into building reinforcement learning environments. These environments let models continuously self-improve on verifiable tasks — Recursive Self-Improvement, or RSI.
🎙️ Host
Back to the software thread: what have you already done? What do you want to do next?
👱♂️ Neil
Surprisingly, people think the kernel work is done. There are indeed exceptional people like Tri Dao who've written excellent kernels — modern deep learning and Transformers are built on FlashAttention.
Translator's note: Tri Dao is a Stanford PhD in computer science, currently an assistant professor at Princeton University and chief scientist at Together AI. He is the lead author of FlashAttention — an algorithm that redesigns how attention reads and writes memory, without changing the computational result, only how data moves between GPU memory hierarchies. This dramatically speeds up Transformer training and inference while significantly reducing VRAM overhead. Neil mentions him here because FlashAttention is the most famous achievement at the intersection of "model algorithms" and "chip memory efficiency," and serves as the starting point for this discussion of kernel engineering.
But stray even slightly from that happy path — say a new model changes how it encodes positional information, modifies RoPE (Rotary Position Embedding, a method for letting the model know each token's sequential position) — and your existing kernel suddenly doesn't apply, needs patching. We're not yet at the point of inventing new kernels from scratch, but the ability to rapidly modify existing ones is essential.
"Kernel" is a broad term for any program running on a GPU. In the past each kernel had a narrow purpose: matrix multiplication was one, adding two tensors was another. Now we're increasingly fusing them — do a matrix multiplication, add it to another multiplied matrix, maybe combine it all into one kernel without writing data back to DRAM only to read it back for the addition.
🎙️ Host
"Writing more efficient kernels" sounds like exactly the kind of engineering AI should be especially good at. Maybe we're heading that direction, just not there yet. But if we're not there yet, why is this still done by humans today?
👱♂️ Neil
I don't want to speak for Tri, but what I learned from him: kernels don't necessarily need to be handwritten anymore. We're more designing kernels on a whiteboard. First figure out what the machine should do, then tell the model in concise natural language: what are the inputs and outputs, how should the task be distributed across the GPU. Let the model handle the actual code.
I'm not sure why models aren't especially good at this yet, and I don't think it's any kind of moat. But I believe in six months there will be much better kernel engineering models, and labs will tell you they've already largely automated their own kernel engineering.
Anthropic R&D Automation Index. Scored on Epoch AI's AL0 (no AI involvement) to AL5 (fully autonomous AI) automation scale. As of August 2026, Claude-led work (AL4: end-to-end task completion from high-level instructions with human oversight) accounts for 26% of R&D; work at "collaborative" level (AL3) or above exceeds 90%. Data source: Anthropic official
🎙️ Host
So for "software as advantage": if software is about pushing hardware utilization as close to Lightspeed as possible, will it stop being an advantage for a company like yours over time?
👱♂️ Neil
Yes. Models like Mythos or GPT-6 Astra are a rising tide that lifts all boats. I also don't think it makes sense to specifically train a model that's only good at kernel engineering — kernel engineering is just a tiny subset of programming overall. Maybe you can inject some privileged information in the prompt to help the model write better kernels, but broadly speaking, we're all downstream of frontier models on this capability.
🎙️ Host
There's a pendulum in energy history I've always loved. How much energy in a lump of coal can we actually extract? Much of energy history is pushing that ratio from 10% to 95%. If Blackwell is that lump of coal, where are we today?
👱♂️ Neil
When a GPU is doing what it loves most — large-dimension matrix multiplication — our optimizations are already quite efficient, hitting 70-80% of peak utilization, near saturation. That's pretty good. What's limiting it isn't software, but power. NVIDIA's stated peak FLOPS is a bit optimistic because of power throttling — you'll never actually hit that number.
But most of the time running Transformers, you're not on that happy path of large-batch matrix multiplication. So our work is essentially building an engine around the chip to keep the GPU fed with large batches of work at every moment.
A major shift in the GPU space over the past year: the programming target is no longer just one GPU, but an entire rack, even an entire data center. NVIDIA now delivers complete rack systems. The latest GB300 ships as NVL72 with 72 GPUs per rack. Now companies are competing over who can program rack-scale computers more efficiently. I think this approach represents the future of computational efficiency and speed. NVIDIA is doing very well here: for lowest latency, you should choose them; for highest throughput, probably them too.
🎙️ Host
People say everyone is pulling out all the stops to get more of the best chips like Blackwell. If I'm willing to accept a slightly worse, or even significantly worse chip, what does that market look like?
👱♂️ Neil
First point: NVIDIA takes a long-term view on all its chips. Faced with massive Blackwell demand, they could have raised prices like suppliers in the past and let the supply-demand curve intersect. But NVIDIA understands that if only the deepest-pocketed customers buy all the chips, and that customer accumulates too much power, it could hurt NVIDIA in the long run. Compute is power today, so they're quite strategic in allocation.
Relationships matter enormously too. No one wants to take an order from a startup that's been around for a few months saying they want to rent 10,000 Blackwells for 3-5 years — who knows if you can actually pay. To get compute today, you either have excellent relationships or very strong financial backing.
As for other chips, I wouldn't call them "inferior chips." There's no absolute good or bad in chips, only price. At the right price, I can find a use for anything. That's one of our company's core beliefs.
Take AMD: broadly good chips, the difficulty is people don't know how to program them well. NVIDIA has done a decent job optimizing for its own chips, and we can squeeze out a bit more alpha; there's far more to do on other chips because the vendor has put less effort into out-of-the-box kernels. The common perception that "AMD is worse than NVIDIA" — I'm happy for people to keep underestimating it, so I can buy as much as possible.
Though this is already becoming less true — publicly Meta and OpenAI have both bought massive amounts of AMD, and AMD supply is getting booked up too. But there's a long line of emerging companies behind them: Etched, SambaNova, d-Matrix. Their biggest challenge is scale, whether they can get enough wafers from TSMC to mass-produce. When new chips come out, I want to know immediately and evaluate whether I can buy a substantial portion of their supply.
🎙️ Host
So this is essentially arbitrage? You're better than others at extracting performance from chips no one is paying attention to, then reselling with margin.
👱♂️ Neil
We're very good at working with multiple chip architectures simultaneously, and willing to mine performance where others aren't looking. Another advantage: we can rebuild our stack around new chips very quickly. We don't have much legacy baggage, and we have creative, fast-moving data center partners. If TPU makes sense, use TPU; if Trainium makes sense, use Trainium; if neither works well alone, put them in a heterogeneous system.
Every chip has its own strengths. Our job is to find them and exploit them fully.
🎙️ Host
Let me insert an investor concern.
Look at memory stocks, or one of my favorite charts recently: semiconductors as a percentage of the S&P 500. Historically 2%, 3%, 4%. Now it's 19%, 20%, 21%. If you know market history, you see lots of things that spiked to some crazy short-term high, then collapsed back to long-term averages. Micron, SK Hynix — these companies have made people a lot of money, but the long-term consensus is still that "compute" is ultimately a commodity, that it shouldn't permanently occupy a quarter or even a fifth of global market cap. Everyone admits extreme scarcity right now, but many people secretly think "we'll figure this out eventually, these things will return to where they were." What do you think of this narrative?
👱♂️ Neil
I'm less a student of market history than a participant in it. I was born in 1997. My mom worked at Intel around the dot-com bubble in 2000, and I remember when Cisco was the most valuable company in the world, with Intel right behind it.
The biggest difference is that a lot of the networking equipment investments back then were betting on future demand that never materialized. AI consumption isn't like that. People buy tokens because they can use them right now. Tokens don't get stockpiled; they get burned immediately.
This is also very different from two years ago. During the Hopper shortage in 2023 and 2024, all the money went into training, which is inherently speculative. Now everyone's setting spending caps on Claude Code, and I think inference spend will go up monotonically, with no speculative component in it.
The Scavenger Strategy: Scavenge Chips First, Then Scavenge Power
🎙️ Interviewer
Back to hardware. We've moved from chips to racks to clusters, and the next layer is data centers. You mentioned some partners doing cool things—how do you see data centers today and in the future? To support this massive inference demand, this is clearly a link that needs continuous innovation.
👱♂️ Neil
There's been a thread running through this whole conversation: what's the difference between training and inference? Why was everyone designing around training two years ago, and why is the shift happening to inference now? I think the most conservative players in the entire AI stack are definitely the infrastructure folks like data centers, and even more conservative are the chip infrastructure companies like TSMC.
In the past, AI data centers were built primarily for training. Training demands higher network bandwidth and larger cluster scale, so training clusters can usually do inference, but inference-optimized clusters can't necessarily do training.
Data centers have diseconomies of scale in a sense—deploying 100,000 GPUs in one location is much harder than 10,000, and 10,000 is harder than 1,000. Everyone talks about megawatts (MW) and gigawatts (GW) now, but in the United States, you basically can't build a 1-gigawatt data center anymore. 100 megawatts is getting increasingly difficult too, unless you're one of a tiny handful of special customers. 10 megawatts is probably the edge of what's feasible today, and 1 megawatt—I think those are everywhere.
So there's plenty of total power in the market, but it's very distributed. The people building training centers aren't interested in this, because nobody wants to do training across multiple data centers. The market hasn't fully adapted to this yet—data center developers are still running around looking for 100-megawatt or 10-megawatt sites. But more and more people are realizing that inference is actually well-suited to distributed 1-megawatt data centers. We agree with that, and we're happy to buy little slices of compute all across the United States and stitch together our own inference network.
🎙️ Interviewer
Give us a visceral sense: physically, how big are 1-megawatt versus 10-megawatt data centers?
👱♂️ Neil
Liquid cooling has made this pretty wild. For 1 megawatt of compute, you'd normally imagine a giant warehouse. Now it fits in about 8 racks, each roughly the size of a refrigerator. Picture 8 refrigerators lined up—that's 1 megawatt.
So what happens when you stand up 1,000 small data centers instead of one 1-gigawatt mega-facility?
You often don't have power redundancy, no diesel backup generators—those are expensive, we cut all of that. A lot of times you don't even have redundant networking. Find a place with good power access, single feed, run one fiber line in—not three fibers with failover and SLAs. These data centers will go down occasionally. Some sites might only have 95% availability, and that's completely within my expectations.
🎙️ Interviewer
A data center with 95% availability would be fatal for anyone else. Why are you buying it?
👱♂️ Neil
Two reasons. First, we have a very robust control plane. As long as the failure isn't correlated with other data centers, it can handle any single point of failure and just move the load elsewhere. For me, 95%, 98%, 99% availability—the goodness is roughly linear.
Second, the workloads are asynchronous. We're serving long-horizon agents. When a request fails, you just resume on another GPU. The agent has been running for an hour; if one round takes an extra minute or even ten minutes because of a GPU swap, customers usually don't care. So we're very explicit with customers: average throughput is very competitive, but P99—the slowest 1% of requests—we can't guarantee latency. In exchange, they get very low cost.
🎙️ Interviewer
Let's talk more about power. What interesting or innovative things are you seeing? Where do you think this is headed?
👱♂️ Neil
I said I want 95% availability data centers—well, at the right price, could I take 80%? Probably.
I grew up in California. I love solar and wind. I think U.S. wind and solar are massively undervalued. The problem has always been intermittency: you've got continuous baseload paired with an intermittent power source, what do you do? I think we're actually not far from solving this. I can absolutely tolerate outages measured in days or even weeks, which is a data center operator's worst nightmare—no wind, clouds overhead, fog hanging in the valley for a while, and you're down for an extended period.
But this is highly predictable. I model the weather, figure out when the data center will be offline, and move the load somewhere else ahead of time. The trick is, I can get power that nobody else would touch because dealing with these outages is too annoying. And if the chips are cheap enough—probably not NVIDIA racks—the idle capital cost doesn't bother me.
🎙️ Interviewer
I heard you call this whole approach the "scavenger strategy." Unpack that for us.
👱♂️ Neil
We scavenge chips first, then scavenge power for those chips.
The logic behind both is the same: I don't want to compete with Anthropic or OpenAI for compute, because I can't beat them and I don't need to. I can't get large, concentrated supply, but I can aggregate distributed supply from everywhere. Over time, this distributed factory can build a very strong cost advantage. We want to build the best steel mill in the world, but instead of one giant factory, we connect many small steel mills together.
🎙️ Interviewer
So you're basically doing a real-life version of Factorio (a 2D production and construction simulation game developed by Czech indie studio Wube Software, where players use conveyor belts, robotic arms, assemblers, and power systems to build automated production complexes while fending off hostile creatures).
👱♂️ Neil
Very much so.
🎙️ Interviewer
Following this logic, how deep can vertical integration go? At one extreme, you own everything—power, data centers, custom chips, software, then sell finished tokens to users. At the other extreme, you're extremely asset-light, just a coordination layer. What do you think?
👱♂️ Neil
I actually have two selves inside: the CEO who needs to keep the company running and growing sustainably, and the founder. The latter is more imaginative, but from the CEO perspective, we need to be more pragmatic.
The capital required to "own everything" is insane. Software has high leverage, so you have to start with software. As for whether we eventually own generation assets or just sign power purchase agreements with utilities—I lean toward letting others specialize in what they've historically been good at, and only bringing it in-house once we've grown to a scale where we've earned the right.
There's still a lot of efficiency to be gained across the entire stack. The key is breaking suppliers' fixed assumptions about what customers need. We're betting on the largest compute market in history, with tens of billions to trillions of dollars going into inference eventually. So it makes sense to customize hardware and software specifically for inference. My job is to identify every link worth customizing. If partners can do it, we let them; if they can't, we do it ourselves.
🎙️ Interviewer
Pulling back to look at the whole system—software, hardware, energy—where do you think the biggest efficiency losses are in producing useful AI tokens today? Rank them for us.
👱♂️ Neil
I think compute scaling is actually very efficient. Give me more FLOPS and I'll use them, and we're already pretty disciplined about it. Modern models rarely exceed 10% density—meaning no more than 10% of all experts are activated at once—and frontier models are closer to 1%. The MoE (Mixture of Experts) side is squeezed pretty well; I don't think there's much waste there.
Where we're bad is attention and its memory usage, specifically that KV cache is nowhere near compressed enough. Looking at its information entropy, it doesn't deserve the space it takes. We're storing several kilobytes per token now, probably off by 1-2 orders of magnitude. I don't know how far frontier labs have gotten internally, but DeepSeek has been publishing openly on shrinking it, making fast progress—about one order of magnitude per year—which itself suggests there's still a long road ahead.
These are still local optimizations. Zooming out, I don't think the industry is scheduling compute effectively at all. NVIDIA shipped 5 million Blackwells this year. Where did they all go? Are they all being used? I doubt it.
A lot of compute disappears into private pools that never see the light of day—those GPUs just sit idle. The silicon and power are already invested but not actually being used. I want to orchestrate the world's compute as a shared resource. I suspect people love making jokes about xAI cluster utilization, but the reality elsewhere is much worse. GPU clusters are generally very underutilized; piles of GPUs are just lying in warehouses.
Figure: VentureBeat, May 8, 2026. Citing Cast AI's "2026 State of Kubernetes Optimization Report": enterprise GPU average utilization is around 5%; procurement cycles and depreciation make it difficult to release idle GPUs.
🎙️ Interviewer
And your business is essentially attacking this efficiency loss head-on. What about fabs? Everyone's wondering how memory companies, TSMC, Intel, and others will expand capacity. Will more production shift to U.S. soil? If we could snap our fingers and make existing chip inventory 100x, tokens would presumably get much cheaper. So I'd love to hear your take on this piece.
👱♂️ Neil
The links constrain each other. Even if you could instantly double existing chip supply and solve TSMC's capacity bottleneck, you'd immediately hit the next bottleneck.
But what's interesting to me is what fabs consider "must-deliver" versus "invariants that customers will always want." I think that's more of a fluid relationship than it appears. The spread between the worst and best chips on any production line is huge. TSMC works very hard to tighten process corners, making the worst chip as close as possible to the best one, adding lots of controls for that—and maybe I don't need those controls. Maybe I'm willing to find a place for that worst chip, willing to accept far more defective units.
The Most Important Hiring Traits: Curiosity and Passion
🎙️ Interviewer
Let's talk about how you've designed this company. You mentioned learning a lot from NVIDIA—can you walk us through how you're building the team and company while staying true to that North Star?
👱♂️ Neil
We place a lot of value on "limit thinking." When we first start supporting a model, efficiency is usually low. But we don't just look at the present — we think about what we'll be able to do in a month, six months, a year. We also don't treat the capabilities of existing machines as fixed. Even with Blackwell, whenever we hit a bottleneck, we dig in and document the cause. On one hand, this lets us feed back to NVIDIA or our partners; on the other, it helps us decide what chips to buy going forward.
The patterns we believe are durable for our company get baked into every future decision.
We also emphasize collaboration heavily. One of the most important traits we look for when hiring: someone who is both a good student and a good teacher. Many people on the team were TAs in college and genuinely enjoy sharing knowledge that way. We do whiteboard sessions constantly, and I think this academic atmosphere — where everyone has something to teach and something to learn — is incredibly important to us.
🎙️ Interviewer
What kind of people do you want to hire? In three years, machines will take on more work. What traits will give people resilience in that environment?
👱♂️ Neil
Curiosity. 100% curiosity. There's one thing I can't teach: a love of performance, the willingness to burrow into every microsecond of a machine's execution and figure out exactly what's happening in that moment.
For performance engineers, that's the most important trait in my view, and it's what I'm actually looking for when I hire. I don't weight a lot of AI experience heavily, and I completely disregard CUDA experience — that's actually a massive red herring. CUDA and GPU concepts have changed so much in the past five years that demanding ten years of experience is meaningless. I can teach the technology, but I can't teach a love of performance engineering. That's what I'm after.
Open Source Won't Go Away — If Someone Steps Back, Someone Else Steps In
🎙️ Interviewer
Can you go through the major labs one by one, and then talk about the relationship between closed-source and open-source camps, and what you think is happening now and what will happen?
👱♂️ Neil
Labs pay an extraordinarily high premium to be ahead by 3-6 months. I think it's probably still worth it — what OpenAI and Anthropic are doing makes complete sense.
Distillation is a sensitive topic, and it's central to the relationship between closed-source and open-source models. Some people argue that "distillation is theft" because using a frontier model's outputs to train a new model amounts to stealing its capabilities. But even if you never actively scrape Anthropic's data, AI-generated content on the internet is only increasing. How many new repos on GitHub over the past year were created with Claude Code? If you train a model on that code, is that distillation?
I wouldn't be surprised if you could train a Fable-level model using only good open-source code from GitHub. If we accept that users own their outputs from interacting with AI, and many of them upload those outputs to GitHub, then there's a latent form of distillation that will persist long-term. In my view, stopping the diffusion of information or model capabilities is impossible. It will happen — the only question is how fast.
🎙️ Interviewer
If Scaling Law holds long-term, does being ahead by 3-6 months create value? Can they charge a premium for their tokens over extremely cheap open-source tokens?
👱♂️ Neil
Possibly, but I'm not sure how long that premium lasts.
Enterprise deployment doesn't move on a 3-6 month cycle. Many companies might still be on Opus 4.6 or 4.7, with a mountain of questions about any change. I don't think this race produces a winner, or even that it has a day when it's decided. It's an ongoing process. Fundamentally, open source won't disappear. Whichever leader steps back and leaves a vacuum, someone new will fill it. The incentives here are too strong, and training a frontier-level model gets easier every day.
🎙️ Interviewer
What future do you want to see?
👱♂️ Neil
I want every company, even every user, to build their own harness and make the agent truly theirs. This "customization" doesn't have to come from fine-tuning weights — it's more about in-context learning, and the condition for that is cheap tokens.
The fact that machines can think is remarkable in itself. We should put that capability into as many hands as possible.
🎙️ Interviewer
You sit in a very unique position with a very unique perspective. Compared to friends who are similarly well-informed and deeply engaged in this space, what views do you hold that would make them think you have three heads?
👱♂️ Neil
Mostly around chips. When I talk about custom silicon, I'm basically always talking about how to work around HBM shortages by offloading memory more aggressively to flash and other forms. Everyone on the team knows I've been beating this drum — what would need to change in model architecture to make massive KV cache offloading to flash actually work? I'm always whiteboarding this.
In the inference world, our view differs from others on "what becomes possible if you design an entire serving system around 1-10 tokens per second?" That's our North Star. One layer out is the bigger question: how do people consume a trillion tokens per day?
That's the capability we want to build for the world.
🎙️ Interviewer
How much is a trillion tokens? Give people a feel for it.
👱♂️ Neil
At OpenAI's pricing, running on GPT-5.5 or 5.6, it would cost at least $5 million.
🎙️ Interviewer
What does a world look like where "every person consumes $5 million worth of tokens per day"?
👱♂️ Neil
We need to improve per-token cost by at least 3-6 orders of magnitude, bringing $5 million down to $5,000, before you probably have customers. In fact, for certain model sizes, a trillion tokens is already approaching the tens of thousands of dollars — that's already in the realm of what you can imagine for a single task run.
🎙️ Interviewer
Would you worry that ordinary people simply won't use it this way — that there isn't actually that much demand for intelligence in the world?
👱♂️ Neil
I will never believe that. There is always demand for intelligence in the world. The entry point to it is the challenge our product community needs to solve. I'm not a product person — I can't claim to have the best vision — but I want to ensure that those who can use it are never blocked by "my free users can't afford this" or "I can't give out this many tokens." I keep hearing this from customers, and we want to solve it.
Losing TSMC Wouldn't Be as Bad as People Think
🎙️ Interviewer
Flipping the question: what consensus do you think is wrong?
👱♂️ Neil
You can't talk about this without talking about NVIDIA. In the near term I'm bullish on them, and I wouldn't bet against them — this company always finds new ways to grow.
But one thing that surprises people: from Hopper to Blackwell to Rubin, comparing BF16 multiply unit power efficiency on an apples-to-apples basis, it hasn't improved that much. Step back further: TSMC 5 nm, 4 nm, 3 nm, to 2 nm — the power efficiency improvements haven't been that dramatic either.
The consequence is that people have lost their minds with geopolitical anxiety, worried about what happens if we lose TSMC. My view is: it wouldn't be that bad. Supply would absolutely take a hit, but the best Western process, say Intel's, in the worst case is probably only a ~2x gap in power efficiency. That's far smaller than chip-industry discourse would have you imagine.
🎙️ Interviewer
What in the AI world is most interesting to you that's not directly on your path?
👱♂️ Neil
We're entirely downstream of models. We just do our best to predict where the model builders are heading, then build our serving architecture accordingly — software and hardware choices follow.
The most interesting game is still how profound the fact of machine thinking itself is. Sparse attention versus dense attention, switching data types — these all matter enormously. We used to train with BF16, and now we can use lower-precision formats like FP8, FP4. It seems like an arbitrary choice, but it affects what chips I can use in the future, how I should build hardware, how I think about compute.
🎙️ Interviewer
Assume there are 100 entrepreneurs in this room who want to start new compute companies — specialized chips, systems, racks, that kind of hardware. What advice would you give them?
I'm not asking which technical bet to make. I'm asking how they should position their companies to have a better chance of succeeding in the future of compute. I suspect every approach will be tried, and some will succeed — which would be a good thing for the world.
👱♂️ Neil
The most important thing is understanding supply chain bottlenecks. You have to first convince me or an investor: you know the key constraints that determine modern chip supply — TSMC wafer capacity, HBM capacity, advanced packaging, power — and you have a solution for each. At the end of the day, building chips is about finding opportunity within these constraints.
You build a chip because you believe NVIDIA made some hard-to-change choices. They're not perfect — they just balance things very well. So you need to get sharp, pick a direction, and say: "I think they underestimated how short HBM would be, and we're going to push hard in the opposite direction."
🎙️ Interviewer
Why?
👱♂️ Neil
Because there's no easy way to suddenly build a lot of memory fabs. The folks in Boise (Micron, headquartered in Boise) don't love sinking massive capex into a cyclical business — they've been burned on that many times.
🎙️ Interviewer
Could "HBM shortage" force the world to work around it by making everything else more efficient?
👱♂️ Neil
I think they'll make everything else more expensive. It's like how iPhones cut memory configurations and raise prices — we just have to live with it.
🎙️ Interviewer
Why doesn't NVIDIA just go all the way and sell tokens themselves?
👱♂️ Neil
NVIDIA is smart — they don't compete with their customers, and they play the long game on everything. Why don't they just build their own NeoCloud and sell compute out the back door?
Because Jensen Huang is exceptionally good at making friends into billionaires. He turned CoreWeave into a multi-billion-dollar company — why destroy that goodwill? He wants a diverse ecosystem of NeoCloud and inference providers, all competing to create demand for NVIDIA. That way, if any one of them decides to vertically integrate or switch to AMD, there are three others hungry to take their place.
There's Beauty in Understanding Every Piece of the Puzzle
🎙️ Interviewer
I ask every guest the same closing question: What's the kindest thing someone has ever done for you?
👱♂️ Neil
My first thought is all the mentors I've had over the years.
They're a precious group of people who carved out significant time from their own schedules and made "making sure you actually understand something" their own problem. The people at NVIDIA planted in me a love of performance engineering.
My undergraduate advisor was the same way. I'm an extremely impatient person. I remember during my sophomore year, I once showed up to his office hours and said, "I want to work on AI chips. I know exactly what I want to do. Why waste time on these networking and operating systems fundamentals?"
He laid out the entire technology stack for me. At the time, I only wanted to study one tiny point in the system. But he told me that very few people truly understand the full picture — from gate-level silicon all the way up to how you build excellent internet-scale services — and that I should spend my life pursuing that complete understanding. He showed me what a beautiful thing it is to understand every piece of the puzzle.
That stayed with me for a very long time.
Recommended Reading
Join Our Membership Community
