Thinking is a mechanical process, AI are going to do it｜5Y View

Recommender

Peter, Partner at 5Y Capital

The operating space in the AGI era is vast, but the truly long-term worthwhile bets are scarce. On the timeline of accelerating intelligence evolution, I believe the most important ability for founders is to make differentiated (and right) technical bets.

Nat and Daniel discussed some of the most important questions of today, such as the importance of context length. They see the bet on long-context windows as a canonical example — not just providing a good model, but delivering innovation along one axis that is orders of magnitude beyond what anyone else has offered to date, and one that actually seems to work.

What other controversial bets today will matter for AGI in the long run?

This article is republished from the WeChat public account "Fan Yang"

The article we're sharing today comes from the well-known "technology and business strategy" blog Stratechery, by its founder Ben Thompson. The interview subjects, Nat Friedman and Daniel Gross, are an investing duo who very much resemble the "Marc Andreessen and Ben Horowitz of the AI era" — the latter pair became Silicon Valley's preeminent investors in the post-2008 mobile internet era through their founding of A16Z.

These two are remarkably energetic and have their own original perspectives on the world. Starting in 2017, Nat and Daniel began partnering on AI investments, establishing an organization called AI Grant — both a "distributed AI lab" and a novel type of investment fund. This kind of "technology and research-driven organization with venture capital capabilities" will only become more common in the AI era.

The most important points Nat Friedman and Daniel Gross make in this article are:

1. "Superhuman reasoning entities" are the next big thing. If someone can create something that actively reasons and thinks through problems in any domain of its choosing, just as humans do, they will become the industry leader. This is analogous to how Google created the PageRank algorithm in its early days, then built out a complete product, business model, and set of values around that technology, ultimately staying ahead of everyone else.

2. Path dependency on Transformers / large language models / scaling laws is real — the same pattern existed in the development of past computing and internet technologies. But there are also innovations not based on transformer architecture happening now; interesting and promising research areas are emerging, even if they're hard to make sense of.

3. The real way to create value and true differentiation is through a product. The most valuable "eval" is user evaluation.

Reasoning About AI: An Interview with Nat Friedman and Daniel Gross

Author: Ben Thompson

Editor: Fan Yang

Date: February 29, 2024

Good morning,

I'm delighted to welcome Daniel Gross and Nat Friedman back to Stratechery for our ongoing interview series — this is our sixth conversation (we previously spoke in October 2022, December 2022, March 2023, August 2023, and December 2023).

This series is somewhat unique on my blog Stratechery in that my interview subjects are typically not investors; however, this series began in October 2022 when Friedman and Gross were launching a grant program, and we came together because we shared a view that there simply wasn't enough activity or discussion around AI. A month later, ChatGPT arrived and the world changed dramatically — including for Friedman and Gross as two of the leading investors in the space. So I've found it valuable and popular to continue this conversation series between us, but please keep in mind that Friedman and Gross may have invested in some of the companies we discuss.

It's worth noting that the past month has been particularly significant in terms of "AI world" news. We did our best to cover everything in our conversation, from Gemini to Sora to Groq, Mistral, and Nvidia, as well as our usual philosophical questions about the AI space and what might come next — particularly the current moment where tech companies are combining different types of models with transformer architectures in the pursuit of reasoning.

Interview

Topics covered:

Vesuvius Challenge | Sora and the Attention Pyramid | Groq and the Speed Step-Change | Devices and Robotics | Gemini 1.5 and Large Context Windows | The Gemini Disaster | Mistral, Evals, and OpenAI

Vesuvius Challenge

Nat and Daniel, welcome back to Stratechery.

DG (Daniel Gross): Thanks for having us, Ben.

NF (Nat Friedman): Glad to be back.

Ben: I feel like it's been forever since I've had you on, which is how you measure things now given everything that's happened in AI — it's actually only been about two and a half months. And I rushed to get you back on considering the flurry of announcements from AI companies last week, even before the Google "Gemini incident" we'll get to later (referring to Google releasing their most advanced AI model Gemini, which generated images with heavily "politically correct" content tendencies, drawing fierce criticism). But before we get to AI, Nat, give us an update on the Vesuvius Challenge.

Editor's note from Fan Yang: The Vesuvius Challenge's official website is scrollprize.org. Mount Vesuvius in Italy (near the ancient Roman city of Pompeii) erupted 2,000 years ago, burying and carbonizing numerous papyrus scrolls from the Herculaneum library. After archaeologists discovered these ancient scrolls, scientists attempted to unroll and decipher them, but lacking appropriate technology, destroyed some scrolls while others crumbled into fragments. Silicon Valley entrepreneur Friedman, after watching a presentation online by University of Kentucky computer scientist Brent Seales on restoration techniques for the Herculaneum scrolls, became deeply interested and proactively proposed co-launching this challenge, soliciting technical solutions from around the world to recover the scrolls' contents — particularly using AI to solve this problem.

NF: Oh my god, okay. Poor Daniel has to endure me repeating this monologue all the time, but I just get so excited talking about this project.

Ben: Refresh our memory — what is the Vesuvius Challenge, and where do things stand now?

DG: Ha, and when did it start?

NF: Thank you for the reminder, Daniel. The event happened in 79 AD, when Mount Vesuvius erupted, sending a massive surge of hot gas and hot mud — a pyroclastic flow — that covered the entire Bay of Naples and completely buried the towns of Pompeii and Herculaneum. As it turns out, just outside Herculaneum, there was an incredibly luxurious villa built by Julius Caesar's father-in-law, extremely spacious and opulent, that was also buried under 60 feet of mud.

When farmers accidentally discovered this buried villa while digging a well in the 1700s, and as they excavated underground through walls and rooms, they mostly looted various statues and artifacts. They also found a set of strange gray objects — carbonized lumps — that turned out to be papyrus scrolls. What's unique about this is that no other ancient library from that period survived. If you leave papyrus in the humidity of the Bay of Naples, it tends to completely rot away within about 100 years. So the ancient writings we have were all preserved through chains of monks copying them sequentially during the Middle Ages. No complete ancient library had survived.

That's basically it, but the mystery and difficulty is that these carbonized scrolls that remain cannot be opened. They're incredibly fragile — I've experienced this myself — if you carbonize papyrus and try to unroll it, it crumbles into pieces in your hands. You can't read them. The very act of trying to open them destroys them. So since their discovery in the 18th century, there's been nearly 300 years of attempts to open and read these artifacts.

Early last year, Daniel and I decided to launch a project to try to crack this problem. The general approach was to scan the scrolls at really high resolution in 3D without opening them and without damaging them. To do this, you need to use a particle accelerator to get very high resolution, and then hopefully through these scans, reduce this problem to a software problem where you can use computer vision and machine learning algorithms to virtually unwrap and read these scrolls.

A little over a year ago, or less than a year ago when we launched the Vesuvius Challenge, I really didn't know if it would work. But it absolutely seemed worth trying, and we decided to launch it as a global competition to get more smart people involved. I just thought this was so incredibly cool, and it seemed like almost nobody knew about it — that there were thousands of ancient scrolls that couldn't be opened — and we thought, maybe we could motivate 1,000 people at home on their laptops to crack it. And that's basically what's happening now.

Just last month, we were incredibly excited to announce that a $700,000 grand prize went to a team that had just completed this landmark work — they were able to reveal and read a large portion of a scroll. We have 2,000 Greek characters that have never been seen before, completely new to the world.

So the project worked, it succeeded. Now, I think what we need to do next is scale this up. What we can currently read is only 5% of one scroll, and we have hundreds of scrolls remaining, with possibly thousands more underground. So what we need to do is scale the algorithms up so we can read through entire scrolls one by one, and then hopefully read all the scrolls.

Ben: Do we have high confidence that we'll actually be able to read these scrolls?

NF: I'm very confident now. Look, we've only read 5% of one scroll, there's still a long way to go.

Ben: In a way, this is like a zero-to-one moment.

NF: Yes, exactly. If we can read 5%, then we can likely read an entire scroll. If we can read one scroll, we can probably read most of the others too. So yes, I've gone from "oh my god, I don't know if this will work" to "this is definitely going to work, it's just a matter of time and how efficiently we can do it."

Ben: I mean you have to be careful. You've gone from low expectations and high hopes to high expectations. You might be setting yourself up for disappointment.

NF: (laughs) That's possible too!

Ben: This sounds incredibly exciting. You've mentioned in several interviews, and another cool thing is the speed — basically less than a year, which is a good sign of our ability to figure out what the problem is and scale it up.

NF: Yes, I mean, if you could run it quickly, with what we know now, you could do all of this in a month. So most of the work is figuring out what to do, figuring out what the approach is, where to start, and what algorithms work well. Now that's all accumulated knowledge we have, all the code is on GitHub, it's all straightforward. The data is public, we have a really excellent group of people working on this. The next important step is what we call Auto Segmentation. Basically, you have this 3D scan of a scroll, and you need to trace the spiraling papyrus surface within it, and this process is still fairly labor-intensive. We're basically doing it with human annotation — people go in and manually click on cross-sections of the papyrus in the X-rays.

Ben: And then you select the ink markings or other substances on top to distinguish them.

NF: Yes, exactly. As it turns out, machine learning algorithms are able to pick up subtle patterns in the ink. The major breakthrough came when Casey Handmer, using his own eyes to manually inspect the data — what he calls persistent direct observation — was actually able to notice some ink markings emerging.

Ben: And then you let the algorithm work on that section?

NF: Yes. It turned out he could see cracked mud patterns, and you could see this represented about 2% of the ink, and then there's 98% of the ink that's very difficult to see, and the machine learning algorithm learns over time because you're adding more ink regions to the training data. But now, with machine assistance, you can reveal the entire ink region — it's a very simple model, it just needs good training and the right architecture, and it works very well. So I'm quite confident that this year we'll have a whole book. Maybe next year we'll find hundreds of books, provided we can get permission to scan all the scrolls that have already been found.

Ben: Wow, that's amazing. Congratulations.

NF: Thank you. Yes, it's been fun.

Sora and the Attention Pyramid

Ben: Starting from known data and trying to decipher and extract information from it — that's fascinating. Sora is the complete opposite. It starts from basically random noise and generates videos up to a minute long.

Speaking of time expectations — when DALL-E 2 came out, I wrote: "Look, this is obviously going to eventually be about video." I think video is a hugely important problem. I still believe in VR, but I think generative AI is crucial to that medium's success. At the time I thought, this is obviously still several years of iteration away.

And yet, time fast-forwarded just 19 months, during which we actually already had Stable Diffusion-powered video, and now we've fast-forwarded all the way here, with Sora obviously being far more powerful and high fidelity. This reflects how fast things generally move these days, but when you saw these videos, Daniel, what was your reaction? Was it like "yes, this is it"? Did you anticipate a model like Sora? Was this mid-journey, or were you surprised too?

DG: Well, we observed at the beginning of the podcast that there's a time dilation dynamic in AI — your sense of how fast things are developing changes dramatically. There's a feeling that it's actually a relatively calm period, and then three or four things happen in succession, mostly all on the same day.

Ben: Yeah.

DG: I think Google's Gemini's context window expansion came out right then, Sora and two or three other things all happened that same day.

Ben: We're going to try to cover all of this below, see if we can get through it in an hour.

DG: I think everyone suddenly realized, "Wow, the acceleration trend in tech is back." For me, with Sora specifically — and I think Nat mentioned something similar, I'm not sure who brought it up first? Probably him — Sora was really just a belief that scaling does work at the end of the day. Now people are debating whether Sora already has a world model and what that even means. To me, that's secondary. I think people tend to get overly philosophical about this kind of discussion.

My main observation is simply from the perspective of pure aesthetic enjoyment, economic value — scaling continues to work, and we'd previously really seen it work in the domain of text. Then we gradually started running out of text tokens, which I think is roughly where the industry is now. The beauty of video, particularly with their approach, is that you can genuinely generate infinite training data. Your end goal is trying to make an auto encoder that has the same logic pairings you have in a game engine, but using diffusion and transformer architectures. Either way, you can generate massive amounts of data, and you can actually demonstrate...

Ben: Infinite data, because you can use game engines to generate it?

DG: Yes. When I say "infinite," I mean the number of video tokens is vastly larger than text. Of course, in terms of the amount of logical information it contains, video is less dense, but there's vastly more of it. It doesn't fully understand how glass shatters, but it absolutely understands how water ripples through video.

Ben: And how light diffuses, which is pretty incredible. But wait, I want to press on this point. Why is there more video than text? It seems counterintuitive given that text has been far cheaper to produce throughout human history? When you analyze why the TV market developed differently from music and differently from text, it's actually the opposite — text is cheap and easy to distribute. So why is there more video?

DG: Obviously video contains vastly more information than text, and I would argue, especially with the advent of the internet, it's actually cheaper to capture and distribute than text. Currently, that information is not as entropy rich or as logically rich as text. Of course, in text there's also a gradient in distribution, and anyone pre-training a model will tell you that most of the text is useless. Actually, the number of very high quality tokens in text or video is quite small. Probably in music too. There's an interesting Pareto distribution here.

Fan Yang's note: The Pareto distribution is a statistical phenomenon, also known as the 80/20 rule. It describes a common imbalance where, in many cases, most results come from a small portion of causes or resources. In other words, the Pareto distribution states that most output comes from a few important inputs or factors. For example, a restaurant menu contains a Pareto distribution. Imagine a restaurant with 100 dishes, but only 20 of them are what most customers order — they bring in 80% of sales, and these are the 20 dishes you need to focus on preparing well.

Ben: Yes, in the early days of generative AI, I think I got one thing wrong in my mental model: it turns out that high quality, well-labeled data is actually much better than just scraping from the internet.

DG: The magical thing about the transformer architecture is that it works even when the data's bad, so I think there was a mirage-like period where people thought data quality didn't matter since it worked anyway. A friend of ours compared previous technology to trying to balance a pole on your fingertip, while the transformer just wants to work. But I think people forgot that if the data is high quality, it only gets better. So the real miracle is that it barely works even with bad data, but with much better data, it gets much better.

I mean, high quality tokens are in some sense a form of stored computation, so I think the math work that many researchers are doing now is: you can spend hundreds of millions on Nvidia trying to get highly refined tokens, or you can acquire them from humans and there's a mathematical modeling you'd want to do basically on the cost per token, and how you value that. But, theoretically there's very high value data that you could get there without it by spending an infinite amount of compute, but you can also bypass a lot of flops and just eliciting high quality information from humans in some domains is easier.

Ben: It's interesting you bring up the physics aspect. You've already touched on the philosophical debate. My take on this is that scaling is an important point — the scaling works, and we're far from hitting bottlenecks on the transformer architecture. But even if you scale infinitely, would you use Sora-type models to model how an aircraft wing works? I'm deeply skeptical of that. But actually, for what you're describing, this becomes a moot point in the context of virtual reality examples. When you're in VR or in any entertainment scenario, nobody actually cares whether the physics of air over a wing perfectly matches reality. If you went frame by frame analyzing CGI physics in today's films, you'd probably find all kinds of holes, but it simply doesn't matter. To me, that's what matters. The "physics" of these models is good enough, and good enough "physics" is sufficient in many cases.

DG: Yes, I think so. I think you've pointed out that this is a downmarket disruption — you're not going to use this model to replace "putting things in a wind tunnel for testing." But you might use it to replace the modeling or sketching for a video game you want to make, or movie scenes and such, so I think it's an incredible tool.

As for Sora, as OpenAI's tweet about releasing it showed, it takes a very long time to render content. So right now I'd analogize it to the early days of LucasArts, where rendering a single frame took hours and was very expensive. Of course, we've now reached the point where we can have better products on our computers with Unreal Engine than they had back then.

Ben: So basically now when you look back at Toy Story 1, you know the rendering took days and it looks terrible.

DG: It looks terrible, and Sora's starting position isn't that bad. It needs minutes rather than days, but it is indeed very expensive — though that will get better over time.

I think there's still an open question — we think it's possible, but it has to be answered: "How much can these models be distilled?" I think this is an interesting abstract question for any domain in life, whether it's doing very high-quality math, writing high-quality code or English, or generating music. What is actually the terminal model size that you could have for that particular task? This is worth thinking about because it's possible the latent space is ultimately governed by a few simple laws. Once you figure them out, they're actually quite small in size. That's an experiment we'll be running with our cheap physics simulator, and as for what you mentioned — not just whether we can make them, but can we distill them and compress them to something that has the basic laws of reality so much that a human can watch it and enjoy it, but also run it on a single GPU, and maybe one day even on a single MacBook, who knows?

Ben: You just mentioned this, and I think it's a very interesting observation about text. Text is more logically dense, but videos are more — I don't remember how you put it — information dense, or rather a picture is worth a thousand words, and video is worth millions more. We perceive much more information from video, even though a ten-minute passage in a film can be summarized in a paragraph of text.

There's a very interesting fork here between how much logic is embedded in a particular segment and the perception of how much information is there. And I suddenly realized there's this aspect in video too — we've already seen this with image models. Even though images may be more impressive to humans and contain more information, they can be much smaller than language models. The same is true for video. From my perspective, there's no reason it wouldn't be, given how it works. This only highlights the massive market for entertainment / virtual reality, even if this is different from "I have an assistant that understands meaning and can be agentic and act."

NF: Yeah, I find it interesting. I mentioned this on Twitter, but I find it fascinating that many AI labs are supposedly aimed at creating AGI, some "superhuman reasoning entity," and they seem to have collectively decided that the path to AGI includes creating entertaining images and videos, because they're basically all doing it. Maybe achieving AGI does require doing this, who knows?

But we can see an interesting consensus in this industry, which is that for humans, video is at the top of the attention hierarchy — it's the thing that grabs you the most. We do see this, because Gemini 1.5 made quite a significant breakthrough in reasoning capability through a large number of tokens, and Gemini 1.5 was released just hours before Sora, yet Sora was far more compelling and captured people's imagination because you just watch it, it's that simple. Once you see what Sora can do, it's hard to unsee.

Ben: Yeah, I think that's another insight I took away. I think Daniel is right that OpenAI has the courage to scale and their faith is rewarded, and I think perhaps Sora's launch, in many ways, was a wake-up call for the last remaining skeptics of "scaling." If you were ever on the sidelines saying scale doesn't do it, it's hard to raise that objection in the face of Sora. Another thing is it reminds us that for humans, video is at the top of the attention pyramid — social media has been teaching us this for years, but now I think we're seeing it in AI too.

Groq and the Speed Step-Change

Groq and the Speed Step-Change

Ben: Another release, I think on the same day, was Groq putting online a demo using their processors. This is about the processor, not model innovation. They used Mistral and Llama as the available models, but the speed was truly striking. I think this matters, not because of what it means for Groq — that's a separate question, and I'm actually curious about your views on certain issues — but because for a long time, there is a user experience issue when it comes to AI. Many of the use cases we discuss, because it is human-like, the breadth of the uncanny valley is so large that basically any friction in this experience matters far more than when you're using a phone.

When using a phone, when you pull it out of your pocket or you're using the device, you never forget you're using a phone or computer. You never go, "Wow, I thought I was talking to a real person, but actually I was speaking into my phone." No, that never happens, so you actually have more tolerance for friction in user experience. However, when it comes to using AI, because it can sound human, speed matters, it matters hugely, and I think that's why that demo mattered — setting aside Groq's commercial prospects, it really made you feel, yes, this is the right direction. Speed actually makes an astronomical difference, and it felt like validation of my view.

Fan Yang's note: Groq is a technically leading "machine learning inference accelerator" company. Groq claims its LPU (Language Processing Unit) delivers 10x the inference performance of NVIDIA GPUs (Graphics Processing Units) at one-tenth the cost.

DG: Yeah, I think we humans have pretty fast response times from our minds, and I think the brain runs at a pretty high hertz — depending on your mood, you have alpha, beta, gamma frequency states, but at the end of the day, we perceive reality very quickly, and we hadn't quite had an experience where something was that instant and that fast and that fluid. But honestly, I think this is just the beginning. Some people are going to have to work hard to fully realize this concept, whether on Groq's hardware or elsewhere, and polish it into a very refined, elegant product that can handle interruptions and things like that.

But once someone does it, if I had to guess — if we try to make predictions on the next podcast or a future one — what is the big new thing? One of my views is that we'll enter a more agentic world of models, and what we have now is still in the pre-Cambrian explosion period.

You go to chat.openai.com, input a bunch of words, and some words come out, and this model ultimately ends up rhyming (cracking jokes) more than it's thinking (the model is rhyming more than it's thinking), and it's a bit slow. I think the next era is about real AI agents executing tasks for you on the internet, conversing with you at human speed, and I think the economy and market prices don't factor this in at all (the economy and market prices don't factor this in at all).

Ben: Well, that's the reason to be optimistic about Groq. If you actually calculate the cost of their systems, part of why they're so fast is that each chip has a tiny amount of SRAM, SRAM that can keep data in place, and it's super expensive, but it's deterministic — they know exactly where the data is — but this means they need big systems to have enough memory (they need big systems to have enough memory). This means they would need a large market to develop (they would need a large market to develop). So they're pushing this cost per token idea (cost per token idea), but you need to have astronomical numbers of tokens flowing through the system for that pricing to make sense. Still, my feeling is that speed actually matters, and it's a use case unlocker (a use case unlocker).

NF: Speed is a user interface unlocker too (a user interface unlocker too). Because the model outputs slowly, you have to adopt streaming tokenization — the token stream basically just blasts at you — and now with speed, speed has always been a feature, I think in many ways this just reminds us of a long-standing rule of UI design, which is that speed matters, latency matters (speed matters, latency matters). It's an interesting thing because users typically don't ask for it, but they definitely feel that they prefer responsive things over sluggish ones.

Ben: I think, as I said, this speed difference matters much more for this class of models.

NF: But in this case, I think it also unlocks new types of UI (it unlocks new types of UI), whereas before you could only sit there watching the model just stream tokens at you (the model just stream tokens at you).

Ben: Well, in this situation, you can actually have a conversation with the model, and it feels like a normal conversation. Not strange at all.

NF: Yeah. And actually, I think, to some extent, it feels more superhuman (feels more superhuman), because you can get an essay in seconds, you can get a book written in minutes, and to some degree, the superhuman feeling is stronger (the superhuman feeling is stronger). But I also think it becomes more reasonable to have the model, if you're willing to spend the money, explore several paths — maybe it tries ten approaches and picks the most effective one — because it can do that so quickly.

Ben: Yeah, it has enough time to explore.

NF: There might be more exploration time. So we've gotten used to seeing these UI hack techniques, like with Bing, where it outputs some text, then deletes it and says, "Sorry, I said something I shouldn't have," or some other nonsense. At slow speeds, it's almost comical — it gives you this strong feeling that we're still in the early days of AI. But at high AI speeds, this kind of flaw might not be noticed at all, so I think the speed improvement unlocks a whole bunch of new experiences that were previously impossible, which is exciting.

Groq is fascinating because this company has been around for a long time. The founder, Jonathan Ross, invented the TPU at Google, then set out to improve on it and do something better in some ways. I feel like they were almost done for, and then LLMs suddenly appeared, and they had this specialized chip architecture that seemed to work well. Again, you find that beneath the surface, it's quite deterministic (it's quite deterministic), which matches their approach.

Ben: You mentioned scaling earlier, Daniel. I think a related question is about chip design in general — at what point does it make sense to specialize even more than the GPU (at what point does it make sense to specialize even more than the GPU)? GPUs are more specialized than CPUs, but they're still general-purpose technology, and that brings real costs when it comes to things like latency. Are these two complementary and inseparable? If scale is actually the ultimate answer to almost everything, does that mean the opportunity for more specialized chip architectures might arrive sooner than we expected?

DG: I think so. Sitting here, I feel like we're on the cusp of the AI ASIC era (Application-specific integrated circuit). Maybe Groq is a bit early because it's been around a little too long, but if I had to guess, ASICs will be a major part of the future.

Fanyang's note: An ASIC is a customized integrated circuit specifically designed to perform particular tasks or specialized functions, unlike general-purpose processors such as CPUs or graphics processors such as GPUs that have broad application ranges. It's like a customized toolbox with various tools inside, each designed for a specific problem — for ASICs, these problems include accelerating computation speed, encryption, image processing, and so on. The design and production costs of ASICs are often higher, closer to craftsmanship.

I think one of the main changes is, I remember calling Jonathan the day after Llama was released, and I told him the industry was finally going to standardize, that the industry would standardize around something you could show people to demonstrate how good you are, because before, his problem was that he was parading around a bunch of these benchmarks and people had a tough time translating that into something that was so economically valuable (he was parading around a bunch of these benchmarks and people had a tough time translating that into something that was so economically valuable), so economically valuable that they'd reconfigured their entire architecture for a specialized chip (they'd reconfigured their entire architecture for a specialized chip). This wasn't just Jonathan's problem — the entire AI company ecosystem in 2016, 2017 had this problem. In fact, Meta created a standard by open-sourcing Llama, and tokens per second basically became a metric that everyone was thinking about. This became an industry standard you could execute against, and much more importantly, you can measure your balance sheet by (much more importantly, you can measure your balance sheet by).

AI companies went through two cycles when training models: they were relatively less concerned about margins, they just wanted the best GPUs, they didn't want to take any risks. You spend $300 million, you just want your model to "tape out" properly (you just want your model to "tape out" properly), and then if you find product market fit (product market fit, meaning someone pays for your product and it grows organically), you naturally enter the inference era. Now, in the inference era (in the inference era), you end up staring at your COGS, you're staring at your COGS every month, and you're thinking, "God, we're paying so much per hour, per GPU. We have every reason to assign five engineers to redesign this completely alien platform that's totally different from before." This is essentially an ASIC — if I call their chip an ASIC, people might be unhappy, but you know what I mean.

Ben: Yeah, exactly, this situation is closer to ASIC than GPU.

DG: It's an ASIC, and it makes complete sense because you just need to stare at your costs. It's a bit like asking: if you could lower your interchange rate, as a fintech company, how much would you be willing to spend to build your own infrastructure to achieve that? Well, the answer is usually a lot of money. And NVIDIA's margin is a kind of interchange rate for tokens. I think people are absolutely willing to do the build work and bear the heavy lifting for custom architectures — in a way they weren't willing to accept in 2017, when very few companies even had revenue.

Fanyang's note: In financial markets, the interchange rate refers to the fee charged by banks or payment networks for processing credit or debit card transactions. As an analogy, it's like opening a store in a shopping mall and paying rent to the operator — financial companies pay interchange fees to conduct transactions on payment networks. Today's NVIDIA is essentially collecting rent on flowing tokens.

Ben: The inference market is smaller than the (AI) training market.

DG: By the way, the only companies that have this technology are the ad companies, like Meta and Google — they have their own chips. So I think what ends up happening is, you're now able to monetize these models in a way where you can do the math yourself on why it makes sense to rewrite these models for custom architectures. If I had to guess, NVIDIA's dominance in training, as far as I can tell, remains strong as ever. Over time, I don't think they'll lose market share, but the pie gets bigger — the inference pie gets bigger, and that includes some ASICs. To some extent, TPUs and Meta already have their own internal custom inference chips. I think, over time, that pie gets bigger because there's economic value in doing so.

When thinking about terminal numbers, I think one thing that hasn't been fully factored in is that when we think about terminals, we usually think about them in terms of AI's power requirements and all of that. The denominator is usually the number of NVIDIA chips produced in a year — about two to three million, so maybe 20 to 30 million watts of power demand. But if the denominator is chips produced through TSMC, because there are all these AI ASIC companies in the market, anyone with product-market fit decides to make their own chips — that's 20 million, 30 million, 40 million chips per year. Most of what's produced today is obviously iPhone chips, which are very low power. But anyway, I think the dynamics shift when infrastructure migrates to more specialized domains.

One thing that could break this situation, which I should mention, is that we're in a very unstable environment right now, because if the architecture changes — if someone achieves an architectural breakthrough, and plain transformers perform terribly, and you actually want something else, then everyone will rush to the new domain, and you're actually going to want something that's a little bit more general and not specific. So NVIDIA, AMD would even become the choice for inference chips. But absent this kind of disruption — and every day that passes, I think the odds of that decrease — not because the transformer architecture is a miracle and is the best architecture, but because the amount of ecosystem around transformers keeps growing. I think it makes sense for these companies to specialize with their own chips.

Devices and Robotics

Ben: An interesting question about interface speed-ups is whether we're on the verge of really unlocking actual new devices. I think about that AI hardware product called Rabbit R1 or something similar that was shown at CES. I haven't gotten my hands on it, but I think it's going to be a terrible product. It connects to the cloud for computing, the latency generated there will make the experience bad, it uses GPUs to run the device — it won't work very well. I've already realized that.

But you can imagine a world where, if it connects to the cloud, what happens if it connects to this Groq interface? Sometimes it gets faster, a bit more interesting. What if we could actually run a relatively small but data-input-heavy model locally? This has been an ongoing question, but at least from a public perspective, there hasn't been massive development work on this front so far. When does this start crossing over into devices other than a browser and a chatbot?

DG: What do you think?

NF: I've been waiting for someone to develop these things, because I think the necessary technology already exists — it just needs to be put together in the right way. But over the past year, no one has developed an AI that passes a conversational Turing test for a one-minute conversation or a two-minute conversation. You just fuse an automatic speech recognition (ASR) model with an LLM and a text-to-speech model in some way, and you can get low enough latency to achieve a pretty magical user experience.

There are already some people — I met one of them last week — who are using truly high-quality voice models from places like ElevenLabs and stitching these things together with low latency, and it's working. I've seen a project called Retail.AI. It didn't fully hit the mark 100%, but it's the closest team I've heard to this goal. You do feel something when you use it. There's the sense that there is a personality on the other side. As people train models that really understand prosody, and are able to invoke tone appropriately, and achieve a really full duplex — so they're not waiting for pauses in your conversation but can jump in at any time — I think there will be a magical feeling. We'll get closer and closer to that, and I expect someone to make a run at this this year.

So I think this is the AI experience from the sci-fi movie Her. Everyone knows it's possible and coming. I'm a bit surprised it's taken this long, but I think we'll definitely get there. I don't know if it's a device, but it's an experience. I don't know if it has to be local, I don't think it does. Retail.AI, I think, shows it doesn't have to be local to work.

Ben: That's part of the problem too. Part of the implication of all the LLM discussion is that it's great for virtual physics — this good-enough physics. But will it really cross over into the physical world, or will there be more and more bifurcation, where the online world is entirely virtual, who knows what's real and what's fake? But there's also a very clear line, or one aspect where — let's look at robotics, where the core physical attributes are still very deterministic, and it has to actually work. But because it speaks with an LLM, can that actually help you cross the divide from a perception perspective?

NF: It seems inevitable that there's going to be some kind of local and remote processing. If you have a robot, it needs fairly high-hertz processing to move around, react to things, not fall over. That has to be localized — maybe a large part of it can be localized. But when it makes bigger decisions that require consulting a huge amount of data, say it's your personal helper robot, it knows everything about your life, not all of that information is stored locally, or it knows it needs to look up information about the world. So I think there will always be some kind of big brain in the cloud that's used for something. And I think this split is the big question. But we're already seeing this layered model in robotics, where you'll have 50 Hz or 100 Hz models handling robot kinematics, helping the robot move through the world.

Ben: Those are still deterministic approaches, right? That's not running on a transformer?

NF: Actually, there are some learned approaches to kinematics that are working too. We're starting to see this end-to-end training. In fact, I think Daniel and I recently spoke with a company doing this. There seems to be a wave of excitement building around robotic foundation models. We haven't had our GPT-3 moment for robotics yet, where you put a few hands on a table and it can tie shoelaces, decorate cakes, or assemble Lego — and do all these things relatively well, or what feels like the beginnings of robotic intelligence. But that seems like it's coming in the next 12 or 18 months. We're going to see those demos.

What's enabling it is this belief in scaling and a few breakthroughs on the model architecture side, and what's holding it back is data. You don't have the common crawl of robotics data. You can't search the internet for robotics instruction data. So all the effort is going into collecting these datasets. The early demos are genuinely impressive, and in some cases they do involve local learned models for motion and kinematics and balance.

Ben: Do you think data is going to be the real differentiator? Will there be a scramble to acquire exclusive data sets, or will datasets become commoditized too, and everyone will realize that the way you actually differentiate is with the product — and that having the best dataset actually benefits everyone, so there will be more collective action?

NF: I think it's a really good question. If this were a few years ago, I would have thought open common data sets were more likely. There are some open robotics datasets now, but they're small and low quality. But now we're in an AI gold rush, and those expensive projects to collect large amounts of data, whether through teleoperation or other means, are likely to happen inside well-funded companies — both large and small.

Ben: Does this apply to data generally? Because in theory, everyone would be better off with a collective approach, a high-minded approach about where we're going to actually differentiate. But the stakes are too high now, and everyone's saying, "No, this is my data, I'm not sharing it"?

NF: The walls are going up. Definitely the shutters are down on data. It used to be much easier to crawl the internet. Scraping has gotten harder overall — you can see this across the board. So I think some companies that didn't used to think of user-generated content as an asset have suddenly realized it. They're saying, "Wait, we have all this data we could train on."

Ben: Speaking of Reddit's IPO.

NF: Yeah, exactly. We shouldn't just let people take the data and train on it. Make it harder to scrape, and treat it as an asset that may have value over time. So that's happening generally. And robotic data is so expensive to collect. There's some question about how much can be done in simulation, but either way, you have to do a lot of work to collect data. My bet would be that there ends up being lots of competing private data sets.

---

Gemini 1.5 and Large Context Windows

Ben: Anyway, we have to get to the main topic. Google's Gemini models — there's good news and bad news. Let's start with the good news.

I found Gemini 1.5 surprising, and validation from companies like Groq expanded my expectations of what these models can do. The idea is: look, just throw everything you want into the context window. You don't need to build some RAG system. You don't need to figure out what goes in and what doesn't. For me, that convenience — yes, it may be relatively slow — but in a way, it's a huge change. You can do stupid things. I linked a tweet where someone inserted a line into The Great Gatsby and saw if it could find it. Like, "Who's going to ever do that?" That sentence — "Who's going to ever do that?" — defines new products that eventually become big deals. And I think that possibility, for me — yes, there's a spectrum from small context windows to large ones, but for me, 1.5 crossed over. It became a huge change where you can just do whatever you want.

NF: Yeah, I completely agree. I think the world was surprised because they not only delivered a good model, but they delivered innovation along an axis that was several orders of magnitude beyond what anyone else had offered to date, and it seems to actually work. The fact that it's a multimodal model with long context also gives you the opportunity to do things like throw in an hour-long video and reason over it, or throw in a thousand examples — now you're not fine-tuning a model, you're just prompting it with a bunch of examples, and it can learn to do incredible things.

Fan Yang's note: Why is long context valuable for large language models? LLMs sometimes need more contextual information to make more accurate predictions or generate more meaningful text. Imagine if you only told your friend a fragment of a story instead of the whole thing — your friend would likely feel confused and unable to understand what you're trying to convey. Similarly, if an LLM can only see very short passages or sentences, it may fail to accurately grasp the full context, leading to generated text that lacks coherence or precision. Additionally, "long context" gives the model better "memory capabilities," enabling it to maintain consistent semantic understanding across longer stretches of text. Such models can also perform better reasoning and logical inference because they can take into account more information and background knowledge — much like encountering someone in real life with a good memory, clear logic, and clear expression.

Ben: It's like Excel. Excel let normal people program. Gemini lets normal people fine-tune a model. You don't actually have to do anything — you just throw all your stuff in and it figures it out.

NF: The bet on long context is very important. We think being able to not just retrieve massive amounts of information but reason over massive amounts of information is a superpower. I mean, it's a human capability to some extent. We humans have episodic memory and procedural memory, the ability to retain skills or memories over time, and there's always been this question of "How do AI models do this? How will they develop episodic or procedural memory?" In context, you can do both.

You can put episodes in that the model will remember, and you can put skills in, like what Google actually did teaching it new languages inside a single prompt and then asking it to use those skills. So this has always been an important missing skill. This may not be the final way it shows up in AI systems, but it's a new way we can do this, and I think it's very meaningful.

You can also do things that approximate superintelligence. Reasoning over massive codebases, showing it hours of surveillance footage and asking it to correlate across that footage. I think this is an incredible breakthrough. Google has clearly discovered some secrets, and we've been looking for clues, going through the literature, trying to figure out what the secret sauce is. But this is absolutely a differentiating factor.

Ben: The question I'm most interested in is, how much of this is the model and how much of this is infrastructure? Because last year they did a demo at their enterprise event — it was weird, I can't find any record of this demo, I spent hours looking for it last week. I distinctly remember this when I was writing about Gemini 1.5. They talked about this sharding capability — we know sharding as a database thing and what problems it solves and what challenges it introduces, but they were talking about it, I think, in the context of training. But it seems like they're also using sharding at inference time, they have this ability to distribute workloads not just across chips, not just across clusters, but at least in theory, across data centers, which introduces huge challenges as far as you're constrained by the speed of light.

Google's networking capabilities have always been well-known, but I'm not sure people realize how that advantage can be applied to solving these problems. Daniel, you talked about how much can you make a sparse model — to do that, you take a mixture-of-experts sort of approach and spread it out.

This is the exact opposite of Groq. Groq is massively serial, incredibly fast. What if we can spread it out all over the place and because the use case is tolerable of latency — we can take that extreme to its logical conclusion. It seems like only Google can do what Gemini 1.5 is doing right now, and no one else even seems close.

DG: Do you think anyone else is close to Google's level, Nat?

NF: Well, we know of one other company that has this capability.

DG: Yes.

NF: Last week, Daniel and I invested in a company called Magic. They have a very nice, very efficient, even longer-than-Gemini context mechanism, and it's working. Honestly, we thought there was only one company with this capability, and now we know there are two.

Fan Yang's note: magic.dev — Magic's self-introduction on their official website.

Ben: Interesting.

NF: So maybe there's a third. Who knows?

Ben: Interesting. So maybe this Google capability isn't as powerful as it seems.

NF: Well, when Magic showed it to us, we still thought it was an incredible achievement.

Ben: From a use-case perspective, this is a big deal. If someone other than Google can do this, all the better — that's clear.

DG: It's a different type of long context. I think, like human memory, the ability to use it effectively in some scenarios is different from just repeating something you heard a year ago. So I think over time, like all benchmarks, we'll realize, "Oh, token size isn't always bigger-is-better, and it's not the same for everything." But it's a very, very high quality reasoning engine — that's my take on it. One component of this reasoning engine is a very large context window. But I think that's only part of the equation.

Anyway, besides Magic, there are some others who are either ahead of this or not far from it, and I do think the idea that we're still constrained by context length is something we'll look back on like kids today looking back at having to swap out floppy disk drives halfway through playing a computer game.

I think these things are going to become very important, there are many different approaches, and then I think the next step — this is actually something Magic is very secretive about, so we should probably be careful how much we share — but it's something they and many others are thinking about, which is the ability to do active reasoning.

Today's ChatGPT, even Gemini — these models are a little bit closer to someone rhyming and not thinking. So the Magic team is looking for what feels right, what's a good sort of vibe, for lack of a better word.

Ben: Yeah, there's not much logic to current models.

DG: Yeah, AI today is closer to Jay-Z in the studio talking into the microphone as quickly as possible, trying to get the thing that sounds right out, as opposed to John von Neumann. It turns out if you just do that over the entire corpus of human knowledge, you end up getting something that seems smart, but we're not actually sure if it's really smart. That's why it struggles with things like programming and math.

So, active reasoning is the important thing that I think a lot of people are working toward, and yes, we've seen some pretty remarkable stuff. Everything is still very early stage, but if there's a big breakthrough of the year in AI, if I had to guess, it won't be context window, but very large context combined with active reasoning and thinking.

Ben: Does this still run into a situation where, I mean, going back to context window, you can tie it to the scaling question. Maybe the transformer architecture, you can scale it more than you think, and that's enough to get you close [to reasoning capability]. Same with context window. Just make the context window bigger and bigger, doesn't that solve the memory problem? Because persistence can be maintained there. Is active reasoning the same, is it still a one-shot process, or because we've moved away from the von Neumann architecture, where things are in memory and retrieved and shuttled back and forth, all this one-shot aspect. I don't even know how this will evolve, am I thinking about this the right way?

DG: Yeah, I think you're exactly right, I think there are many different approaches. One idea is, if you could infer things fairly quickly, you can have the model, this is the most straightforward idea, have the models just read their own output, think about it, write a little bit more.

Then there's an idea where you think, "Gosh, if we're doing that, why are we bothering to emit all this text and then read all this text? Can't we just do this active thinking process in the model weights themselves?" This is the frontier of research and the trade secrets, and I think this will make or break these companies.

I do think if someone actually does that, it would be equivalent to Google launching PageRank in a competitive search engine era. You have to make a great product, PageRank alone doesn't make Google, but it gives them the chance to be number one, which they've maintained in search to this day.

I think if someone could create something that had active reasoning and actively thought-through problems the way humans do in whatever domain they choose, they would be ahead.

Ben: Do you think [achieving active reasoning and active thinking] is a software problem, not a hardware problem?

DG: At Apple, there's an interesting saying: "Hardware people think everything is a software problem. And software people think everything is a hardware problem." I think it's fair to say this can be solved more easily, I think it can be solved in software, not in hardware.

Ben: From an innovation perspective, that's good, because for startups, solving problems through software is easier to get started with than tackling infrastructure constraints, otherwise it would be tough.

DG: If we were a sufficiently advanced civilization that could conjure up any node-size chip at will in seconds, maybe we could all just do these things. But practically, this problem is solvable now too, I think it will be solved in software, because that is the more malleable piece of the system, I think it is just a math problem at the end of the day, people don't like hearing that because they like to believe there's something deeply human in it, but even these ideas, I mean ultimately can be represented as a math problem, it's possible. So we think it might be solved this year, if [active reasoning] actually happens, this could become the defining event of the year.

Ben: We're back to the philosophical debate, Nat.

NF: Yeah. I mean "thinking" is a mechanical process and machines are going to do it, I still firmly believe that. Things I've seen recently have made me believe it even more, if that's possible.

Ben: I look forward to seeing thinking machines.

The Gemini Disaster

The Gemini Disaster

Fanyang's note: Someone used Google Gemini 1.5 to generate an image related to Elon Musk. The result from Google Gemini 1.5 is shown on the right.

Ben: We saved the juiciest topic for last. Another aspect of Gemini — another aspect is, actually, I was on Dithering with John Gruber today, and I thought he put it well. Google's AI large model Gemini 1.5 feels so distasteful as it shipped because it feels like bad faith, blatantly "We're not actually doing our best job to give you an answer." This is displayed directly, and it feels like an aspect we would forgive AI for getting wrong, we've been forgiving OpenAI, their early versions obviously had bias issues, but they've addressed that. But Gemini 1.5 doesn't feel like it's in good faith, maybe it was an accident, but it crossed a line in people's perception, this looks very problematic.

It's baffling how this happened? How did we get this product from a company that was otherwise very cautious about product releases, ultimately becoming a public disaster?

NF: Well, I think you're right. They shouldn't be given as much grace as OpenAI, one reason being they saw what came before and learned nothing from precedent. OpenAI's image generation AI Dall-E 2 had its own crazy woke image creation problem, they had to fix and fine-tune it, and learned from it, all forgivable because they were pioneers in this space, ChatGPT went through similar things, so Google should have seen all of this happen and learned from it to do better.

Fanyang's note: Wokeism is a term used to describe a social and political ideology. It derives from the colloquial English term "woke," originally referring to a state of awareness or recognition of social injustice and racial discrimination. In Western society today, due to overcorrection, many also use it to criticize excessive emphasis on political correctness and identity politics, or express disdain for cancel culture and extreme political correctness.

Ben: That's a great point. This is a huge advantage of going first in any field, because you get more grace.

NF: Yeah, you get more grace, because no one had solved these problems before. But Google clearly wasn't first, and still made mistakes that feel like they're from 2021 or 2022, which is less forgivable.

How did this happen? I think culture is a very important factor. You've written about this, it's obvious that inside Google it's hard for someone to raise their hand and say, "Hey, I don't think we should ship it in this form, we should do something to address this."

NF: We also heard from some Google employees that with these models, it's unlikely to be a deep problem in model training. It was more like a decision someone made later in the productization process. So there may be a set of system prompts or templates, or something like that, imposing a set of rules and guidelines on the model, while the original internal model wasn't built this way.

I think that's where the challenge lies. Google has always had an interesting term for product launches — they call it "externalization." I've always thought this is a very telling term in Google's culture, because it captures, in a way, how Google sees itself. They develop breakthrough technology internally, then "externalize" the magic. This isn't product-first thinking, or even customer-first thinking — it's technology-first thinking. And I think the mistake lies right there, in the process of "externalizing" the technology.

So in some ways this problem is also easy to fix — probably just editing a file could dramatically improve things. But on the other hand, editing that file might mean going through multiple layers of product people and policy people who will have many opinions about it. There's a gulf between the brilliant minds creating the models and the users, and these "middlemen" are where the challenge lies.

Ben: What do you think actually happened here, Daniel? Was it a data problem, or the model, or the RLHF process, or prompt engineering — where did things go wrong?

DG: Well, we had a good discussion about this earlier. I think there are traditionally some things people misunderstand a bit. Pre-training and fine-tuning a model are not distinct ideas — they're somewhat the same thing. Fine-tuning is just more pre-training at the end. As you're training the model, and I think this is what we believed and is now scientifically well-validated, the ordering of the information is extremely important. Because look, for basic things like how to properly punctuate a sentence, you can solve that any which way. But for higher-sensitivity things — the aesthetic of the model, the model's political preferences, and so on — those aren't completely binary domains, and it turns out the ordering of how you show the information matters a lot.

In my mind, I always imagine it as trying to pull a very tight bedsheet across a bed — that's your embedding space. You pull the sheet to the top right corner, the bottom left pops out. You do this, then the top right pops out. That's what you're doing. You're trying to align this high-dimensional space to a particular set of mathematical values, but at some point you never get a perfect answer or zero loss. So order matters, and traditionally fine-tuning has been more of the final stage of pre-training.

I think this originally emerged with the liberal leanings of OpenAI's ChatGPT models. I think this was a relatively harmless byproduct, because the model becomes very sensitive to the final data points shown to it, and these data points can easily accidentally create bias. For example, if you set up a few words in your internal software prompting human raters on which tokens to write into the model, those words bias them. If raters can see other raters' results, you get these reflexive processes. It's like resonant frequencies — they compound very quickly. Errors compound over time. I think you might inadvertently get a slightly politically left-leaning model, because a lot of online text is slightly politically left-leaning.

Ben: What's interesting is that even with stupid opinions, like about meat or selling goldfish — I think "selling goldfish" might be one of my favorite opinions. It's like, "No, I'm not going to sell a being." It's hilarious. I used Nate Silver's tweet as an anchor, like the San Francisco Board of Supervisors, but it's not that any one opinion is inappropriate. This illustrates your point — this is how these models work. If a specific small set of beliefs is input at the final stage of model training, it seamlessly expands across the entire set.

DG: Exactly right. Whatever happened with Gemini or other models, we observe these models, and they all exist on a kind of subterranean Jungian plane where they automatically calibrate to each other. The model isn't doing anything wrong — it's just reflecting what we humans do, and it turns out these things cluster into similar buckets.

Fan Yang's note: I saw this meme on Xiaohongshu earlier. The top part is "Jungian psychology" (on Xiaohongshu it's colloquially called Jung's "Red Book"). The conversation above mentioned "a Jungian plane in the latent model," which I found quite interesting. The reference to Jung here describes a hidden, latent, deep-level connection within AI large language models. It originates from psychologist Carl Jung's theory of the "collective unconscious" — the idea that humans share a subconscious level containing universal symbols, images, and experiences. Because large language models learn from humanity's knowledge base, there exists some kind of deep-level connection within the model itself, like human subconsciousness, where they automatically adjust to each other. And this connection may be beneath "their" or "our" awareness.

Ben: That's how human politics works, right?

DG: Yes. No one wants to say this publicly.

Ben: Well, no one can research and understand every topic in the world.

DG: Of course.

Ben: So you might have a few ideas you truly understand. And your core ideas need to align with other people's core ideas. We've seen this — it's like Politics 101. These models work the same way. I think this is the first time I've used my political science degree in a Stratechery article, but anyway, please continue.

DG: (Laughs) This is becoming very important! I'm thinking about the Reformation, because in 1517, Martin Luther wrote his 95 Theses, and through the printing press, he managed to create a new religion that spread across Europe. In a way, everyone's been trying to draw analogies between ChatGPT and the printing press, but they actually function almost oppositely.

The whole process works in the opposite direction. The printing press was a technology to disseminate information through a book basically and convince people to do things, while the LLM agent is the kind of antibook — it very concisely summarizes things. If that's indeed the case, it can awaken people to the fact that they've long been complicit in religion, because it very concisely summarizes these things for you and puts everything in the hidden space, and suddenly you realize, "Wait, this veganism concept is connected to that other concept." In a way, LLM technology is a kind of Reformation in reverse, where everyone suddenly realizes how many things were wrong.

Ben: That's a brilliant insight, Daniel. I mean, it captures — there's a joke going around now, "Look, you know the main reason some right-wing provocateur didn't break all this is that they're simply not smart enough to do it this effectively." Because it's like, "Look, you thought all these things were unrelated — let me put them all in one package and present them to you. Now, what do you think?"

DG: Exactly right. So it strips away any nuance of ideology and lays it bare, and yes, people react to that. I think the most interesting message is that Google lacks a very basic process. That's your point — maybe people thought about it before launching the model, maybe they didn't think at all. I'm thinking of that famous Steve Jobs interview where he said, "The problem with Microsoft is they have no taste." I think what surprised people about AI — we discussed this on the podcast — but I think what people generally didn't anticipate is that fine-tuning a model is just as aesthetic an art as making a beautiful landing page for your website.

In hindsight, it shouldn't be surprising that the Borg, who built the Google Cloud Platform (GCP) interface, also produced a very mechanical AI model. And it should be equally unsurprising that Mistral, a French AI startup with French culture and a French-style product, was able to produce a commendable model. I mean, it may not be the smartest model, but at least in my personal testing, it was relatively well-behaved, and its political tone was very neutral.

Ben: Alright, actually, I want to talk about Mistral later, but Nat, what should Google do now?

DG: Besides call you?

NF: (laughs) Yes, I mean, I think this is a leadership challenge. They have a missing editor, a missing product editor, a missing person with taste and judgment, someone in the company with the authority to veto anyone and ensure the right things happen. I think leadership change has to happen. Culture is the hardest thing to change in a company. You can do strategic changes, product changes, operational changes. Cultural change is the most difficult and can only be achieved through leadership. We either need to see markedly different behavioral change from Google's leadership, or we need to see completely different leaders.

Ben: That's why I wrote this piece. There are reasons to question whether Google CEO Pichai is the right person for the job. There's the classic distinction between "wartime CEO" and "peacetime CEO," and he's the epitome of a peacetime CEO. He did that well. This isn't to diminish his leadership of Google in the 2010s, the Aggregator era, keeping the incumbents happy, not wanting to antagonize them — that was in their interest. They managed to never get criticized on things like censorship. On the internet, they'd get some criticism, but in Congress, Facebook took all the arrows, and Google just floated through smoothly. Pichai was very effective at these things. We'd complain about him standing before Congress, no one could pronounce his name, but that was an advantage, actually a good thing.

That's not what Google needs now. Google needs a wartime CEO, and that's roughly the situation now. I think our conversation this episode provides a very specific point in time where this particular leadership style has completely failed the company. This hasn't just failed their future, it's endangered their present.

NF: Actually, this is a real opportunity, because when you make a big enough mistake, you have the opportunity to truly correct it. A series of small mistakes makes it hard to make big changes, but when you've obviously failed, I think AI is an exciting thing because it makes some things that were previously hard to notice or easy to hide become clearly visible. In the past week, AI has made some of Google's cultural problems very clearly visible, so now it's like, "God, there's a bunch of dirty laundry exposed in the sunlight, and now someone, I don't know who, but someone has the opportunity to really use it to drive the change that's necessary." This isn't just about shipping product, it's about shipping product that people want to use. That's the big problem.

Ben: The product itself isn't their priority. This gets to the bad faith issue.

NF: Well, I do have to commend the Google team for their ability to ship product, I do acknowledge that. I think what's interesting is how quickly Gemini 1.5 came after Gemini 1.0, their choice to label it 1.5 instead of 2.0, releasing Pro before Ultra was ready, they shipped a lot of things. I think they've at least broken the static friction, but clearly, they're off target, and their being off target has deep cultural and organizational reasons. I'm sure the DeepMind team and the Gemini team had nothing to do with this, and it's actually quite a shame that this happened.

Ben: Yeah, I can't imagine how frustrating that must be. Well, that's another problem — they'll lose some employees over this. "I built this incredible AI model, and another layer of the company was allowed to destroy it. Why should I spend all my time working here?"

NF: Well, the challenge is on all these fronts, who is their common reporting line? I think it's probably Google CEO Sundar Pichai, so he really has to exercise leadership. When you have a company this large, the challenge is that things easily fall through the cracks, and you see organizational divergence.

I also think some of this is because the field is still young. The challenge we face is that tone is a design problem, the industry is still early, we lack design tools, we lack personalized Photoshop-like tools that would make these problems more visibly apparent in the work. But these problems should have been obvious — this was too conspicuous. So I think this is a leadership opportunity for someone. It could be Sundar Pichai, it could be someone else.

Mistral, Evals, and OpenAI

Ben: You mentioned Mistral, they just released a new large model this week. I haven't written much about them, but first, you've done an initial evaluation of their product. I also found it interesting that they announced a new investor in the process, and that's Microsoft. This is a great response to the Microsoft-OpenAI situation, because what does Microsoft have? They have money, they can very explicitly hedge their bets, and now they've done so, and also sent a signal to OpenAI that, "Look, while we're very dependent on you now, we're going to work hard to make sure that's not the case going forward."

NF: Yes, I think Microsoft CEO Satya knows all his eggs are in one basket, and he's very wisely working to make sure that doesn't happen again. We've seen him do this before. He partnered with Mark Zuckerberg on the original Llama model open-sourcing, and provided cloud services through Azure. Now he has Mistral, which is currently the most prominent open-source AI model leader, and I think their execution speed and taste are very impressive, as Daniel said.

Ben: Why are they doing so much better than Llama?

NF: This is interesting. I think they were essentially the Llama team originally, so I think they have several advantages. Mistral has startup agility, and I think that's important. Maybe they also have some "beneficial" constraints. They only have limited capital, only limited compute, so they set out to solve within those constraints.

Ben: Maybe also because they can't afford to hire that many fine-tuners.

NF: Right? Well, that's for sure. Without a doubt, one thing Mistral cares about very much, as we discussed earlier, is data quality. We know they've worked very hard to clean their training data, and by doing so effectively obtained a "compute multiplier," thereby obtaining a "quality multiplier." But now their model is performing far beyond its weight, it almost feels like magic. Their new Mistral large model performed very well on evals, they haven't fully revealed what it is, maybe a mixture-of-experts of Mistral mediums or something. But damn, it's really impressive, so I think this is just a lot of agility, a real hardcore team, they have good taste and judgment, and so far they've made very good decisions.

Ben: By the way, how do evals work? Everyone publishes these test scores. What makes a good model? What makes a bad model?

NF: Well, this is an interesting topic that I care about. It's amusing to see all these company CEOs touting their MMLU number, and MMLU is an evaluation that Dan Hendrycks developed himself when he was an undergraduate. So you're basically seeing trillion-dollar company CEOs talking about their scores on a test launched by an undergrad, and this is one of the premier reasoning evals.

I think if you look at progress across AI as a whole, we've seen incredible progress on models, incredible progress on alignment tools like RLHF. Product has finally really developed, we're seeing a lot of product, even policymakers are very excited. But one area where progress seems to be lagging furthest behind is evals. Evals are basically testing models to see what they can and can't do, see how they behave, so you can have some understanding before release. It's a classic low-prestige activity. Like benchmarking, right? But it has enormous impact in the industry. When new benchmarks emerge, everyone wants to match them.

I think we currently face a series of challenges. First, there are very few good public evals, and even when there are, like MMLU, they either don't actually predict certain types of capabilities, or they're approaching saturation. I mean, on MMLU, someone has already hit 80 now, and with multi-shot, sometimes you can even get into the 90s, so your benchmark is saturated. In fact, all benchmarks tend to saturate a few years after release. This is really a deficiency.

I remember Andrej Karpathy once told us that the only model evaluation he trusts is Twitter evaluation. After a model is released, you can check the user sentiment on Twitter a few weeks later to see if people like it. But every company's CEO, while training these large models, says: "We have to be at the top of the leaderboard." By the way, what is this leaderboard? It's something thrown together by some undergrads, organized by people at UC Berkeley — Chatbot Arena.

Ben: These evaluation methods don't include the parts that Daniel has always considered important, like taste, tone, and feel.

DG: By the way, evaluating models with humans isn't easy. Why do we think evaluating models with models is easy? We've reached a stage where we're no longer discussing whether GPT-2 can write more than three paragraphs — models are good enough that evaluation has become a deep problem. Pearson is a massive company, and part of what they do is creating systems and methods for evaluating humans, but these don't exist for AI models, though they probably should.

Fanyang note: Pearson is a global education company focused on educational materials, educational technology, and assessment services, providing various educational resources and solutions including textbooks, online learning platforms, exams, and assessment tools.

Ben: Within OpenAI as an organization, have we underestimated Sam Altman? And are there others who have been underestimated — is there some sort of product sense in that organization that people can feel but perhaps can't measure, yet we still underestimate its importance?

NF: I'd pick Greg Brockman. I think Greg is actually a major driving force there, with excellent taste.

Ben: He was formerly at Stripe, which is a well-known company for being excellent at product and taste.

NF: Yes, he worked at Stripe and did many remarkable things there. One thing Stripe did in its early recruiting that most people probably don't remember — Stripe launched a CTF (capture the flag) for security, which Greg Brockman orchestrated. It was very carefully planned and really attracted very smart talent. Greg has excellent taste, judgment, and product execution ability. I think he also went deep on their CUDA kernels and training code, but I think he's a very energetic person with strong product sense. Of course, I think many others are involved too. Sam Altman obviously knows when a company is running well, given his involvement in many startups. But if you ask who is underestimated in this regard, I'd probably say Greg.

Ben: Well, these past few months have been significant moments. What big events should we be watching for next? Is GPT-5 coming soon, or are we not even sure if there's a GPT-5? Will something like Groq from last week suddenly emerge? Again, Groq is a classic example — this company had existed for many years but suddenly broke through. My question is "who knows?" What do you think will happen next?

DG: Well, I'm curious to hear what Nat would say, but let me offer one thought. I think we're currently in an odd tweener era. I think that for the AI models we have today, to use them in truly economically valuable ways requires building products around them, requires the kind of extensive time and product attention that Nat and his team put in during the GitHub era, to optimize models to make them faster and smaller, to really tailor models for products like Copilot. That's the paradigm we're in now, and I think this still holds true.

Fanyang note: Nat Friedman was formerly CEO of GitHub. Copilot is an AI-based programming assistance tool, like a personal assistant for programmers while they code. It uses large language models such as GPT (Generative Pre-trained Transformer) to provide developers with real-time suggestions and code snippets to help them write code more efficiently.

On the other end is a world where the AI models we have act as agents, without needing human-like interfaces. We interact with colleagues through Slack and Gmail, and that's perfectly fine. If I had to guess, at some point this year we'll take a bigger step in this direction — we'll have more coworkers than copilots. That's Nat's phrase, which I'm borrowing here, and I think it's very prescient. This will be a major event, and I don't know where it will come from. Maybe from OpenAI, maybe from Magic, maybe from DeepMind, but I think this is the grand goal the industry is working toward. Do you agree, Nat?

NF: Yes, I think that's right, I completely agree. What I'm interested in seeing is reasoning, some better definition of reasoning, a way to measure reasoning, and market improvements in reasoning — whether it's the active reasoning Daniel mentioned, or somehow a way of generating training data for the models that allow them to learn patterns of reasoning more directly, or methods for measuring these models' reasoning capabilities.

Right now we're trying to mine for it and see how their reasoning is, but I think making direct progress on this would be very important, and there's a belief in the field that this may stem from figuring out how to do reinforcement learning on text. I think that could be a potential path, but overall, I think some direct method of improving model reasoning capabilities, and then seeing actual improvement — getting real logical thinking and good results — that's very important.

Ben: And all of this is still happening within the broader transformer architecture context, just working at the edges and recombining some things?

NF: Yes, it seems the evolution of the computer industry has always been like this — you start somewhere and then keep building and evolving over time.

Ben: I was going to mention x86 (the processor architecture Intel launched for personal computers), like CISC (Complex Instruction Set Computer) architecture versus RISC (Reduced Instruction Set Computer) architecture, right?

NF: Exactly.

Ben: CISC architecture wasn't the best approach, but once you're two years ahead, you're two years ahead.

NF: Yes. I think it might be the same in the AI era.

Ben: Path dependency is huge.

NF: Yes. We're path-dependent. People just find ways to add what's needed before or after training, or they bolt on some model architectures. I don't know, that's where most of the AI research is right now. Most of the research is not fundamental new architectures, so probabilistically, you'd expect progress to happen in areas that scale existing architectures. But we should also pay attention — there are some interesting and promising areas being researched that aren't based on transformer architectures. But the direction of progress I'm most focused on is definitely reasoning.

Ben: Nat, Daniel, great to have you both. I look forward to inviting you again at some point in the future when there may be enough changes worth talking about.

DG: Wonderful. Thanks for having us, Ben.

NF: Thanks, Ben.

Original link:

https://stratechery.com/2024/an-interview-with-nat-friedman-and-daniel-gross-reasoning-about-ai

Thanks to Stratechery for high-quality information. If you want to read the original article, please subscribe to Stratechery Plus to support the original author.

5Y Capital seeks out, supports, and inspires lone entrepreneurs, providing them with everything from spiritual backing to operational support. We believe that if the "crazy" you — as others see it — starts to be believed in, the world will become a different place.

BEIJING · SHANGHAI · SHENZHEN · HONG KONG

WWW.5YCAP.COM