Code Brain | Bill Gates and Sam Altman's Conversation

If you asked people to name the leading figures in artificial intelligence, one name would probably come up more than any other: Sam Altman. His team at OpenAI is pushing the boundaries of AI with ChatGPT, and Bill Gates sat down with him on a recent podcast to talk about what's next. The conversation covers why today's AI models are the dumbest they'll ever be, how society will adapt to technological change, and even where humans will find purpose once we've perfected AI.

Bill Gates: My guest today is Sam Altman. Of course, he's the CEO of OpenAI. For a long time, he's been an entrepreneur and leader in the tech industry, including running Y Combinator, which funded many remarkable companies like Reddit, Dropbox, and Airbnb. Shortly after I recorded this episode, he was removed as CEO of OpenAI, which completely surprised me, at least briefly. A lot happened in the days following the firing, including nearly all OpenAI employees signing a letter in support of Sam's return, and now, Sam is back. So before you hear our conversation, let's check in with Sam and see how he's doing.

Bill Gates: Hey, Sam.

Sam Altman: Hey, Bill.

Bill Gates: How are you?

Sam Altman: Oh, man. It's so crazy. I'm okay. It's a very exciting time.

Bill Gates: How's the team?

Sam Altman: I think, you know, many people have noticed this fact, which is that the team has never been more productive, optimistic, or excellent. So I guess that's the silver lining behind all of this.

In a sense, this was our real moment of growing up, and we're very motivated to become better, to become a company prepared for the challenges we face.

Bill Gates: That's great.

So we won't discuss that matter in our conversation; however, you will hear Sam's commitment to building safe and responsible AI. I hope you enjoy this conversation.

Welcome to Unconfuse Me. I'm Bill Gates.

Bill Gates: Today we'll focus primarily on AI, because it's so exciting and people are also concerned about it. Welcome, Sam.

Sam Altman: Thank you so much for having me.

Bill Gates: I've had the privilege of watching your work progress, but I was very skeptical at the beginning, and I didn't expect ChatGPT to work so well. It surprised me a great deal. We don't actually understand this kind of encoding. We know the numbers, we can see it multiplying, but how do you encode Shakespeare's works? Do you think we can have a deeper understanding of this representation?

Sam Altman: Absolutely. It's very hard to do this in the human brain, and you could say it's a similar problem — you have these neurons, they're connected to each other. But their connections are changing, and we can't slice open your brain to observe how it evolves, whereas we can perfectly inspect [the model]. There's already some very good work on interpretability, and I think more explanations will emerge over time. I think we'll be able to understand these networks, but our current understanding is very low. And as you've seen with pleasure, the little bit we do understand has already been very helpful for improving these things. Setting aside scientific curiosity, we all have motivation to truly understand them, despite their massive scale. We could also ask, where in your brain is Shakespeare encoded, and how is it represented?

Bill Gates: We don't know.

Sam Altman: We really don't, and you could say it's even more unsatisfying that among all these numbers that we should be able to perfectly inspect, observe, and run any test on, we still can't find the answer.

Bill Gates: I'm quite convinced that within the next five years, we'll understand it. That understanding will let us do much better than we can today in terms of training efficiency and accuracy.

Sam Altman: Totally agree. You see this throughout the history of technology with many empirical discoveries. They didn't know what was happening, but clearly it worked. Then, as scientific understanding deepened, they could make it much better.

Bill Gates: Yes, in physics, biology, sometimes you just kind of bumble around and then — wow — how did that happen?

Sam Altman: In our case, the person who built GPT-1 solved that "wow" himself, which is somewhat impressive, but without deep understanding of how it worked or why it was effective. Then we had scaling laws that let us predict how good it would get. That's why when we told you we could do a demo, we were fairly confident it would work. We hadn't trained the model yet, but we were quite confident. That led us to try many things and develop an increasingly scientific understanding of what was happening. But it really did start with empirical results first.

Bill Gates: As you look ahead two years, what do you think will be the key milestones?

Sam Altman: Multimodality will certainly be important.

Bill Gates: You mean speech in, speech out?

Sam Altman: Speech in, speech out, then images, eventually video. Clearly, people really want this. We've launched images and audio, and the response has been much stronger than we expected. We can push that much further, but probably the most important area of progress will be around reasoning ability. Right now, GPT-4's reasoning ability is very limited. And reliability — if you ask GPT-4 most questions 10,000 times, one of those 10,000 times it might answer quite well, but it doesn't know which one. And you want to get the best answer every single time, so improvements in reliability will be very important.

Customizability and personalization will also be very important. People want different things from GPT-4: different styles, different sets of assumptions, and we'll make all of that possible, and then also let it use your own data. It can know about you, your email, your calendar, how you like appointments scheduled, and connect with other external data sources — all of that will be among the most important areas of improvement.

Bill Gates: In the current fundamental algorithm, it's just doing simple feedforward multiplication, so to generate each new token, it's essentially doing the same thing. I'd be interested to see if you could, like solving complex math equations, apply transformations an arbitrary number of times, so the control logic for reasoning might need to be much more complex than what we do today.

Sam Altman: At a minimum, it seems like we need some form of adaptive computation. Right now, we spend the same amount of compute on every token, whether it's a simple token or solving some complex math problem.

Bill Gates: Yes, like, "Solve the Riemann hypothesis..."

Sam Altman: That would take a lot of compute.

Bill Gates: But it uses the same compute as saying "The."

Sam Altman: Right, we at least need to make that usable. We may need something more complex on top of that.

Bill Gates: You and I both attended an education session at the Senate. I was pleased that about 30 senators attended and got up to speed quickly, because this is a major change driver. I don't think we've overdone it to attract politicians. However, when they say, "Oh, we messed up on social media, we should do better" — that's a huge challenge, with very negative factors in terms of polarization. Even now, I'm not sure how we should respond.

Sam Altman: I don't understand why governments haven't been more effective on social media, but that seems worth studying as a case to understand, because they're now going to face challenges related to AI.

Bill Gates: That's a good case study. So when you talk about regulation, are you clear about what type of regulation should be built?

Sam Altman: I think we're starting to figure it out. It's very easy to overregulate in this space, and you can see this has happened with many things in the past. But similarly, if we're right — and maybe it turns out we're wrong — but if in the end we're right, and this technology develops to the level we think it will, it will affect society, affect the balance of geopolitical power, and many other things. For these still-hypothetical but extremely powerful future systems — not GPT-4, but systems with 100,000x or a million times the compute — we've embraced the idea of a global regulatory body that would keep a close eye on these super-powerful systems, because they would indeed have such massive global impact. One model we've discussed is something like the International Atomic Energy Agency. With nuclear energy, we made that decision too. Because of its potential global impact, it needs a global body, and I think that's reasonable. There will be many short-term issues, like what these models can and cannot say? How do we think about copyright? Different countries will have different considerations, and that's fine.

Bill Gates: Some people think that if models become very powerful, we'll be scared of them — global nuclear regulation works basically because at least on the civilian side, everyone wants to share safety practices, and that's worked quite well. When you get to the weapons side, that's not the case. If the key is preventing the whole world from doing dangerous things, you'd want a global government, but today on many issues like climate, terrorism, you can see we have trouble cooperating. People even invoke US-China competition to explain why any idea of slowing down is inappropriate. Is any idea of slowing down, or slowing down enough to be careful, hard to implement?

Sam Altman: Yes, I think asking for a slowdown is extremely difficult. If instead you said, "Do what you want, but no compute cluster can exceed a particular, extremely high power threshold" — given the costs here, we'd probably only see five such clusters — any cluster like that would have to submit to something like international weapons inspectors. The models there would have to undergo safety audits, pass some tests during training, and pass audits and tests before deployment. That seems possible to me. I wasn't sure before, but I took a trip around the world this year and spoke with heads of state in many countries that would need to participate in something like this, and they almost all expressed support. This wouldn't save us from everything — there would still be issues that show up on smaller systems, and some things could go pretty wrong — but I think it could help us with the highest-level risks.

Bill Gates: I do think that in the best case, AI can help us with some hard problems.

Sam Altman: Certainly.

Bill Gates: Including the polarization problem, because it could undermine democracy, and that would be an extremely bad thing. Right now, we're seeing a lot of productivity gains from AI, which is a very good thing. What are the areas you're most excited about?

Sam Altman: First, I always think it's worth remembering that we're on this long, continuous curve. Right now, we have AI systems that can do tasks. They certainly can't do a complete job (what a position entails), but they can do some tasks, and there's productivity improvement there. Eventually, they'll be able to do more things like what humans do for work today, and we humans will of course find new and better jobs. I fully believe that if you give people more powerful tools, they don't just work faster — they can do fundamentally different things. Right now, we can maybe triple a programmer's speed. That's what we're seeing, and it's one of the areas we're most excited about, and it's working very well. But if you can make programmers three times more efficient, it's not just that they can do three times as much — it's that they can think at a higher level of abstraction, use more brainpower to think about completely different things. It's like going from punch cards to higher-level languages — it wasn't just that we programmed a little faster, it was a qualitative leap. And we are seeing that.

When we look at these next-generation AIs that can complete more whole tasks, you can imagine it as a little agent where you say, "Help me write this entire program, I'll ask you a few questions along the way, but it's not just writing a few functions at a time" — and so many new things will emerge from that. Then it can do even more complex things. Someday, maybe there's an AI where you can say, "Help me start and run this company." And then someday, maybe there's an AI where you can say, "Go discover new physics." What we're seeing now is exciting and wonderful, but I think it's worth putting in the context of this technology being on a very steep growth curve for at least the next five or ten years. These existing models will all become the dumbest models.

Programming is probably the productivity area we're most excited about today. It's already being deployed and used at massive scale. Healthcare and education are two other areas where we're very much looking forward to rapid development.

Bill Gates: What's a bit daunting is that unlike previous technology improvements, this one improves very fast and has no ceiling. It can reach human level in many job categories — even if it can't do unique scientific research, it can make customer service calls and sales calls. I think you and I do have some concerns that, although this is a good thing, it will force us to adapt faster than ever before.

Sam Altman: That's the scary part. Not that we have to adapt — it's not that humans don't have incredible adaptability. We've been through these massive technological shifts before, where large amounts of what people do for work change over the course of a generation, and over a generation we seem to absorb that quite well. We've seen that in all the great technological revolutions of the past. Each one gets faster, and this will be the fastest by far. That's what I find a bit scary — the speed at which our society will need to adapt to this, and what will happen in the labor market.

Bill Gates: One aspect of AI is robotics, or blue-collar work, when you get hands and feet at human-level capability. The incredible breakthrough of ChatGPT was getting us focused on white-collar work, which is totally fine, but I worry that people are losing sight of blue-collar work. What do you think about robotics?

Sam Altman: I'm very excited about it. We started working on robotics too early and had to shelve that project. It was also difficult for the wrong reasons, without helping us make progress on the hard parts of machine learning research. We were dealing with bad simulators and tendon ruptures and things like that. Over time, we also increasingly realized that we first need intelligence and cognition, and then we can figure out how to adapt it to physicality. Starting from how we build these language models, it's easier to go from there. But we've always planned to come back to this.

We've started making investments in some robotics companies. On the physical hardware side, I'm finally seeing genuinely exciting new platforms being built for the first time. When the time comes, we'll be able to leverage our models, as you just said, use their language understanding capabilities and future video understanding capabilities, and say, "Alright, let's do something amazing with robotics."

Bill Gates: If the people who have already done legs really well actually do arms, palms, and fingers, and then we put them together, and the price isn't outrageously expensive, then this will rapidly change the job market for a lot of blue-collar-type work.

Sam Altman: Yes. Of course, if we go back seven to ten years, the consensus prediction was that it would hit blue-collar work first, then white-collar work, and creativity might never, or at least be last, because that was magic and humanity's strong suit.

Obviously, it's exactly the opposite now. I think there are many interesting reasons that explain why this happened. With creative work, hallucination in GPT models is a feature, not a bug — it lets you discover new things. Whereas if you're going to have a robot move heavy machinery, you better be able to be very precise. I think it's just a case of having to follow where the technology goes. You might have preconceived notions, but sometimes science doesn't go in that direction.

Bill Gates: So what's the app you use most on your phone?

Sam Altman: Slack.

Bill Gates: Really?

Sam Altman: Yes, I wish I could say ChatGPT.

Bill Gates: [laughs] More than even email?

Sam Altman: Way more than email. I think the only thing that might beat it is iMessages, but yes, Slack is more than iMessages.

Bill Gates: There's a lot of coordination to do inside OpenAI.

Sam Altman: Yes, what about you?

Bill Gates: I'm Outlook. I'm old-school email, or else browser, because of course a lot of my news comes through the browser.

Sam Altman: I didn't count browser as an app, and it's possible I use it more, but I'd still bet on Slack — I use it all day.

Bill Gates: Incredible.

Bill Gates: Well, we have a vinyl record player here. As I do with other guests, I asked Sam to bring one of his favorite records. So, what did you bring today?

Sam Altman: I brought Max Richter's recomposition of Vivaldi's The New Four Seasons. I like music without lyrics when I work, and this record retains the comfort of Vivaldi's original, and pieces I'm very familiar with, but has enough new notes to deliver a completely different experience. There are musical works where you develop strong emotional attachments because you listened to them heavily during key periods of your life, and The New Four Seasons is what I listened to a lot when we were starting OpenAI.

I think it's just beautiful music, it's soaring and optimistic, perfectly suited to what I need when I work, and I think the new version is fantastic.

Bill Gates: Is this performed by an orchestra?

Sam Altman: Yes, it's performed by the Chineke! Orchestra.

Bill Gates: Wonderful.

Sam Altman: Shall we play it now?

Bill Gates: Yes, let's listen.

Sam Altman: This is the prelude to the movement we're going to hear.

Bill Gates: Do you wear headphones?

Sam Altman: I do.

Bill Gates: Do your colleagues tease you about listening to classical music?

Sam Altman: I don't think they know what I'm listening to, since I do wear headphones. Working in silence is very difficult for me — I can do it, but it's not my natural state.

Bill Gates: That's interesting. I agree that songs with lyrics feel distracting to me, but this is more of a mood-type thing.

Sam Altman: Yes, and I keep it very quiet — I can't listen to loud music either, somehow that's always been my habit.

Bill Gates: Wonderful, thank you for bringing the beautiful music.

Bill Gates: Now, for me, if you really get to incredible capability through AI, AGI, AGI+ — there are three things I worry about: one is bad actors controlling the system, and if we have good people with equally powerful systems, hopefully that minimizes that problem; two is the possibility of the system controlling everything, which for some reason I worry less about, but I'm glad others are focused on it; what confuses me most is human purpose, and I'm very excited about this — I'm very good at working on malaria and eradicating malaria, and very good at bringing smart people together and putting resources into it. When the robot says to me, "Bill, go play pickleball, I can eradicate malaria. You're just a slow-thinking person." At that point it's a philosophically confusing thing. How do we organize society? Yes, we want to improve education, but what is education for, if you take it to the extreme — we still have great uncertainty. For the first time, the chance of this happening within 20 years is non-zero.

Sam Altman: There are many psychological difficulties of working in technology, but what you're describing is the most difficult for me, because I also get so much satisfaction from it.

Bill Gates: You do bring value.

Sam Altman: In a sense, this might be the last hard thing I do.

Bill Gates: Our thinking is so shaped by scarcity — scarcity of teachers, doctors, and good ideas. Partly because, I do wonder, if a generation grows up without that scarcity, what would they think about how to organize society and what to do with this philosophical concept? Maybe they'd come up with a solution. I worry that my thinking is so influenced by scarcity that I can barely even think about this.

Sam Altman: That's what I tell myself too, and I genuinely believe it — that although we're giving up something in a sense, we're going to have something smarter than us. If we can get to this "post-scarcity" world, we'll find new things to do. They'll feel very different. Maybe instead of solving malaria, you're deciding which galaxy you like and what you plan to do with it. I believe we'll never run out of problems, never run out of ways to find fulfillment and do things for each other, never run out of understanding of how we play the human game for other people — that will still matter enormously. It'll definitely be different, but I think the only way out is through. We have to do this, it's going to happen, and it's already an unstoppable technological process because the value is too great. I'm very, very confident we'll succeed, but it will feel quite different.

Bill Gates: Applying this technology to certain current problems — like tutoring kids, helping motivate them, or discovering drugs for Alzheimer's — I think how to do that is very clear. Whether AI can help us reduce war, reduce polarization. You'd think that with increased intelligence, not polarizing is common sense, not going to war is common sense, but I do think many people would be skeptical. I'd love to have people work on the most difficult human problems, like whether we can get along. If we think AI can help humans get along better, I think that would be very positive.

Sam Altman: I believe it'll surprise us in that regard. This technology will surprise us with how much it can do. We'll have to wait and see, but I'm very optimistic. I agree with you, that would be a very large contribution.

Bill Gates: On the equity side, technology is usually expensive — personal computers, internet connections — and it takes time to bring costs down. I'm thinking, the cost of running these AI systems looks pretty good. Will the cost per evaluation come down a lot?

Sam Altman: It already has come down a lot. GPT-3 was our longest-running, most-optimized model, and over the more than three years since we launched it, we've brought the cost down by a factor of 40. That's a good start for three years. For GPT-3.5, I'd bet we've brought the cost down by almost a factor of 10 by now. GPT-4 is new, so we haven't had as much time to reduce costs, but we will continue. I think we have the steepest cost-reduction curve of any technology I know of, better than Moore's Law. This is not just because we've figured out how to make models more efficient, but because we have a better understanding of the research, and we can get more knowledge and capability into smaller models. I think we will drive the cost of intelligence down toward zero, and that will be a transformative change for society.

Now, my basic model of the world consists of the cost of intelligence and the cost of energy. These are the two biggest inputs to quality of life, especially for poor people, but overall as well. If you can reduce both of those costs, you can have more of everything, you can bring more improvement to people. We're on a curve where, at least for intelligence, we're really going to deliver on this promise. Even at the current price — which is the highest it will ever be, and far more than we expected — for $20 a month, you get a huge amount of GPT-4 access, and the value is way more than $20. We've already gotten quite low.

Bill Gates: What about competition? Is it interesting that so many people are piling into this space all at once?

Sam Altman: It's both annoying and motivating and fun, [Bill laughs] and I'm sure you've had similar feelings. It does push us to move faster and do better, and we're confident in our approach. There are a lot of people, I think, who are skating to where the puck is, and we're skating to where the puck is going, and that feels good.

Bill Gates: I think people would be surprised by how small OpenAI is. How many employees do you have?

Sam Altman: About 500, so we're slightly larger than we used to be.

Bill Gates: But that's small, by the standards of Google, Microsoft, Apple.

Sam Altman: It really is small, and we have to run not just a research lab but now a real business and two products.

Bill Gates: The expansion of all your capabilities, including talking to people all over the world, listening to all the supporters — that must be interesting for you.

Sam Altman: Absolutely fascinating.

Bill Gates: Is this a company of young employees?

Sam Altman: Older than average.

Bill Gates: Okay.

Sam Altman: It's not a bunch of 24-year-old programmers.

Bill Gates: Indeed, my perspective is somewhat distorted because I'm in my 60s. I look at you, you're younger than me, but you're right, you have a lot of people in their 40s.

Sam Altman: People in their 30s, 40s, 50s.

Bill Gates: It's not like early Apple or Microsoft, where we really were kids.

Sam Altman: No, and I've reflected on this. I think companies have gotten older in general, and I don't know what to make of that. I think it's in some ways a bad sign for society, but I tracked this at YC. Over time, the best founders have trended older.

Bill Gates: That's interesting.

Sam Altman: In our case, we're even older than that average.

Bill Gates: Your role at YC helped those companies learn a lot, and I imagine that was good training for what you're doing now.

Sam Altman: It was incredibly helpful.

Bill Gates: Including seeing the mistakes.

Sam Altman: You could definitely say that. OpenAI did many things opposite to the standard YC advice. We spent four and a half years before launching our first product. When we started the company, we had no concept of a product, we weren't talking to users. I still wouldn't recommend most companies do this, but having learned and seen these rules at YC, I felt like I understood when, how, and why we could break them, and what we did was really quite different from anything else I've seen.

Bill Gates: The key is that you assembled a talented team, got them focused on the big problems, not some short-term revenue issue.

Sam Altman: I don't think Silicon Valley investors would have supported us at the level we needed, because we had to spend so much on research before we could launch a product. We just said: "Eventually the model will be good enough, and we know it'll be valuable to people." But we're very grateful for the partnership with Microsoft, because this kind of ahead-of-time investment isn't what the venture capital industry is good at.

Bill Gates: It's really not, and the capital requirements were substantial, almost at the limit of what venture capital could handle.

Sam Altman: Probably past it.

Bill Gates: Probably so. I very much agree with Satya's thinking about "how do you combine this brilliant AI organization with a large software company?" You could even say one plus one is far greater than two.

Sam Altman: Yes, it's been great. You really hit on something there, and this is what I learned from YC too. We can say we want to find the best people in the world to do this. We want to make sure our goals and the AGI mission are aligned. But beyond that, we let people do their thing. We recognize this will have twists and turns, and it'll take time.

We had a roughly correct theory, but a lot of the strategy along the way proved to be wildly wrong, and we just tried to follow the science.

Bill Gates: I remember coming to see the demo, and really wondering what the revenue path was for this project? What does it look like? In this frenzied time, you still have an incredible team.

Sam Altman: Yes, great people want to work with great colleagues.

Bill Gates: That's a draw.

Sam Altman: There's a deep gravity center there. Also, it sounds cliché, every company says this, but people feel a deep sense of mission. Everyone wants to be part of building AGI.

Bill Gates: That must be exciting. When you blew me away again with the demo, I could feel that energy. I saw new people, new ideas, and you're still moving at an incredibly fast pace.

Sam Altman: What's the advice you give most often?

Bill Gates: Talent comes in many forms. Early in my career, I thought it was just raw IQ, like engineering IQ, and sure, you could apply it to finance and sales. But that turned out to be so wrong. Building a team with the right combination of skills is so important. Walking them through their problem, guiding them to think about how they should build a team with all the different skills — that's probably one of the most helpful pieces of advice I give. Yes, telling kids math and science are cool if you like them, but what really surprised me was the mix of talents.

What about you? What advice do you give?

Sam Altman: About how most people miscalculate risk. They're afraid to leave their comfortable job to do what they really want to do. In reality, if they don't, they'll look back on their lives and think, "God, I never went to start that company I wanted to start, or I never tried to become an AI researcher." I think that's actually the riskier path.

Related to that, being clear about what you want to do and asking others for it has unexpected benefits. A lot of people get stuck spending time on things they don't want to do, and the advice I give most often is probably to figure out how to solve that.

Bill Gates: If you can get people into a job where they feel purpose, it's so much more fun. Sometimes that's how they have the huge impact.

Sam Altman: Absolutely.

Bill Gates: Thanks for coming, this was a fantastic conversation. In the days ahead, I'm sure we'll have much more to discuss as we work to shape AI in the best possible way.

Sam Altman: Thank you so much for having me, I really enjoyed talking with you.

Bill Gates: Unconfuse Me is a Gates Notes production. Special thanks to my guest today, Sam Altman.

Bill Gates: Tell me, what was your first computer?

Sam Altman: It was the Mac LC2.

Bill Gates: Good choice.

Sam Altman: It was a great machine. I still have it, and it still works.