Breakfast with the OpenAI Founder: 5 Principles for Building a Great Tech Company | Yunqi Capital Insights
Those who set rules for AI must first set rules for themselves.
Greatness can't be planned, but every great company deserves to be retraced. OpenAI's current success belongs not just to one team, but to the Silicon Valley ecosystem. Yet the most crucial precondition was that OpenAI understood the rules of this ecosystem and found its most suitable place within it.
In 2016, journalist Steven Levy had his first breakfast with Greg Brockman, one of OpenAI's core co-founders and its CTO. Over the following six years, Steven maintained close contact with Brockman as well as OpenAI's key figures: CEO Sam Altman, Chief Scientist Ilya Sutskever, and researcher Alec Radford. To track OpenAI's progress, Steven also stayed in touch with Elon Musk, who had abandoned OpenAI and predicted its failure, as well as Microsoft CEO Satya Nadella and CTO Kevin Scott, who would later drive a pivotal partnership.
Steven meticulously documented several "life-or-death decisions" that made OpenAI what it is. We've excerpted and compiled the following content. Through this nearly 10,000-word piece, we can understand how this team thought through adversity and developed a set of principles for itself to reach that seemingly "impossible" goal.
This edition of "Yunqi Kepu" is excerpted from the October issue of Weird, compiled with assistance from DeepL. We hope it provides some reference for tech entrepreneurs currently on their journey and facing critical choices.
When You Can't Find the Specific Path, Trust the Specialists
OpenAI was founded in December 2015. At the time, I interviewed Musk and Altman. According to their introduction, the project "aims to make AI safe and accessible by sharing it with the world." In other words, "open source." But Altman cautioned me not to expect quick results. "For a very long time, this project will look like a research lab," he said.
More than a year after OpenAI's founding, I had lunch in San Francisco with Greg Brockman, OpenAI's CTO. "Our goal, and what we're really pushing, is to get systems to do things that humans couldn't do before," he said. But at the time, OpenAI appeared to be just a group of researchers publishing papers, even though $1 billion had already been invested (mainly from Musk). "Nothing was going well," Altman said. "We knew what we wanted to do and why we wanted to do it, but we didn't know how." Altman still remembers a moment when the small team gathered in Brockman's apartment — they didn't have an office yet. "I was thinking, what are we going to do?"
In 2016, OpenAI hired a then-unknown researcher, Alec Radford. Radford joined OpenAI in 2016, "sort of like joining a graduate program." In his view, it was an open place where he could research AI without pressure.
Radford's first experiment was to scan 2 billion Reddit comments and train a language model. Like many of OpenAI's early experiments, this one failed. But at OpenAI, this 23-year-old researcher was allowed to keep trying, to fail again.
"We just felt that Radford was really excellent at AI, so let him do his thing," Brockman said.
Later, Radford trained a language model on roughly 100 million Amazon product reviews, simply predicting the next character in a generated user review. Surprisingly, the model figured out negativity and positivity on its own. When asked to create positive or negative content, it generated praise or harsh reviews as requested (though the writing was somewhat stiff: "I love the look of this weapon... A must see for men who like chess!").
"It was a complete accident," Radford said. The positive and negative valence of reviews is a complex function of semantics, yet somehow Radford's system had partially mastered it. Within OpenAI, this neural network component became known as the "unsupervised sentiment neuron."
In early 2017, a preprint of a research paper co-authored by eight Google researchers was released. Its title was "Attention Is All You Need." It would later be called the "Transformer paper" — named both to reflect the game-changing nature of the idea and as an homage to toys that transform from trucks into giant robots. But at the time, only a handful of people like Sutskever understood how powerful this breakthrough was.
Brockman said: "The Transformer was our team's real eureka moment. This is what we'd been waiting for. More progress was made in the two weeks after than in the previous two years."
This was also OpenAI's strategy at the time — work hard on solving the problem, and believe that we or someone in the field would figure out the missing piece.
Adam D'Angelo, CEO of Quora and an OpenAI board member, said: "To take advantage of the Transformer, you needed to scale it up and operate more like an engineering organization. You can't have every researcher doing their own thing, training their own models, only doing elegant things that can get published. Someone has to do the more tedious, less elegant work." He added, "This is what OpenAI could do and others couldn't."
Radford and his collaborators named the model they created "generatively pretrained transformer" — the acronym GPT-1. The model contained 117 million parameters, or variables, and outperformed all previous models in language understanding and answer generation. Radford still remembers one late night at the OpenAI office. "I just kept saying, 'Well, this is cool, but I'm pretty sure it can't do X.' Then I'd quickly write some evaluation code, and sure enough, it could do X."
The world didn't know it yet, but Altman and Musk's research lab had begun an ascent that might slowly approach the summit of artificial general intelligence (AGI). OpenAI's crazy idea suddenly didn't seem so crazy anymore.
OpenAI Development Timeline | TechTarget
All Rules Were Formulated Around "Mission and Vision"
Each iteration of GPT performed better, partly because each iteration devoured an order of magnitude more data than its predecessor. Just one year later, OpenAI trained GPT-2 on the open internet with 1.5 billion parameters. Like a toddler mastering language, its responses became better and more coherent — so much so that OpenAI hesitated to release the program to the public. Radford worried it would be used to generate spam. Ultimately, OpenAI temporarily withheld the full version, offering the public a less capable one.
In early 2018, Musk became dissatisfied with OpenAI's progress — or, feeling that since OpenAI was making progress, it needed leadership to seize the opportunity. Or, as he later explained, he felt safety should be more important. In any case, whatever the problem, he had a solution: turn everything over to him. He proposed taking a majority stake in the company, adding it to his portfolio of multiple full-time jobs (Tesla, SpaceX) and oversight obligations (Neuralink and Boring Company).
Musk felt entitled to own OpenAI. "It wouldn't exist without me," he later told CNBC. "I came up with the name!" But Altman and the rest of OpenAI's brain trust had no interest in becoming part of the Musk universe. When they made this clear, Musk cut ties and offered the public an incomplete explanation: he was leaving the board to avoid conflicts with Tesla's AI work. He bid farewell at an all-hands meeting early that year, where he predicted OpenAI would end in failure. He also called at least one researcher a "jackass." And he took his money with him.
With no company revenue, this was an existential crisis. "Musk is cutting off his support," Altman asked in a panicked call to Reid Hoffman. "What do we do?" Hoffman volunteered to keep the company running, covering administrative costs and salaries.
But this was only a stopgap. OpenAI still had to find money elsewhere. Silicon Valley loves throwing money at talented people working on trendy technology. But less so if they're working at a nonprofit. For OpenAI, getting the first billion had already been a huge leap. To train and test new generations of GPT, then acquire the computing power needed to deploy them, the company needed new "billions." And fast.
So in March 2019, OpenAI came up with a bizarre solution: it would remain nonprofit, wholeheartedly committed to its mission; but it would also create a for-profit entity. The actual structure of this arrangement was extremely complex, but essentially, the entire company was now engaged in "capped" profit-making. If the cap was reached — the number wasn't public, but if you read the company's charter carefully, the cap might be in the trillions — everything beyond the cap would return to the nonprofit research lab.
This novel scheme was almost a quantum approach to corporate structure: depending on your frame of reference, the company was simultaneously for-profit and not-for-profit. The details lived in charts full of boxes and arrows—the kind in the middle of scientific papers that only PhDs or college-dropout prodigies dare to enter. When I suggested to Sutskever that this looked like something a not-yet-conceived GPT-6 might propose when prompted to evade taxes, he wasn't enthusiastic about my analogy. "This has nothing to do with accounting," he said.
But accounting matters. The optimization target of a for-profit company is profit. There's a reason companies like Meta feel pressure from shareholders when they pour billions into R&D. How could that not affect how the company operates? But wasn't avoiding commercialization exactly why Altman made OpenAI a nonprofit in the first place? According to chief operating officer Brad Lightcap, the company's leadership believed that the board—still part of the nonprofit controlling entity—would ensure that the drive for revenue and profit wouldn't overwhelm the original idea. "We need to maintain the sense of mission as our reason for being," he said. "It shouldn't just be spiritual; it should be reflected in the company's structure." Board member Adam D'Angelo said he took this responsibility seriously: "My job and the job of the rest of the board is to make sure OpenAI stays true to its mission."
Lightcap explained that potential investors were warned about these boundaries. "We have a legal disclaimer that says as an investor, you might lose all your money. We're not here to generate returns for you. We're here first to accomplish a technological mission. And, oh, by the way, we really don't know what role money is going to play in a post-AGI world."
That last line was no joke. Buried somewhere in the restructuring documents was a provision: if the company succeeded in creating AGI, all financial arrangements would be reconsidered. After all, from that point on, it would be a whole new world. Humanity would have an alien partner that could do much of what we do, only better. So previous arrangements might effectively be null and void.
There was one small problem, though: right now, OpenAI didn't actually know what AGI was.
OpenAI's office in San Francisco's Pioneer Building | MIT Technology Review
Choosing the Right Partner, the Right Way
When it came to defining AGI, Altman's answer was no clearer. "It's not a single Turing test—it's many things that we might use," he said. "I know that's unsatisfyingly vague. But we genuinely don't know what it will look like when we get there."
To realize OpenAI's vision, billions of dollars in venture capital barely qualified as a "bet." The Transformer architecture that created LLMs required massive hardware. Each iteration of the GPT series demanded exponentially more power—GPT-2 had over 1 billion parameters, while GPT-3 would use 175 billion. OpenAI was now like Quint in Jaws, after the shark hunter sees the size of the great white. "It turns out we didn't know how big a boat we needed," Altman said.
Obviously, only a handful of companies had the resources OpenAI required. "We very quickly locked in on Microsoft," Altman said. Microsoft CEO Satya Nadella and CTO Kevin Scott deserved credit for the software giant's ability to swallow an uncomfortable reality: after spending more than 20 years and billions of dollars building what was supposed to be a cutting-edge AI research division, Microsoft needed a jolt of innovation from a company that had existed for only a few years.
Scott said that it wasn't just Microsoft that had fallen behind—"everyone had fallen behind." OpenAI's singular focus on pursuing AGI, he said, had allowed it to achieve something like a moon landing, while the big companies weren't even aiming for that target. It also proved that not pursuing generative AI was a miss Microsoft needed to fix. Microsoft's initial $1 billion investment was paid back in compute time on its servers. But as confidence grew on both sides, the deal expanded. Now Microsoft had poured $13 billion into OpenAI. "Investing at the frontier is very expensive," Scott said.
Of course, since OpenAI's existence depended on the support of a major cloud provider, Microsoft had negotiated plenty for itself. Through hard bargaining, Microsoft secured what Nadella called a "non-controlling equity stake" in OpenAI's for-profit arm—reportedly 49%. Under the terms of the deal, some of OpenAI's original ideals—equal access for everyone—seemed to get dragged into the trash. Now Microsoft had exclusive licensing rights to commercialize OpenAI's technology. And OpenAI committed to using only Microsoft's cloud services.
In other words, without even taking a cut of OpenAI's profits (Microsoft reportedly gets 75% of profits until it recoups its investment), Microsoft had locked in one of the world's hottest new customers for its Azure network. Though Nadella said, this might be humanity's last invention: once machines are smarter than us, we may have bigger things to worry about.
When Microsoft began pouring serious cash into OpenAI ($2 billion in 2021, $10 billion earlier this year), OpenAI had already completed GPT-3, which was, of course, far more impressive than its predecessor. When Nadella saw what GPT-3 could do, he said, it was the first time he deeply understood that Microsoft had caught hold of something truly transformative. GPT, for example, had taught itself how to code.
"We didn't train it to code—it just got good at coding on its own!" he said. Leveraging its ownership of GitHub, Microsoft released a product called Copilot, which used GPT to generate code on command. Microsoft later integrated OpenAI's technology into new versions of its productivity software. Users would pay extra for these products, and a portion of that revenue would be recorded on OpenAI's books.
Some observers were shocked by OpenAI's one-two punch: creating a for-profit arm and striking an exclusive deal with Microsoft. How could a company that had promised to remain patent-free, open-source, and fully transparent exclusively license its technology to the world's largest software company? Musk's comments were particularly biting. He posted on Twitter: "This seems like the opposite of open—essentially, OpenAI has been captured by Microsoft." On CNBC, he offered an analogy: "Suppose you founded an organization to save the Amazon rainforest, but you became a lumber company, cut down the forest, and sold it."
Musk's sniping might be dismissed as "the resentment of a spurned suitor," but he wasn't alone. John Carmack, founder of id Software, said: "The way Musk's whole vision evolved into this feels a little gross." Another prominent industry insider, who requested anonymity, said: "OpenAI has gone from a small, open research institute to a secretive product development company with an unearned sense of superiority."
Even some employees were repelled by OpenAI's venture into the for-profit world. In 2019, several key executives, including research director Dario Amodei, left to found a rival AI company called Anthropic. They recently told The New York Times that OpenAI had become too commercialized, a victim of mission drift. Another OpenAI defector is Rewon Child, a major technical contributor to the GPT-2 and GPT-3 projects. He left in late 2021 and now works at Inflection AI, a company led by former DeepMind co-founder Mustafa Suleyman.
Altman claimed not to be bothered by the defections, framing them as simply how Silicon Valley works. "Some people are going to want to go do great work elsewhere, and that will move society forward," he said. "And that's absolutely in line with our mission."
Sam Altman | Analytics Insight
Engaging with the Negative Conversation, Offering Solutions
Before last November, what people knew about OpenAI was largely confined to the world of technology and software development. But now the whole world knew that OpenAI had released a consumer product based on the latest version of GPT-3.5 in late November. Even so, OpenAI was stunned by the response to ChatGPT. "Internally, the excitement was much more centered on GPT-4," said CTO Murati. "So we didn't really think ChatGPT was going to change everything. On the contrary, it made the public realize that the reality of AI had to be dealt with now." ChatGPT became the fastest-growing consumer software in history, reportedly accumulating 100 million users.
But OpenAI has been reluctant to confirm this, saying only that it has "millions of users." Radford said: "I didn't fully appreciate that making an easy-to-use conversational interface for an LLM would make it so much more intuitive for everyone to use."
Inside OpenAI, people debated whether to release a tool with such unprecedented capabilities. But Altman was in favor. He explained that the launch was part of a strategy to "get the public used to the reality that AI was destined to change their daily lives," and presumably for the better. Internally, this was called the "iterative deployment hypothesis."
Some said the new product's release was tied to promises the company had made to investors and equity-holding employees, because the company needed to make money. OpenAI was already charging fees to customers who used its products frequently. But OpenAI insisted that its true strategy was "to provide a soft landing for the singularity." OpenAI policy researcher Sandhini Agarwal said: "Think back to the Industrial Revolution — everyone thinks it was great for the world. But the first 50 years were really painful. A lot of people lost their jobs, a lot of people were impoverished, and then the world adapted. We're trying to think about how to make the period before AI adaptation as painless as possible."
Sutskever put it differently: "You build a more powerful intelligence, and you're going to keep it in the basement?"
In February, Microsoft leveraged its multi-billion-dollar partnership to release a version of its Bing search engine powered by ChatGPT, sending public opinion into another frenzy. CEO Nadella was ecstatic, having beaten Google at bringing AI into Microsoft products. He essentially told Google, the king of search: you've been cautious about releasing your own LLM products, now you have to do it too. He said: "I want people to know that we made them jump."
In doing so, Nadella triggered a race in which companies large and small rushed to release AI products before they had been adequately vetted. He also set off a new round of media coverage that kept more and more people awake at night. Plus, it had an unseemly habit of hallucinating misinformation all on its own.
But Altman believed it would be all the better if OpenAI's products forced people to confront the implications of AI. When discussing how AI might affect humanity's future, the majority of humanity should have a voice.
As society began prioritizing every potential downside of AI — job losses, misinformation, human extinction — OpenAI started positioning itself at the center of the conversation. Because if regulators, legislators, and doomsayers launched a charge to strangle this nascent alien intelligence in its cloud cradle, OpenAI would be their primary target regardless. OpenAI's Chief Policy Officer Anna Makanju said: "Given our current visibility, when things go wrong, even if those things are made by another company, it's still a problem for us, because we're now seen as the face of this technology."
Senate Judiciary Committee Chairman Richard Blumenthal said: "Altman's approach to dealing with members of Congress has been very upfront and very shrewd." He contrasted Altman's behavior with that of a young Bill Gates, who had foolishly avoided legislators during Microsoft's antitrust investigation in the 1990s. Blumenthal said: "By contrast, Altman was happy to sit with me for over an hour and try to educate me. He didn't come with a huge phalanx of lobbyists or entourage. He just demoed ChatGPT, and it blew my mind. I was excited about its prospects and its potential dangers."
Rather than shying away from discussions of these dangers, OpenAI described itself as "the force most capable of mitigating them." When Altman made his first appearance at a congressional hearing, battling a severe migraine, he encountered little of the hostile questioning and arrogant grandstanding that tech CEOs often face under oath. Instead, senators asked Altman for advice on how to regulate AI, and Altman enthusiastically agreed.
Altman's positioning himself as a champion of regulation made sense; after all, his mission was AGI, but safe AGI. Critics accused him of gaming the regulatory process so that rules would hinder small startups and give OpenAI and other large companies an advantage. Altman denied this. While he was open in principle to the idea of an international body overseeing AI, he did feel that some proposed rules — such as banning all copyrighted material from datasets — created unfair obstacles. He made clear that he had not signed a widely circulated letter urging a six-month pause on the development of more powerful AI systems.
But he and other OpenAI leaders did put their names to a single-sentence statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Altman emphasized: "Yes, I agree with that." Altman and his team argued that working on and releasing cutting-edge products was the path to addressing societal risks. Only by analyzing how ChatGPT and GPT-4 users responded to millions of prompts could they gain the knowledge to make future products ethically sound.
Greg Brockman (left) and Ilya Sutskever (right) | Wired
Facing Complexity
It was clear that OpenAI's "Open" was no longer the radical "Open" the company had proposed at its founding. When I mentioned this to Sutskever, he shrugged. "Obviously, times have changed." But he cautioned that this didn't mean the end result had changed. He said: "We're dealing with a massive, catastrophic technological shift, and even if we all do our best, there's no guarantee of success. But if things go well, we get to live an incredible life."
Altman said: "I cannot emphasize enough: we have no master plan. It's like we're going around every corner, shining a flashlight. We're willing to navigate the maze to get to the end. The maze has gotten twisty, but the goal hasn't changed. Our core mission remains — believing that safe AGI is an incredibly important thing, and the world doesn't value it enough."
Meanwhile, OpenAI was clearly slowly developing the next version of its large language model. Incredibly, the company insisted it had not yet begun work on GPT-5, a product that people either salivated over or dreaded, depending on their perspective. Clearly, OpenAI was making exponentially powerful improvements on existing technology. Brockman said: "Our biggest regret is that we don't have new ideas. Having something that can be a virtual assistant is nice. But that's not our dream. Our dream is to help humanity solve problems it currently can't."
Given OpenAI's history, the next series of major innovations might have to wait for a breakthrough on the scale of the Transformer. Altman hoped OpenAI would achieve this — "We want to be the best research lab in the world," he said, "but even if we can't, the company will leverage others' advances, just as it leveraged Google's work. A lot of people around the world are going to do important work."
It would help if generative AI itself didn't create so many new problems. For instance, LLMs need to be trained on massive datasets; obviously, the most powerful LLMs ingest the entire internet. Some creators and ordinary people weren't happy about this — they had unwittingly supplied content for these datasets, only to find themselves somehow contributing to ChatGPT's outputs.
Tom Rubin, an elite intellectual property lawyer who officially joined OpenAI in March, was optimistic that the company would eventually find a balance that served both its own needs and creators' needs — including creators like comedian Sarah Silverman, who was suing OpenAI for using her content to train its models. One direction for OpenAI: partnering with news and image agencies like the Associated Press and Shutterstock to supply content for its models, eliminating the question of who owns what.
When I asked Rubin whether he was convinced AGI would definitely happen, and whether he was eager for it to happen, he looked blank. He paused and said: "I can't even answer that." When pressed further, he clarified: "As an IP lawyer, his job wasn't to accelerate the arrival of terrifyingly intelligent computers. But from where I sit, I look forward to it."
In just eight years, the company had transformed from a struggling group of researchers into a world-changing Promethean beast. OpenAI was evolving into a more "standard" Silicon Valley unicorn, joining the ranks of Big Tech companies that shape people's daily lives.
Title: What OpenAI Really Wants
Link: https://www.wired.com/story/what-openai-really-wants/
Author: Steven Levy
