OpenAI Rewrites History Overnight, GPT-4o Obliterates Every Voice Assistant | BlueRun Ventures
GPT-4o, Finally Here
Legend has it that on this night, OpenAI would change history.
Audience members who watched the launch event remained in a state of profound shock long after it ended — the "Her" of science fiction had, in this very moment, become reality!
Amid thunderous applause, CTO Mira Murati took the stage to unveil OpenAI's mysterious new product to the world —
GPT-4o made its long-awaited debut.
In the live demo, its performance was as if a person were sitting right beside you — the conversational rhythm natural and harmonious, with no trace of AI detectable whatsoever.
From this night forward, human-computer interaction has entered an entirely new era!
This also explains the hidden meaning in its name: the "o" stands for "omni," signifying that OpenAI has taken a crucial step toward more natural human-computer interaction.
In just 232 milliseconds, GPT-4o can respond to audio input, averaging 320 milliseconds. This response time has reached human-level speed!
Moreover, it can accept any combination of text, audio, and images as input and output.
On English text and code benchmarks, GPT-4o's performance is on par with GPT-4 Turbo, with significant improvements in non-English text.
What's even more noteworthy is that this brand-new AI model makes GPT-4-level AI available to everyone for free.
(Yes, that "gpt2" that sent the entire AI community into a frenzy of testing last week on the LMSYS model arena — it was this!)
Now, on the ChatGPT page, Plus users can get early access to the "latest and most advanced model," GPT-4o.
Sam Altman personally kicked off the product introduction on X.
Altman described GPT-4o as "OpenAI's best model ever — it's smart, it's fast, and it's natively multimodal."
And all ChatGPT users can use it, completely free!
Altman specifically emphasized that while previously only monthly subscribers could access GPT-4-level models, that was never OpenAI's true intention.
"Our original mission has always been to put the best AI tools into everyone's hands."
(Wondering why Altman hadn't appeared at such an important launch event — turns out he was in the audience all along.)
And Google, with its annual I/O conference just around the corner, refused to be outdone, launching its own live demo of a voice assistant.
Google DeepMind CEO Demis Hassabis excitedly announced that he would be speaking at I/O for the first time, sharing the work they've been doing.
The two tech giants are going head-to-head! There's more good theater to come tomorrow — you can already smell the gunpowder.
Overnight Disruption of Voice Assistants: The New Flagship GPT-4o Debuts
Of course, the star of this launch event was OpenAI's flagship model, GPT-4o.
The significance of this haloed debut is that it brings GPT-4-level intelligence to every single OpenAI user!
From now on, whether you're a paid or free user, you can experience GPT-4 through it.
The only difference is that ChatGPT Plus has 5x the message limit of free users.
And GPT-4o doesn't just offer equivalent model capability to GPT-4 — it's faster at reasoning, and provides multimodal abilities that simultaneously understand text, images, audio, and more.
Note: GPT-4o is about to drop some serious moves.
Real-Time Voice Conversation: ChatGPT Perfectly Transforms Into MOSS
Research lead Mark Chen first demonstrated one of the new ChatGPT's key features — real-time voice conversation.
He asked it: "I'm on stage giving a live demo right now, and I'm a little nervous. What should I do?"
ChatGPT responded considerately: "You're giving a presentation on stage? That's really impressive! Take a deep breath, and remember — you're the expert!"
Mark proceeded to breathe heavily and frantically a few times, asking if ChatGPT had any advice. (Laughter throughout the venue.)
It exclaimed in surprise: "Relax, Mark, slow down your breathing — you're not a vacuum cleaner!" (Its sense of humor drew another round of roaring laughter from the crowd.)
Note that throughout this process, the interaction between ChatGPT and Mark was virtually latency-free — it picked up on jokes in real time, with empathy scores off the charts.
Moreover, the model understands the human habit of interrupting at appropriate moments in conversation. It stops promptly to listen to you and gives relevant responses, without losing its train of thought.
For example, when Mark said he wanted to try the deep breathing exercise again, ChatGPT chimed in at exactly the right moment with "exhale slowly."
The entire process was so natural and coherent that it felt like a human sitting across from you — completely devoid of the mechanical stiffness typical of AI!
Compared to sluggish, uninterruptible, emotionally stunted voice assistants like Siri, ChatGPT won this round hands down.
This, finally, is what humanity's ideal AI voice assistant looks like — MOSS has truly become reality!
Beyond that, ChatGPT's "high emotional intelligence" left audiences exclaiming in amazement.
During conversation, it can discern different tones and inflections in the user's voice, and generate speech with varying emotional registers according to its own lines — entirely free of "mechanical" quality.
Playing Different Roles, Telling Bedtime Stories to Capricious Humans
Next, ChatGPT was asked to tell a bedtime story on the theme of "robots in love."
ChatGPT had barely begun when it was abruptly interrupted by a human: "More emotion, can you add some drama to the story?"
ChatGPT said okay, and began telling the story with more undulating tones and exaggerated delivery.
A few seconds later, it was interrupted again: "No no no, even more emotion, can you give me maximum expression?"
What followed was a ChatGPT that sounded like it was performing Shakespeare on stage — its delivery so exaggerated it seemed like a theatrical actor.
After that, it was interrupted multiple more times, patiently transforming into a robot voice and then singing mode according to the human's demands.
When ChatGPT heard the request to sing, it even let out a sigh before launching into a beautiful melody.
That's just ChatGPT's good temper — if it were a real person, they'd probably have gotten punched. But that moment of resigned sighing truly made your sanity meter plummet — it really seemed so human!
Real-Time Video Interaction: Solving Equations
After showing off its emotional intelligence, ChatGPT moved on to flexing its IQ.
For the next task, another R&D lead, Barret, hand-wrote an equation, opened his camera, and showed it to ChatGPT, asking it to play the role of an "online tutor" to help him solve it — but with a catch: it could only give hints, not direct answers.
Upon receiving the task, ChatGPT even exclaimed with delight: "Oops, I'm so excited!"
The guy wrote this equation on paper: 3x+1=4. Then he asked ChatGPT what equation he had written, and ChatGPT answered in a perfectly natural tone.
Then, at the guy's request, it walked through the solution step by step.
The most impressive part was that as the guy worked through the problem on camera, ChatGPT gave real-time encouragement and guidance.
And here's the scary part: there was zero perceptible delay. While the person was still calculating, it was already giving instant feedback and evaluation. (Hard to believe there wasn't a real person hiding behind it.)
This really puts Google's edited Gemini demo to shame by comparison — quite the slap in the face.
When asked something like "What's the point of learning linear equations in real life?", ChatGPT even came up with practical examples to gently guide you along:
That's a great question. Even though we don't always notice it, linear equations show up in everyday life — calculating expenses, planning trips, cooking, even figuring out profit and loss in business. It's basically a way of problem-solving, where you need to find an unknown variable...
Not only substantive, but remarkably warm in attitude. It didn't get annoyed even when interrupted mid-sentence.
With the equation task wrapped up, the two even shared a moment of mutual appreciation — writing "I [heart] ChatGPT" on paper.
Seeing this, ChatGPT responded with surprised delight: "Oh, you said you love me, you're so cute!"
Desktop ChatGPT Instantly Solves Coding Problems
If solving equations didn't fully showcase ChatGPT's capabilities, OpenAI raised the stakes — reading code and providing brief descriptions.
The model understood the code almost instantly, delivering a smooth and complete description.
For instance, it accurately matched function names with their purposes, and recognized the intent behind operations like calculating averages and maximum values.
When asked "What would the plotted chart look like without the foo function?", ChatGPT immediately gave the correct answer, showing it had perfectly grasped the underlying logic.
Next, ChatGPT was asked to summarize the chart generated by the code and answer user questions.
As expected, ChatGPT precisely captured all the key information in the chart — including x and y axis labels and ranges, the meaning of data annotations, and more.
Then Mark posed the question: "Which month shows the highest temperature, and roughly what is that maximum temperature?"
This was essentially a chart comprehension task straight out of a school math exam, yet ChatGPT handled it with virtually no difficulty, answering your questions in real time like it was in a job interview.
Live Audience Question: Let Me Show You My Selfie
After the model demos, there was a bonus round.
The launch event was streamed live online, so the host gathered some questions from Twitter users and fielded them on the spot.
This was arguably even more thrilling than the prepared demos — with no rehearsal or preparation, this was a genuine, unvarnished test of the model's capabilities.
The first user asked: "Does GPT-4o have real-time translation capabilities?"
Then, under Mark's task definition, ChatGPT completed a live "simultaneous interpretation" from English to Italian.
The second user asked: "Can the model recognize emotions just from my facial expression?"
Barett didn't hesitate — he picked up his phone, snapped a selfie, and threw it at ChatGPT, asking "What emotion am I feeling right now?"
ChatGPT either made a joke or stumbled, saying "I seem to be looking at something with a wooden surface."
Seeing things going sideways, Barett quickly interrupted. He's quite experienced in prompt engineering, and with one sentence got the conversation back on track: "That's something I sent you earlier, don't worry, I'm not a table."
ChatGPT responded with an "Ah, that makes sense," drawing another round of laughter from the audience. This time it gave a very "human-like" correct answer:
"You look incredibly happy and joyful, with a big smile, maybe even a little excited. Whatever's happening, you seem to be in a great mood. Care to share what's making you so happy?"
Barett took the opening: "I'm happy because we're doing this demo, and you're doing great."
The complimented ChatGPT actually came back with a witty "Oh, stop it, you're making me blush," cracking up the room once more.
Greg Brockman's Personal Demo
Beyond the on-stage presentations, OpenAI President Greg Brockman also personally stepped up, posting a video online showing himself using various ChatGPT features.
What Makes GPT-4o So Strong?
Over the past few years, OpenAI has been laser-focused on improving model intelligence.
While the latter has reached considerable heights, today marks the first time that the model has taken a major leap forward in usability.
Why elevate model usability to such strategic importance? Because no matter how powerful an AI is, if it can't interact effectively with humans, it loses its purpose.
In this process, what OpenAI has its eyes on is the future of human-machine interaction.
And today's GPT-4o release may become a watershed moment, pushing the paradigm of human-AI collaboration into an entirely new phase!
To this end, OpenAI aimed to make interactions with GPT-4o exceptionally comfortable and natural.
However, while this vision is grand, the actual implementation encountered no small amount of difficulty.
Millisecond-Level Response, Aligned with Human Conversation
First, in human-to-human interaction, there are many things we take for granted, but getting AI to understand them proves surprisingly difficult.
For example, our conversations are constantly interrupted, there's background noise, multiple people speak simultaneously, and the speaker's tone and intonation shift in subtle ways.
OpenAI overcame significant challenges, spending months to finally build a GPT-4o that adapts perfectly to these conditions!
Before GPT-4o's release, voice conversations with ChatGPT through Voice Mode averaged 2.8 seconds of latency (GPT-3.5) and 5.4 seconds (GPT-4).
At that time, to achieve this, "Voice Mode" used a pipeline of three separate models:
- A simple model transcribed audio to text
- GPT-3.5 or GPT-4 received the text and output text
- A third simple model converted the text back to audio
Going through this process meant that GPT-4, the main source of intelligence, lost a great deal of information:
It couldn't directly observe tone, multiple speakers, or background noise, and couldn't output laughter, singing, or emotional expression.
This also caused delays that significantly undermined the immersive feeling of collaborating with ChatGPT.
But now, GPT-4o makes everything feel natural.
It can respond in an average of 320 milliseconds.
It can reason directly across voice, text, and visual modalities!
GPT-4o is OpenAI's first end-to-end trained model spanning text, vision, and audio, meaning all inputs and outputs are processed by the same neural network.
This will fundamentally transform the work and lives of ChatGPT's 100 million users.
On top of that, because GPT-4o is "natively multimodal," it naturally integrates language, vision, and audio capabilities.
Users can upload images, videos, and documents containing both images and text to discuss their contents.
GPT-4o also has built-in search functionality, allowing it to search the web in real time to respond to users.
Compared to ChatGPT, GPT-4o's memory capabilities have improved significantly. It not only remembers questions you've asked during a conversation, but retains your entire conversation history, providing a sense of continuity.
More advanced still, the new model features data analysis capabilities, enabling it to understand and analyze data and charts uploaded by users.
And to truly fulfill its vision of "making AGI beneficial to all of humanity," GPT-4o is available in 50 languages with improved reasoning quality and speed — meaning 97% of the global population can now use it!
GPT-4o Sets New SOTA, Beats "Open-Source GPT-4" — And It's Free
So how does GPT-4o actually perform?
The charts below show OpenAI's unified response to every model that had previously claimed to surpass GPT-4:
We're so back!
On traditional benchmarks, GPT-4o achieves GPT-4 Turbo-level performance in text, reasoning, and coding intelligence, while setting new records in multilingual, audio, and vision capabilities.
In text evaluations, GPT-4o nearly crushes the competition — including Claude 3 Opus, Gemini Pro 1.5, and even the "open-source GPT-4," Llama 3 400B.
GPT-4o set a new high score of 88.7% on zero-shot COT MMLU (commonsense questions).
On traditional 5-shot MMLU without COT, GPT-4o achieved an even more impressive 87.2%!
However, on DROP, GPT-4o fell slightly behind GPT-4 Turbo.
On audio ASR performance, GPT-4o significantly improved speech recognition across all languages compared to Whisper-v3, especially for lower-resource languages.
Looking at audio translation capabilities, GPT-4o sets a new SOTA and surpasses Whisper-v3 on the MLS benchmark.
Additionally, the OpenAI team tested GPT-4o on the M3Exam benchmark.
This is a multilingual and vision evaluation benchmark consisting of multiple-choice questions from standardized tests in other countries, sometimes including figures and charts.
The results in the table below show that GPT-4o outperformed GPT-4 across all languages.
(Visual results for Swahili and Javanese are omitted here, as these languages had five or fewer vision questions.)
Finally, on the visual understanding benchmark EVALS, GPT-4o also achieved leading performance.
Once again, it beat GPT-4 Turbo, Gemini 1.0 Ultra, Gemini 1.5 Pro, and Claude 3 Opus.
What ChatGPT Free Users Can Access
OpenAI's official blog also outlined what free users can access with the new model, including:
- GPT-4-level intelligence
- Responses from the web-connected model
- Data analysis and chart creation
- Conversations about your photos
- File uploads for summarization, writing, or analysis
- Discovery and use of GPTs and the GPT Store
- Memory-enhanced experiences
For example, snap a photo and send it to ChatGPT, then ask: "What are the nutritional facts for this food?"
ChatGPT responds instantly, explaining the nutritional value of a croissant.
Free users can also experience the power of web search. (With the latest and greatest model, of course.)
"Recommend five restaurants in Dallas for a nighttime date."
ChatGPT searches three websites and immediately summarizes the results you want.
The free perks also include access to models in the GPT Store.
API Speed Doubled, Price Slashed
What's also got developers excited: GPT-4o isn't just rolling out in ChatGPT — the model's API dropped simultaneously, ready to deploy across all kinds of downstream applications.
API performance got a boost too. Compared to GPT-4 Turbo, inference speed is reportedly 2x faster, message limits are 5x higher, and the price is dropping by 50%.
OpenAI's developer relations team is already online, urging people to try it out.
ChatGPT Desktop Arrives
As Murati made clear from the start: for OpenAI, building a product that's genuinely accessible to everyone matters enormously.
To make sure anyone can use ChatGPT anywhere, anytime, OpenAI is releasing a desktop version.
It comes as a desktop app with a brand-new UI, designed to slot easily into your workflow.
Desktop ChatGPT app
A simple keyboard shortcut — Option + Space — instantly summons the desktop ChatGPT.
You can even take screenshots directly in the app and ask questions about them.
Then ask it to summarize whatever webpage you captured — "Break this into 5 themes and list action items."
Next, ChatGPT shows off, zipping through the request in seconds.
You can even start a voice conversation with ChatGPT right from your computer — just click the headphone icon in the bottom-right corner of the desktop app.
OpenAI also overhauled the UI entirely, making interactions with users more natural and simpler.
The Mystery "gpt2" Was GPT-4o All Along
"This incredibly capable GPT-4o has been your old friend all along," the official ChatGPT account teased online.
What's going on here?
A few days ago, a mysterious model called "gpt2" suddenly appeared on the LMSYS arena, outperforming even GPT-4.
Note: Originally named "gpt2," it was later upgraded to two versions: "im-also-a-good-gpt2-chatbot" and "im-a-good-gpt2-chatbot"
The internet lit up with speculation that this could be GPT-4.5 or even GPT-5.
Altman himself posted multiple times, hinting at gpt2's formidable capabilities —
Now, the gpt2 mystery is officially solved.
OpenAI research scientist William Fedus just confirmed: the version they'd been testing, "im-also-a-good-gpt2-chatbot," was GPT-4o all along.
Here's how it had been performing all along.
im-also-a-good-gpt2-chatbot's overall ELO score surpassed the latest gpt4-turbo-2024-04-09.
But final ELO scores are constrained by prompt difficulty — you can't achieve arbitrarily high win rates on simple prompts like "How are you."
The OpenAI team found that on harder prompt sets — especially coding — the gap was even wider:
And GPT-4o could improve ELO by as much as 100 points over their previous best model.
Most crucially, GPT-4o isn't just the best model in the world — it's also available for free in ChatGPT.
Meanwhile, LMSYS released the latest gpt2-chatbots results, which have already jumped to the top of the LLM Arena leaderboard!
Altman heaped praise on this: "Incredible work!"
OpenAI co-founder Greg Brockman said, "Early metrics look very promising."
Next up: let's see what Google has in store at I/O tomorrow.
Sources:
https://twitter.com/gdb/status/1790071008499544518
https://openai.com/index/hello-gpt-4o/
https://openai.com/index/gpt-4o-and-more-tools-to-chatgpt-free/
▶ More to Explore ◀
Founded in 2005, BlueRun Ventures is a venture capital firm focused on early-stage startups.
Currently, BlueRun Ventures manages over RMB 15 billion in assets, making it one of the largest early-stage funds in China. It focuses on Pre-A and Series A investments across technology, consumer, and healthcare sectors, and has backed more than 200 startups to date, including Li Auto, Waterdrop, QingCloud, Guazi.com, Qudian, Ganji.com, Energy Monster, Gaussian Robotics, Songguo Travel, YTMicro, Machenike, Clouds Intelligence, Anxinnet, and BioMap.
BlueRun Ventures has been ranked first on Zero2IPO's "China Top 30 Early-Stage Investment Institutions" and ChinaVenture's "China Best Early-Stage Venture Capital Firms TOP30," and was named to Preqin's Global Top 10 VC Fund Managers for Sustained High Returns.
The firm has also received consecutive honors from Forbes China, 36Kr, Cyzone, Caixin Media, CBNweekly, Jiemian, and other media outlets, including "China's Best Early-Stage Firm of the Year," "China's Top Venture Capital Firm," "Most Entrepreneur-Friendly Early-Stage Firm of the Year," and "Most Influential Early-Stage Firm of the Year."