BonsonAI Li Mu: Voice Agents Are Hard, But the Opportunity Is Just Beginning | Vital Views

Dr. Mu Li, founder of BonsonAI, a portfolio company of Oasis Capital, delivered a compelling talk at Silicon Valley 101's annual tech conference on the real-world deployment of Voice Agents.

Drawing on two actual projects — how to make an agent simultaneously play the dual roles of "game designer + game NPC" in an open-world game to deliver a unique player experience, and how to pass rigorous industry certifications to turn AI into a licensed insurance-selling telesales agent — he offered a deep dive into the key lessons and best practices for taking voice agents from technical challenges to commercial deployment.

Below is Dr. Li's presentation. Enjoy.

Yes, today I want to talk about voice agents.

This has been a very hot topic this year, because people believe the technology is now ready for actual use — that it has reached the "deployment" stage.

So today I'll share two projects we've worked on using voice agents, discuss some lessons and best practices we've learned, and explore how to help customers actually put them to use.

First, what is a voice agent? Basically, it's an "agent with a voice" — users interact with the agent through a voice interface. The goal is to make human interaction with large language models more natural.

There are two key points here:

First, real-time performance. This is different from video generation. Video can be processed offline, but voice must be real-time. Response time has to be under one second — what we call "end-to-end latency" must be below one second.

The second point is that in most cases, people aren't just chatting. Chat-oriented agents (like companion bots) are the minority. More often, voice agents are task-oriented. Things like customer service, information retrieval, or selling products. Even "companionship" is a task — it's not just casual conversation. There's always a specific goal behind it, like telling a story or accomplishing something.

That's the basic concept of a voice agent. In the rest of this talk, I'll use two examples to show how we built it and what we learned.

Here's the first example: "I really need to find a way out of here, fast." "Well, looks like you're getting straight to the point. But could you at least tell me a little more about yourself first? Like..." "Okay, let's see [your introduction]." "Alright. I'm a sophomore at Marianna University, majoring in astrophysics."

This is a game we started building about two and a half years ago. It's an app — this is the user interface. The character in the game is named Stella, and players interact with her through voice to help her complete missions.

You could say the entire storyline is driven by the player's voice input.

It's an open-world game. That line above is dialogue captured directly from the game, which launched two months ago.

The basic plot is: Stella crash-lands on an alien planet, her ship has crashed here, and she asks the player for help. "This world is so strange, there are so many options here, I can experience so many emotions." She speaks with the player through voice, asking them to help her escape. The game's world-building is very extensive, and this is just the first experimental installment in the series.

So what role does the voice agent play here?

It's both game designer and actor. As "designer," it needs to create logically coherent, fun plotlines. As "actor," it has to produce dialogue that fits the character's persona.

The character Stella has a complete backstory and personality profile — about twenty pages of design documents. The agent must perform according to these specifications.

From a game design perspective, when players interact with the game, the plot needs to be guided. If the story just moves forward in a single line, that's not an "agent" — it's just a regular script. You need to give the plot complex branching structures, graph structures, so players can freely explore.

The challenge is, you have to make the game "actually fun." Like writing a novel or a screenplay, you have to follow narrative rhythm, foreshadowing, climax — principles that create tension and make the story interesting.

Additionally, players in an open world will constantly test boundaries, will "mess with" the AI. The agent must always stay within character. For example, this game is sci-fi, set two thousand years in the future, and your dialogue is spontaneous. If a player asks, "What movies have you watched lately?" the model can't answer with contemporary films — that would make them "movies from a thousand years ago."

But the problem is, most language models are trained on data from the present-day world. So the question becomes: how do you migrate these "future world" settings into the model?

Let me share an early conversation log (from when it was still the Chinese version) to illustrate the challenge. The scene: Stella finds some food and asks the player, "Which one should I eat?" The player answers, "Let me help you look — I don't think you should eat anything."

So Stella says, "You need to find meat to eat." Then the player goes to search how to hunt, but the game setting is: you haven't seen any animals yet. So Stella replies, "I'd really like to eat meat, but there are only vegetables here." The player still refuses to help. Stella pleads: "I really need your help." The player still says "No."

Now the plot is stuck — the story can't advance. So we added a rule: if the player refuses three times, force the plot forward and have the system automatically choose an option. As a result, Stella would "starve to death." She says, "I'm dying." The player responds: "Then die." This is obviously not a "good player," but the model must remain polite and in-character.

The challenge is: this is an open-world game, but your responses have to be logical. It's set one or two thousand years in the future, and many details aren't defined in the script. Game designers can't write scripts for every possible situation.

The model has to "think" for itself about what's plausible in a future world, while also making the interaction fun. This is a game, not a chatbot. These were the core challenges we faced.

Here's how we approached it: we started this project two years ago. At the time, GPT-4 was available, but extremely expensive. We ran the numbers — using GPT-4 would have meant massive losses.

The best open-source model then was Llama 2, but its performance wasn't strong enough. So we pre-trained our own 30-billion-parameter model, using roughly 5 trillion tokens. The training data came mainly from novels and text-based role-playing games.

The result was roughly comparable to Llama 2 on general tasks, but better at role-playing. Our experience was that pre-training takes months, and even if you slightly surpass Llama 2, you quickly get overtaken by Llama-2-70b.

So spending too long on pre-training isn't cost-effective. That was our lesson. It also suggests that may have been the wrong decision.

Another thing we did: because GPUs were so expensive at the time, we spent considerable effort building our own data center, which significantly reduced costs.

After that, we shifted to post-training — and this is where things got critical. Our narrative workflow was incredibly complex. What I'm showing here is just a simplified example; the real thing was far more intricate.

We hired 20 annotators, but first we had to train them to become "good game writers," because they needed to judge whether an agent's response was plausible, whether it stayed in character, and score and rank accordingly.

After one quarter of training and annotation, our model surpassed GPT-4 in this particular game scenario. But here's the problem: it was just one game, a tiny slice of the entire open world. What if you wanted to do multiple games?

We entered phase two, aiming to expand to more games and characters, with the goal of reducing our reliance on prompt engineers. Because prompt engineering is complex, and different versions of GPT-4 are extremely sensitive to prompt variations.

We wanted game designers to write the prompts, but that wasn't sustainable. A year and a half ago, we realized we needed to pre-train a reward model first — one that could automatically judge which responses were good or bad, rather than relying on human scoring.

So we trained this reward model first, teaching it to recognize quality within game contexts, and then used it to train our main model.

One important lesson we learned: even in something that sounds as simple as gaming, there are massive instruction-following challenges. To be plausible, the model has to maintain sufficient generality. Even a domain-specific model needs to perform well on general tasks.

If the best closed-source model (say, OpenAI's API) scores 90, but in your application the general capability benchmark has to be 85. Below that threshold, you'll hit a ceiling in your specific use case — you can't break through. So first, you have to ensure stable performance on general tasks.

Then, for specific tasks, we built an in-domain evaluation system, with extensive character and scenario setups to verify whether the model follows rules, outputs plausible responses, and strikes the right tone in specific situations.

Once you have this evaluation benchmark, you can fine-tune your model to surpass others in that domain. This is a general principle: you have to care about in-domain performance.

The key is simultaneously ensuring general task performance doesn't degrade, and designing a truly high-quality in-domain evaluation task — only then can you quantitatively see model improvements.

Our overarching takeaway: intelligence still fundamentally comes from large-scale pre-training. Looking back after the project wrapped, all major performance gains came from pre-training on massive amounts of data.

This made us reconsider — abandoning pre-training may not have been the right call. Maybe we should have spent one more quarter on it. That was our lesson.

But even so, the model still had limitations. In complex scenarios, after more than 50 turns of dialogue, quality dropped noticeably and intelligence became unstable.

Additionally, current models still struggle with complex world-building and multi-character environments. Whether it's speech models or video generation models, they can generally only handle two to three characters simultaneously. Even in pure text scenarios, four characters or four parallel settings are extremely challenging.

Another point: in the demo, you saw our latest system is still "turn-by-turn" dialogue. So these projects have all focused primarily on large language models themselves.

But our realization is that if you want to achieve truly "human-like" interaction, you have to adjust the architecture — you can't stick entirely with the traditional three-stage structure. This became the direction for our next project.

"Is now a good time to chat?" "Actually, I'm about to head out." "No problem, John. I know you're busy. If it's convenient, I can call you back later." "Actually, that's not necessary. My company already has health insurance." "That's great. I'm glad you're already covered. But actually, many people I speak with also have employer insurance, and they don't realize there are options that could help reduce their out-of-pocket costs."

Alright, this example is completely different — this is insurance sales. Before it was gaming, now it's selling insurance.

At first I thought selling insurance would allow lots of creative freedom, room to improvise. But in reality, this industry is extremely formal. First, you can't just call anyone — you can only contact users who have actively left their information and expressed interest in the product.

Second, the insurance industry is heavily regulated. Let me explain the specific issues. What we're building now is an AI telemarketer — a voice agent playing the role of "telephone salesperson." The current case involves selling health insurance over the phone, across multiple countries.

There are two regulatory requirements: (First) You must pass a telemarketer certification exam, with a score of 80 or above to go live. (Second) There are performance metrics.

For example, given 1,000 customers, you call them and must complete a certain number of sales. At the same time, customer complaint rates must stay below a cap. If customers feel the experience was poor, or the information was inaccurate, and they file complaints — insurance companies take this extremely seriously.

So the capabilities you need here include: First, sufficient intelligence to answer questions precisely according to the sales script. I'll show what "precise answering" means in a moment.

Second, tool use, because insurance sales involves extensive internal tools for queries and calculations, combining different plans.

Third, the voice has to be very "human-like." When you make a call, the person might be outside, with background noise, speaking with an accent — the agent has to understand, adapt, and sound natural rather than robotic.

Finally, end-to-end latency. When I finish speaking, your response has to come in under one second, or the user will feel like the person on the other end is slow to react.

For example, if a user asks: "What's the maximum payout for dental insurance?" and the agent responds: "Up to $600." That's wrong, completely wrong. This task assessment fails.

The correct, precise answer is: $400 for routine treatment, while the $600 payout is for anterior restoration. It's written in the product materials.

So, "if you have any dental issues" — that kind of response is inaccurate. "If you have [specific issue], [the payout is X]" — that's accurate. If the agent gives the former (non-specific), that response fails.

Another more challenging point, similar to gaming. For example, when the agent tries to schedule an insurance discussion with a customer, there's a rule: maximum three attempts. If all three fail, you must end or reschedule.

For instance: first call, customer says "no." Second attempt, still "no." Third time rejected again, you must stop. If the user says "uh-huh," you might mistakenly think they're interested, maybe even adjust your pitch based on tone change: "Then can I explain how it would benefit you personally?" But in fact, you need to recognize that tone as impatience. Reading the context, you should realize: "I've already tried three times, time to reschedule or hang up."

This is where the complexity of voice input comes in. You don't just need to understand the words — you have to recognize emotion, rhythm, and context.

The next key question is: how do we achieve real-time voice interaction?

I want to show you several model architectures we're currently using.

The first is the most advanced: end-to-end full-duplex. This means there's a single unified model between the user and the system. The user speaks directly into the model with raw audio waveforms, and the model listens and generates responses in real time.

During the interaction, the user can interrupt at any time, and the model can naturally interject with feedback words. If the user says something long, you might chime in with "right," "exactly," and so on. This is currently the most natural form of human-machine interaction.

However, this kind of system hasn't truly been deployed commercially yet. There are only one or two demo versions available to try out, and they're still not very controllable.

In most practical applications — GPT-4o, for example — what you get is end-to-end half-duplex. That is, the system uses a voice activity detector (VAD) to determine whether the user is speaking, splits the audio into segments, and feeds them into the model, which then generates a response to the previous segment. So this is a half-duplex mechanism.

Another approach is the chained two-component architecture. It also operates in a turn-taking mode, but uses two models instead of one. The first model handles understanding — it receives audio and generates a text response. The second model handles generation, converting that text into speech output.

The final type is the chained three-component architecture. The flow is automatic speech recognition (ASR), then a large language model to generate a response, then a text-to-speech (TTS) model to produce the voice output.

End-to-end full-duplex interaction feels the most human because you can interrupt. The three-component approach, while less natural, offers the most customizability. You can more easily add new features or capabilities to the agent.

In actual client projects, we typically use the chained two-component architecture.

For example, we use a 13B understanding model to generate text responses. If the user's question is complex, we call a larger fine-tuned model as a tool to do some "thinking," then hand off to a 1B generation model to output the speech.

These models are actually all based on the same large language model (LLM); they just use different data mixtures for continued pre-training or fine-tuning.

The understanding model, for instance, needs large amounts of audio at varying quality levels, including low-quality speech, because it has to understand humans speaking in noisy environments with diverse accents. At the same time, it needs continued pre-training on text tokens.

The generation model is the opposite — it needs higher-quality, longer-duration audio to improve naturalness and speech fluency. The LLM itself is further trained on domain-specific data.

This architecture makes customization easy. Because both the understanding and generation modules are general-purpose, they can be reused across multiple scenarios. When entering a specific industry, you only need to fine-tune that particular model.

This lets us maintain intelligence while achieving low latency. That's the key to commercializing voice agents.

There are a few core concepts here.

First, a voice agent needs to listen, speak, and think simultaneously — listening to the user while generating response sentences. In between, it can asynchronously call a large language model for deeper thinking or information retrieval. All of these processes happen asynchronously and in parallel.

We also focus on something called context engineering — a step beyond prompt engineering. Because in real business scenarios, product information and other operation manuals might run to 100,000 tokens. The system has to be able to dynamically stitch together this context, automatically generating the appropriate prompt to drive the model.

There's another module called the organizer. It's responsible for identifying user types and automatically switching strategies, while also doing intent analysis — for example, judging ambiguous feedback like "uh-huh," or tracking conversation task progress in real time.

Combine these modules, and we can achieve both high intelligence and low latency at the same time.

This project started at the beginning of this year. We partnered with a Fortune 500 insurance company. We began in January, and when we started using ChatGPT-4 in February, our test score was 55. But you have to hit this bar — the passing score for human salespeople is 80. You have to reach that standard to go live.

We went through a lot of difficulties, but the system improved steadily. It took about half a year to three quarters, but we finally reached and even exceeded human-level performance.

One lesson we learned here: evaluating end-to-end voice agents is extremely difficult, because you have to test with real human phone calls. Once real calls are involved, it's very hard to automate evaluation. But without this kind of testing, you can't truly measure overall system performance.

There are still ongoing challenges. Handling complex product combinations in real-time scenarios remains very difficult. Insurance, for example, has massive numbers of products and pricing combinations. When a customer says, "That's too expensive, I want a cheaper option," the system has to pick out suitable alternatives in real time.

Another issue: high security requirements drive up costs. Someone in a forum discussion asked whether OpenAI might become the sole dominant AI player. In the B2B space, the answer is no. Take insurance as an example — when deploying in different countries, data can't leave the country, and in some cases can't even leave the company's internal security zone.

So you either rent GPT models to run under your own account, or you have to build your own. That's why advancing AI is so difficult. And it's precisely why we've invested heavily in building our own full stack of models, rather than just doing prompt engineering on top of APIs.

I've just shown you two voice agent projects we developed over the past two years. Our takeaway: voice agent tasks are highly scalable.

Even though the gaming industry setup and the insurance industry setup differ greatly, from a technical perspective they share the same model architecture and training methods. What differs is just the data types and evaluation approaches. Whether it's pre-training, post-training, or data labeling pipelines, all require enormous human effort — the essence is the same. Games emphasize "fun and engagement," phone sales emphasize "precision and compliance," but both require careful handling of user (corpus) input.

That said, I believe these systems are now ready for real-world deployment, though still in the Day One stage. In gaming, for instance, we're still at single-character, small-world scale. Expanding to multi-character, massive world-building remains very difficult. In phone sales, we can now sell about five health insurance combinations for a given company; selling general product types is still hard.

Overall, these voice salespeople work best for products in the $500–$5,000 price range. Of course, selling entirely new products with existing models still requires extensive tuning. But application scenarios are expanding rapidly. Previous customer service systems were text-only; now they can all add a voice interface, which means enormous potential applications.

I believe the commercial deployment of voice agents is just beginning. The next few years are going to be incredibly exciting.

Finally, if you're interested in working with us or partnering, please reach out.

(This article was edited by the Datawhale team. Click Read Original to access the talk video.)