NetEase Youdao's Open-Source Real-Time Speech Suite Hits No. 1 on Hugging Face

The Voice Agent Track Gets a New Open-Source Model.

👦🏻 Author: GaKi

🥷 Editor: Koji

🧑‍🎨 Layout: NCon

On July 24, Jensen Huang posted his first tweet on X, attaching a joint open letter titled Open Weights and American Leadership in AI. Within days, the list of signatories grew to more than 150, including OpenAI and Google.

One immediate backdrop for this letter was the progress of Chinese open-source models over the past year.

In recent months, China has released several noteworthy open-source models, including Kimi K3 with 2.8 trillion parameters, Qwen3.8-27B, and Step 5 Preview (expected to open weights in October). These are all large language models, and they've captured most of the industry's attention.

But the truly notable shift has been happening beyond large language models.

In mid-September, NetEase Youdao open-sourced two real-time voice models: R2T2 and T3PO. R2T2 handles real-time speech-to-text transcription, while T3PO handles real-time translation. Within days of release, both models climbed to #1 on Hugging Face's ASR (Automatic Speech Recognition) and Translation trending lists, respectively.

That same month, OpenAI opened API access to its full-duplex voice model GPT-Live-1, and Google released Gemini 3.8 Live.

At a time when major tech companies at home and abroad were simultaneously fixating on the "real-time voice" niche, NetEase Youdao achieved a fairly impressive result — no small feat.

Chinese open-source models are beginning to move beyond the most visible battleground of large language models, pushing deeper into more granular, specific technical layers like speech recognition and real-time translation. And they've already reached the top of the leaderboards.

So what makes these two models' rankings truly worth attention isn't simply that "NetEase Youdao built two good models."

The more significant change is that the influence of Chinese open-source models is permeating layer by layer through the entire AI technology stack.

---

This article will use these two models as a starting point to discuss two questions:

[1] Why does "real-time voice" — a niche track that receives far less attention than large language models — matter?

[2] Why has NetEase Youdao persisted in this area?

Chinese Open-Source Models Begin Entering Specialized Directions

Hugging Face released a spring open-source report in March 2026, showing that Chinese models accounted for 41% of platform downloads over the past year, surpassing the United States in both monthly and total downloads.

If you look at when Chinese models overtook the US in download volume, it happened even earlier — starting in July 2025.

In September 2025, Alibaba's Qwen surpassed Meta's Llama to become the most-downloaded language model family on Hugging Face. According to official Hugging Face statistics, more than 150,000 derivative models were built on Qwen — more than the combined total for Google's and Meta's foundation models.

Beyond downloads, on the model aggregation platform OpenRouter, Chinese open-source models' share of actual token calls was under 1.2% at the end of 2024; by April 2026, it had approached 50%.

However, most of these numbers still come from general-purpose large models like DeepSeek, Qwen, Moonshot AI, and GLM. The conversation mostly revolves around parameter scale, whether it's a dense model, whether it has 1M context length, and scores on coding and reasoning benchmarks.

But if you follow model developments regularly, you'll notice that the situation in specialized directions is somewhat different, particularly for "real-time voice models."

After all, when a developer downloads a model from specialized ASR or translation leaderboards, they're usually already planning to integrate it into their product.

So, in a sense, specialized leaderboards better reflect real demand in specific scenarios.

The developer community's response to R2T2 and T3PO illustrates this point.

First, a brief introduction to these two models.

R2T2 is a streaming speech recognition model — meaning it transcribes speech to text in real-time as a person speaks, without revising already-displayed words.

T3PO is a simultaneous interpretation model that provides real-time bidirectional translation between Chinese and English.

After R2T2's release, Apolinario from the Hugging Face open-source team posted a comment on the model's discussion page.

In the post, they said they used an Agent to build an interactive demo for R2T2, hosted it on Spaces running on free ZeroGPU infrastructure, and offered to transfer the Space to Youdao.

A third-party developer ported R2T2 to audio.cpp, a pure C++ local voice inference framework, reimplementing its streaming state machine and adding local microphone real-time recognition.

This code was merged into the main repository on September 19.

The community also created multiple GGUF quantized versions of R2T2 and T3PO, from Q2 to Q8. Developers can run both models on their own computers using llama.cpp or Ollama.

Porting, quantization, and adapting to local frameworks all require meaningful time investment from developers. That people are willing to do this indicates teams are already preparing to use them in real projects.

So specialized-direction models are also among the most intensively redeveloped by developers.

Having covered the performance of open-source models, especially "real-time voice" specialized models, among developers, let's now turn to "commercial scenarios."

The Hardest Part of Voice Agent Is "Real-Time"

One of the biggest commercial scenarios for real-time voice models is Voice Agent.

Ten months ago, the Crossing team published an article titled Inventory: 18 AI Voice Agent Startups Worth Watching in 2025, providing a detailed overview of startups in this track. At the time, we made a prediction:

"People are seriously underestimating the potential of Voice as an AI interaction interface."

A Voice Agent refers to an AI Agent that can listen, speak, and execute tasks — completing multi-step operations like booking appointments, processing refunds, or placing orders during a single phone call.

In 2026, this direction is even hotter than it was in 2025.

a16z listed "Voice Agent moving from tech demo to scaled deployment" as one of its three predictions for 2026. Andrew Ng has also publicly stated that many people underestimate the importance of the voice technology stack.

The technical changes have also concentrated this year.

In July, OpenAI released the full-duplex model GPT-Live-1, replacing ChatGPT's previous advanced voice mode. Full-duplex means the model can listen and speak simultaneously, responding with "mm-hmm" and "right," and distinguishing whether the user is pausing to think or has finished speaking.

Testing by language learning company Speak showed that GPT-Live-1 reduced false interruptions during learner thinking pauses by nearly 80% compared to turn-based systems.

This is a major upgrade, and as a result, many people have been "complaining" across various community platforms:

GPT-Live-1 is, honestly, a bit uncanny valley.

In September, GPT-Live-1 opened API access, priced at approximately $0.05 per minute. Including phone lines and other components, the comprehensive cost of a one-minute AI call is already below that of human customer service.

Real-time voice models also have substantial potential on the AI hardware side.

AI glasses and headphones mostly lack screens; voice is the primary input and output method. Meta and EssilorLuxottica sold over 7 million pairs of glasses in 2025, more than triple the previous year.

EssilorLuxottica 2025 financial report, original text

One of the most commonly used features on such hardware is real-time translation. Meta's glasses, Google and Samsung's Android XR glasses, and Apple's AirPods all feature real-time translation as a headline capability.

Translation scenarios are extremely latency-sensitive — after all, the person speaking in front of you won't wait for you to "think and receive the translation."

So the challenges of Voice Agent converge on "real-time."

Duan Yitao, NetEase Youdao's Chief Scientist, made a point at the Youdao AI Open Day on September 16:

In the past, human-AI communication was "I say one sentence, you say one sentence" — the system could judge when the user finished speaking and when to respond. In real-time voice scenarios, the model faces incomplete information at every moment.

Because a real-time voice pipeline typically divides into three segments:

[1] Speech recognition converts sound to text

[2] AI comprehends the text and decides how to respond or translate

[3] Speech synthesis reads the result aloud

The output of the first segment determines whether the latter two can begin working promptly — and this is precisely R2T2's domain.

R2T2's most distinctive feature is "append-only," meaning: already-output characters are never revised; newly recognized content is only appended at the end.

Notably, accuracy did not decline after adopting "append-only."

In NetEase Youdao's internal tests with 160-millisecond chunks, as well as on the AMI meeting dataset that more closely resembles real-world scenarios, R2T2 achieved quite strong scores:

On latency, R2T2's average recognition delay falls between 200 and 600 milliseconds.

The model has 2B parameters and can run on personal computers after quantization.

Glasses and headphones have limited battery and compute; a voice model of this size holds practical value for hardware manufacturers.

On the other hand, T3PO is more adept at trading off between quality and latency.

T3PO is a 15B-parameter simultaneous interpretation model supporting Chinese-to-English and English-to-Chinese translation, offering three modes: low-latency, native, and high-quality.

Importantly, it uses the Apache 2.0 license, allowing developers free commercial use.

Used in sequence in real tasks, the two models perform better together — for example, after voice input, R2T2 transcribes to text in real-time, T3PO translates in real-time, and finally subtitles or synthesized speech are output.

Why has NetEase Youdao kept investing in "real-time voice"?

Looking at the overseas real-time voice track, capital concentrates at two ends.

One end is underlying voice model and real-time audio/video technology companies. ElevenLabs raised $500 million at an $11 billion valuation in February 2026; Deepgram and LiveKit each completed over $100 million in funding earlier this year.

The other end is enterprise customer service applications. Sierra, Decagon, and Parloa raised a combined approximately $1.55 billion in the first half of 2026.

Major tech companies are also moving downstream. OpenAI and Google directly offer low-cost integrated voice APIs; Apple and Meta have made real-time translation a free feature on headphones and glasses.

So independent companies in the middle layer face the most pressure — for example, Palabra, a startup focused on real-time simultaneous interpretation, raised $8.4 million in January this year, which some consider undervalued.

In this landscape, latecomers need to choose a specific position and build technical distinctiveness within it.

NetEase Youdao's chosen position of "real-time recognition and real-time translation" appears connected to its accumulated experience.

Youdao has long worked on dictionary, translation, and translation pen products, serving language scenarios like word lookup, translation, and oral practice. So it has long-standing accumulation in language data, translation quality, and user needs in real-world scenarios.

In 2023, Youdao released its education-focused large model "Ziyue." Now, its model matrix covers multimodal reasoning, translation, and voice interaction.

At the September 16 Open Day, Youdao categorized these two newly open-sourced models under the "Ziyue Live" series. This series is positioned as a real-time interaction model matrix, with R2T2 and T3PO as the first two releases.

In the past, external impressions of Youdao were mostly tied to dictionaries, translation, and learning apps. What reached the top of Hugging Face's specialized leaderboards this time were two underlying models, reflecting Youdao's long-term model investment in the voice direction.

This body of work has received relatively little external attention in the past.

On the other hand, major tech companies tend to provide end-to-end solutions — covering voice input and output, with the middle process largely inaccessible to developers.

What Youdao open-sourced are two separable components. Developers can freely combine R2T2 and T3PO with their chosen large models and speech synthesis models, or deploy them on their own servers.

Enterprises have demand for this kind of controllability. For example, voice Agent development platform Vapi won an Amazon Ring contract this year among 40 competitors. Vapi's approach is letting developers independently swap out recognition, large model, and speech synthesis modules.

Amazon Ring's decision not to use a major tech company's integrated API directly indicates that large enterprises still value control over each individual component.

---

Looking back at this year, the influence of Chinese open-source models continues to rise.

Models like Kimi K3, Qwen 3.8 27B, and Step 5 Preview have given China its place at the frontier of open-source large models.

Now, this influence is beginning to extend into more specialized directions.

Voice Agent requires real-time recognition, real-time translation, full-duplex dialogue, and on-device deployment. Each is a specific technical problem, and each requires specialized models to solve.

R2T2 and T3PO reaching #1 on two Hugging Face specialized leaderboards is a highly representative milestone.

In recent years, overseas developers came to know Chinese AI first through general-purpose models like DeepSeek, Qwen, and Moonshot AI.

Going forward, they will likely begin to know more Chinese AI teams through a speech recognition model, a translation model, an on-device model.

Chinese open-source models are no longer just competing on the most visible main tracks. Now, even in those narrower, harder-for-ordinary-users-to-see technical links that directly determine product experience, Chinese models are beginning to appear.

And as AI truly enters industry, competition will focus on one specific problem after another, one specific model after another.

R2T2 and T3PO are just two of them.

Join the membership group