This post introduces a NAVER Cloud paper presented at ACL 2026, the flagship conference in natural language processing (NLP) and artificial intelligence (AI), held this July in San Diego, California: “Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models.”
Jung Kyudan (NAVER Cloud*, KAIST)
Kim Jihwan (NAVER Cloud*, KAIST)
Kim Soyoon (NAVER Cloud)
Kim Jeonghoon (NAVER Cloud)
Choo Jaegul (KAIST)
Park Cheonbok (NAVER Cloud)
*NAVER Cloud Residency Program
The first step toward voice AI that converses in real time
What does it take to build a Korean-language voice AI that converses as naturally as a person?
In a previous tech blog post, we described how HCX-OMNI is extending HyperCLOVA X to understand speech more accurately and speak more naturally. But even as models get better at listening and more fluent at speaking, much of today’s voice AI still works in one direction: the user asks → the AI answers → the user waits again.
Real human conversation doesn’t work that way. We break in mid-sentence with “Hold on,” or drop a quick “Yeah” or “Right” while listening to an explanation. Sometimes we change what we’re asking for before an answer is finished, or fire off a follow-up question right in the middle of what we’ve just heard.
We’re now moving past this limit to prepare a more human kind of voice conversation, one where the AI and the user interact in real time. As a first step, we want to share a data preprocessing pipeline for voice AI that can be interrupted mid-answer and can take new questions while it’s still speaking—just like a real conversation.
From an AI that answers and waits to one that responds mid-conversation
Until now, much of voice AI has followed a fixed structure: the user speaks → the model listens → the model answers → and only then does the user speak again. This is the half-duplex approach. Like a walkie-talkie, one side has to wait while the other is talking.
By contrast, a full-duplex voice AI is much closer to how people actually talk. Even while it’s speaking, it keeps listening to the user’s voice, judging in real time whether they’re happy with the answer, asking something new, or breaking in to revise what they asked for.
Suppose a user asks for suggestions for a weekend date.
In the half-duplex approach, while the AI is saying “In the morning you could take a walk…,” the user just has to wait. Even if the user cuts in with “Wait, focus on indoor options,” the AI can’t act on it until it has finished the answer it was giving.
In the full-duplex approach, the moment the AI hears the user’s “Wait,” it stops its current explanation and changes course: “Got it, let me redo this around indoor options.” When the user just gives a short “Yeah” or “Sounds good,” on the other hand, the AI doesn’t seize the floor—it simply keeps its explanation going.
The point of a full-duplex voice AI isn’t simply to answer quickly. It’s to hold a human-like conversation: knowing when to wait for the user, when to take in what they say, and when to stop speaking.
So how does a model learn to converse like this? The starting point lies not in the model architecture but in the training data. To converse like a person—chiming in, breaking in—a model has to learn from data that captures exactly that kind of conversation. So before building the model, we decided to start with the data that would become its foundation.
The Sommelier pipeline we’re introducing today refines real conversational speech into training data that keeps natural, real-time conversation intact, and we’re releasing part of it as an open-source pipeline for research use. It’s the first step we’re taking on the long road toward real-time voice AI.
Noise to delete, or signal to keep?
Training a full-duplex model calls for data that captures how people actually talk. And here lies a fundamental problem: most existing speech training data consists of a single person reading text or explaining something at length. Even when multi-speaker conversation data exists, it’s often recorded at the audio quality of an old phone call, or it lacks the scale and environmental diversity needed to train today’s large speech-language models.
The deeper problem is that conventional speech preprocessing has treated overlapping speech as an error or as noise. In the name of producing clean audio, it deletes the parts where two people talk at once, or keeps only one speaker’s voice. But look closely at real conversation and overlap isn’t just noise to be removed.
Let’s take the case of someone getting advice on a travel itinerary.
Here, the user’s first “Okay” is a signal for the AI to keep going—a brief acknowledgment that they’re following along. The second, “Wait,” is an interruption that changes the direction of the conversation. Because the user has added a new condition, the AI needs to stop its current answer and re-plan to include the show.
Both utterances may overlap with the AI’s voice, but they play completely different roles in the conversation. Train a model on data that has deleted all of these segments, and the AI struggles to learn both the timing of natural backchanneling and how to respond to interruptions.
Conventional preprocessing methods built for half-duplex data removed behaviors like these natural interruptions, or preprocessed only the final answer, when building a dataset. So our core principle is clear: break away from the conventional preprocessing approach and preserve this natural backchanneling and these interruptions in a form a full-duplex model can learn from.
Sommelier: A dataset pipeline that revives conversation in three stages
Sommelier is an open-source pipeline that turns raw real-world conversation into training data where the flow of human conversation stays intact. The whole process breaks down into three main stages:
- Identifying who spoke and when,
- Restoring overlapping voices speaker by speaker, and
- Cross-checking multiple speech-recognition outputs to verify the transcription.
This lets us preserve in the data the way human conversation unfolds and the timing of it—from short backchannels like “Yeah” to interruptions like “Hold on.” It’s a process that turns the interactions arising naturally in real-life conversation into training signal.
Stage 1: Identifying who spoke and when
The first stage in building full-duplex training data is figuring out “who spoke and when.”
A long audio file contains not just the actual conversation but also silence, background sound, and meaningless stretches. We first find the segments where a voice is present, then split long audio at the silences so the conversational flow isn’t broken.
Next, we identify which speaker each utterance belongs to. This isn’t simply turning speech into text; it’s a process of building a timeline of the conversation, like this:
- 00:01.2 — Speaker A
- 00:02.0 — Speaker B
- 00:02.4 — Speaker A
- 00:03.1 — Speaker B
This timeline is what lets the model learn not just “who said this sentence” but “at what moment they broke in.” Short utterances like “Yeah,” “Mm,” and “Right” matter especially in a full-duplex model, because they may not be new questions at all but signals that set the rhythm of the conversation. So we designed Sommelier’s speaker-separation stage to capture short utterances and quick speaker changes more reliably. Specifically, we use voice activity detection to find the segments of the original audio where speech is actually present, then run speaker diarization to determine which speaker each segment belongs to. This produces a conversational timeline showing who spoke from when to when.
Stage 2: Restoring overlapping voices instead of erasing them
In human conversation, two people often talk at the same time—one person is explaining something while the other chimes in with “Ah, right” or breaks in with “Hold on.”
Conventional methods often cut these overlapping segments out or kept only one speaker’s words. But for a full-duplex model, it’s precisely these overlapping segments that matter, because they’re the segments where it can learn how a user reacts while the AI is speaking.
Consider this situation, for example.
In this exchange, Speaker B’s “Mm” is a short reaction, while “Wait” is an interruption that changes the direction of the conversation. Even though both utterances overlap with Speaker A’s speech, neither should be thrown away. To handle this, Sommelier uses overlap separation to restore the segments where two people’s speech overlaps. It first extracts each speaker’s voice characteristics—their speaker embedding—from the non-overlapping segments. Then it separates only the parts where voices actually overlap, using a speech separation model, matches each separated voice back to its speaker, and stitches it onto the original utterance.
Next, we check for background sound, since real conversation data can contain all kinds of background audio on top of human voices. Because this can interfere with training, a BGM (background music) detector first finds the segments that contain background sound, and speech separation is applied again only to those segments to extract just the speech. In short, overlapping utterances are salvaged with overlap separation, while background sound is cleaned up—only where needed—with the BGM detector and speech separation.
This keeps Speaker A’s sentence intact while also preserving Speaker B’s backchannels and interruptions. So the data the model learns from records not just “what was said” but “when, and over whose words, it was said.”
Stage 3: Combining multiple speech-recognition outputs to extract the text
The final stage, ensembled ASR, adds text to the audio for training. Training a full-duplex model requires not just the audio but also the text that corresponds to each utterance.
But relying on a single speech-recognition model can cause problems. Every model has its weak spots, so it may invent words that were never said or repeat the same words over and over.
For example, the original audio may have contained only a brief bit of noise, yet the speech-recognition output ends up with a repetition or hallucination like this:
“Yeah, yeah, yeah, yeah, yeah…”
If a transcription like this goes straight into the training data, a model trained on it can end up behaving the same way. To prevent this, Sommelier uses an ensemble method that combines the outputs of several speech-recognition models and picks out the most reliable transcription. This detects and heads off the repetitive generation that these models frequently produce, so that one recognizer’s error doesn’t turn into a bad conversational habit in the full-duplex model.
Training results: A model that waits better and interrupts better
We also checked directly, through a full-duplex benchmark, whether the data we built actually helps in training a full-duplex model. We ran 83 hours of conversation data through the Sommelier pipeline, used it to further train the full-duplex Moshi model, and evaluated backchanneling, interruptions, and natural turn-taking during conversation.
The results were clear. First, in backchannel segments—where the user can react briefly with “Yeah” or “Right”—the rate at which the model started an unnecessarily long utterance fell from 1.000 to 0.291, and the frequency of short backchannels rose from 0.001 to 0.052.
The rate at which the model naturally took over after the user finished speaking also rose, from 0.941 to 1.000. Finally, the score measuring how well the model responded to the content of a user’s interruption mid-utterance improved sharply, from 0.765 to 3.684.
In other words, a model trained on this data is better at actually listening to the user, stopping its own speech when needed, and producing a response that fits what has just come in.
The first step toward a full-duplex model
This post has laid out how we’ve thought about what data a full-duplex AI model needs and which features of real conversation have to be preserved.
Human conversation doesn’t unfold in a fixed order. We react with “Yeah” mid-sentence, and when a new condition comes to mind we break in with “Hold on.” Sometimes two people talk at once, and ambient noise and background sound mix into the conversation.
Sommelier preserves this complexity instead of erasing it. It identifies who spoke and when, restores overlapping voices speaker by speaker, and combines multiple speech-recognition outputs to build more reliable training data. As a result, the model learns far better when to wait, when to respond with a short acknowledgment, and when to stop speaking.
Releasing the Sommelier pipeline is a starting point. The pipeline, together with NAVER Cloud’s own know-how in building full-duplex datasets, will form the foundation of the full-duplex models we’ll unveil going forward. We’ll keep sharing the technical insights and results we gain along the way.
You can find the code and a demo for the Sommelier pipeline in the open-source repository and on the project page.