Conversational artificial intelligence has taken a qualitative leap with voice-to-voice (V2V) models. No longer are we dealing with systems that transcribe, process text, and then synthesize; now audio goes in and out directly, preserving nuances like pauses, intonation, and even interruptions. Understanding how these architectures work is key for companies looking to integrate voice assistants into their processes, whether in customer service, task automation, or custom software.
The simplest form of V2V is the cascade: an automatic speech recognizer (ASR) converts audio to text, a language model (LLM) generates the response, and a text-to-speech (TTS) engine voices it. This approach is transparent, easy to debug, and allows component swapping, but it loses all prosodic information — tone, rhythm, sighs — that does not fit into text. Native voice systems, on the other hand, work directly with discrete representations of sound, such as tokens from a neural codec. For example, Mimi (from Kyutai) compresses one second of audio into 100 tokens distributed across eight parallel streams. The predictive model — an autoregressive transformer — learns to generate those tokens sequentially, later reconstructing the waveform through a decoder.
Audio representation is crucial. A waveform sampled at 24 kHz requires 24,000 points per second, which is impractical for a transformer due to quadratic attention. Instead, a codec like EnCodec or DAC reduces the frame rate to 50–75 frames per second, each carrying several residual vector quantization (RVQ) indices. This yields bitrates around 2 kbit/s, 192 times lower than the original PCM, but with losses: whispers, rare accents, or background noise may degrade. However, the codec's quality limits the final system's acoustic quality, not its understanding capability.
Architecturally, a typical V2V model consists of four blocks: an input encoder (converting the waveform to embeddings or tokens), a transformer backbone (managing context and history), output heads (predicting audio or text tokens), and an audio decoder (reconstructing the waveform). The backbone is usually a text LLM adapted to audio, with causal attention and a KV cache to avoid recomputing past states. In full-duplex systems like Moshi, two audio streams (user and assistant) are modeled in parallel using a temporal transformer and a depth transformer, achieving latencies around 200 ms.
Training these models is expensive. First, the codec is trained with reconstruction objectives, adversarial and perceptual losses. Then the voice language model is pretrained on hundreds of thousands of hours of audio, often with automatic transcriptions. Next, the sound representation is aligned with text using audio-transcription pairs, and finally the model is fine-tuned with dialogues and human preferences (RLHF). Safety is also addressed in this phase, although production control requires additional layers of filters and tool policies.
In real time, the biggest challenge is end-to-end latency. Voice activity detection (VAD) decides when the user has finished; a turn-ending model prevents premature cuts. Streaming sends chunks (20–240 ms) that the encoder processes incrementally while the decoder already starts generating output. Barge-in allows the user to interrupt playback; full-duplex goes further by keeping both channels active, modeling overlaps and backchannels. These design decisions directly affect user experience and computational cost.
Prosody and emotion are another front. A native voice model can capture tone, speed, and pauses, but it does not interpret emotions: it only correlates acoustic patterns with training responses. Gemini Live, for example, offers affective dialogue in specific versions, but most closed APIs do not guarantee a correct reading of mood. For critical applications (banking, healthcare), it is safer to ignore emotional inference and focus on neutral, verifiable responses. This is where cybersecurity comes in: preventing an attacker from injecting voice commands or cloning an identity. Cybersecurity solutions must integrate watermarks and authentication policies.
In the business domain, V2V models are transforming contact centers, virtual assistants, and automation systems. A company can deploy a voice agent that understands complex queries, accesses databases, executes actions, and communicates naturally. However, evaluation must be rigorous: P95 latency, accuracy on names and numbers, task success rate, and cost per session. It is not enough that it sounds good; it must solve the customer's problem.
At Q2BSTUDIO we develop custom software solutions that integrate voice models, artificial intelligence, cloud computing (AWS/Azure), Business Intelligence with Power BI, and process automation. Our team combines technical knowledge of V2V architectures with experience in production deployments, ensuring that the voice not only sounds human but meets the security, latency, and auditability requirements of the business. If you are evaluating incorporating voice into your applications, we help you choose between a transparent cascade or a native system, always with objective metrics and a pragmatic approach.
The future of voice-to-voice lies in multimodal models (audio, text, image), token efficiency and faster decoders, and better long-context management. Cascades will not disappear because they offer control and observability; native systems will gain in naturalness. The key is to measure, not to believe.
