Back to Blog

Qwen3.5-Omni: An Omni Model That Acts in Real Time

Most "omni" models are, underneath, very good perception-response machines: you hand them audio or video, they hand you text back. Qwen3.5-Omni is built around the claim that this is the wrong shape for the job. Real interaction is not a sequence of turns with a passive observer — it involves interrupting, being interrupted, calling tools mid-conversation, and speaking while still thinking. The report frames the model as a native omni agent: it perceives text, images, audio, and audio-visual streams, and it also acts, autonomously invoking WebSearch, executing FunctionCall, and speaking in real time. The most interesting engineering in the paper is what that "real time" costs.

The Qwen3.5-Omni architecture. A Vision Encoder and the AuT audio encoder feed interleaved multimodal tokens with explicit timestamps into the Hybrid MoE Thinker. The Talker reads hidden states extracted from the Thinker's middle layers and autoregressively emits a multi-codebook RVQ sequence; an MTP module fills in residual codebooks per frame and a streaming codec decoder renders the waveform frame by frame.The Qwen3.5-Omni architecture. A Vision Encoder and the AuT audio encoder feed interleaved multimodal tokens with explicit timestamps into the Hybrid MoE Thinker. The Talker reads hidden states extracted from the Thinker's middle layers and autoregressively emits a multi-codebook RVQ sequence; an MTP module fills in residual codebooks per frame and a streaming codec decoder renders the waveform frame by frame.

Thinker and Talker, now both sparse

The Thinker–Talker split goes back to Qwen2.5-Omni: the Thinker does the reasoning and emits text, and the Talker turns that into speech while reading the Thinker's hidden states directly rather than its finished words. What changes in 3.5 is that both halves become Hybrid-Attention Mixture-of-Experts models borrowed from Qwen3.5. That matters more for the Talker than it sounds — a speech decoder that has to keep pace with a live conversation is exactly where a Gated Delta Net's cheap long-sequence handling pays off, because it cuts the KV-cache I/O that otherwise dominates long-context decoding and caps how many conversations you can serve at once.

The context budget is the other headline: 256k tokens, which the report cashes out as more than 10 hours of audio or over 400 seconds of 720P video sampled at 1 FPS. Audio and video are interleaved into one sequence rather than processed in separate towers, which is the only arrangement that makes sense if you believe — as the authors argue — that audio-visual streams are the most naturalistic representation of the world, with the two modalities intrinsically coupled rather than independently observed.

The AuT audio encoder. Trained from scratch on 40 million hours of audio-text pairs, it downsamples filter-bank features 16× through four Conv2D blocks before self-attention, producing audio tokens at a 6.25 Hz rate.The AuT audio encoder. Trained from scratch on 40 million hours of audio-text pairs, it downsamples filter-bank features 16× through four Conv2D blocks before self-attention, producing audio tokens at a 6.25 Hz rate.

The audio front end, AuT, is trained from scratch on 40 million hours of audio-text pairs — labels generated by Qwen3-ASR rather than humans. Filter-bank features are downsampled 16× through four Conv2D blocks, giving a token rate of 6.25 Hz, or about 160 ms of signal per frame. Two details are worth noting: the language mix is deliberately balanced at 3.5 : 3.5 : 3 across Chinese, English, and everything else, covering more than 20 languages; and the encoder is trained with a dynamic attention window, so the same weights behave well both under real-time prefill caching and on offline understanding tasks where the whole clip is available.

The clock mismatch between text and speech

Here is the problem I hadn't seen stated so cleanly before. A text tokenizer and a speech tokenizer compress at completely different, language-dependent rates. When a model generates text and speech tokens on two parallel tracks, those tracks drift — and in streaming synthesis the drift surfaces as skipped words, mispronunciations, and mangled numbers. It gets worse for languages the text tokenizer encodes inefficiently. Qwen3-Omni handled this with a dual-track Talker; the report is candid that this was a source of instability.

ARIA (Adaptive Rate Interleave Alignment) replaces the two tracks with a single interleaved stream of text and speech tokens, governed by one constraint: for any prefix of the generated sequence, the cumulative speech-to-text token ratio must not exceed the item-level global ratio. That's it. No forced alignments from MFA, no fixed interleaving rate hand-tuned per language. The ratio constraint adapts on its own to whatever encoding efficiency a given language happens to have, and it naturally permits an arbitrary text prefix followed by coherent speech continuation — which is precisely the shape of streaming generation.

The Talker also gains a dedicated system prompt describing the target voice. Compared with a speaker embedding, a prompt can carry textual descriptions and codec sequences, which is what makes zero-shot voice cloning from a user-supplied sample and end-to-end control over volume, speed, and emotion fall out of the same mechanism rather than needing three separate ones.

Timestamps, written out as text

Qwen3-Omni used TM-RoPE to tie temporal position IDs to absolute time. The report identifies two failure modes: for long audio-video inputs the position IDs become enormous and sparse, which weakens long-range temporal modeling; and learning under that scheme demands training data sampled uniformly across many frame rates, which is expensive to build.

The fix is almost aggressively simple. Prepend each video or audio-video temporal patch with a plain formatted text string giving its timestamp in seconds, and sprinkle timestamps at random intervals through audio sequences too. The model learns to read timecodes the way it learns to read anything else. It costs a little context length and buys back precise temporal perception that extrapolates to long inputs — and it's what makes the model's screenplay-level captioning, with automatic scene segmentation and timestamped descriptions of who is speaking and how they relate to the audio, actually work.

Training: build specialists, then close the audio gap

Pre-training runs in three stages. First the LLM is frozen while the vision and audio encoders (and their adapters) are aligned to it. Then everything unfreezes for a general stage of roughly 4 trillion tokens at 32k sequence length — and the modality split is the interesting part: audio 1.99T, image 0.95T, text 0.92T, video-audio 0.29T, video 0.14T. Audio is the single largest slice, which is not how most multimodal models are built. Finally the window extends to 262,144 tokens with long audio and video upweighted.

Post-training the Thinker takes three stages, and the middle one solves a problem specific to omni models. After distilling a suite of domain specialists (text, reasoning, coding, agentic, vision, audio) into one model, a gap remains: the same question asked by voice gets a worse answer than the question asked in text. So they run on-policy distillation across modalities — generate the response under the text condition, then use it as the distillation target for the audio-conditioned version of the same query. The model is taught to answer as well when listening as when reading. A third stage of interaction-aligned RL then targets the failures that only appear in real conversations: unintended language code-switching, persona drift, and instruction-following that decays over long contexts.

The Talker gets four stages of its own: pre-training on over 20 million hours of multilingual speech with multimodal context; a long-context stage that does continual pre-training on a quality-stratified subset (curated with help from Qwen3-Omni-Captioner) to scrub hallucinations picked up from noisy data, extending to 64k; DPO against human-annotated multilingual preference pairs, plus rule-based rewards under GSPO; and finally light speaker fine-tuning.

Where it lands

On audio understanding, Qwen3.5-Omni-Plus edges past Gemini-3.1 Pro on MMAU (82.2 vs 81.1) and MMSU (82.8 vs 81.3), and opens a wide gap on music: RUL-MuchoMusic 72.4 vs 59.6. On end-to-end speech dialogue it takes VoiceBench 93.1 vs 88.9. Gemini keeps MMAR (83.7 vs 80.0), SpeechRole, and WildSpeech-Bench, so the audio-understanding win is real but not a sweep.

Recognition and translation are the most lopsided results in the paper. On ASR, Plus reaches 6.55 WER on Fleurs across 60 languages against Gemini's 7.32, 1.11 on LibriSpeech clean against 3.36, and — most strikingly — 3.46 on KeSpeech against 23.67, a Chinese-dialect benchmark where Gemini essentially falls over. Speech-to-text translation across the top 59 Fleurs languages goes to Qwen in both directions. Forty million hours of ASR-labelled pre-training buys exactly what you would expect it to buy.

Audio-visual understanding is where the paper's own tables argue against its abstract. Plus takes DailyOmni (84.6 vs 82.7), the real-world interactive Qualcomm IVD (68.5 vs 66.2), and captioning on Omni-Cloze (64.8 vs 57.2). But Gemini-3.1 Pro leads WorldSense (65.5 vs 62.8), AV-SpeakerBench (75.1 vs 71.3), VideoMME with audio (89.0 vs 83.7), and OmniGAIA tool use by a wide margin (68.9 vs 57.2). "Reaches the level of Gemini-3.1 Pro" is a fair summary of the aggregate; benchmark by benchmark it trades wins and losses, and agentic tool use over audio-visual input is a clear gap.

The quieter achievement is the one that's easy to skip: Plus does not pay for omnimodality in text. Against the text-only Qwen3.5-Plus-Instruct it scores 85.9 vs 86.8 on MMLU-Pro, 94.2 vs 94.3 on MMLU-Redux, 83.9 vs 85.9 on GPQA, 81.0 vs 82.7 on TAU2Bench — small, consistent regressions — and it actually ties on IFEval at 89.7. Historically, bolting on audio has cost text models much more than a point.

On generation, Plus reaches 1.26 WER on SEED test-en, the best in the comparison, and takes the lowest WER in 22 of 29 languages on the multilingual TTS set while leading most languages on speaker similarity. First-packet latency lands at 435 ms for audio input and 651 ms for video on Plus, and 235 / 426 ms on Flash, with a generation real-time factor between 0.178 and 0.334 across concurrency levels — meaning speech is synthesized several times faster than it is played, which is the actual requirement for a conversation that doesn't stutter.

Why it matters

The through-line here isn't a single architectural idea, it's a set of alignment problems that only appear once you insist on real time. Text and speech tokenize at different rates, so ARIA constrains their ratio instead of forcing an alignment. Absolute-time position IDs go sparse over long video, so timestamps become text the model can read. Answers degrade when the question arrives as sound, so the text-conditioned answer becomes the distillation target for the audio-conditioned one. Each fix is small; together they're most of what separates a demo from something you can hold a conversation with. The emergent capability the report is most excited about — Audio-Visual Vibe Coding, writing executable code straight from a spoken-and-shown instruction — is the sort of thing that only becomes possible once perception, reasoning, and action share one context and one clock.

Links: arXiv:2604.15804 · Hugging Face demo · ModelScope demo