Turn a vision-language model into a speech-centric omni model without
changing its architecture, updating its parameters, or running multimodal re-alignment.
Moving from a strong VLM to a capable Omni model is not a free upgrade. It
introduces recurring alignment costs,
difficult temporal signals, and a risk of losing capabilities the backbone already learned.
Advances in stronger VLM backbones do not automatically transfer to existing Omni models, requiring the audio pathway to be repeatedly adapted and realigned for each new backbone.
Reliable audio-visual integration remains difficult because audio is temporally dense, often noisy, and must be precisely aligned with speech and visual events.
Jointly modifying the backbone can weaken mature abilities in image and video understanding, visual grounding, coding, mathematics, and domain knowledge.
Plug-and-play by design Speech is transformed into reliable language context outside the VLM. Everything mature inside the backbone stays frozen.
Accept speech or audio alongside images, video frames, and the user query.
Extract transcript segments, language, confidence, and start/end timestamps.
Fuse transcript, timestamps, visual input, and query through the standard prompt interface.
Weights frozenGenerate text directly, with optional text-to-speech for a spoken response.
Matched native-omni and frozen-backbone comparisons separate the capability gained from the capability retained.
Can audio routing recover omni understanding without native training?
TFO is competitive on speech-centric audio-visual tasks, raises the audio-only average in every matched model setting, and improves multilingual CoVoST2 performance across all five comparisons.
Does freezing the VLM preserve capabilities that native omni training can weaken?
TFO retains stronger image averages in all five comparisons, stronger video averages in four of five, higher medical QA averages in all five, and higher coding-and-math averages in three of four reported settings.
Where do richer acoustic representations remain necessary?
Music, environmental sounds, vocal tone, emotion, and fine-grained audio-visual matching still benefit from native acoustic representations. TFO also shifts part of the cost from training to sequential ASR inference.
Each pair compares a native omni model with TFO applied to its corresponding frozen VLM backbone.