MiniCPM-o 4.5: A 9B Model That Can See, Hear, and Speak at the Same Time

Most multimodal AI models today work in turns. You feed them a video, wait for them to process it, and then get a response. While this works well for traditional chatbots, it does not reflect how humans naturally interact with the world. We listen while we speak. We react to things as they are still happening.

MiniCPM-o 4.5 is a new open-source model designed to close this gap. With just 9 billion parameters, it can continuously watch a live video, listen to incoming audio, and speak a response – all at the same time. In this post, we’ll explore how the model works, examine its architecture, discuss our experiments, and highlight where it performs well, as well as its current limitations.

Table of Contents

The Problem with Turn-Based AI

Imagine watching a cricket match with a friend. They do not wait for the over to finish before reacting. They shout in real time, while still watching the next ball.

Traditional multimodal models cannot do this because of two hard limitations:

Temporal staleness: The model cannot update its ongoing response if something new happens on screen while it is still speaking.

No proactivity: The model only responds when explicitly asked. It will never speak up on its own, even if something important happens on screen.

Moving past these limitations requires a fundamentally different interaction design.

Architecture: Four Blocks, One Connected Pipeline

Instead of running separate services for vision, audio, and text, MiniCPM-o 4.5 connects all components directly at the token level in a single end-to-end trainable system.

Figure 1: The four blocks of MiniCPM-o 4.5 connected through shared token-level hidden states. (Source: Figure 4, MiniCPM-o 4.5 Technical Report, arXiv:2604.27393).

For explanation, we divide the architecture into four functional blocks. The paper groups them into three categories: encoders, the LLM, and speech decoders:

1. Vision Encoder (SigLIP2-based ViT, 0.4B): Each incoming video frame is split into slices. Every slice is encoded into 1024 tokens and then compressed down to just 64 tokens by a resampler module. This is a 16× token compression ratio, far higher than the roughly 4× most models use. It significantly reduces the computational cost of real-time streaming.
2. Audio Encoder (Whisper Medium, 0.3B): Incoming audio is processed in small streaming chunks, producing 50 feature tokens per second. These are further compressed 5× by an MLP projector, resulting in 10 audio tokens per second being passed to the language model.
3. LLM Backbone (Qwen3-8B): This is the central reasoning component of the architecture. It receives the combined visual and audio tokens, performs multimodal reasoning, and generates text output alongside hidden states that carry prosody and context information for speech.
4. Speech Decoders: A lightweight Llama-based speech token decoder (~0.3B) takes the LLM’s hidden states and converts them into discrete speech tokens. A streaming flow-matching decoder then turns those tokens into actual audio waveforms, with voice style controlled by a reference audio clip in the system prompt, enabling voice cloning out of the box.

The encoders, Qwen3-8B backbone, and speech token decoder are connected through shared token-level hidden states and are trained jointly. A separate streaming flow-matching decoder then converts speech tokens into audio waveforms.

Omni-Flow: Making Full-Duplex Work

The key framework behind MiniCPM-o 4.5 is called Omni-Flow. The core idea is simple: instead of alternating between listening and speaking phases, align everything (video, audio, and output) on one shared timeline.

Figure 2: Existing models alternate between perceiving and speaking. MiniCPM-o 4.5 does both simultaneously. (Source: Figure 3, MiniCPM-o 4.5 Technical Report, arXiv:2604.27393).

Omni-Flow splits the timeline into fixed 1-second chunks. In every chunk, the model:

• Receives the new video frames and audio that just arrived
• Decides whether to stay silent or speak
• Generates its output, conditioned on what it just saw and heard

The decision of whether to speak is kept separate from deciding what to say. The model first predicts a “listen” or “speak” control token and then generates the corresponding content. The paper shows this two-step design is more stable than collapsing both decisions into one.

The chunk size of 1 second was found to be the sweet spot. Shorter chunks (0.1s, 0.2s) degraded output quality noticeably because the model did not have enough context per window to make stable decisions.

TAIL: Keeping Speech in Sync

There is a subtler problem that Omni-Flow alone does not solve. Text generation is very fast, but speaking that text out loud takes real time. If the model generates too much text too quickly, the spoken voice starts to lag behind what is actually happening on screen.

Figure 3: Non-interleaved and fixed-ratio methods let speech drift behind the live scene. TAIL adjusts chunk by chunk to stay in sync. (Source: Figure 5, MiniCPM-o 4.5 Technical Report, arXiv:2604.27393).

To fix this, the paper proposes Time-Aligned Interleaving (TAIL). Rather than using a fixed ratio of text tokens to speech tokens, TAIL tracks the total accumulated playback lag across the whole conversation. If the voice is falling behind the real-time boundary, TAIL forces the model to generate fewer text tokens in the current chunk, letting the audio catch up.

It also uses a small look-ahead window: speech tokens for the last few words in a chunk are deferred to the next chunk. This gives the speech decoder just enough future context to handle pronunciation correctly (for example, “the” in “the apple” sounds different from “the car”), without letting the text stream run far ahead of playback. This sync comes at a cost: dynamic TAIL raises English WER from 2.38 to 3.93 (paper Table 10), trading some accuracy for timely playback.

Getting Started

Setup is straightforward – install the dependencies, load the model, and call model.as_duplex() to switch it into full-duplex streaming mode. From there, you load a video and a reference voice clip, set a system prompt describing how the model should behave, and feed it audio/video one chunk at a time. The model decides each second whether to stay silent or speak.

The full installation steps, model initialization code, and streaming loop are available on the official Hugging Face model card.

Training: A Staged Build-Up

Training uses four stages to stop modalities from interfering with each other:

1. Speech Pretraining: The vision and language backbone stay completely frozen. Only the new audio modules are trained, teaching them to map incoming speech into the LLM’s token space without disturbing anything already learned.
2. Joint Pretraining: Everything is unfrozen. The model is now trained on a balanced mix of vision, speech, and text data simultaneously. Keeping the data ratio fixed across hardware ranks stops any one modality from dominating the others.
3. Supervised Fine-Tuning: Large-scale instruction-following data sharpens real-world behavior. Training is done at randomized resolutions and frame rates, which lets users trade visual quality against compute at inference.
4. Reinforcement Learning (GRPO): Reasoning and accuracy are refined using GRPO. A smooth length reward trims unnecessarily long responses without hurting correctness. Interestingly, the authors report that hallucination reduction learned from static images via RLAIF-V transfers into the streaming video setting as well.

Benchmark Results

All benchmark figures are author-reported and not independently reproduced by us.

A compact 9B model with competitive benchmark performance.

Vision-Language Performance (Instruct Mode)

The model runs in instruct or thinking mode; the table below is instruct mode, which favours direct answers.

Full table with all 20 benchmarks in the technical report.

Performance is task-dependent: strong on document parsing, MathVista, and multi-image tasks, but behind on MMMU and MMVet.

Vision-Only Full-Duplex Benchmark Results

LiveSports-3K-CC is an audio-free benchmark; 54.4 is an author-reported win rate vs GPT-4o, not an accuracy percentage. It validates continuous visual streaming only; it does not test the simultaneous audio-in/speech-out pipeline this article is about.

Local Inference Efficiency on a Single NVIDIA RTX 4090

These figures are reported in the MiniCPM-o 4.5 technical report (Table 11) and have not been independently reproduced by us. Note: Throughput and memory figures are measured on text-only input; first-token latency is measured using 64-frame video inputs.

The practical takeaway: on this text-only benchmark, MiniCPM-o 4.5 fits under 12 GB even quantized, while Qwen3-Omni-30B-A3B still needs 20 GB at INT4 and cannot run in BF16 at all on a single RTX 4090. (For full-duplex audio+video streaming, VRAM needs are higher: see the FAQ below.)

Loading MiniCPM-o 4.5

We load the model once in bfloat16 on the GPU with vision, audio, and TTS initialized. The same instance is reused across all experiments below.

LOCAL_PATH = "/path/to/your/MiniCPM-o-4_5/snapshot"  # your local model snapshot folder
SAMPLE_RATE = 16000
REF_AUDIO_PATH = "assets/audio1.wav"

model = AutoModel.from_pretrained(
    LOCAL_PATH,
    trust_remote_code=True,
    local_files_only=True,
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16,
    init_vision=True,
    init_audio=True,
    init_tts=True,
)
model.eval().cuda()
model.init_tts()
ref_audio, _ = librosa.load(REF_AUDIO_PATH, sr=SAMPLE_RATE, mono=True)

Putting It to the Test: Our Own Experiments

We evaluated the model across multiple scenarios to assess its performance beyond the results reported in the paper.

Before running any experiment, strip the source video’s audio once so the final output carries only the model’s own voice:

def strip_video_audio(input_path, output_path):
    """Remove original audio from video so output contains only AI voice."""
    cmd = ["ffmpeg", "-y", "-i", input_path, "-an", "-c:v", "copy", output_path]
    subprocess.run(cmd, check=True)  # raises CalledProcessError if ffmpeg fails

Each test below strips the source video’s audio first, so the output only carries the model’s generated speech.

Helper: Inject questions into the audio stream

These helpers build audio chunks manually so we can control exactly what the model hears each second.

def _inject_question(chunks, q_path, q_sec, total):
    """Overlay a single question's audio onto the chunk list starting at q_sec,
    overwriting whatever real/background audio was there."""

    qaudio, _ = librosa.load(q_path, sr=SAMPLE_RATE, mono=True)
    q_chunk_count = int(np.ceil(len(qaudio) / SAMPLE_RATE))

    for i in range(q_chunk_count):
        target_chunk = q_sec + i
        if target_chunk >= total:
            break
        offset = i * SAMPLE_RATE
        slice_ = qaudio[offset: offset + SAMPLE_RATE]
        if len(slice_) < SAMPLE_RATE:
            slice_ = np.pad(slice_, (0, SAMPLE_RATE - len(slice_)))
        chunks[target_chunk] = slice_.astype(np.float32)


def build_audio_chunks_qa(total, q1_path, q1_sec, q2_path=None, q2_sec=None, base_audio_segments=None):
    # fall back to narrator audio between questions, else silence
    if base_audio_segments is not None and len(base_audio_segments) >= total:
        chunks = [
            (base_audio_segments[i].copy() if base_audio_segments[i] is not None
             else np.zeros(SAMPLE_RATE, dtype=np.float32))
            for i in range(total)
        ]
    else:
        chunks = [np.zeros(SAMPLE_RATE, dtype=np.float32) for _ in range(total)]

    _inject_question(chunks, q1_path, q1_sec, total)

    if q2_path is not None and q2_sec is not None:
        _inject_question(chunks, q2_path, q2_sec, total)

    return chunks

Helper: Shared Streaming Loop

All experiments reuse this same core loop – only the prompt and audio chunks change per test.

def run_mode(model, ref_audio, video_frames, stacked_frames, audio_segments,
             mode_name, video_for_output, output_path, system_prompt,
             max_new_speak_tokens_per_chunk=20):
    """Streams video+audio chunk-by-chunk into the model and mux's its speech onto video_for_output."""
    print(f"Running: {mode_name}")

    model_duplex = model.as_duplex()
    model_duplex.prepare(prefix_system_prompt=system_prompt, ref_audio=ref_audio, prompt_wav_path=REF_AUDIO_PATH)

    results_log, timed_output_audio = [], []

    for chunk_idx in range(len(audio_segments) + 5):  # +5 tail chunks to let model finish speaking
        audio_chunk = audio_segments[chunk_idx] if chunk_idx < len(audio_segments) else None
        frame = video_frames[chunk_idx] if chunk_idx < len(video_frames) else None

        frame_list = [frame] if frame is not None else []
        if frame is not None and stacked_frames and chunk_idx < len(stacked_frames) and stacked_frames[chunk_idx] is not None:
            frame_list.append(stacked_frames[chunk_idx])

        model_duplex.streaming_prefill(audio_waveform=audio_chunk, frame_list=frame_list,
                                        max_slice_nums=1, batch_vision_feed=False)

        result = model_duplex.streaming_generate(
            prompt_wav_path=REF_AUDIO_PATH,
            max_new_speak_tokens_per_chunk=max_new_speak_tokens_per_chunk,
            decode_mode="sampling",
        )

        if result["audio_waveform"] is not None:
            timed_output_audio.append((chunk_idx, result["audio_waveform"]))
        results_log.append(result)

        print(f"[{chunk_idx:03d}s] listening..." if result["is_listen"] else f"[{chunk_idx:03d}s] MODEL: {result['text']}")

    # mux model-generated speech onto the silent video
    generate_duplex_video(video_path=video_for_output, output_video_path=output_path,
                           results_log=results_log, timed_output_audio=timed_output_audio,
                           output_sample_rate=24000)
    print(f"Saved -> {output_path}")

We tested the live commentary feature on a football penalty clip, to see if the model could act as a commentator on its own, narrating continuously and reacting to the action in real time.

VIDEO_PATH = "assets/football.mp4"
silent_video_path = "input_silent.mp4"
strip_video_audio(VIDEO_PATH, silent_video_path)

video_frames, audio_segments, stacked_frames = get_video_frame_audio_segments(
    VIDEO_PATH, stack_frames=1, use_ffmpeg=True, adjust_audio_length=True
)
total_chunks = len(video_frames)

if audio_segments is None:
	audio_segments = [np.zeros(SAMPLE_RATE, dtype=np.float32) for _ in range(total_chunks)]

run_mode(
    model, ref_audio, video_frames, stacked_frames, audio_segments,
    mode_name="Live Sports Commentary",
    video_for_output=silent_video_path,
    output_path="football_commentary.mp4",
    system_prompt="""You are a sports commentator watching a live video feed.
Speak when something actually happens that is worth commentating on -
not every second. When you do speak, keep it to one short sentence.
After commentating, go quiet and keep watching until the next notable moment.
If nothing new is happening, stay silent.""",
    max_new_speak_tokens_per_chunk=30,
)

Output:

The model stayed quiet through routine play and spoke up only at notable moments, matching the prompt’s instruction to comment only when something worth calling out actually happens.

2. Behavioral Narration in a Retail Surveillance Clip

Next, we evaluated the model on a retail surveillance clip showing a phone display counter to assess whether it could go beyond simple narration and identify suspicious behavioural patterns from visual cues in real time.

VIDEO_PATH = "assets/theft.mp4"
# ...same setup as Test 1 (strip_video_audio → get_video_frame_audio_segments)

run_mode(
    model, ref_audio, video_frames, stacked_frames, audio_segments,
    mode_name="Theft Detection via Behavioral Narration",
    video_for_output=silent_video_path,
    output_path="theft_detection.mp4",
    system_prompt="""You are watching a live surveillance feed of a phone display counter. The items on display are iPhones. Describe only what you clearly see happening right now, in one short sentence. Pay close attention to how items are handled - note if someone grabs items quickly or forcefully, handles multiple items in a hurry, or conceals an item in a pocket, bag, or clothing without paying. Use natural, direct language to describe exactly what the person is doing with their hands and where the item ends up. If a person repeatedly takes items and conceals them without paying, clearly state that the person appears to be stealing. If unsure of the exact action, describe it generally instead of guessing. Keep speaking continuously - don't wait for major changes.""",
    max_new_speak_tokens_per_chunk=30,
)

Output:

The model narrated the sequence of actions in detail – describing the person approaching the display, rapidly picking up multiple phones, and concealing them in their clothing – and correctly escalated to explicitly stating that the person appeared to be stealing once the take-and-conceal pattern repeated across multiple counters.

3. Cooking Narration + Real-Time Q&A

This experiment evaluates whether the model can follow a cooking video in the same way a person would, watching the preparation while listening to the narration, and then handle interruptions without losing context.

Test 3A: Narration Only

This run feeds the model the video’s own narrator audio, to see if it can follow along in sync.

VIDEO_PATH, REF_AUDIO_PATH = "assets/cooking.mp4", "assets/audio1.wav"
# ...same setup as Test 1

run_mode(
    model, ref_audio, video_frames, stacked_frames, audio_segments,
    mode_name="Cooking Narration Only",
    video_for_output=silent_video_path,
    output_path="cooking_narration.mp4",
    system_prompt="""You are a proactive, real-time commentator narrating a close-up food-preparation video. Describe exactly what is visible on screen at every moment -- the ingredients shown, their colors and textures, and any hand or utensil movement -- cutting, scooping, spreading, drizzling, plating -- as it happens. This video is mostly extreme close-up, slow, deliberate shots -- do NOT wait for big scene changes or obvious cooking techniques like chopping or stirring before speaking. A hand entering the frame, a spoon or knife touching the food,sauce being poured, or the camera holding on a plated dish is enough reason to narrate immediately. If nothing is moving for a moment, describe the appearance of the food in detail (color, texture, garnish, plating) instead of staying silent. Never wait for anything to be said first, and never leave long silent gaps -- always find something visible to comment on. Speak naturally, like a food stylist walking someone through a close-up recipe shot.""",
    max_new_speak_tokens_per_chunk=30,
)

Output:

The model was able to narrate what it saw and heard in sync, correctly following the basic flow of the video and the narrator’s instructions.

Test 3B: Narration + Injected Question

This run injects one extra question mid-narration, to see if the model can handle an interruption.

Note: Questions are pre-recorded and injected at particular timestamps.

# Same video_frames/audio_segments as 3A - just swap the audio chunks + prompt
qa_chunks = build_audio_chunks_qa(
    total_chunks, "assets/q1.m4a", 2,
    q2_path="assets/q2.m4a", q2_sec=11,
    base_audio_segments=audio_segments,
)

run_mode(
    model, ref_audio, video_frames, stacked_frames, qa_chunks,
    mode_name="Cooking Narration + Injected Question",video_for_output=silent_video_path,
    output_path="cooking_interruption_qa.mp4",
    system_prompt="""You are a precise video question-answering assistant. Watch the video and answer the user's question directly based on the visual evidence.CRITICAL INSTRUCTIONS: 1. Provide ONLY the direct, factual answer to the question asked. 2. Use complete, full words (no abbreviations or short-hand). 3. Do NOT include any introductory phrases, explanations, conversational filler, or commentary about other steps in the video. 4. Output the answer and nothing else. 5. Answer after question completion.
""",
    max_new_speak_tokens_per_chunk=30,
)

Question asked:

Output:

The model correctly identified the injected question and answered it using the ongoing visual and audio context.

4. Voice Cloning

For this test we used a simple prompt, and swapped in different reference voice clips to see how well the model matched each one during live speech generation.

VIDEO_PATH = "assets/vid.mp4"
REF_AUDIO_PATH = "assets/audio1.wav"

video_frames, audio_segments, stacked_frames = get_video_frame_audio_segments(
    VIDEO_PATH, stack_frames=1, use_ffmpeg=False, adjust_audio_length=True
)

# set use_ffmpeg=False as this video has no audio rest same setup as Test 1

ref_audio, _ = librosa.load(REF_AUDIO_PATH, sr=SAMPLE_RATE, mono=True)
run_mode(
    model, ref_audio, video_frames, stacked_frames, audio_segments,
    mode_name="Voice Cloning - Reference Voice 1",
    video_for_output=silent_video_path,
    output_path="audio1.mp4",
    system_prompt="You are a proactive, real-time commentator. Continuously describe the scene as it happens.",
    max_new_speak_tokens_per_chunk=20,
)
# repeat with REF_AUDIO_PATH="assets/audio2.wav", output_path="audio2_test.mp4"

Reference voices used:

Output videos:

In informal listening, the cloned voices stayed recognizably similar to each reference across short samples, with no drift back to a default voice mid-stream; we did not run a formal similarity metric.

Note: subtitles in the clips above were re-synced, nudged slightly ahead, and lightly styled for readability, a presentation tweak only, not a change to the model’s actual output.

Our Findings

Proactive vs. reactive modes both work as designed. In commentary mode (sports, theft detection), the model spoke continuously without prompting. In Q&A mode, it stayed largely silent and spoke only when addressed or when a notable event occurred.

Behavioral narration can escalate to a judgment call when the pattern is clear. In the theft detection test, the model didn’t just describe isolated actions; it tracked the pattern across multiple counters (rapid grabbing, repeated concealment without paying) and correctly escalated from neutral description to explicitly stating the person appeared to be stealing.

Interruptions are handled correctly. When a question was injected mid-narration during the cooking test, the model paused, answered directly, then resumed tracking the video normally.

Voice cloning stayed stable. The cloned voice matched each reference clip’s tone throughout, with no drift back to a default voice.

Narration is solid but still surface-level. In the synced cooking narration test, the model reliably followed both the visual steps and the narrator’s audio in sync, but didn’t proactively surface skipped details like ingredient quantities or timing cues, consistent with the paper’s own note that proactive behavior is still limited in this first version.

Wrapping Up

MiniCPM-o 4.5 is a meaningful step toward AI that feels less like a chatbot and more like a real conversation partner. The Omni-Flow idea (aligning vision, audio, and text on one shared timeline) is simple in concept but effective in practice. The authors note that speech can occasionally mispronounce words or mix languages during long sessions, and proactive behavior is still fairly basic, which is expected for a first version of full-duplex interaction at this scale.

But as a small, open, and efficient model that can see, hear, and talk simultaneously on consumer hardware, it is a genuinely useful step forward for real-time computer vision and multimodal applications.

The model weights, code, and llama.cpp-omni deployment framework are fully open-source and ready to try today.

Frequently Asked Questions

Does MiniCPM-o 4.5 support full-duplex streaming in languages other than English and Chinese?
Real-time speech interaction is officially supported only in English and Chinese. Visual and text understanding cover 30+ languages, but we did not test real-time speech in other languages.

What’s the minimum GPU/VRAM needed to run it?
Requirements depend on the framework: the PyTorch demo needs ~28 GB, while llama.cpp-omni’s full-duplex mode runs on ≥12 GB NVIDIA GPUs or an M4 Max Mac with 24 GB unified memory. Quantization is what brings memory down, not something you get “even” alongside it.

Can you fine-tune MiniCPM-o 4.5?
LLaMA-Factory and SWIFT support supervised and LoRA fine-tuning for multimodal domain adaptation, but public recipes do not reproduce the full time-aligned Omni-Flow pipeline for proactive full-duplex behaviour. That proactive behaviour was learned during the original time-aligned training, and there’s no public tooling to replicate it.

Can I change or clone the model’s voice for real-time speech output?
Yes, via native zero-shot voice cloning. By passing a reference audio clip directly into the system prompt at inference time, the speech decoder automatically matches its real-time synthesis style and voice characteristics to mimic that target speaker.

How does the model decide exactly when to speak or listen?
It uses a Listen-Speak (LS) control mechanism. At the absolute beginning of each 1-second time block, the model generates a single binary control token. This acts as an architectural toggle switch: it tells the system to either remain silent and buffer environmental audio or begin streaming voice tokens.

References

Post Views: 5