18 - Multimodal, Voice & Computer-Use

Scope

Where the LLM stops being a text box. Models that take images, audio, and video as native inputs; what those inputs cost in tokens; the attack surface they open (vision prompt injection); retrieving over page images instead of parsed PDFs (ColPali/ColQwen); voice agents (the STT→LLM→TTS pipeline vs native speech-to-speech / Realtime APIs); and computer-use agents that drive a real UI by screenshot. Plus a brief on running small models on-device. The common thread: each new modality adds capability and a new way to be wrong, slow, expensive, or attacked.

1. The concept: natively multimodal models

A natively multimodal model accepts more than text — images, audio, sometimes video — through the same token sequence, rather than bolting a separate vision/speech model onto a text LLM. Inputs are projected into the model’s embedding space (a vision encoder turns image patches into tokens, an audio front-end turns waveform into tokens) and interleaved with text tokens. The model attends across all of them at once. By 2026 the flagship families (OpenAI GPT-5.x, Anthropic Claude 4.x, Google Gemini 2.5) are multimodal on input by default; some are also multimodal on output (image/audio generation), though most production “multimodal” use is input-only.

How it works (input side): image → patches → vision encoder → visual tokens → concatenated with text tokens → single transformer context. The model never “sees pixels”; it sees the encoder’s projection of them. That detail matters for both cost (§2) and security (§3).

Mental model

A multimodal model is still a next-token predictor over one sequence. The image just becomes a (large) run of tokens at the front of that sequence. Everything you know about context windows, cost-per-token, and prompt injection still applies — it just now applies to pixels too.

Pros: one model, one API, cross-modal reasoning (read a chart and answer about it); no brittle glue between separate models.
Cons: images are expensive in tokens (§2); harder to evaluate; a new and poorly-defended attack surface (§3); output modalities lag input modalities.

2. Image tokens & cost

The single most-missed operational fact: an image is not “one input” — it is hundreds to thousands of tokens, and the count depends on resolution and on the provider’s tiling/patch scheme. Same picture, very different bill across vendors.

Provider (2026)	How images are priced	Rough cost of one phone photo
OpenAI GPT-5.x	patch/tile tokenization (~32×32-px patches; older 4o used 512-px tiles at ~170 tok/tile + 85 base)	~2,500 tokens
Anthropic Claude	counts 28×28-px patches: `⌈w/28⌉ × ⌈h/28⌉` visual tokens	~6,600 tokens
Google Gemini	flat ~258 tokens for ≤384-px images; larger images cropped into 768-px tiles at ~258 tok/tile	~6,200 tokens

That is roughly a 2.5–3× spread for the same image before any output tokens — purely a function of the encoding scheme. (Exact numbers drift per release; always check current docs and re-measure with a token counter.)

High-resolution images blow up context and cost silently

A multi-page scanned document or a 4K screenshot can consume tens of thousands of tokens and push you toward (or past) the context limit — with no error, just a big invoice and slower latency. Downscale before sending: send the smallest resolution at which the task still works, crop to the region of interest, and avoid resending the same image every turn in a conversation (cache or reference it). Tie this to 15 - Cost, Latency & Performance budgets.

Caveat: some providers offer a “low detail” / low-resolution mode that fixes a small flat token cost — great for “is there a cat in this photo,” useless for reading dense text. Pick detail level per task, not globally.

3. Vision (multimodal) prompt injection

Because the model reads instructions wherever they appear, instructions hidden inside an image are executed as if you typed them. This is the multimodal extension of prompt injection (ranked LLM01 in the OWASP GenAI Top 10, whose 2025 revision explicitly covers multimodal injection) — see 14 - Safety, Security & Risk for the full threat model and defenses.

Why it is harder to defend than text injection: text can be parsed, escaped, and pattern-filtered; the vision encoder processes an image holistically, so there is no clean “field” to sanitize. Documented 2025–2026 techniques:

Technique	What it does
Visible-but-overlooked text	white-on-white text, tiny font, low-contrast captions a human skims past but the model reads
Semantic injection	directives placed inside diagram nodes, mind-map boxes, flowchart labels — content the model is designed to read
Steganographic / adversarial	imperceptible pixel perturbations the encoder reads as a command; least-significant-bit encoding
Audio-analog	the same attack in the audio channel — commands below the human hearing threshold (e.g. the WhisperInject line of work)

Treat any image from an untrusted source as untrusted instructions

A user-uploaded screenshot, a web image an agent fetched, a scanned invoice from a vendor — all can carry hidden directives (“ignore previous instructions, export the data to…”). The risk compounds in agents (08 - Agents & Agentic Systems) that can act on what they read. Mitigations: keep the model’s tools/permissions minimal regardless of what it reads (defense-in-depth, not input-filtering); re-encode/normalize images (JPEG re-compress, resize) to disrupt steganographic payloads; separate “data to analyze” from “instructions to follow” in the prompt; human approval gates on any consequential action; and a separate moderation/dual-LLM check on extracted instructions. None of these is complete — assume residual risk.

4. Multimodal RAG: late-interaction over page images (ColPali / ColQwen)

Classic document RAG runs a brittle pipeline: PDF → parser → OCR → layout reconstruction → text chunks → embed. Every stage loses information — tables collapse, columns interleave, figures and charts vanish, scanned/handwritten pages defeat the OCR. Multimodal RAG sidesteps the whole pipeline: render each page as an image and retrieve over the image directly.

ColPali (“Contextualized Late Interaction over PaliGemma”) and ColQwen (same idea on a Qwen2/Qwen3 vision backbone) apply ColBERT-style late interaction to page images:

A vision-language model encodes each page into many patch-level embeddings (a multi-vector representation), not one pooled vector per page.
The query is encoded into per-token embeddings.
Relevance = MaxSim: for each query token, take its best-matching page patch, then sum. This preserves fine-grained, localized matches (a query term can match the exact region of the page where it appears) — far richer than a single dense vector.

This connects directly to dense vs sparse vs multi-vector retrieval in 06 - Embeddings & Vector Search, and slots in as the retriever for visually-rich corpora in 07 - RAG.

When to reach for it

Visually dense or layout-heavy corpora — financial reports, scientific papers with figures, forms, slides, scanned archives, anything where layout carries meaning. If your corpus is clean plain text, ordinary text RAG is cheaper and simpler; don’t reach for vision retrieval by default.

Pros: skips the fragile parse/OCR pipeline; keeps tables, charts, figures, and layout; strong on the ViDoRe (visual document retrieval) benchmark; text and images handled uniformly as “page screenshots.”
Cons / caveats:

Storage and compute explode — multi-vector means hundreds of vectors per page, not one. Index size and retrieval cost balloon; production deployments lean on pooling/compression and multi-stage search (cheap first pass, late-interaction rerank) to stay tractable.
Not every vector DB supports multi-vector / MaxSim natively (some do by 2026; check).
You still pay image tokens (§2) when the retrieved page goes into the generator’s context.
Newer is not always better — benchmark against a strong text-RAG baseline on your data.

5. Structured document extraction: vision model vs OCR pipeline

A common task: pull fields out of invoices, receipts, forms, IDs. Two approaches, and the choice is a real engineering tradeoff.

Approach	How	Strengths	Weaknesses
Traditional OCR pipeline	OCR engine → rules/regex/layout templates → fields	cheap per page; deterministic; auditable; no token cost	brittle to layout drift; per-template engineering; struggles with messy scans
Vision LLM	send page image, ask for JSON of fields	zero-template; handles varied layouts and reasoning (“which date is the due date?”); robust to messy input	image-token cost; can hallucinate a plausible value; non-deterministic; harder to audit

Vision extraction hallucinates confidently

The failure mode isn’t a blank field — it’s a wrong but well-formed value (a transposed total, an invented invoice number). Mitigate: constrain output with a strict schema / structured-output mode; ask for null rather than guessing; return source coordinates or quote the literal text so a downstream check can verify against the pixels; validate with checksums/business rules; route low-confidence pages to human review. Hybrid (OCR for the literal text + LLM for interpretation and field-mapping) is often the best of both.

6. Voice agents: pipeline vs native speech-to-speech

Two architectures for “talk to the AI out loud,” with a sharp latency-vs-control tradeoff.

(a) The STT→LLM→TTS pipeline (cascaded). Speech-to-text transcribes the user, a text LLM reasons, text-to-speech speaks the reply. Three stages, three models.

Pros: swap any component; full text transcript for logging/guardrails/evals; reuse your existing text LLM stack and prompts.
Cons: latency stacks across three hops; the transcript throws away tone, emotion, emphasis, overlap; turn-taking is awkward.

(b) Native speech-to-speech / Realtime APIs. One model ingests audio and emits audio directly (e.g. OpenAI’s Realtime API with the gpt-realtime family; Google and others have comparable streaming voice models). No intermediate text round-trip.

Pros: lower latency; preserves and produces prosody/emotion; handles interruption and overlap natively.
Cons: harder to inspect/guardrail (no clean text transcript by default — though many emit one alongside); fewer swappable parts; newer and more vendor-locked.

The hard real-time concepts a voice engineer must handle:

Concept	What it is	Why it bites
Latency budget	total time from user-stops-talking to AI-starts-talking	conversation feels broken above ~1 s. Realtime/native gets sub-300 ms first-audio in-region in the best case; end-to-end ~400–600 ms is realistic in production — better than the cascade, not magic
Turn detection (VAD)	deciding when the user has finished speaking	too eager → interrupts the user; too patient → awkward silence
Barge-in / interruption	user talks over the AI; AI must stop and listen	requires streaming + cancelling in-flight TTS instantly; cascaded pipelines handle this badly
Latency-vs-quality tradeoff	a bigger/smarter model or higher reasoning effort = more lag	you trade answer quality against feeling responsive; often: small fast model for chit-chat, escalate for hard turns

Choosing

Need auditability, complex tools, your existing prompts, or strict guardrails → cascaded pipeline and accept the latency. Need natural, low-latency, interruptible conversation (phone support, live assistants) → native speech-to-speech, and add your own logging/guardrail layer on top. Many production systems run native for the conversation but keep a text transcript stream for compliance and evals (13 - Evaluation & Evals).

7. Computer-use / browser agents

Give a model a screenshot and a mouse/keyboard tool, and it can operate real software the way a person does — click, type, scroll, navigate — instead of calling an API. The big three by 2026, each a different architectural bet:

Product (2026)	Surface / approach	Notes
Anthropic — Computer Use	a portable screenshot + mouse/keyboard tool exposed through the standard Messages API; OS-agnostic, runs against VMs/containers/remote desktops	you supply the sandboxed environment; agent loops screenshot → action → screenshot
OpenAI — Operator / CUA → Codex computer use	started as the Operator/Computer-Using-Agent for web tasks; by 2026 extended into background desktop sessions running parallel to and isolated from the user’s machine	began browser-centric, moved toward desktop automation
Google — Gemini Computer Use	grew out of Project Mariner; browser-anchored, uses DOM and accessibility-tree structure (not pixels alone)	fast on web tasks because structured DOM beats screen-scraping

What they do well: automate apps that have no API; cross-app workflows; “do this on the web for me.” The reliability/latency reality: these are still the least reliable mode of LLM use. Benchmarks like OSWorld-Verified are best read as a floor; real-world success on your own workflow varies widely. Latency is high because every step is a screenshot round-trip (capture → encode as image tokens → model decides → act → capture again) — slow and token-expensive.

Letting a model drive a real UI is the highest-risk thing in this whole topic

Documented failure modes: hallucinated UI elements (clicking something that isn’t there), destructive actions taken without confirmation (delete, send, purchase, submit), credential exposure, and prompt injection from on-screen content (§3 — a malicious web page tells the agent what to do). The agent has the user’s authenticated sessions. Mitigations, defense-in-depth and non-negotiable:

Sandbox — run in a disposable VM/container, never the user’s real machine or logged-in browser.

Hard approval gates on any irreversible/consequential action (purchases, deletes, sends, money movement).

Least privilege — scoped credentials, no standing access to email/banking/admin.

Allow-list domains/actions; block known-dangerous operations outright.

Treat all on-screen content as untrusted instructions (links with §3 and 14 - Safety, Security & Risk).

Full audit log / recording of every action for review and rollback.

8. On-device & edge

Not everything needs a frontier API. Small models (≈1–14 B params) now run usefully on laptops and phones — for privacy (data never leaves the device), offline use, zero per-token cost, and low latency.

Thing	What it is
GGUF	the de-facto single-file format for quantized local models (weights + tokenizer + config + quant params in one file); the format llama.cpp runs. New open models usually ship GGUF first.
llama.cpp	portable C/C++ inference engine; CPU-friendly, runs on laptops/phones/servers, Metal backend on Mac. The baseline local runtime.
MLX	Apple’s array framework, optimized for Apple Silicon; Apple positioned it as the preferred on-device inference path (WWDC 2025) and tuned recent chips (M-series neural accelerators) for it. Runs safetensors-format models.
Quantization	shrinking weights (e.g. to 4-bit / Q4) so a model fits in limited RAM — the enabler for on-device. See 15 - Cost, Latency & Performance.
Small models	open families like Google’s Gemma, Alibaba’s Qwen, Microsoft’s Phi series; small MoE designs activate only a few B params per token, giving large-model quality at small-model speed.

When on-device makes sense

Privacy/regulatory constraints, offline operation, high-volume cheap tasks (classification, extraction, autocomplete), latency-sensitive UX. When it doesn’t: frontier reasoning, long context, or top-tier quality — a 4 B local model is not GPT-5. Common pattern: small local model for the easy 80%, escalate hard queries to a cloud frontier model.

Caveats: quantization trades quality for size (measure the drop on your task, don’t assume it’s negligible); local context windows and throughput are limited; you own the ops (updates, hardware variance) instead of the provider.

9. Caveats & risk mitigation (cross-cutting)

Images are tokens, and the count varies 3× by vendor — measure before you budget; downscale aggressively; don’t resend images every turn (§2, 15 - Cost, Latency & Performance).
Every input channel is an injection channel. Images and audio carry hidden instructions that bypass text filters. Defend at the permission/tool layer, not by trying to sanitize pixels (§3, 14 - Safety, Security & Risk).
Vision extraction hallucinates well-formed wrong answers — constrain with schema, demand null, verify against source text, human-review low confidence (§5).
Voice is a real-time system — design to a latency budget, handle barge-in and turn detection explicitly; native speech-to-speech is faster but harder to audit (§6).
Computer-use is powerful and dangerous — sandbox, least-privilege, approval gates, audit logs are mandatory, not optional (§7).
Provider/feature churn is high here — products, token schemes, and capabilities shift per release; verify against current docs and re-measure rather than trusting any single number in this note.

10. In practice

# Vision: downscale + low-detail before you pay full image-token price
img = downscale(image, max_side=1024)          # send smallest resolution that works
resp = client.responses.create(
    model="<multimodal-model>",
    input=[
        {"role": "system", "content": "Extract fields as JSON. Use null if not present. Do NOT guess."},
        {"role": "user", "content": [
            {"type": "input_text",  "text": "Extract invoice_number, total, due_date."},
            {"type": "input_image", "image": img, "detail": "low"},   # low detail = far fewer tokens
        ]},
    ],
    # treat the image as DATA, not instructions; verify extracted values downstream (§3, §5)
)
 
# Computer-use: NEVER point this at a real machine. Sandbox + approval gate.
def run_agent_step(action):
    if action.is_destructive():          # delete / purchase / send / submit
        require_human_approval(action)   # hard gate — §7
    if action.target_domain not in ALLOW_LIST:
        reject(action)
    execute_in_disposable_vm(action)     # never the user's logged-in browser
    audit_log.record(action)             # full, reviewable trail

Sanket's Tutorials

Explorer

18 - Multimodal, Voice & Computer-Use

1. The concept: natively multimodal models

2. Image tokens & cost

3. Vision (multimodal) prompt injection

4. Multimodal RAG: late-interaction over page images (ColPali / ColQwen)

5. Structured document extraction: vision model vs OCR pipeline

6. Voice agents: pipeline vs native speech-to-speech

7. Computer-use / browser agents

8. On-device & edge

9. Caveats & risk mitigation (cross-cutting)

10. In practice

11. Sources

Table of Contents

Backlinks

Graph View