Vision Models

Vision-language models accept a user message that contains text plus one or more images, then stream a text answer from the local runtime. Xybrid models this as a normal chat turn: build image envelopes, combine them with prompt text in a user-message envelope, and run the selected VLM through the same model loading and streaming APIs used for text-only LLMs.

Supported Models

Each model must include its language model and its vision projector (mmproj) or equivalent vision encoder. If the sibling artifact is missing, model load fails before generation with a MissingArtifact error naming the referenced path.

Build With Vision Enabled

The default platform presets remain text-only. Add llm-llamacpp-vision to the build that will run VLMs:

cargo build -p xybrid --features platform-macos,llm-llamacpp-vision

For apps using the Rust crates directly, compose the target platform preset with llm-llamacpp-vision. For Flutter, Kotlin, Swift, and Unity packages, use the release artifact or local native build that was produced with the same feature.

CLI

Use --input-image with --input-text to send a single multimodal user turn:

xybrid run \
  --model gemma-4-e2b \
  --input-text "Describe this image." \
  --input-image ./cat.png

For local fixture validation before the registry entry is published:

./integration-tests/download.sh gemma-4-e2b
cargo test -p xybrid-cli --features llm-llamacpp-vision \
  run_input_image_captions_gemma_4_e2b_fixture

The REPL also supports staging images for the next turn:

/image ./cat.png
what is in this picture?

The staged image list is cleared after the prompt is sent.

Flutter

import 'dart:io';
import 'package:xybrid_flutter/xybrid_flutter.dart';

await Xybrid.init();

final model = await Xybrid.model('gemma-4-e2b').load();
final image = XybridEnvelope.image(
  bytes: await File('cat.jpg').readAsBytes(),
  format: 'jpeg',
);
final prompt = XybridEnvelope.userMessage(
  text: 'Describe this image.',
  images: [image],
);

await for (final token in model.runStreaming(prompt)) {
  stdout.write(token.token);
}

Studio uses the same envelope shape. Attach controls should be enabled only when the selected model supports image input, the native build includes a vision backend, the mmproj artifact is present, and the device memory floor is satisfied.

Kotlin

import ai.xybrid.Envelope
import java.io.File

val model = XybridModelLoader.fromRegistry("gemma-4-e2b").load()
val image = Envelope.image(File("cat.jpg").readBytes(), format = "jpeg")
val prompt = Envelope.userMessage(
    text = "Describe this image.",
    images = listOf(image),
)

val result = model.run(prompt)
println(result.text)

Swift

import Foundation
import Xybrid

let model = try ModelLoader.fromRegistry(modelId: "gemma-4-e2b").load()
let imageData = try Data(contentsOf: URL(fileURLWithPath: "cat.jpg"))
let image = try XybridEnvelope.image(imageData, format: "jpeg")
let prompt = try XybridEnvelope.userMessage(
    "Describe this image.",
    images: [image]
)

let result = try model.run(envelope: prompt)
print(result.text ?? "")

For the Q8 Gemma 4 E2B path, validate on iPhone 15 Pro or newer before claiming mobile support. Lower-memory devices should be blocked by capability checks before loading the bundle.

Unity

using System.IO;
using Xybrid;

using var model = XybridClient.LoadModel("gemma-4-e2b");
byte[] imageBytes = File.ReadAllBytes("cat.jpg");
using var image = Envelope.Image(imageBytes, "jpeg");
using var prompt = Envelope.UserMessage("Describe this image.", new[] { image });

using var result = model.Run(prompt);
result.ThrowIfFailed();

Debug.Log(result.Text);

Multi-Turn Chats

Conversation history can include prior text and image-bearing user messages. When replaying history, preserve the ordered text/image parts instead of flattening image turns into text. Backends are allowed to re-prefill image tokens on later turns; v0.2.0 does not introduce a cross-backend image embedding cache.

Real-Time / Live Camera Vision

A live camera feed is not a video model. On-device VLMs cannot caption every frame, so the design point is understanding events, not frames per second. Xybrid models a live camera view as a cascade: a cheap always-on gate decides when the scene has meaningfully changed, and only then wakes the expensive VLM for a single frame. The viewfinder keeps running regardless; understanding lags the camera by design.

The cascade

camera frames ──► cheap luma change-gate ──► [fires only on change]
                                               │
                                  single-flight, latest-frame-wins
                                               │
                            one frame ──► VLM (mtmd) ──► streamed answer
                                               │
                                  replace-in-place answer + history log

The gate computes a format-aware luma signature over a downsampled grid and compares successive frames. A static scene produces near-zero VLM fires (idle ≈ no inference); the gate is what prevents duplicate-answer spam and keeps token-per-frame cost — the dominant cost driver — bounded. The VLM only ever sees gated, single frames.

Expectations: best-effort, never block

There is no smooth-video promise and no latency SLA. A realistic on-device rate is roughly 1–5 understanding events per second, and that drops further under thermal load or on lower-end devices. The contract is:

• The viewfinder never freezes while the VLM is busy.
• Answers refresh when the scene changes and the model finishes a turn.
• Under heat or memory pressure, fires slow down (graceful throttle) rather than freezing the UI or crashing.

This is a best-effort understanding loop layered on top of the existing stateless streaming VLM path, not a real-time perception engine.

Latest-frame-wins (drop-if-busy → cancel-and-replace)

The loop is single-flight — only one VLM generation runs at a time. Two strategies bound how a new gated frame interacts with an in-flight one:

• Drop-if-busy (app-only baseline): a new frame is ignored while a generation is in flight. Simple, but the answer can lag the scene because the loop waits for the current turn to finish.
• Cancel-and-replace (with the runtime primitives): a newer gated frame preempts the in-flight generation. This is built on reachable streaming cancellation — cancelling the Dart stream now drives a real runtime abort (the generation halts at the next token and releases the model lock) rather than just unsubscribing in Dart — plus a preemptive cancel-current-run slot on the model handle. The net effect is latest-frame-wins: the freshest scene takes priority and stale frames don't head-of-line-block.

Capability and device floors

Live vision entry is gated the same way single-image vision is — the selected model must declare image input, the native build must include a vision backend, the mmproj artifact must be present, and the device must meet the memory floor. A device or model below those floors is blocked with a typed capability error before the camera loop starts, not mid-session.

Raw-frame path (imageRaw)

The baseline live loop JPEG-encodes each fired frame before handing it to the VLM. The runtime additionally exposes a raw-frame path that skips per-frame JPEG encoding: camera pixel buffers (packed RGB, with conversion from common camera formats) flow through mtmd directly via the imageRaw envelope binding instead of image. The encoded path is unchanged and remains the fallback.

In Studio the raw path is flag-gated and off by default (--dart-define=XYBRID_STUDIO_RAW_FRAMES=true). It stays off until raw-vs-encoded caption parity is confirmed on-device; with the flag unset, Studio uses the encoded path. See the validation runbook for the parity check.

// Raw-frame envelope (skips per-frame JPEG encode).
final image = XybridEnvelope.imageRaw(
  bytes: rgbBytes,      // packed RGB pixel buffer
  width: width,
  height: height,
);
final prompt = XybridEnvelope.userMessage(
  text: 'What does this label say?',
  images: [image],
);

On-device only

Image-bearing turns run on-device or fail — there is no cloud fallback for image turns, in live mode just as in single-image vision. The thermal / memory-pressure abort machinery is shared with the routing layer, but in live mode it throttles or stops the local loop rather than spilling to cloud.

Live-mode telemetry

A naive live loop would emit one telemetry row per fired frame and flood the traces backend. Instead, live-mode inferences are tagged (a live_mode flag plus a frame_session_id that groups one camera session) and rate-limited by a per-session sampler (roughly one row per second per session, TTL-bounded), so a live session produces a sampled rollup rather than per-frame spam.

Troubleshooting

MissingArtifact

The model metadata references an mmproj or vision encoder file that is not present next to the language model. Re-fetch the model bundle or verify the registry entry includes every sibling artifact.

UnsupportedModelCapability

The selected model is text-only, but the input envelope contains an image or a multi-part user message. Pick a VLM such as lfm2-vl-450m or gemma-4-e2b.

UnsupportedBackendCapability

The model declares vision input, but the loaded backend/build cannot satisfy it. Rebuild with llm-llamacpp-vision or switch to a release artifact that includes the vision backend.

Device Below Floor

The selected VLM needs more RAM than the current device reports. For Gemma 4 E2B Q8, use an 8 GB class device or choose a smaller VLM. Apps should surface this as a capability gate before generation starts.

Invalid Image Format

Use encoded PNG, JPEG, or WebP bytes. HEIC and camera-native raw pixel buffers are separate ingestion paths and may require platform-specific conversion before calling Envelope.image.