Streaming Inference

Xybrid supports two types of streaming inference: LLM token streaming (available now) and real-time audio streaming (planned).

LLM Token Streaming

Stream LLM responses token-by-token as they are generated, enabling real-time chat UIs.

Flutter

final model = await Xybrid.model('gemma-3-1b').load();

final envelope = XybridEnvelope.text('Tell me a joke');

await for (final token in model.runStreaming(envelope)) {
  stdout.write(token.token); // Print each token as it arrives
}

With Conversation Context

final context = ConversationContext();
context.setSystem('You are a helpful assistant.');
context.pushText('Tell me a joke', MessageRole.user);

final envelope = XybridEnvelope.text('Tell me a joke');

await for (final token in model.runStreamingWithContext(envelope, context)) {
  stdout.write(token.token);
}

With Generation Config

final stream = model.runStreaming(
  XybridEnvelope.text('Write a poem'),
  config: GenerationConfig.creative(),
);

await for (final token in stream) {
  stdout.write(token.token);
}

Unity (C#)

// Callback-based streaming
using (var envelope = Envelope.Text("Tell me a story."))
using (var result = model.RunStreaming(envelope, token =>
{
    Debug.Log(token.Token);
}))
{
    result.ThrowIfFailed();
}

// Convenience method
string response = model.RunStreamingText("Tell me a story.", token =>
{
    dialogueText.text += token.Token;
});

StreamToken Properties

Property	Type	Description
`token`	`String`	Generated token text
`index`	`int`	Token position in sequence
`cumulativeText`	`String`	All text generated so far
`isFinal`	`bool`	Whether this is the last token
`finishReason`	`String?`	`"stop"`, `"length"`, or `"error"`

SDK Support

Feature	Flutter	Unity	Kotlin	Swift
`runStreaming()`	Supported	Supported	Planned	Planned
`runStreamingWithContext()`	Supported	Supported	Planned	Planned
Generation config	Supported	Supported	Planned	Planned

When debugging model load failures, unexpected streaming output, or other backend issues, set the XYBRID_LLAMACPP_VERBOSITY environment variable before your process starts. This raises the verbosity of llama.cpp's internal C++ logger from the default (suppressed at normal log levels).

Value	Meaning
`0`	Silent (no llama.cpp logs)
`1`	Errors only
`2`	Warnings and errors
`3`	Info, warnings, and errors
`4`	Full debug output (all library logs)

Example invocation:

XYBRID_LLAMACPP_VERBOSITY=4 ./my-xybrid-app

Use 4 when investigating model load failures: it surfaces the underlying llama.cpp diagnostic that the Rust-side error wraps. The variable is read once when the llama.cpp backend is initialized, so set it before launching the process. Available since v0.1.0-beta5.

Real-Time Audio Streaming (Planned)

Real-time audio transcription with partial results (chunked Whisper inference) is planned for a future release. This will enable:

Continuous microphone input with chunked processing
Partial transcription results as audio is received
Voice Activity Detection (VAD) for automatic segmentation
Configurable chunk sizes and overlap

This feature will be available across all SDKs when implemented.