Streaming Inference
Real-time token streaming for LLM and audio transcription
Xybrid supports two types of streaming inference: LLM token streaming and real-time audio streaming (live transcription).
LLM Token Streaming
Stream LLM responses token-by-token as they are generated, enabling real-time chat UIs.
Flutter
final model = await Xybrid.model('gemma-3-1b').load();
final envelope = XybridEnvelope.text('Tell me a joke');
await for (final token in model.runStreaming(envelope)) {
stdout.write(token.token); // Print each token as it arrives
}With Conversation Context
final context = ConversationContext();
context.setSystem('You are a helpful assistant.');
context.pushText('Tell me a joke', MessageRole.user);
final envelope = XybridEnvelope.text('Tell me a joke');
await for (final token in model.runStreamingWithContext(envelope, context)) {
stdout.write(token.token);
}With Generation Config
final stream = model.runStreaming(
XybridEnvelope.text('Write a poem'),
config: GenerationConfig.creative(),
);
await for (final token in stream) {
stdout.write(token.token);
}Swift
let model = try await Xybrid.model("gemma-3-1b").load()
for try await token in model.streamTokens(envelope: .text("Tell me a joke")) {
print(token.token, terminator: "") // Print each token as it arrives
}streamTokens returns an AsyncSequence. Cancelling the task that consumes it,
or breaking out of the loop, stops generation at the next token. Pass
options: .make(generationConfig: config) for generation settings. With a
conversation context, use the pull API: runStreamWithContext(envelope:context:options:)
returns a stream id to drain with streamNext(streamId:) and close with
streamClose(streamId:).
Unity (C#)
// Callback-based streaming
using (var envelope = Envelope.Text("Tell me a story."))
using (var result = model.RunStreaming(envelope, token =>
{
Debug.Log(token.Token);
}))
{
result.ThrowIfFailed();
}
// Convenience method
string response = model.RunStreamingText("Tell me a story.", token =>
{
dialogueText.text += token.Token;
});StreamToken Properties
| Property | Type | Description |
|---|---|---|
token | String | Generated token text |
tokenId | integer or null | Raw token ID, when the backend exposes one. Swift Int64?, Kotlin Long?, Unity long?. Dart has no such field |
index | non-negative integer | Zero-based token position in the sequence. Dart int, Swift UInt64, Kotlin ULong, Unity uint |
cumulativeText | String | The text emitted so far. Tool-call blocks are left out, so this can differ from the model's raw output (see rawText) |
isFinal | bool | Whether this is the last token. Dart always ends the stream with one. Unity's IsFinal, and a non-null finishReason in Swift and Kotlin, mark the terminal token, but a run that generates no text ends without one: treat the end of the loop as the end of the stream |
finishReason | String? | Set on the terminal token: "stop" (end of sequence or a stop sequence), "length" (hit max_tokens) or "tool_calls" (the turn ended on a tool call; see toolCalls). Cloud-routed runs pass the gateway's reason through unchanged, so other values can appear. In Dart, a failed run reports "error: <message>" here; see Failures |
toolCalls | list | Tool calls the model asked for. Terminal token only: empty on every other token and on turns that call no tool. See Streaming Chat With Tools |
rawText | String? | The turn's raw output, tool-call block included. Pass it as priorAssistantText to continue the turn. Terminal token only, and only when toolCalls is not empty |
Failures
No SDK reports a failure as a bare "error" finish reason. Each SDK reports it differently:
- Flutter: the stream ends with a token whose
finishReasonstarts with"error: "and carries the message. Checktoken.isErrorand readtoken.errorMessage. This token can follow a normal final token. - Swift and Kotlin:
streamTokensthrowsXybridError, so thefor try awaitloop (Swift) or thecollectcall (Kotlin) throws. In the pull API,streamNextthrows. - Unity:
RunStreamingreturns anInferenceResultwithSuccessset to false. Callresult.ThrowIfFailed()to throw.RunStreamingTextcalls it for you.
SDK Support
| Feature | Flutter | Unity | Kotlin | Swift |
|---|---|---|---|---|
runStreaming() | Supported | Supported | Supported (streamTokens; no preempt) | Supported (streamTokens; no preempt) |
runStreamingWithContext() | Supported | Supported | Pull API only (runStreamWithContext) | Pull API only (runStreamWithContext) |
| Generation config | Supported | Supported | Supported | Supported |
Troubleshooting
Surfacing llama.cpp C++ logs
When debugging model load failures, unexpected streaming output, or other backend issues, set the XYBRID_LLAMACPP_VERBOSITY environment variable before your process starts. This raises the verbosity of llama.cpp's internal C++ logger from the default (suppressed at normal log levels).
| Value | Meaning |
|---|---|
0 | Silent (no llama.cpp logs) |
1 | Errors only |
2 | Warnings and errors |
3 | Info, warnings, and errors |
4 | Full debug output (all library logs) |
Example invocation:
XYBRID_LLAMACPP_VERBOSITY=4 ./my-xybrid-appUse 4 when investigating model load failures: it surfaces the underlying llama.cpp diagnostic that the Rust-side error wraps. The variable is read once when the llama.cpp backend is initialized, so set it before launching the process. Available since v0.1.0-beta5.
Two kinds of native call print nothing at any level, on the thread that makes them: tokenization, and the load and passes of a decision model. Their diagnostics can quote the prompt, so xybrid drops them. Logs from other threads, such as another model loading at the same time, are unaffected.
Real-Time Audio Streaming
Live transcription sessions take microphone audio as the app feeds it in and return partial transcripts as speech arrives, with optional voice-activity detection to cut chunks on speech boundaries. See Live Transcription for iOS, Android and Unity, and Speech-to-Text for Flutter. React Native opens a session with model.stream() (guide).