Xybrid

Streaming Inference

Real-time token streaming for LLM and audio transcription

Xybrid supports two types of streaming inference: LLM token streaming and real-time audio streaming (live transcription).

LLM Token Streaming

Stream LLM responses token-by-token as they are generated, enabling real-time chat UIs.

Flutter

final model = await Xybrid.model('gemma-3-1b').load();

final envelope = XybridEnvelope.text('Tell me a joke');

await for (final token in model.runStreaming(envelope)) {
  stdout.write(token.token); // Print each token as it arrives
}

With Conversation Context

final context = ConversationContext();
context.setSystem('You are a helpful assistant.');
context.pushText('Tell me a joke', MessageRole.user);

final envelope = XybridEnvelope.text('Tell me a joke');

await for (final token in model.runStreamingWithContext(envelope, context)) {
  stdout.write(token.token);
}

With Generation Config

final stream = model.runStreaming(
  XybridEnvelope.text('Write a poem'),
  config: GenerationConfig.creative(),
);

await for (final token in stream) {
  stdout.write(token.token);
}

Swift

let model = try await Xybrid.model("gemma-3-1b").load()

for try await token in model.streamTokens(envelope: .text("Tell me a joke")) {
    print(token.token, terminator: "") // Print each token as it arrives
}

streamTokens returns an AsyncSequence. Cancelling the task that consumes it, or breaking out of the loop, stops generation at the next token. Pass options: .make(generationConfig: config) for generation settings. With a conversation context, use the pull API: runStreamWithContext(envelope:context:options:) returns a stream id to drain with streamNext(streamId:) and close with streamClose(streamId:).

Unity (C#)

// Callback-based streaming
using (var envelope = Envelope.Text("Tell me a story."))
using (var result = model.RunStreaming(envelope, token =>
{
    Debug.Log(token.Token);
}))
{
    result.ThrowIfFailed();
}

// Convenience method
string response = model.RunStreamingText("Tell me a story.", token =>
{
    dialogueText.text += token.Token;
});

StreamToken Properties

PropertyTypeDescription
tokenStringGenerated token text
tokenIdinteger or nullRaw token ID, when the backend exposes one. Swift Int64?, Kotlin Long?, Unity long?. Dart has no such field
indexnon-negative integerZero-based token position in the sequence. Dart int, Swift UInt64, Kotlin ULong, Unity uint
cumulativeTextStringThe text emitted so far. Tool-call blocks are left out, so this can differ from the model's raw output (see rawText)
isFinalboolWhether this is the last token. Dart always ends the stream with one. Unity's IsFinal, and a non-null finishReason in Swift and Kotlin, mark the terminal token, but a run that generates no text ends without one: treat the end of the loop as the end of the stream
finishReasonString?Set on the terminal token: "stop" (end of sequence or a stop sequence), "length" (hit max_tokens) or "tool_calls" (the turn ended on a tool call; see toolCalls). Cloud-routed runs pass the gateway's reason through unchanged, so other values can appear. In Dart, a failed run reports "error: <message>" here; see Failures
toolCallslistTool calls the model asked for. Terminal token only: empty on every other token and on turns that call no tool. See Streaming Chat With Tools
rawTextString?The turn's raw output, tool-call block included. Pass it as priorAssistantText to continue the turn. Terminal token only, and only when toolCalls is not empty

Failures

No SDK reports a failure as a bare "error" finish reason. Each SDK reports it differently:

  • Flutter: the stream ends with a token whose finishReason starts with "error: " and carries the message. Check token.isError and read token.errorMessage. This token can follow a normal final token.
  • Swift and Kotlin: streamTokens throws XybridError, so the for try await loop (Swift) or the collect call (Kotlin) throws. In the pull API, streamNext throws.
  • Unity: RunStreaming returns an InferenceResult with Success set to false. Call result.ThrowIfFailed() to throw. RunStreamingText calls it for you.

SDK Support

FeatureFlutterUnityKotlinSwift
runStreaming()SupportedSupportedSupported (streamTokens; no preempt)Supported (streamTokens; no preempt)
runStreamingWithContext()SupportedSupportedPull API only (runStreamWithContext)Pull API only (runStreamWithContext)
Generation configSupportedSupportedSupportedSupported

Troubleshooting

Surfacing llama.cpp C++ logs

When debugging model load failures, unexpected streaming output, or other backend issues, set the XYBRID_LLAMACPP_VERBOSITY environment variable before your process starts. This raises the verbosity of llama.cpp's internal C++ logger from the default (suppressed at normal log levels).

ValueMeaning
0Silent (no llama.cpp logs)
1Errors only
2Warnings and errors
3Info, warnings, and errors
4Full debug output (all library logs)

Example invocation:

XYBRID_LLAMACPP_VERBOSITY=4 ./my-xybrid-app

Use 4 when investigating model load failures: it surfaces the underlying llama.cpp diagnostic that the Rust-side error wraps. The variable is read once when the llama.cpp backend is initialized, so set it before launching the process. Available since v0.1.0-beta5.

Two kinds of native call print nothing at any level, on the thread that makes them: tokenization, and the load and passes of a decision model. Their diagnostics can quote the prompt, so xybrid drops them. Logs from other threads, such as another model loading at the same time, are unaffected.


Real-Time Audio Streaming

Live transcription sessions take microphone audio as the app feeds it in and return partial transcripts as speech arrives, with optional voice-activity detection to cut chunks on speech boundaries. See Live Transcription for iOS, Android and Unity, and Speech-to-Text for Flutter. React Native opens a session with model.stream() (guide).

On this page