Flutter
On-device ML inference for Flutter applications
The Flutter SDK (xybrid_flutter) provides Dart bindings to the Xybrid Rust core via FFI, enabling on-device ML inference in mobile and desktop applications.
Requirements
- iOS 16.0+
- macOS 14.0+ on Apple Silicon
- Android 9 / API 28+
The precompiled native libraries are built for these versions. For releases after 0.16.0 the plugin also declares them, so an app that targets an older version fails at build time. Raise your app's minimums:
- Android: set
minSdk = 28or higher indefaultConfigofandroid/app/build.gradle.kts. Flutter's default,flutter.minSdkVersion, is lower. - iOS: set
platform :ios, '16.0'inios/Podfileand the Runner target's iOS Deployment Target to 16.0 in Xcode. - macOS: set
platform :osx, '14.0'inmacos/Podfileand the Runner target's macOS Deployment Target to 14.0 in Xcode.
Installation
Add to your pubspec.yaml:
dependencies:
xybrid_flutter: ^0.20.0Initialization
Initialize the SDK before using any Xybrid APIs:
import 'package:xybrid_flutter/xybrid_flutter.dart';
void main() async {
WidgetsFlutterBinding.ensureInitialized();
await Xybrid.init();
runApp(MyApp());
}Local inference is available without authentication. An apiKey configures
cloud access. To send execution traces to the Xybrid console,
also configure observability explicitly:
await Xybrid.init(
apiKey: const String.fromEnvironment('XYBRID_API_KEY'),
observability: ObservabilityConfig.xybrid(
apiKey: const String.fromEnvironment('XYBRID_API_KEY'),
metadataOnly: true,
),
);An API key alone does not enable export. The example selects metadata-only
capture; otherwise enabled observability captures governed text by default.
Use ObservabilityConfig.otlp(endpoint: 'http://collector:4318') for a Collector
without Xybrid authentication. CPU/memory sampling is separately configured
with resourceTelemetry. ingestUrl remains a legacy explicit opt-in to OTLP
trace export; new integrations should use observability.
Key prefixes and scopes, what gets exported, and provisional workspaces: see API keys.
Xybrid.init() also subscribes to OS battery state on iOS and Android via battery_plus and forwards each reading to the routing engine.
Core Pattern: Load → Run
All inference follows the same pattern:
- Create a loader — specify the model by registry ID or local bundle
- Load — download (if needed) and initialize the model
- Run — execute inference with an input envelope
final model = await Xybrid.model('kokoro-82m').load();
final result = await model.run(XybridEnvelope.text('Hello world'));
print(result.text);Text-to-Speech
final model = await Xybrid.model('kokoro-82m').load();
final envelope = XybridEnvelope.text('Hello, how are you?', voiceId: 'af_heart');
final result = await model.run(envelope);
// Get WAV-wrapped audio bytes for playback
final wavBytes = result.audioAsWav(sampleRate: 24000);Speech-to-Text
final model = await Xybrid.model('whisper-tiny-ggml').load();
final envelope = XybridEnvelope.audio(bytes: audioBytes, sampleRate: 16000);
final result = await model.run(envelope);
print(result.text); // "Hello, how are you?"For live transcription, open a session and feed PCM float32 mono 16 kHz as it
arrives. flush() ends the session and returns an FfiTranscriptionResult:
the transcript in text, plus durationMs, chunksProcessed and the
session's observation reference.
final session = await XybridStreamSession.fromModel(model);
session.partials.listen((partial) => caption = partial.text); // listen first
session.feed(samples);
final transcript = (await session.flush()).text;Earlier releases returned the transcript from flush() as a String. Read
.text from the result instead.
LLM Chat with Streaming
final model = await Xybrid.model('gemma-3-1b').load();
final context = ConversationContext();
context.setSystem('You are a helpful assistant.');
context.pushText('Tell me a joke', MessageRole.user);
final envelope = XybridEnvelope.text('Tell me a joke');
await for (final token in model.runStreamingWithContext(envelope, context)) {
stdout.write(token.token); // Print each token as it arrives
}Loading with Progress
final loader = XybridModelLoader.fromRegistry('kokoro-82m');
await for (final event in loader.loadWithProgress()) {
switch (event) {
case LoadProgress(:final percentage, :final downloadedBytes, :final totalBytes):
print('Loading: $percentage% ($downloadedBytes / ${totalBytes ?? '?'} bytes)');
case LoadComplete():
print('Model ready!');
case LoadError(:final message):
print('Error: $message');
}
}progress spans every file the model needs — weights plus companions such as a
vision projector — and never moves backwards, so the bar neither restarts per
file nor rewinds when a download retries. totalBytes is null when the source
publishes no size (a Hugging Face repo); downloadedBytes is exact either way,
which is what you need for megabytes, speed, and time remaining.
Loading a Local GGUF File
import 'package:path_provider/path_provider.dart';
// One directory per GGUF: a file the user picked, or an asset copied out of
// the app bundle.
final support = await getApplicationSupportDirectory();
final path = '${support.path}/models/my-model/my-model.gguf';
final model = await XybridModelLoader.fromModelFile(path).load();A raw llama.cpp GGUF needs no model_metadata.json. Creating the loader touches
no disk; load() generates the metadata from the GGUF header and writes it next
to the file, with the context capped at 4,096 tokens (raise its context_length
for longer chats). Put each GGUF in its own writable directory, such as one under
getApplicationSupportDirectory(): assets declared in pubspec.yaml are not
files on disk and the iOS app bundle is read-only, so model_metadata.json
cannot be written next to either. A model_metadata.json already in the
directory is used as is: a second GGUF in the same directory would load the
first one's model. The generated metadata has no vision projector, so a vision
GGUF with an mmproj still needs fromDirectory. A missing file makes load()
throw an XybridException whose message contains GGUF file not found: <path>.
With nothing to download, loadWithProgress() emits only LoadComplete or
LoadError.
API Reference
Xybrid
Main entry point for the SDK.
class Xybrid {
static Future<void> init({
String? apiKey, // cloud authentication; does not enable export
String? gatewayUrl, // override the LLM gateway URL
String? ingestUrl, // override the telemetry ingest URL (self-hosting)
String? resourceTelemetry, // enable CPU/memory sampling
ObservabilityConfig? observability, // explicit OTLP export configuration
});
static bool get isInitialized;
static void setApiKey(String apiKey);
static XybridModelLoader model(String modelId);
static XybridPipeline pipeline({String? yaml, String? filePath});
}XybridModelLoader
Creates and loads models from different sources.
class XybridModelLoader {
factory XybridModelLoader.fromRegistry(String modelId);
factory XybridModelLoader.fromRegistrySpeculative(String modelId);
factory XybridModelLoader.fromBundle(String path);
factory XybridModelLoader.fromDirectory(String path);
factory XybridModelLoader.fromHuggingFace(String repo);
factory XybridModelLoader.fromModelFile(String path);
Future<XybridModel> load();
Stream<LoadEvent> loadWithProgress();
}Load events:
| Event | Properties | Description |
|---|---|---|
LoadProgress | progress (0.0–1.0), percentage (0–100), downloadedBytes, totalBytes (nullable) | Download/load progress |
LoadComplete | — | Model is ready |
LoadError | message | Loading failed |
XybridModel
A loaded model ready for inference.
class XybridModel {
Future<XybridResult> run(XybridEnvelope envelope, {GenerationConfig? config});
Future<XybridResult> runWithContext(XybridEnvelope envelope, ConversationContext context, {GenerationConfig? config});
Stream<StreamToken> runStreaming(XybridEnvelope envelope, {GenerationConfig? config});
Stream<StreamToken> runStreamingWithContext(XybridEnvelope envelope, ConversationContext context, {GenerationConfig? config});
}XybridEnvelope
Input container for model inference.
class XybridEnvelope {
factory XybridEnvelope.audio({
required List<int> bytes,
required int sampleRate,
int channels = 1,
});
factory XybridEnvelope.text(String text, {String? voiceId, double? speed});
factory XybridEnvelope.embedding(List<double> data);
factory XybridEnvelope.textWithRole(String text, MessageRole role);
XybridEnvelope withRole(MessageRole role);
}XybridResult
Output from model inference.
class XybridResult {
bool get success;
String? get text; // ASR / LLM output
Uint8List? get audioBytes; // TTS output (raw PCM)
List<double>? get embedding; // Embedding output
int get latencyMs;
Uint8List? audioAsWav({int sampleRate = 24000, int channels = 1});
}StreamToken
Token from streaming LLM inference.
class StreamToken {
final String token; // Generated token text
final int index; // Zero-based token position
final String cumulativeText; // Text emitted so far, tool-call blocks left out
final bool isFinal; // Last token in sequence
final String? finishReason; // "stop", "length", "tool_calls", or "error: <message>"
final XybridInferenceMetrics? metrics; // Final token only
final ObservationRef? observation; // Trace reference, final token only
final List<ToolCall> toolCalls; // Final token only; empty otherwise
final String? rawText; // Raw turn text with the tool-call block; null without tool calls
bool get hasToolCalls; // toolCalls is not empty
bool get isError; // finishReason starts with "error:"
String? get errorMessage; // The message of an error token
}ConversationContext
Multi-turn conversation management for LLM models.
class ConversationContext {
ConversationContext();
ConversationContext.withId(String id);
String get id;
int get historyLength;
bool get hasSystem;
void setSystem(String text);
void setMaxHistoryLength(int length);
void push(XybridEnvelope envelope);
void pushText(String text, MessageRole role);
void clear(); // Clears history, keeps system prompt
}enum MessageRole { system, user, assistant }XybridPipeline
Multi-stage pipeline for chaining models.
class XybridPipeline {
factory XybridPipeline.fromYaml(String yaml);
factory XybridPipeline.fromFile(String path);
String? get name;
int get stageCount;
List<String> get stageNames;
Future<XybridResult> run(XybridEnvelope envelope, {ObservationOptions? observation});
}Pipeline Example
final yaml = '''
name: voice-assistant
stages:
- whisper-tiny
- gemma-3-1b
- kokoro-82m
''';
final pipeline = XybridPipeline.fromYaml(yaml);
final result = await pipeline.run(
XybridEnvelope.audio(bytes: audioBytes, sampleRate: 16000),
);
// Result is TTS audio from the final stage
final wav = result.audioAsWav();observation: takes the same ObservationOptions as RunOptions.observation.
result.observation is the pipeline's workflow reference, and
result.metrics.stageLatenciesMs[i].observation is each stage's inference reference.
XybridException
class XybridException implements Exception {
final String message;
}All SDK errors throw XybridException, except in the token-streaming methods (runStreaming, runStreamingWithContext, runStreamingWithFallback): those end the stream with an error token instead (StreamToken.isError, StreamToken.errorMessage). Loading errors are surfaced via LoadError events when using loadWithProgress().
Platform Support
Precompiled libraries ship for iOS (device and Simulator, arm64), macOS on Apple Silicon, Android (arm64-v8a, x86_64), Linux (x86_64) and Windows (x86_64). Where a model runs depends on its format:
| Model format | iOS device | iOS Simulator | macOS | Android, Linux, Windows |
|---|---|---|---|---|
| GGUF (llama.cpp: LLMs, vision-language models) | Metal GPU | CPU | Metal GPU | CPU |
| Whisper (whisper.cpp) | CPU; optional Core ML encoder | CPU; optional Core ML encoder | Metal GPU | CPU |
| ONNX (ONNX Runtime) | CPU | CPU | CPU | CPU |
On iOS, whisper.cpp runs the encoder on Core ML when a
<model>-encoder.mlmodelc (e.g. ggml-base.en-encoder.mlmodelc) sits beside
the model file. ONNX Runtime's Core ML execution provider is linked on iOS and
macOS, but the SDK does not select it yet, so ONNX models run on the CPU. No
path uses NNAPI or DirectML.