Tool Calling

Local models can call tools: you declare functions, the model decides when to call them, your code executes the call, and the result feeds the next turn. Everything runs on-device — the only network traffic is whatever your tools themselves do.

Xybrid's design rule is that capabilities are data, not entry points. Tools ride the existing types end to end:

1. Declare tools on GenerationConfig — plain data, any function you want.
2. run the request; parsed calls come back on the result.
3. Execute the calls yourself — your code, your sandboxing, your APIs.
4. Feed results back with Envelope::tool_results and run again.

There is no agent framework and no callback API: the loop is your code, which is why your own tooling is first-class — xybrid never needs to know what a tool does, only its declaration and its JSON result.

Supported Models

The tool-call syntax is auto-detected from the model's own chat template:

A bundle advertises support with an advisory tool_calling: true flag in model_metadata.json (ModelMetadata::supports_tool_calling() / XybridModel::supports_tool_calling() read it). The flag gates UI defaults — enforcement always happens at run time.

Bring Your Own Tools (Rust SDK)

Define a tool with Tool::function — a name, a description the model reads, and a JSON Schema for the arguments:

use xybrid_sdk::{GenerationConfig, ModelLoader, Tool};
use xybrid_sdk::ir::{Envelope, EnvelopeKind, ToolCallResult};

let tools = vec![Tool::function(
    "get_weather",
    "Current weather for a city.",
    serde_json::json!({
        "type": "object",
        "properties": { "city": { "type": "string" } },
        "required": ["city"]
    }),
)];

let config = GenerationConfig::default().with_tools(tools);

Run a turn and execute whatever the model asked for:

let model = ModelLoader::from_registry("lfm2.5-350m").load()?;
let question = "What's the weather in Paris?";

let result = model.run(
    &Envelope::new(EnvelopeKind::Text(question.to_string())),
    Some(&config),
)?;

let calls = result.tool_calls(); // Vec<ToolCall>, empty if none
let mut results = Vec::new();
for call in &calls {
    // call.function.name + call.function.arguments (JSON string).
    // Execute with YOUR code — hit an API, query a database, anything.
    results.push(ToolCallResult {
        call_id: call.id.clone(),
        name: call.function.name.clone(),
        content: serde_json::json!({ "temp_c": 21, "sky": "clear" }),
    });
}

Feed the results back as the next turn — same tools, same system prompt, one run per model turn:

let continuation = Envelope::tool_results(
    question,               // the user message being continued
    result.text().unwrap(), // the prior turn's raw output, tool block included
    &results,
);
let final_turn = model.run(&continuation, Some(&config))?;
println!("{}", final_turn.text().unwrap_or_default());

That's the whole surface. Multi-step loops (budgets, retries, parallel tools) are ordinary application code — see crates/xybrid-core/examples/lfm2_230m_tools.rs and gemma4_e2b_tools.rs for complete reference loops, including streaming. The runnable functiongemma_tools.rs example uses the pinned Q8_0 fixture through the standard GenerationConfig::with_tools and Envelope::tool_results flow:

./integration-tests/download.sh functiongemma-270m-it
cargo run -p xybrid-core --example functiongemma_tools --features llm-llamacpp

Rules the loop must follow (v1)

• One run is one model turn. The continuation replays only the immediately prior assistant turn — for multi-hop chains, fold earlier results into the user text you pass to tool_results (the examples show the pattern).
• Streaming halts at the call boundary. A tools-bearing streaming run suppresses tool-call blocks from the token stream, and the terminal token carries finish_reason: "tool_calls", the parsed calls (PartialToken::tool_calls), and the raw turn text to replay (PartialToken::raw_text) — all surfaced on every binding's stream token, so a streaming caller never parses raw text.
• Continuations stream, and they keep context. Feed the tool_results envelope back through the same call you started with, streaming or not, with or without a ConversationContext. Only image-bearing conversations are refused (see the callout above).
• Push to history after the loop, not between turns. A continuation envelope replays the user turn itself, so pushing it to the context first duplicates it in the prompt. Commit the user message and the final answer once the loop settles. Reference loop: crates/xybrid-core/examples/functiongemma_tools_streaming_context.rs.
• Budget your turns. Small models can search forever without settling; cap the loop and force a tools-off synthesis turn from your accumulated notes when the budget runs out.

Streaming Chat With Tools

A chat UI wants both: tokens as they arrive, and history across turns. The loop is the same three steps as above, run through runStreamingWithContext — the model streams its answer, halts at the call boundary, and the continuation streams too:

Stream<StreamToken> turn(XybridEnvelope input) =>
    model.runStreamingWithContext(input, context,
        config: GenerationConfig.greedy(tools: tools));

var input = XybridEnvelope.text(userText);
var prior = '';

for (var step = 0; step < maxSteps; step++) {
  List<ToolCall> calls = const [];

  await for (final token in turn(input)) {
    buffer.write(token.token);          // paint tokens as they arrive
    if (token.hasToolCalls) {
      calls = token.toolCalls;          // typed, already parsed
      prior = token.rawText!;           // the turn text to replay
    }
  }

  if (calls.isEmpty) break;             // the model answered; done

  input = XybridEnvelope.toolResults(
    userText: userText,
    priorAssistantText: prior,
    results: [
      for (final call in calls)
        ToolResult(
          callId: call.id,
          name: call.name,
          contentJson: await runMyTool(call.name, call.argumentsJson),
        ),
    ],
  );
}

Push to the context once the exchange settles, not between turns: the tool_results envelope replays userText itself, so a mid-loop push puts the question in the prompt twice.

Three things make this work without any parsing on your side:

• Tool-call blocks are suppressed from the token stream, so token.token is always display text. The calls arrive typed on the terminal token instead.
• rawText on that same token is the turn's raw output including the tool-call block — the streaming equivalent of XybridResult.text, and what priorAssistantText needs. Don't use cumulativeText: that is what your UI painted, with the blocks suppressed.
• The tool_results envelope goes back through the same call, so the continuation keeps the conversation context and keeps streaming.

Swift, Kotlin, Python, and Unity C# expose the same fields on their stream token — toolCalls, rawText, finishReason, plus a hasToolCalls convenience — and feed the continuation back through the same streaming call. Python spells it has_tool_calls; Unity C# spells it HasToolCalls.

Tool Calling in the CLI (xybrid repl)

The REPL ships the reference experience. For models whose metadata declares tool_calling: true, three built-ins are on by default:

xybrid repl --model lfm2.5-350m
❯ who won the 2022 fifa world cup final?
  ⚙ web_search("2022 FIFA World Cup final winner") · 614ms
    Search "2022 FIFA World Cup final winner": The 2022 FIFA World Cup final…
  The 2022 FIFA World Cup final was won by Argentina.

• --no-tools disables tools for the session; /tools lists them and /tools on|off toggles at run time.
• Search providers: set XYBRID_SEARCH_PROVIDER=tavily (+TAVILY_API_KEY) or brave (+BRAVE_API_KEY) for open-web coverage; unset defaults to Wikipedia with no key.
• With a reasoning model (e.g. lfm2.5-1.2b-thinking), --show-reasoning prints each turn's chain-of-thought as a dimmed block directly above what the turn produced — its ⚙ tool calls or its answer — the clearest way to watch a model decide whether and which tool to use.

Your own tools in the REPL (--tools-file)

Declare tools in a JSON (or YAML) file; each maps a function to a command you author:

[
  {
    "name": "get_weather",
    "description": "Get the current weather for a city.",
    "parameters": {
      "type": "object",
      "properties": { "city": { "type": "string" } },
      "required": ["city"]
    },
    "command": ["./scripts/weather.sh"],
    "timeout_secs": 30
  }
]

xybrid repl --model-file ./LFM2.5-350M-Q4_K_M.gguf --tools-file my-tools.json
❯ what's the weather in paris?
  ⚙ get_weather({"city":"Paris"}) · 55ms
    get_weather: {"city":"Paris","sky":"clear","temp_c":21}
  The current weather in Paris is clear with a temperature of 21°C.

Contract:

• The model's arguments arrive as a JSON object on stdin; the command's stdout is the tool result (parsed as JSON when possible, otherwise wrapped as {"output": "..."}).
• The executable and its fixed argv come from your file only — model output is never interpolated into the command line and never passes through a shell. If you want shell behavior, write a script.
• Commands are killed after timeout_secs (default 30); output is capped.
• Passing --tools-file is an explicit opt-in: it also enables tools for models whose metadata doesn't declare tool_calling (an unsupported template still fails loudly at run time). A bundle that declares tool_calling: false stays off.

From the Mobile and Desktop SDKs

Tool calling crosses the FFI boundary as of 0.6.0, so Swift, Kotlin, Python, Unity C# and Dart drive the same loop. It stays turn-based: one run is one model turn, your code executes the tools, and a continuation envelope feeds the outcomes back. There is no cross-boundary callback.

Three pieces make up the loop:

1. Put your tool schemas on GenerationConfig.tools.
2. Read XybridResult.toolCalls — each carries id, name and argumentsJson.
3. Build the continuation with XybridEnvelope.toolResults(...) and run it with the same tools as the original turn, so the executor rebuilds an identical chat prefix.

let tools = [XybridToolDefinition(
    name: "get_weather",
    description: "Current weather for a city",
    parametersJson: #"{"type":"object","properties":{"city":{"type":"string"}}}"#
)]
let config = XybridGenerationConfig.greedy(tools: tools)

let options = XybridRunOptions(
    generationConfig: config,
    abortOn: [],
    fallbackToCloud: false,
    maxGraceTokens: 0,
    correlationId: nil
)
let first = try model.run(envelope: .text("Weather in Paris?"), options: options)

if !first.toolCalls.isEmpty {
    let results = first.toolCalls.map { call in
        XybridToolResult(callId: call.id, name: call.name, contentJson: execute(call))
    }
    let continuation = try XybridEnvelope.toolResults(
        "Weather in Paris?",
        priorAssistantText: first.text ?? "",  // raw output, tool-call block included
        results: results
    )
    // same options — the continuation must carry the same tools
    let answer = try model.run(envelope: continuation, options: options)
}

Kotlin mirrors it with XybridEnvelope.toolResults(userText, priorAssistantText, results). Python uses tool_results_envelope(...) and snake_case fields. Dart uses the named-argument factory XybridEnvelope.toolResults(userText:, priorAssistantText:, results:) with GenerationConfig.greedy(tools: ...) and result.hasToolCalls.

priorAssistantText must be the turn's raw text verbatim — tool-call block included. Stripping it breaks the prefix the model continues from.

Gate your tool UI on the bundle's advisory flag rather than assuming:

if model.supportsToolCalling() == true { showToolsToggle() }

nil means the bundle says nothing and never implies support. Enforcement still happens at run time: tool calling is llama.cpp-only, and unsupported paths — no embedded chat template, the mistralrs backend, the cloud fallback leg — reject tool-bearing requests rather than quietly generating without them.

Declaring Support in Your Own Bundles

Add the flag to model_metadata.json for models whose chat template renders tools (the /xybrid-init skill can generate metadata):

{
  "metadata": {
    "tool_calling": true
  }
}

Apps read it before downloading weights via ModelMetadata::supports_tool_calling() — None means the bundle says nothing, and never implies support.

Small-Model Expectations

Sub-1B models emit tool-call syntax reliably but reason loosely about when to call: LFM2.5-230M over-triggers (searches on greetings) while 350M tends to answer from memory without searching. The REPL's default system prompt is a compromise — override it with --system, and prefer larger tool-tuned models when tool discipline matters.

Tiny models also pay a "tool tax": with tool declarations in the prompt, some refuse ordinary requests they handle fine otherwise ("I can't write a poem"). The REPL compensates — a zero-tool-call capability refusal is retried once with the declarations withheld, cued as answered without tools. A model's own refusals earlier in the conversation can also bias later turns; clear resets the history.