Semantic Router for Local LLMs: Fast Edge Tool Routing

Autonomous AI agents that execute local tool calling and multi-step reasoning represent the forefront of on-device machine learning. Yet running interactive agents on consumer Apple Silicon faces an immediate computational wall: injecting multiple JSON tool schemas into an autoregressive model bloats the context window, multiplies Time-to-First-Token latency, and triggers frequent false-positive tool calls on simple conversational prompts. Decoupling tool selection from argument generation via a specialized Semantic Router or Action Layer cuts decision latency to under 45 milliseconds on Apple Silicon, eliminating context bloat and transforming sluggish mobile agents into instantaneous personal software.

The Computational Bottleneck of Autoregressive Tool Selection

Evaluating the mechanical limits of edge AI requires recognizing how autoregressive Large Language Models process tool-augmented prompts. A language model forward pass operates in two distinct phases governed by the Roofline Model: an initial prompt prefill phase that is compute-bound, followed by sequential autoregressive token decoding that is strictly memory-bandwidth bound.

In conventional agentic architectures—such as ReAct loops, OpenAI-compatible function calling schemas, or naive system prompting—every tool the agent might possibly invoke must be serialized as a JSON Schema and injected into the context window. Declaring a modest suite of eight to fifteen capabilities (calendar scheduling, local document searching, reminders, calculator evaluation, contacts management, and local terminal execution) injects between 1,500 and 3,500 tokens of static schema definition into every single conversational turn.

This architectural pattern imposes severe penalties on mobile hardware like the iPhone 16 Pro (A18 Pro) and Mac workstations:

• Prefill Latency Explosion (TTFT): Evaluating a 2,500-token prompt prefill on mobile silicon consumes between 600 ms and 1,200 ms of continuous GPU execution before generating the very first token. In an interactive assistant, this latency introduces perceptible lag before each response.
• KV Cache Memory Bloat: The Key-Value activations corresponding to those static tool definitions must remain pinned in unified DRAM across the entire session. In long multi-turn interactions, unneeded tool schemas consume hundreds of megabytes of resident memory, accelerating pressure toward iOS Jetsam eviction limits.
• High False-Positive Rates in Small Models: While 70-billion-parameter cloud models reliably distinguish when a tool is required versus when a plain conversational response is appropriate, edge-optimized models (ranging from 1.5B to 8B parameters) struggle with complex system prompts. Faced with dense JSON schemas, compact models frequently hallucinate parameter names, emit malformed brackets, or invoke tools impulsively when the user simply submitted a greeting.

Forcing a generative model to simultaneously determine intent, select the appropriate function, and structure JSON arguments creates a compound failure domain. Solving this friction requires separating the decision of what to execute from the generative mechanics of argument formatting.

To analyze how reasoning and action loops execute on device, review the guide to local AI agents for iPhone on Apple MLX.

Semantic Routing Architecture: Vector Proximity vs Distilled Classifiers

Semantic routing intercepts user input at the system boundary, classifying intent and routing the message to the optimal execution target before invoking a heavy generative backbone. In local edge environments, two primary routing paradigms have emerged:

1. Bi-Encoder Cosine Proximity Routing: Popularized by libraries like semantic-router, this approach passes the incoming user query through a compact embedding model (such as MiniLM or BGE, typically 20M to 100M parameters) to produce a dense vector representation. The system then computes the cosine distance against pre-embedded vectors representing sample user utterances for each target tool. If the similarity exceeds a calibrated threshold, the query routes to that specific tool pipeline. While execution is fast (10–25 ms on Metal), static utterance matching struggles with dynamically registered tools (which require re-indexing vectors at runtime) and exhibits high sensitivity when user phrasing diverges from reference examples.
2. Distilled Semantic Action Layers: Rather than relying on vector distances between isolated utterances, a dedicated classification model directly cross-attends the user prompt against dynamic tool manifests in a single forward pass. The network computes softmax probabilities across all active tool classes while maintaining an explicit, calibrated NONE class for standard conversational queries. This enables runtime tool registration and deterministic fallback handling without retraining embedding indices.

The operational trade-offs across these architectures illustrate the dramatic efficiency gains of decoupled routing on Apple Silicon:

By delegating tool selection to a dedicated, sub-200MB model, the primary generative LLM is completely freed from evaluating monolithic system prompt schemas, slashing interactive response latency and preserving memory bandwidth.

To dive deeper into schema definitions and structured function calling with Metal grammars, see the guide on function calling with local LLMs on Apple Silicon.

Lapis-SAL Research: Distillation and 40ms Decision Layers

To investigate the viability of on-device semantic routing, Lapis Labs developed Lapis-SAL (Semantic Action Layer), an open-source research initiative exploring specialized intent and tool routing for Apple Silicon edge devices. The project focuses on solving the tension between dynamic tool availability and strict edge compute budgets.

The research architecture is structured around knowledge distillation from a larger teacher model:

• Teacher Model (421M Parameters): Trained on extensive bilingual corpora (English and Spanish) covering multi-domain tool definitions, parameter requirements, and conversational edge cases. The teacher learns generalized semantic representations across diverse user intents and out-of-scope conversational patterns.
• Distilled Student Model (150M Parameters): Knowledge distillation transfers the classification and routing capabilities of the teacher into a compact 150M parameter student. The student weights are quantized and converted to native Apple MLX format (lapislabsai/lapis-sal-150M-MLX), compiling directly to Metal Performance Shaders for unified memory execution.
• Open Research Availability: Published under the permissive Apache 2.0 license, the model family provides open weights for the MLX, PyTorch, and Hugging Face communities.

On a private synthetic benchmark comprising 1,000 rigorous evaluation scenarios, the distilled 150M MLX student demonstrated a 95.1% routing accuracy, deciding between dynamic tool invocation and the NONE fallback in approximately 40 milliseconds on consumer Apple Silicon hardware.

It is crucial to recognize the architectural boundaries of this work: Lapis-SAL is an intent classification and routing engine, not a generative conversational LLM. It does not produce free-form creative prose or generate synthetic tokens. Instead, it serves as a lightweight, deterministic gatekeeper that sits in front of generative models, ensuring that computational resources are expended only when strictly necessary.

To understand how token overhead and expanding context degrade unified memory, review local LLM context window limits on iOS.

Two-Stage Routed Agent Pipeline: Guided Generation without Context Bloat

Deploying a semantic router transforms local agent execution into a clean, two-stage pipelined architecture. Rather than routing all prompts through an all-or-nothing monolithic prompt, execution branches conditionally based on high-speed classification:

1. Stage 1: Intent Ingress and Routing (0–45 ms): The incoming user prompt is ingested alongside the active tool registry by the 150M Action Layer. Within 40 milliseconds, the router outputs either an explicit tool identifier (e.g., CALENDAR_CREATE_EVENT) or NONE.
2. Stage 2A: Direct Conversational Streaming (When Router = NONE): If the user submitted a general question, brainstorming prompt, or conversational greeting, the request bypasses tool mechanisms entirely. The prompt is forwarded to the generative LLM without a single line of JSON schema. The model begins streaming tokens within 80 to 110 ms, eliminating prefill lag and guaranteeing zero parameter hallucinations.
3. Stage 2B: Targeted Grammar-Constrained Extraction (When Router = Tool ID): If a specific tool is selected, the runtime retrieves only the single schema matching that tool and provides it to the generative model. Furthermore, the generation pass is bound to a finite-state machine (FSM) grammar parser in Apple MLX, enforcing 100% valid JSON argument extraction at the logit level.

This two-stage division delivers transformative operational benefits on iOS and macOS:

• Context Compression: By loading only the single relevant tool schema instead of fifteen unneeded functions, prompt tokens decrease by up to 90%, shrinking prefill latency from over 900 ms down to under 120 ms.
• Thermal and Battery Conservation: Mobile processors spend significantly less time in peak GPU power states. In typical multi-turn sessions where only 10% to 20% of prompts require tool execution, the device avoids executing heavy matrix prefill calculations on 80% of interactions.
• Memory Headroom: The unified DRAM budget remains strictly partitioned. The 150M routing model consumes roughly 185 MB of wired memory, while the generative LLM operates with a lean KV cache that stays comfortably beneath the 5.0 GB iOS Jetsam termination threshold.

To evaluate why token-by-token sequential decoding is governed by DRAM bus bandwidth, see Apple Silicon memory bandwidth for LLMs.

Zero-Egress Security and Local Memory Accounting on Apple Silicon

Beyond latency and memory efficiency, semantic routing on Apple Silicon establishes an uncompromising privacy boundary. Cloud-based agent frameworks route user requests through remote proxies, exposing confidential prompts, calendar events, personal contact names, and proprietary documents to third-party datacenters before any local code executes.

True local intelligence requires mathematical zero-egress execution. In native Swift and Metal runtimes, both the Semantic Action Layer and the primary language model execute entirely in unified DRAM without maintaining open network sockets. Stripping the com.apple.security.network.client entitlement at the Darwin Mach kernel level ensures that neither the intent classifier nor the generative agent can transmit telemetry or token payloads off the physical silicon.

For individuals and organizations seeking private, sovereign intelligence that operates without recurring cloud subscriptions or corporate data retention, local agent execution on Apple Silicon represents the future of personal computing. Lapis brings local language models, on-device document intelligence, and privacy-first tool workflows together in a native application built specifically for iPhone, iPad, and Mac on the App Store.

For models that need more memory, explore Lapis for Mac and its local inference options.