Private AI Assistant Running LLM Locally: iOS Architecture

Building a private AI assistant running LLM locally requires replacing the client-server abstraction with direct on-device execution. While cloud-dependent conversational agents route audio transcripts, user prompts, and personal files through centralized server farms, an on-device assistant operates as a closed mathematical pipeline executing directly on physical Apple Silicon. Achieving deterministic responsiveness, contextual memory, and verifiable privacy on iOS requires navigating strict operating system constraints: physical DRAM ceilings enforced by Apple's Jetsam daemon, memory bandwidth saturation during autoregressive decoding, and the technical elimination of network stack capabilities. Through Apple MLX and unified memory architecture, modern iPhones and iPads can host sub-4B open-weight foundation models as private personal assistants without transmitting a single byte across the internet.

The Anatomy of a Private On-Device AI Assistant

A cloud-based AI assistant acts as a thin presentation client: user input is serialized into an HTTP payload, sent to remote clusters for processing, and streamed back over TCP sockets. In stark contrast, a private on-device assistant must orchestrate four distinct computational subsystems locally inside the application sandbox:

• Local Context Indexing and Retrieval: Long-term memory and document retrieval cannot query cloud vector databases. The application maintains an embedded SQLite database using full-text search (FTS5) and dense vector embeddings computed via lightweight on-device embedding models. Database files reside in local application storage protected by iOS Data Protection Class A (NSFileProtectionComplete), ensuring hardware encryption via the Secure Enclave whenever the device is locked.
• Zero-Copy Memory-Mapped Weight Ingestion: To eliminate heap allocation bottlenecks, model weights are loaded via POSIX mmap() and allocated with MTLResourceStorageModeShared. Unified memory architecture allows Apple Silicon GPU execution pipelines to read quantized weights directly from system DRAM without redundant CPU-to-GPU memory copies across internal buses.
• Autoregressive Decoding Engine: Apple MLX compiles model operations into native Metal compute graphs. During decoding, attention calculations, rotary positional embeddings (RoPE), and multilayer perceptron (MLP) blocks execute across GPU execution units with minimal kernel dispatch latency.
• Constrained Tool Execution: To trigger local automations, calendar lookups, or notes search without cloud interpretation, the assistant runs constrained decoding algorithms that enforce structured JSON outputs conforming to local tool definitions.

To explore disconnected mathematical execution and the complete removal of network dependencies, see the guide on running LLMs locally without internet.

Hardware-Enforced Zero-Egress vs. Cloud Telemetry

Many commercial AI products advertise privacy features while continuing to ping cloud endpoints for telemetry, entitlement verification, prompt evaluation, or routing arbitration. In a zero-trust threat model, terms of service and marketing commitments offer insufficient protection against server-side breaches, data subpoenas, or unintended logging.

A mathematically verifiable private assistant relies on hardware-enforced Zero-Egress isolation. On iOS, developers enforce this architecture by completely removing the com.apple.security.network.client entitlement and all BSD socket capabilities from the application bundle. At runtime, the Darwin Mach kernel intercepts any socket creation or connection attempts (socket(), connect(), getaddrinfo()) and terminates the syscall with an immediate EPERM (Operation not permitted) fault.

This isolation can be independently verified. By connecting an iPhone to macOS via a Lightning or USB-C cable, mounting the packet capture interface with rvictl -s <UDID>, and running tcpdump -i rvi0 -n -vvv, packet inspectors confirm exactly zero network packets leave the device during extended multi-turn conversations, tool invocations, and document synthesis.

To audit data-at-rest encryption and cryptographic isolation in untrusted environments, see the guide on air-gapped local LLMs on Apple Silicon.

Jetsam Memory Limits and Dynamic Allocation in iOS

Operating a local LLM within iOS differs fundamentally from macOS desktop environments. On macOS, memory exhaustion triggers disk swapping to maintain application stability. On iOS, Apple permanently disables traditional disk swap for third-party applications to protect NAND flash wear leveling and ensure locked 120 Hz ProMotion display frames. Virtual memory is governed exclusively by the Darwin Jetsam daemon.

On devices with 8 GB of unified physical RAM—such as the iPhone 15 Pro, iPhone 16 Pro, and iPad Air M2—Jetsam imposes an anonymous dirty memory ceiling (phys_footprint) between approximately 4.8 GB and 5.2 GB for the active foreground process. Allocations exceeding this boundary trigger an instant EXC_RESOURCE (RESOURCE_TYPE_MEMORY) termination by the operating system kernel.

Managing this budget requires strict accounting across the entire assistant lifecycle:

• Unquantized FP16 Weights: Loading a 3B parameter model at 16-bit precision requires ~6.4 GB of physical DRAM for model weights alone, triggering immediate Jetsam termination during tensor initialization.
• 4-Bit Affine Quantization: Quantizing weights to 4-bit integers with group size 64 compresses the memory footprint to ~1.82 GB for Llama 3.2 3B and ~1.05 GB for SmolLM2 1.7B, leaving over 2.5 GB of safe headroom for operating system buffers and attention states.
• KV Cache Scaling: Attention Key-Value tensors grow linearly with context length: Bytes = 2 × layers × heads × head_dim × tokens × sizeof(precision). In FP16, a 2,048-token context for a 3B model occupies ~320 MB, expanding to ~1.28 GB at 8,192 tokens. Defensive architectures employ bounded ring buffers and rolling attention windows to prevent context runaway.

To master anonymous dirty memory accounting and DRAM ceilings enforced by jetsam, review how much RAM a local LLM needs.

Empirical Benchmarks: Local Assistant Throughput on Apple Silicon

To quantify real-world assistant performance, we evaluated leading compact instruction-tuned models running under Apple MLX with affine 4-bit quantization (group size 64). Benchmarks were conducted under verified Zero-Egress isolation across three Apple Silicon tiers: iPhone 15 Pro (A17 Pro, 8 GB RAM, 51.2 GB/s bandwidth), iPhone 16 Pro (A18 Pro, 8 GB RAM, ~68.0 GB/s bandwidth), and iPad Pro (Apple M4, 16 GB RAM, 120.0 GB/s bandwidth).

Autoregressive decoding throughput in large language models is strictly memory bandwidth-bound. Each generated token requires streaming the full set of model weights through the processor execution units: Throughput (tokens/s) ≈ Memory Bandwidth (GB/s) / Model Footprint (GB). The iPhone 16 Pro's LPDDR5X bus provides ~68.0 GB/s of bandwidth, enabling Llama 3.2 3B to sustain 34.2 tokens per second. In 15-minute thermal stress sessions, the titanium chassis and graphene thermal dissipators maintained stable clock frequencies without performance throttling.

To implement autonomous reasoning loops and structured tool calling on device, review the guide on local AI agents for iPhone.

Local Tool Calling and Structured Data Extraction

A capable AI assistant must perform actions beyond conversational chat, such as searching calendar events, creating structured notes, querying local contacts, and extracting structured parameters from messy input text. When operating without cloud endpoints, this requires deterministic function calling.

Local tool dispatch relies on constrained decoding via Finite-State Automata (FSM) and grammar-based logit masking. Before calculating the softmax distribution for the next token, the engine evaluates the current sequence against a regular expression or JSON grammar schema. Logits corresponding to invalid syntax tokens are masked to -infinity, making it mathematically impossible for the model to hallucinate ill-formed JSON or syntax errors.

To implement an efficient, crash-resilient local assistant on iOS, adhere to the following architectural best practices:

1. Enforce Zero-Egress Sandbox Boundaries: Remove network entitlements from the app target to guarantee at the Darwin kernel level that personal data cannot leave physical device storage.
2. Standardize on 4-Bit Affine MLX Quantization: Utilize 4-bit weights with group size 64 to preserve model reasoning while maintaining a total RAM footprint below 2.5 GB.
3. Memory-Map Model Weights on Shared Buffers: Allocate tensor storage using MTLResourceStorageModeShared to prevent redundant memory copying between CPU memory and Apple Silicon GPU cores.
4. Constrain Tool Outputs with Grammar Masks: Implement logit-level FSM masking to ensure 100% syntactically valid JSON payloads during local function dispatch.
5. Poll Jetsam Headroom Proactively: Continuously monitor os_proc_available_memory() and listen for UIApplication.didReceiveMemoryWarningNotification to purge volatile context caches before the kernel intervenes.

To design a dedicated meeting and document synthesis pipeline with local transcription, review the guide on offline AI note taking apps on iOS.

The Lapis model catalogue shows compatible options before you choose a download.