Run Vision Language Models Locally: Architecture & Apple MLX

Processing visual data—confidential financial invoices, medical imagery, proprietary product blueprints, or personal photo libraries—creates a severe privacy vulnerability when routed through remote cloud APIs. Running Vision-Language Models (VLMs) locally directly on Apple Silicon solves this exposure, coupling high-throughput on-device optical comprehension with zero network transmission. By exploiting Apple's Unified Memory Architecture (UMA) and native Metal acceleration via MLX-VLM, modern edge architectures achieve desktop-class visual reasoning with deterministic latency and strict cryptographic privacy.

The Anatomy of an Edge VLM: Vision Encoder, Projector, and Language Decoder

A Vision-Language Model is not a single monolithic neural network. Instead, it couples two distinct modalities—continuous pixel arrays and discrete linguistic tokens—through a tripartite architectural pipeline. To execute visual inference efficiently on edge hardware, each component must be optimized for memory footprint and compute intensity:

1. The Visual Perception Backbone (Vision Encoder): The encoder operates as the sensory frontend. Modern edge models utilize Vision Transformers (ViT) or SigLIP (Sigmoid Loss for Language-Image Pretraining) backbones. Given an input image tensor with height H, width W, and 3 color channels, the encoder decomposes the continuous 2D surface into a grid of non-overlapping square patches of dimension P × P (typically 14×14 or 16×16 pixels). Each patch is flattened into a 1D vector and mapped through a linear projection layer into a patch embedding, supplemented with 2D spatial positional encodings.
2. The Cross-Modal Projector: The visual embedding dimension (e.g., 768 in SigLIP-Base or 1,152 in SigLIP-Large) rarely aligns with the hidden dimension of the target language model (e.g., 2,048 in 3B architectures or 4,096 in 7B models). The projector bridges this geometric divide. Early architectures utilized single linear transformation matrices, whereas modern state-of-the-art models deploy two-layer Multi-Layer Perceptrons (MLPs) with GELU activations or spatial resamplers. This cross-modal projection aligns visual tokens directly into the semantic embedding space of the language backbone.
3. The Autoregressive Language Model Backbone: Once projected, visual patch embeddings are treated by the language model (such as Qwen 2.5, SmolLM2, or Llama 3.2) as an initial prompt prefix. The transformer decoder applies cross-layer self-attention across both the visual prefix tokens and user prompt text tokens, generating reasoning steps and natural language responses token by token.

Because the language decoder is autoregressive, its decoding phase operates identically to pure text models. However, the visual prefill phase introduces distinct performance characteristics that redefine edge resource management.

To contrast compact multimodal model feasibility on mobile form factors, explore our benchmark of local vision models for iPhone.

Dynamic Resolution and Patch Tiling: How Images Become Thousands of Tokens

Early multimodal architectures enforced rigid image preprocessing: input images were cropped or bilinearly interpolated into fixed square dimensions (such as 384×384 or 448×448 pixels). For real-world mobile and desktop use cases—such as parsing wide PDF balance sheets, scanning vertical smartphone receipts, or reading source code screenshots—rigid square interpolation distorts native aspect ratios, blurring small typographic features and crippling Optical Character Recognition (OCR).

Modern edge models address this through Native Dynamic Resolution (as demonstrated in Qwen2-VL and Qwen2.5-VL) and Pixel Shuffle token compression (featured in SmolVLM). Rather than forcing uniform distortion, dynamic tiling slices high-resolution images into variable grids of native patches:

N_visual_tokens = ⌈H / P⌉ × ⌈W / P⌉ + Spatial_Delimiter_Tokens

While dynamic resolution dramatically improves perception fidelity, it introduces a combinatorial challenge for on-device context windows. A 1080×1920 smartphone capture split into 14×14 patches would generate over 10,500 raw spatial patches—an impossible computational burden for mobile DRAM and GPU registers. To balance perceptual acuity with edge memory constraints, contemporary architectures employ spatial downsampling:

• 2D Spatial Merging (Qwen2.5-VL): Groups 2×2 adjacent visual tokens into a single representation through a 2D convolution or strided linear projection, reducing the token count by 4× (e.g., compressing 1,024 patches down to 256 tokens).
• Pixel Shuffle Compression (SmolVLM): Reorganizes high-resolution feature channels into lower spatial dimensions before projecting to the language backbone, enabling high-density OCR on dense infographics while keeping the active context sequence below 512 tokens.
• M-RoPE Positional Decomposition: Multimodal Rotary Position Embeddings decompose positional frequencies across three independent dimensions: temporal, vertical (height), and horizontal (width), allowing the model to interpret arbitrary aspect ratios without spatial confusion.

To analyze pixel-shuffle compression architecture and memory consumption on iOS in depth, read our breakdown on offline SmolVLM on iOS.

The Prefill Memory Spike: Why VLMs Stress Apple Silicon Differently Than Pure LLMs

In pure text language models, user prompts rarely exceed 50 to 100 tokens. The prefill phase (processing the initial prompt) completes in single-digit milliseconds. Under the Roofline performance model, pure text execution is overwhelmingly memory-bandwidth bound during sequential autoregressive decoding.

Vision-Language Models alter this operational dynamic entirely. Attaching a single image immediately injects between 512 and 1,536 visual tokens into the transformer pipeline. This triggers two critical hardware bottlenecks:

1. Compute-Bound TTFT (Time-To-First-Token): While generating sequential text tokens requires an operational intensity of less than 1.0 FLOP per byte transferred, processing the initial prompt matrix across hundreds of visual tokens is strictly compute-bound:

FLOPs_prefill ≈ 2 × N_params × N_visual_tokens

For a 3-billion-parameter model evaluating a 1,024-token image sequence, the Metal GPU must execute over 6.1 trillion floating-point operations before decoding a single character. On an iPhone A18 Pro with ~35 peak GPU TFLOPs (FP16), this prefill phase takes between 180 ms and 320 ms. On an M4 Pro Mac, high GPU core counts reduce this visual TTFT to sub-90 ms latency.

2. KV Cache Allocation Spikes: Before generating the first output token, the key and value activation tensors for all visual tokens must be materialized and stored in the Key-Value (KV) cache across all transformer layers and attention heads:

DRAM_KV = 2 × N_layers × N_heads × D_head × N_tokens × Precision_Bytes

For a 3B model (32 layers, 16 attention heads, 128 head dimension, FP16 precision = 2 bytes per element), a single 1,024-token image instantly locks ~268 MB of active DRAM into the KV cache. If a user attaches multiple images or initiates a multi-turn conversation analyzing charts, the KV cache footprint expands rapidly. On iOS devices constrained by the Darwin kernel's Jetsam memory watchdog, an unmanaged visual prefill spike can trigger immediate application termination (SIGKILL).

Because visual patches inject hundreds of tokens into context instantly, review our guide on KV cache quantization on Apple Silicon.

Quantization Strategies: Why Mixed Precision (FP16 Vision + 4-Bit Decoder) Is Essential

To run large models on edge devices with 6 GB, 8 GB, or 16 GB of unified memory, quantization is mandatory. However, applying uniform 4-bit quantization (such as INT4 or standard AWQ) across the entire multimodal network produces catastrophic failure modes in visual tasks.

Empirical evaluations across open-weight models reveal an essential architectural distinction: Vision Encoders are exceptionally sensitive to quantization error. The weights in a Vision Transformer act as high-frequency spatial feature extractors. Aggressively quantizing these layers to 4 bits introduces geometric quantization noise, blurring text edges, obliterating fine bounding lines on tables, and distorting spatial coordinates.

The industry standard for production on-device VLM execution is Mixed-Precision Quantization:

• Vision Encoder & Projector (FP16 / BF16): Kept at uncompressed 16-bit floating-point precision. Because vision encoders comprise only 150 MB to 400 MB of total parameters (e.g., SigLIP-Base is ~86M parameters; SigLIP-Large is ~300M parameters), preserving full 16-bit fidelity adds negligible memory overhead while guaranteeing pristine visual feature extraction.
• Language Decoder Backbone (4-Bit Affine Quantization): Quantized to 4-bit precision with small block sizes (typically 64 or 128 elements per group, as implemented in Apple MLX). The language model contains 85% to 92% of the parameter count. Quantizing this component slashes the weight footprint of a 3B model from ~6.2 GB down to ~1.9 GB with virtually imperceptible degradation in linguistic reasoning.

To understand how DRAM bus bandwidth dictates decoding speed following visual prefill, review our deep-dive on Apple Silicon memory bandwidth for LLMs.

Edge VLM Architecture Matrix: Memory Budgets, TTFT & Throughput

Selecting the optimal Vision-Language Model depends on the target device's unified memory ceiling and the required visual task (lightweight mobile captioning vs. high-density document OCR). The following matrix summarizes practical allocations, prefill latency, and sustained throughput across leading open-weight architectures on Apple Silicon:

To calculate total DRAM budgets combining model weights, context, and high-resolution images, see our guide on how much RAM a local LLM needs.

Zero-Copy Metal Pipeline: Native Swift MLX-VLM vs. Python/Ollama Runtimes

On traditional x86 and ARM Linux workstations, local vision pipelines suffer severe data transfer penalties. The operating system loads an image from disk into host CPU RAM, decodes the JPEG/PNG via userland libraries, serializes the uncompressed RGB tensor across the PCIe 4.0 bus (capped at ~31.5 GB/s) to discrete GPU VRAM, and triggers CUDA execution. This multi-hop pipeline introduces 50 ms to 150 ms of avoidable bus transfer and synchronization latency.

Apple Silicon fundamentally eliminates this data movement through its Unified Memory Architecture (UMA). When executing on-device VLMs via Apple Metal and native Swift frameworks:

• Zero-Copy Shared Memory (MTLStorageModeShared): The image is decoded into unified system DRAM using Apple's hardware-accelerated ImageIO frameworks. The resulting Metal texture or contiguous array buffer is passed directly to the MLX compute graph without a single byte copied over a system bus. Both the CPU and the Metal GPU access the identical physical addresses.
• No Python Interpreter Overhead: Running models through command-line Python wrappers or Docker containers introduces significant memory bloat—often consuming 600 MB to 1.2 GB of RAM solely for Python runtimes, Torch dependencies, and background daemons. Native Swift compilation eliminates interpreter latency, ensuring cold-start model execution in under 150 ms.
• Metal Performance Shaders Graph (MPSGraph): Visual patch convolutions and projection matrix multiplications are dispatched directly to Apple Silicon's high-efficiency GPU tensor cores, leaving CPU performance cores free to handle asynchronous streaming and UI rendering.

Private On-Device Vision in Practice: OCR, Document RAG & Lapis Integration

The true advantage of edge multimodal AI is privacy. When users analyze tax returns, passport scans, medical health records, or confidential architectural schematics, transmitting high-resolution unencrypted images to third-party cloud data centers introduces severe compliance and data governance risks. A model running on Apple Silicon operates in an air-gapped environment: image pixels and text tokens are processed strictly inside on-device DRAM and discarded the moment the session concludes.

In Lapis for iOS and macOS, this architecture is implemented natively. Utilizing Swift and MLXVLM, Lapis allows users to attach photos, documents, and camera captures directly to conversational threads. The image is processed entirely on the Apple Neural Engine and Metal GPU with dynamic visual token sizing and memory-safe KV cache management. Whether on an iPhone in airplane mode or a high-capacity Mac workstation, Lapis provides instant, private visual intelligence with zero cloud telemetry and zero subscription fees.

The Lapis vision overview explains how to ask about images with a compatible model.