Quantization 4-Bit vs 8-Bit: Mobile LLM Guide

For on-device Large Language Model (LLM) inference on iOS, precision compression is not merely a disk storage convenience; it is the physical boundary between a responsive local application and immediate process termination by the kernel. While 8-bit quantization was long considered the standard for zero-degradation quantization on desktop GPUs, consumer Apple Silicon chips operate under unified memory architectures and aggressive operating system memory ceilings. Understanding the trade-offs between 4-bit and 8-bit quantization reveals why 4-bit is the optimal operating regime for iPhones, delivering twice the token generation throughput while preventing operating system memory kills.

The Mechanics of Post-Training Quantization: INT8 vs INT4

Post-training quantization (PTQ) transforms 16-bit floating-point parameters (FP16 or BF16) into lower bit-width integer representations. Mathematically, linear affine quantization projects continuous weights w into discrete buckets using a scale factor s and a zero-point offset z:

q = clamp(round(w / s) + z, q_min, q_max)

During inference, dequantization reconstructs the floating-point approximation via w_approx = s × (q - z). The fundamental distinction between 8-bit (INT8) and 4-bit (INT4) lies in representation density:

• 8-Bit Quantization (INT8): Provides 256 discrete bins (from -128 to 127). The quantization noise is negligible, easily accommodating the dynamic range of weight matrices. Outlier activations—which naturally occur in transformer attention heads and multi-layer perceptron (MLP) projections—are absorbed without clipping normal weight distributions.
• 4-Bit Quantization (INT4): Provides only 16 discrete bins (from -8 to 7, or 0 to 15). A naive per-tensor or per-channel 4-bit approach causes severe accuracy degradation because an extreme weight value stretches the scale factor s, collapsing all intermediate weights into identical discrete buckets.

To overcome this limitation, Apple MLX and modern inference runtimes implement group-wise affine quantization. Rather than scaling an entire matrix by a single factor, weights are partitioned into independent blocks of 32, 64, or 128 elements. Each group maintains its own floating-point scale and bias. At a group size of 64, the metadata adds only ~0.25 bits per parameter, while localizing outlier distortions and preserving over 98.5% of full-precision numerical fidelity.

The Roofline Model: Why 4-Bit Doubles Generation Speed

A persistent misconception in mobile machine learning is that reducing precision accelerates execution primarily by saving arithmetic compute cycles. In reality, single-user autoregressive token generation at batch size 1 is strictly memory-bandwidth bound, as formalized by the classical Roofline model:

Tokens/second = (Memory Bandwidth in GB/s / Parameter Weight Footprint in GB) × Efficiency Factor

Because generating token N+1 depends on token N, the processor cannot evaluate future tokens concurrently. For every single token produced, the entire weight tensor across all transformer layers must be transferred across the physical memory bus into on-die cache and register files.

Consider a 3.2B parameter model (such as Llama 3.2 3B) running on an iPhone 16 Pro powered by the Apple A18 Pro SoC, which delivers 68 GB/s of memory bandwidth at ~75% sustained Metal efficiency:

• Under 8-bit Quantization: The model weights consume 3.21 GB of DRAM. Every generated token requires streaming 3.21 GB across the bus. At 68 GB/s and 75% efficiency, theoretical decode throughput is (68 / 3.21) × 0.75 ≈ 15.9 tok/s. In real-world MLX Metal execution, it clocks at 15.4 tok/s.
• Under 4-bit Quantization (Group Size 64): The model weights consume 1.82 GB (including scales and zero-points). The bus transfer drops by 43.3%. Theoretical throughput increases to (68 / 1.82) × 0.75 ≈ 28.0 tok/s. In real-world MLX Metal execution, it clocks at 27.6 tok/s.

What about dequantization overhead? Apple Silicon GPU cores feature massive vector SIMD processing power. Unpacking two 4-bit nibbles into 16-bit registers occurs within GPU threadgroup registers while the memory controller prefetches the next cache lines from LPDDR5X DRAM. Because compute units are starved waiting for memory, dequantization introduces zero latency penalty. Moving to 4-bit yields an immediate, near-linear 1.8x to 2.0x acceleration.

The Memory Boundary: The iOS Jetsam Subsystem and KV Cache Growth

While memory bandwidth determines generation latency, operating system memory governance dictates stability. Unlike macOS, which manages memory pressure using compressed RAM and disk-backed NVMe virtual swap files, iOS disables disk swap for third-party applications to protect flash endurance and preserve strict UI responsiveness.

The iOS kernel enforces memory limits through its watchdog: the jetsam subsystem. On an 8 GB iPhone (iPhone 15 Pro, iPhone 16, or iPhone 16 Pro), the hard ceiling for foreground third-party app dirty memory sits between 4.5 GB and 4.8 GB. Exceeding this boundary triggers an immediate, uncatchable EXC_RESOURCE / MEMORY SIGKILL termination.

In a real-world chat application, total resident memory consists of four competing layers:

1. Base Model Weights: Static parameter memory held permanently in RAM.
2. Key-Value (KV) Cache: Dynamic memory that grows linearly with conversation length: 2 × layers × heads × head_dim × tokens × bytes.
3. Prefill Activation Spikes: Ephemeral matrix multiplication tensors allocated during initial prompt evaluation.
4. Application Frameworks & Metal State: Display buffers, UI rendering, tokenizer graphs, and Metal pipeline caches (~250–350 MB).

This reality exposes the fatal vulnerability of 8-bit models on mobile. A 3B model in 8-bit requires 3.21 GB of static RAM. When a user submits an 800-token prompt and the conversation expands to 2,048 tokens, an FP16 KV cache adds ~260 MB, and prefill peak allocations add ~450 MB. Total dirty memory reaches 3.92 GB. If the context expands toward 4,096 tokens (~520 MB cache) or the user attaches an image, memory exceeds 4.6 GB. Jetsam instantly terminates the app.

Under 4-bit quantization, the same 3B model occupies only 1.82 GB of baseline RAM. Even with an 8,192-token context (~1.05 GB cache) and prefill spikes, peak consumption remains under 3.3 GB, preserving a safe 1.2+ GB buffer beneath the jetsam threshold.

Technical Comparison: 4-Bit vs 8-Bit Across Mobile Architectures

The following table provides verified engineering metrics across representative small language models, comparing memory footprint, generation latency, Wikitext-2 perplexity, and iOS memory stability:

Perplexity and Reasoning Quality: The Empirical Accuracy Trade-off

The historical argument against 4-bit precision has centered on perplexity degradation. However, empirical scaling laws established by Dettmers & Zettlemoyer (2023) demonstrated that as parameter count increases, 4-bit precision consistently defines the Pareto-optimal frontier between memory bits consumed and task accuracy.

On a 3B parameter architecture, moving from 8-bit to group-wise 4-bit quantization (group size 64) increases Wikitext-2 perplexity from 6.87 to 7.02 (a modest delta of +0.15). On standardized downstream reasoning benchmarks, the performance delta is negligible:

• MMLU (5-shot General Knowledge): Drops from 63.4% (INT8) to 62.1% (INT4)—a minor 1.3% variance.
• GSM8K (8-shot Mathematical Reasoning): Drops from 44.8% (INT8) to 43.5% (INT4)—a 1.3% variance.
• HumanEval (0-shot Code Generation): Drops from 31.1% (INT8) to 30.5% (INT4)—a 0.6% variance.

This empirical evidence leads to a decisive architectural conclusion: allocating parameter budget to model architecture at 4-bit precision vastly outperforms running a smaller model at 8-bit precision.

A 1.7B model in 8-bit occupies approximately 1.85 GB of RAM and scores ~49.8% on MMLU. A 3.2B model in 4-bit occupies virtually the same footprint (1.82 GB) yet scores 62.1% on MMLU—a 12.3-point leap in cognitive capability. In mobile constraints where RAM is the scarce resource, 4-bit quantization provides substantially higher intelligence per megabyte.

Engineering Best Practices for Deploying Local Models

To maximize speed, context stability, and output accuracy when deploying local models on Apple Silicon devices, follow these core engineering principles:

1. Standardize on 4-Bit with Group Size 64 for Sub-7B Models: Group size 64 strikes the ideal balance between minimizing outlier quantization noise and avoiding metadata memory overhead. Avoid per-channel quantization without grouping on sub-7B models.
2. Maintain a 2.5 GB Safety Headroom on 8 GB iPhones: Never allow static model weights to exceed 2.2 GB on consumer iOS devices. Reserving 2.5 GB ensures room for KV cache expansion, prefill tensor allocations, and background iOS memory pressure.
3. Quantize the Key-Value Cache for Extended Context: When conversational threads exceed 4,096 tokens, compress the KV cache to 8-bit or 4-bit integer format. In Apple MLX, KV cache quantization cuts attention memory by 50% to 75% with zero degradation in retrieval accuracy.
4. Avoid 8-Bit Precision for Models Above 2B Parameters: While 8-bit offers comfort for developers paranoid about precision loss, on iOS it causes immediate memory risk on 3B models and fatal crashes on 7B models. Reserve 8-bit exclusively for sub-1.5B utility networks.
5. Leverage Zero-Copy Metal Buffers with Lapis: By binding directly to Apple MLX and Metal unified memory, Lapis eliminates CPU-GPU copy bottlenecks, schedules concurrent dequantization, and runs 100% offline with zero cloud latency.