SmolLM2-135M at 155 Tokens per Second on Apple Silicon

Published: August 28, 2026 · Read Time: 6 min read · Category: Machine Learning

Author: Abhishek Shivakumar (Systems & Audio Engineering)

A small C++ engine for on-device decode. Architecture, ARM NEON SIMD packing, int8 accuracy, and Snapdragon 8 Gen 3 thermal behaviour.

The Decode Bottleneck

Small language models spend most of their execution time waiting on memory bandwidth during single-token autoregressive generation. At 135M parameters, invoking GPU kernels introduces submission overhead that exceeds the compute time of each layer.

Razor is a compact C++ engine built for CPU-first execution on ARM architectures. Running on the CPU cores keeps the weights in fast cache hierarchies and avoids driver round-trips for small matrix-vector operations.

Throughput Baseline
On an Apple M4 Max, Razor decodes SmolLM2-135M int8 at 155 tokens per second on CPU. On a Snapdragon 8 Gen 3, it achieves 151 tokens per second on a cold device.

C++ SDK Integration

The engine ships as a static library with single-header C++ bindings. Weight checkpoints load directly from disk without external runtime dependencies or background daemons.

Generation supports streaming token callbacks, temperature configuration, top-p nucleus sampling, and state reset hooks.

#include <razor/razor.h>
#include <iostream>

int main() {
    razor::Engine engine;
    engine.load_weights("smollm2-135m-int8.rzr");

    razor::SamplingParams params;
    params.temperature = 0.7f;
    params.top_p = 0.9f;

    engine.generate("Explain the Doppler effect in one sentence:", params,
        [](std::string_view token) {
            std::cout << token << std::flush;
            return true;
        }
    );
    std::cout << "\n";
    return 0;
}

Quantization and Validation

Every quantized model in Razor undergoes verification against HuggingFace Transformers greedy decoding. fp32 output produces byte-identical results across 32 consecutive tokens for SmolLM2 and Qwen3 checkpoints.

On a 68-question factual test suite with identical greedy prompts, Razor int8 scores 58 of 68, while mlx-lm fp16 scores 60 of 68. The confidence intervals overlap across the test run.

Model Scale	Format	Memory Footprint	Decode Speed (M4 Max)	Greedy Token Match
SmolLM2-135M	fp32	651 MB	99.8 tok/s	32 / 32
SmolLM2-135M	int8	249 MB	155.0 tok/s	25 / 32
SmolLM2-135M	int4	189 MB	41.1 tok/s	1 / 32
SmolLM2-360M	fp32	1636 MB	37.5 tok/s	32 / 32
SmolLM2-360M	int8	552 MB	29.5 tok/s	3 / 32
Gemma 3 1B	int8	1180 MB	45.4 tok/s	5 / 5 top-1

Measured Decode Throughput

Measured decode numbers compare dedicated C++ routines against general mobile runtimes. LiteRT q8 achieves 48.9 tok/s on M4 Max and 64.0 tok/s on Snapdragon 8 Gen 3.

Razor achieves 155.0 tok/s on M4 Max and 151.0 tok/s on Snapdragon 8 Gen 3. The CPU-first approach removes driver synchronization barriers and keeps per-token latency consistent.

Thermal Behaviour on Android

Passive mobile hardware exhibits thermal throttling under sustained computation. On the Snapdragon 8 Gen 3, peak performance is reached on a cold device. After 3 to 4 minutes of uninterrupted generation, clock frequencies adjust to manage thermal dissipation.

SmolLM2-135M int8 drops from 151 tok/s cold to 35 tok/s under extended load. Active cooling on the Apple M4 Max prevents thermal decline, sustaining 155 tok/s across 400-token generations. For interactive on-device prompts, cold peak throughput dictates the user response window.

Batched Prefill Throughput

Prompt processing benefits from batched matrix multiplications rather than autoregressive vector passes. Processing a 212-token prompt on Qwen3 0.6B achieves 132 tok/s under batched GEMM compared to 45 tok/s single-token decode.

The SDK includes a verification harness and standalone static libraries for macOS, Android arm64 via the NDK, and iOS XCFramework builds.

Back to Quilio Blog