Razor is a small C++ engine for on-device decode. On an Apple M4 Max, it decodes SmolLM2-135M at 155 tokens per second. On a Snapdragon 8 Gen 3, it reaches 151 on a cold device, where LiteRT q8 manages 64. Answer quality matches mlx-lm on a measured suite.
Razor int8 reaches 155 tok/s on M4 Max and 151 on Snapdragon 8 Gen 3 (cold peak). LiteRT q8 reaches 48.9 on M4 Max and 64 on Snapdragon.
A 68-question factual suite, identical prompts, greedy decode. Razor scores 58 of 68 and mlx-lm scores 60 of 68. The 95% confidence intervals overlap.
Every model Razor runs has been checked against HuggingFace transformers greedy decode. fp32 output is byte-identical: 32 of 32 tokens match for SmolLM2 and Qwen3. Gemma 3 1B is int8 and matches top-1 on 5 of 5 prompts with 0.95 logit correlation.
SmolLM2-135M: fp32 651 MB, 99.8 tok/s, 32/32 match. int8 249 MB, 155 tok/s, 25/32 match. int4 189 MB, 41.1 tok/s, 1/32 match. SmolLM2-360M: fp32 1636 MB, 37.5 tok/s, 32/32 match. int8 552 MB, 29.5 tok/s, 3/32 match. SmolLM2-1.7B: fp32 7248 MB, 7.3 tok/s, 32/32 match. int8 2117 MB, 7.4 tok/s, 13/32 match. fp32 is exact at every scale. int8 holds well at 135M but diverges early at 360M and 1.7B. int4 diverges immediately.
Tested on SmolLM2-135M. Output was correct in every case: 32 of 32 tokens matched normal decode. No configuration improved throughput. Sequential with 25 draft layers and K=4 reached 46.3 percent accept rate but only 29.6 tok/s versus 73.3 baseline. Batched with 25 draft layers and K=4 reached 32.3 percent accept at 13.8 tok/s. At 135M scale, single-token decode is already fast and the verification pass costs more than it saves. Larger models should benefit. That is future work.
The Snapdragon 8 Gen 3 throttles under sustained LLM decode. Peak numbers are from a cold device. After 3 to 4 minutes of continuous generation, throughput drops 3 to 5x. SmolLM2 135M int8: 151 tok/s cold, 35 tok/s throttled, 4.3x drop. Qwen3 0.6B int8: 47 tok/s cold, 12 tok/s throttled, 3.9x drop. Gemma 3 1B int8: 24 tok/s cold, 7 tok/s throttled, 3.4x drop. This is a silicon and cooling constraint. Every CPU-based runtime on this chip faces the same curve. Apple Silicon does not show this curve under the same load. The M4 Max sustains 155 tok/s across a 400-token generation with no measurable drop. The difference is active cooling versus passive. For interactive use on Android, the cold peak is the relevant number: a user sends a prompt, the model generates 20 to 50 tokens, and the phone returns to idle before heat accumulates.
Four phases: fp32 baseline 73.3 tok/s, fp32 phase 2 99.8, int8 quantization 145.5, int8 phase 4 155.
int8 at 249 MB reaches 155 tok/s. fp32 at 651 MB reaches 99.8. int4 at 189 MB drops to 41.1 with quality loss.
CPU wins at 135M. GPU takes over at 360M and 1.7B. Razor picks automatically.
A 212-token prompt on Qwen3 0.6B reaches 132 tok/s batched versus 45 tok/s per-token, a 2.9x speedup. SmolLM2 135M gains 1.35x. Output is identical.
SmolLM2-135M decode: Razor int8 155 tok/s on M4 Max, 151 on Snapdragon (cold peak). LiteRT q8 48.9 on M4 Max, 64 on Snapdragon. LiteRT fp32 44.8 on M4 Max, 39.7 on Snapdragon. mlx-lm 8-bit 95.2 on M4 Max, no Android path.
Gemma 3 1B IT on M4 Max: Razor int8 45.4 tok/s, HuggingFace transformers fp32 27.6, mlx-lm 8-bit GPU 77. Razor wins on CPU by 1.64x over the HuggingFace reference.
Apple M4 Max (macOS 15, shipping). Snapdragon 8 Gen 3 (Android 14, shipping, Honor FCP-N49). Apple A-series (iOS 16.4+, compiles, XCFramework built, needs a connected iPhone for numbers).
Every model is checked against its HuggingFace reference. fp32 weights must match 32 of 32 tokens. int8 weights must match top-1 argmax with top-50 overlap above 0.8. A 68-question factual suite scores Razor 58 of 68 and mlx-lm 60 of 68, with overlapping confidence intervals. The SDK ships with a benchmark harness so every number can be re-run on your hardware.
The SDK is a tarball with static libraries, headers, weight export tools, a benchmark harness, and a JNI shared library for Android. macOS arm64 links system frameworks. Android arm64 uses the NDK. iOS ships a static XCFramework. Weights are on HuggingFace under the base model licence. Licence verification is offline. Every file has a SHA256 checksum.
Token speed across engines. SmolLM2-135M on M4 Max: Razor int8 CPU 155 tok/s, Razor fp32 CPU 99.8, mlx-lm 8-bit GPU 95.2, LiteRT q8 CPU 48.9, LiteRT fp32 CPU 44.8, Xybrid not published. SmolLM2-135M on Snapdragon 8 Gen 3: Razor int8 CPU 151, LiteRT q8 CPU 64, LiteRT fp32 CPU 39.7, Xybrid Q4_K_M CPU 9. SmolLM2-360M on M4 Max: Xybrid Q4 Metal GPU 112, Razor fp32 CPU 37.5, Razor int8 CPU 29.5, LiteRT not published. Gemma 3 1B on M4 Max: mlx-lm 8-bit GPU 77, Razor int8 CPU 45.4, HF transformers fp32 CPU 27.6, Xybrid and LiteRT not published. Gemma 3 1B on Snapdragon: Razor int8 CPU 24, Xybrid and LiteRT not published. Qwen3 0.6B on M4 Max: Razor int8 CPU 59.6, Xybrid and LiteRT not published. Qwen3 0.6B on Snapdragon: Razor int8 CPU 47, Xybrid and LiteRT not published. SmolLM2-1.7B on M4 Max: Razor fp32 CPU 7.3, Xybrid and LiteRT not published. Each bar labels its quantization and compute path. Xybrid's 360M number runs through llama.cpp on Metal GPU with Q4 weights. Razor's 360M number is fp32 on CPU. Different quantization, different compute. Xybrid's 135M Snapdragon number runs through llama.cpp on CPU with Q4_K_M weights.
Four engines compared across five groups: Performance, Validation, Platform support, Scope and features, Distribution. Razor is a purpose-built C++ engine with no runtime dependencies. Xybrid wraps llama.cpp, ONNX, and Core ML. LiteRT uses XNNPACK. mlx-lm uses Apple's MLX framework. Razor validates every model against HuggingFace (32/32 fp32 match, 5/5 int8 top-1, 0.95 logit correlation). Xybrid, LiteRT, and mlx-lm publish no parity data. Razor ships on macOS and Android, with iOS compiling but unmeasured. Xybrid ships on macOS, Linux, and Windows; Android is broken (Issue #16). LiteRT ships on all platforms. mlx-lm is macOS only. Razor is LLM-only, on-device, CPU. Xybrid adds ASR, TTS, vision, cloud routing, fleet management, and OTA updates. Razor ships a benchmark harness in the SDK. Xybrid ships telemetry. Razor is proprietary with a free evaluation. Xybrid and LiteRT are Apache-2.0. mlx-lm is MIT.
Fourteen milestones shipped: fp32 CPU backend (73.3 tok/s), fp32 phase 2 (99.8), int8 quantization (145.5), Android arm64 port (151 tok/s on Snapdragon), int8 optimization (155 tok/s), accuracy suite (85.3 percent, 58/68), temperature and top-p sampling, 360M validation (37.5 tok/s, 32/32 match), int8 phase 4 (182 tok/s, 35 percent faster), 1.7B validation (7.3 tok/s, 32/32 match), Qwen3 0.6B (47 tok/s on Snapdragon), batched prefill (2.9x prompt speedup), Locale app (shipping on device), Gemma 3 1B (45.4 tok/s, 5/5 top-1 match). Next: iOS harness.