Razor changelog

Razor uses semantic versioning. Figures were measured on an Apple M4 Max unless an entry names another device.

0.4.0 (2026-11-15)

The razor command. Text in, text out, with no Python anywhere.

Added

  • razor, one native binary with seven subcommands: list, run, chat, predict, plan, bench and calibrate. razor --version prints the version.
  • Models are chosen by name. Each model has a profile with its files, bit width and short aliases, and you can add your own.
  • A built-in tokenizer that reads a model's tokenizer.json directly, for Qwen3, SmolLM2 and Gemma 3.
  • Built-in chat templates for Qwen3, SmolLM2, ChatML and Gemma 3. The Qwen3 template covers tool definitions, tool calls, tool results and think blocks.
  • Tool calling for Qwen3 0.6B. --tools takes OpenAI-style function specs. In razor chat, /result hands the tool's output back and the model carries on.
  • razor chat: line editing and saved history, multi-line input, and the commands /clear, /reset, /undo, /retry, /system, /think, /tools, /result, /temp, /gen, /stats, /history, /save, /model and /quit. Ctrl-C stops a reply and keeps the session.
  • Later chat turns reuse the conversation so far, so only the new text is processed.
  • razor run streams one reply. Piped input joins the prompt: cat notes.txt | razor run qwen "summarize this".
  • razor predict takes a workload (--prompt, --gen, --context) and requirements (--decode, --ttft, --total, --memory), predicts every model, bit width and backend on this device, and lists which meet them, by how much, and what each of the others misses by.
  • The Razor version is available to programs through the C++ API, and the benchmark tool reports it in its JSON.
  • A check tool that compares Razor's tokenizer and templates with the tokenizers library.

Changed

  • Fast decoding is set per model, from the 0.3.0 measurements: on for SmolLM2 and Qwen3 0.6B, off for Gemma 3.
  • The first reply after loading starts sooner. First token on the first reply fell from 715 to 95 ms on Qwen3 0.6B and from 565 to 22 ms on Gemma 3 1B int4.

Removed

  • The earlier Python wrapper, which needed HuggingFace transformers for tokenizing.

Verified

  • Tokenizer and templates match tokenizer.json exactly on Qwen3 0.6B, SmolLM2 135M, 360M and 1.7B, and Gemma 3 1B: 21 hand-picked texts (accents, CJK, emoji, odd whitespace, code, inline special tokens), three real files of up to 20,000 characters, and four to six chats per model.
  • Replies from razor run are word for word the same as from the earlier Python wrapper on all six profiles.

Known issues

  • Unicode NFC normalization is not applied before tokenizing. Text that is already NFC, which typed text nearly always is, tokenizes the same.
  • Tool calling needs a model whose template has a tool format. Of the shipped profiles only Qwen3 0.6B has one, and at 0.6B it calls a tool only when the request clearly points at it.
  • The memory figure in predict and plan is an estimate. Measured peak memory on the GPU differs from it by about 30% in either direction (Qwen3 0.6B 811 MB measured against 1.07 GB estimated, Gemma 3 1B int4 1092 MB against 806 MB).
  • transformers 5.23 rebuilds SmolLM2's tokenizer without its digit split. Razor follows tokenizer.json, which is what the model was trained with, so its ids differ from that version of transformers on digits after whitespace.

0.3.0 (2026-11-15)

Faster and lighter on the GPU, faster int4 on the CPU, and fast decoding with identical output.

Added

  • Optional fast decoding on the GPU. Output is token for token the same as without it: 128 of 128 tokens in all 16 runs on SmolLM2, Qwen3 and Gemma 3 at int8 and int4.
  • A fast int4 path on the CPU.
  • Requirements and predicted speed from a model's config.json alone, before any download. The Android and iOS bindings expose the same estimate.
  • Calibration accepts several models. With three of different sizes, the planner predicted their decode steps to within -5% to +8%.
  • Checks that confirm optimized and plain decoding give the same tokens.

Changed

  • int8 accuracy on the CPU improved. Against the fp32 HuggingFace model, logits correlate better on all six model and prompt pairs measured: SmolLM2 135M 0.9953 to 0.9988 and 0.9913 to 0.9975, Qwen3 0.6B 0.9958 to 0.9991 and 0.9860 to 0.9979, Gemma 3 1B 0.9947 to 0.9993 and 0.9657 to 0.9969. SmolLM2's first token on the test prompt now matches the reference.
  • The GPU backend loads a model with less memory. The figures are under Performance.

Performance

  • Peak memory on the GPU backend: SmolLM2 135M 524 to 314 MB, Qwen3 0.6B 1274 to 811 MB, Gemma 3 1B int8 2065 to 1399 MB, SmolLM2 1.7B 4333 to 2295 MB.
  • GPU decode at a 64-token prompt: SmolLM2 135M 509 to 564 tok/s, Qwen3 0.6B 274 to 292, Gemma 3 1B int8 181 to 200, Gemma 3 1B int4 212 to 248. Prefill rises 11% on SmolLM2 and 18% on both Gemma builds.
  • A second GPU change makes decode 7 to 17% faster, with identical tokens.
  • int4 on the CPU, Gemma 3 1B: decode 8.9 to 63 tok/s, prefill 6 to 191 tok/s.
  • Fast decoding on a code prompt: SmolLM2 135M 534 to 586 tok/s (+10%), Qwen3 0.6B 295 to 348 (+18%). Gemma 3 1B is slower with it: int8 200 to 167, int4 255 to 200.
  • Against other engines, 2026-11-15 arena, rows where every engine ran at load under 6.6: SmolLM2 135M at a 64-token prompt decodes at 563 tok/s on Razor MLX, 509 on MLX-LM and 474 on llama.cpp, with first token in 4.3, 11.3 and 5.4 ms. SmolLM2 360M at a 512-token prompt decodes at 321 on Razor MLX, 362 on MLX-LM and 333 on llama.cpp. Razor loses that row.

Fixed

  • Builds without MLX (Android, Linux) compile again. A config that asks for the GPU there is refused at start.
  • A 2.1 GB weight file on a slow disk took over 15 minutes to load. The loader now reads large files ahead of use.

Known issues

  • Fast decoding slows Gemma 3 down. The Gemma profiles turn it off (0.4.0).
  • The CPU speed cost of the int8 accuracy change was measured only on a busy machine, where it was inside the noise. A quiet measurement is outstanding.

0.2.0 (2026-11-14)

Razor counts what a model needs, measures the device, and plans how to run it. Real models run on the GPU through MLX.

Added

  • Requirements counted from a model's shape, with no weights: memory, compute and cache growth for each phase. Counted weight bytes match all four weight files to within 0.01%.
  • A device profile measured in about a second and cached per chip.
  • Calibration, which runs a real model once so that predictions fit this machine.
  • A planner that predicts every way to run a model, picks the fastest the engine can run, and says why in numbers.
  • The MLX GPU backend: SmolLM2, Qwen3 and Gemma 3 on the GPU from the same weight files as the CPU.
  • A backend setting: cpu, mlx, or auto, which plans and runs the fastest.
  • razor_bench, which times prefill and decode the way llama-bench and mlx_lm.benchmark do, a parity tool that compares backends token by token, and the arena, which runs Razor, llama.cpp and MLX-LM on the same checkpoints, interleaved, waiting for a quiet machine before each run.

Fixed

  • The GPU backend now refuses a model it cannot run correctly. Before, it could have given wrong numbers without an error.

Verified

  • Correlation with the fp32 HuggingFace model at the first decode step, CPU then GPU: SmolLM2 135M int8 0.9958 and 0.9993, Qwen3 0.6B int8 0.9788 and 0.9973, Gemma 3 1B int8 0.9902 and 0.9994, Gemma 3 1B int4 0.9021 and 0.9022.

0.1.0 (2026-08-14)

The first tracked release: the CPU engine as it stood when the tree entered version control, and the work of the weeks after.

Added

  • A C++17 inference engine with int8 and fp32 paths, for Apple Silicon and Android arm64.
  • SmolLM2 at 135M, 360M and 1.7B, Qwen3 0.6B, and Gemma 3 1B, including Gemma's sliding-window attention.
  • Batched prefill, for a faster first token on long prompts.
  • Temperature and top-p sampling with a seed.
  • Gemma 3 at 4 bits: 696 MB against 1003 MB at int8.
  • The SDK: a macOS shared library, an Android arm64 library with JNI bindings, and a static XCFramework for iOS device and simulator. Offline licence keys, issued per platform.
  • ios/RazorBench, an app that loads a weight file and reports load time, prefill and decode.

Performance

  • Batched prefill of a 241-token prompt: 63 to 274 tok/s, with output byte-identical at every batch size.
  • Load time fell from 8.4 to 0.5 s, with about 430 MB less memory in use.
  • About 1 GB less memory on Gemma 3 1B once the model has loaded.

Fixed

  • Attention was slow on long contexts.
  • A memory error at the end of the context window.
  • The library printed to the host program's stdout.

Verified

  • Qwen3 0.6B and Gemma 3 1B match HuggingFace on top-1 across five prompts, correlating at 0.976 and 0.950 over the reference's top 50 tokens. Gemma's check includes a 607-token prompt that reaches the sliding window: 6 of 6 top-1.
  • The iOS simulator build gives the same tokens as macOS on a 31-token and a 1007-token prompt.

Known issues

  • Gemma 3 at int4 decodes at 9.5 tok/s against 43.3 at int8. Fixed in 0.3.0.
  • iOS is untested on hardware.