Lab

This page is the provenance for everything published here. When a post cites a number, it was measured on this machine, with this configuration.

How I measure

A throughput number only means something if it comes with the procedure. Mine is this, and it’s the same across every post:

Model loaded and warmed up before measuring. The first run after loading includes reading the model from disk and compiling kernels. It gets discarded.

Context declared explicitly. The runtime’s default window is not the window I use, and the KV cache changes both memory and performance. Context is stated in every measurement.

Three runs, median reported. An average hides outliers; a single run is worthless. When spread across runs exceeds 5%, that becomes a note in the post — because high variance usually means thermals or GPU contention, and the reader needs to know.

Fixed prompt and output length. Throughput measured with different prompts isn’t comparable. I use the same prompt and cap output at the same token count.

Dedicated GPU during measurement. No video, no second model loaded, no heavy compositor competing for memory.

Prefill and decode are not the same metric

Two different numbers that a lot of people report as one:

Prefill (or prompt eval) is processing the input prompt — parallel, bound by compute. Decode (eval) is token-by-token generation — sequential, bound by memory bandwidth.

A model can have excellent prefill and mediocre decode, or the reverse. Reporting a bare “tokens/s” without saying which one is the most common source of useless model comparisons. Here both appear separately.

Limits of this bench

Worth stating what these numbers don’t cover: this is a single-user machine with one GPU. I don’t measure concurrency, I don’t measure server-side batching, I don’t measure horizontal scale. Anyone putting this into production with many simultaneous users needs vLLM or TensorRT-LLM and their own test — the numbers here are for deciding what to run on your desk, not for sizing a cluster.