How these benchmarks are run

A benchmark you cannot reproduce is an opinion with decimal places. This page exists so every figure on this site can be checked, argued with, or repeated on your own hardware.

What gets recorded

Every run on this site stores the GPU, its power limit, the host CPU and RAM, the driver version, the CUDA version, the inference runtime and its version, the operating system, the model, the quantisation, the context length, the batch size, the input and output token counts, and how many runs were averaged.

None of those fields are optional. A run missing its driver or runtime fails the site build rather than publishing with a gap.

The workload

Unless a run says otherwise, the standard shape is:

• 1024 input tokens, 512 output tokens
• 4096-token context window
• batch size 1 — the single-user interactive case, not a throughput-maximising server load

Batch size 1 is a deliberate choice. Most people reading this want to know what the card feels like to use, not what it can do saturated.

How figures are sampled

Each published figure is the median of repeated runs, with warmup passes discarded so the first-run compile and cache-fill cost does not land in the result. The number of samples is recorded per run and shown in the table.

Medians rather than means: a single scheduling hiccup should not move a published number.

The derived columns

Two columns are computed rather than measured:

• tok/W — decode throughput divided by measured average board power. It is a crude efficiency figure and it is not the same as total system power.
• tok/$1k — decode throughput per $1,000 of launch MSRP. Launch MSRP is used because street prices move constantly and would make old runs unreadable. It is not what the card costs today.

Both are rounded once, at build time, so every table and chart on the site shows the same figure.

What these numbers do not tell you

• Nothing about output quality. Throughput says how fast tokens arrive, not whether they are worth reading. A more aggressive quantisation will often look better here and worse in use.
• Nothing about multi-user serving. Batch size 1 is the single-user case.
• Nothing about your machine. A different CPU, cooler, power limit, or driver will move these numbers.
• Nothing about long-context behaviour beyond the context length recorded.

Corrections

If a figure here looks wrong, it may be. Send the run you cannot reproduce through the contact page and the correction will be published rather than quietly edited.