Skip to content

Measurement

Speed claims for inference engines are easy to inflate by accident. Our lookup store once held the benchmark's own prompts and answers, and it added about 21 % to the benchmark row without the engine getting any faster. This page describes how TandemLLM measures itself, from the benchmark row and the teacher-forced bench to the release gate.

The row

The benchmark row is serve-single-i256-o256-v1: one request at a time, 256 prompt tokens, 256 generated tokens, thinking off, temperature 0, seed 42, 50 requests after 3 warm-ups. Every configuration runs with the same runner and the same prompt set.

Each request's speed is computed from its own record:

tok/s = (completion_tokens - 1) / (end-to-end seconds - time to first token)

The first token belongs to the prefill, so it is subtracted, and the decode time is the request's wall clock after its first token. A row reports the mean, p50, p90 and max over the 50 requests, with the median time to first token.

The client measures the row's speed and time to first token itself. The server's own numbers come with every response, in timings and metrics (server.md lists them), and its [req] log line holds the same record. The Setup Guide's card cells take two of them from that line: tok_round is tokens_per_block, the mean accepted length, and ms_round is the decode time over blocks (tools/exl3_fastrun.py). No row reports draft_acceptance_rate. It depends on how many tokens a round drafts, and StairCut sets that number again every round.

Two readings of a row are easy to get wrong.

  • The gaps between tokens. A verified round writes several tokens into the socket at once, so half the gaps are microseconds and the other half a whole round. The median inter-token latency therefore says nothing about round time.
  • Mean against median. Some prompts ask the model to repeat text that is already in the context, and the drafters commit 15 tokens a round on them. That lifts the mean. The median reflects fresh text.

Two views of every row

A row runs twice, in two views of the lookup store:

  • Store off. The engine's persistent suffix store is off, so nothing the engine has seen before can help it.
  • Clean store. The store is on but holds no text of the benchmark.

A row with the benchmark's own text in the store measures the store, not the engine. Benchmarks also run with --suffix-store-readonly when they read a store of real traffic, so they cannot write themselves into it.

Noise, and what a row can decide

Two server processes of the same configuration can differ by several per cent. StairCut learns its costs and copy rates online, and the earlier length router decided each request's block width from timing taken inside that request, so two runs can make different decisions on the same prompts. Under the length router, rows taken from three separate server processes spread 10.4 % on the mean and 6.7 % on the median, and three rows through one process spread 5.9 % and 3.1 %.

tools/row3.py runs N rows per configuration through one server, recomputes each from its own records, and reports the median with the spread of the runs. --compare calls a difference resolved only when it is larger than the noise and the run ranges of the two sides do not overlap. A change worth less than about 3 % of the median cannot be settled by the row at any number of repeats.

Speed is a quotient: tokens a round over time a round. A change can trade one factor for the other and leave the quotient flat. The served loop therefore counts its own rounds, and every row reports tokens per round and milliseconds per round next to tok/s. Both are gated. A kernel change must not move tokens per round, and a drafter change must move it up.

A server run can hold the costs still: --no-learn-cost (0.3.24-test) keeps the length router's served price table, so the same text gets the same choices whatever the blocks before it took. In 0.3.23's narrator A/B, with learning on, one prompt took 366, 468 and 504 blocks for the same 2,000 greedy tokens at 93.5 to 94.1 ms a block, and the off runs spread 37 %. With the flag the first two requests of a process still wrote other texts (398 and 505 blocks); the next six wrote one text in 514 blocks each, and the off runs spread 0.19 % (41.64 to 41.72 tok/s). So a server A/B gets the flag, two warm-ups an arm, and compares only runs that wrote the same text. A row pair with the flag on both sides (f0e4931 with only the flag added, and 0.3.24-test) read 50.21 and 49.37 tok/s, 2,288 and 2,297 blocks: the rows still differ a little between two processes.

Every report names the code it measured with code_sha256, a hash over every file of the four code folders. The same commit hashes the same everywhere, so a report can be matched to a commit.

Same hour, both sides

Nobody compares a candidate with a number from another day. It runs against the current release in the same session on the same box, alternated: base, candidate, base, candidate. A board drifts over hours (its temperature, its memory, whatever else runs), and alternation spreads that drift over both sides.

The adoption rule is "never worse". A candidate ships only if nothing resolves worse against the same-hour base, in both store views: not the mean, p50, p90 or max, not time to first token, and not tokens or milliseconds per round.

The block A/B, for changes that keep the bits

Most kernel changes are worth 0.5 to 3 ms of a round of about 93 ms, below what the row can resolve. For a change whose output is bit-identical, only time can differ. tools/block_ab.py loads the served engine once, switches each state's flags inside the process, warms each state, freezes the router's learned costs, and alternates base and candidate over three workloads for N pairs. Every run's tokens must equal the base's first run, and the rule is applied to milliseconds per round. Two states of the same code read "not resolved" with a spread of 0.35 to 0.87 %. Turning the verify graphs off resolved 3.15 ms worse on every workload.

The release gate

ops/gate.sh is the whole protocol in one command, run on the box inside a hold (operations.md). It stops at the first failing step:

  1. The engine's CPU tests (tools/engine_tests.py), in a clean environment; TandemApp's run in its own repository.
  2. Every GPU test, tests/gpu/test_*.py.
  3. Identity: the served configuration with the new flags off must be bit-identical to the build it replaces (tools/flagoff_identity.py).
  4. Agreement: tools/verify_spec.py with the served flags and the candidate's, under the near-tie rule (exactness.md).
  5. Optionally, the block A/B above.
  6. Rows: two store-off rows and one clean-store row with the candidate's flags.
  7. Compare: tools/gatecheck.py against the base reports, with the never-worse rule.

It writes a dated report with every command and its output to results/gate/<label>/report.md. results/ is not under version control.

Rows are not enough on their own. A release candidate once passed every row and broke a real client's request shape. Before a deploy the gate is followed by:

  • tools/client_smoke.py: the request shapes real clients send, through the official openai client. It covers an agent client's tools with arguments validated against their JSON schemas, a 12k-token prompt with penalties and tools, every tool_choice mode, streamed tool-call deltas, n=2, and prompts from 8k to over 100k tokens. It passes with zero failures.
  • tools/api_text_check.py: 13 plain requests byte-identical to the current release.
  • A real agent session with its own tool calls, all of which must go through.

The teacher-forced bench

A change of the verified tree's shape moves near ties (exactness.md), so two configurations rarely write the same text on the same prompt, and their rows measure different texts. tools/forced_bench.py removes that. Each workload has one fixed reference text (the greedy output of the earlier length-router configuration; any fixed text would do), and every configuration drafts and verifies with the real kernels at the real context but accepts along the reference, so all of them commit the same tokens and differ only in rounds and time per round.

The paper's run covers 25 workloads: prose, chat, code, a 48-token answer and a prose-then-code prompt; twelve prompts of the benchmark row; three edit tasks; a 256-token quotation and a 1,024-token copy; and, at 32k tokens of context, fresh text, a copy and a code edit. A workload's rate is the mean of three measured repeats after a warm-up. StairCut averages 90.45 tok/s against 81.95 for the fixed block 16 and 59.31 for the fixed block 8. It is above block 16 on 23 of the 25 workloads and above block 8 on 22. One router served every workload and repeat in that run; in a second run with a new router before every request, StairCut averages 89.88 tok/s against 91.16 for the kept router in that run.

The weight cards' speed rows

Each weight card in the setup guide carries one speed row of the current build (SPD-245): the parallel rows of models/<id>/model.yaml. tools/parallel_card.py measures a row against the server running the profile and writes its raw file to models/<id>/measurements/. The file holds every request's record, the /health samples of every round, the server's limit knobs and the row itself. The registry refuses a row that differs from its raw file's row.

A row measures the engine (SPD-249). Every request goes to /v1/completions with its prompt as text and X-Tandem: raw, so no conductor or chat-layer step runs. The prompt is the model's chat template over the card's messages with thinking off: the tool asks /v1/tokenize for the template's ids ("ids": true), turns them into text with the model's tokenizer (--tokenizer, the weights' directory), and checks that /v1/tokenize reads that text as the same ids, which are the ids /v1/completions reads. A prompt whose ids differ stops the run, and the raw file counts the prompts checked (prompts_checked). /v1/chat/completions works and is in beta; no published speed number comes from it.

  • Prompts: bench/card-prompts-v1.json, a code and a prose workload of about 190 to 250 tokens. The prefill filler is bench/heldout_prose.txt, repeated to the length. Every request starts with a nonce line and sends cache_prompt: false, with thinking off and a seed of its own. The row records the prompt set's version and hash, and the prompt tokens of each workload.
  • Decode: each workload alone, three runs after one warm-up each. A run's rate is (completion tokens - 1) / (last token - first token), and the row gives the median per workload.
  • Prefill: cold prompts of 2,048, 8,192 and 32,768 tokens, counted with /v1/tokenize and checked against the request's own prompt_tokens, three runs each, one token out. The row gives the median time to first token and prompt tokens over that time.
  • At once: 1, 2, 4 and 8 prose requests released together, one discarded warm-up round and three measured rounds a level, 512 tokens a request. The wall rate is all completion tokens over the time from the send to the last end, so it includes the first tokens. The decode rate counts the tokens after each request's first one, over the time from the first first token to the last token. The row gives the median of each over the rounds, and the median time to first token.
  • Proof: /health every 0.2 s during every round. A round counts only when the most slots decoding at once equal N and no more than N requests ran (none came from elsewhere). An invalid round is run again. A level's at_once is the smallest maximum over its rounds, and the registry refuses a level whose at_once is not N or whose valid rounds are fewer than the row's. Before the levels the tool reads the server's live slot limit (parallel.max_running) and refuses to start when it is below the largest N.
  • Setup: the row names the setup file, the lanes, the window, the tokens a request, the sampling, the suffix store and the response cache (both off for a card row), the build and the commit.

Baselines

The paper compares StairCut with the same engine in two other configurations, on the same weights and the same box: plain greedy decoding (speculation off), and one drafter at a fixed block width (8 or 16) with its released tree budgets (speculative-decoding.md). On the benchmark row with the store off that is 49.89 tok/s for StairCut, 47.13 for block 8, 44.80 for block 16 and 13.69 for plain greedy decoding.

One measuring method

Release pages, weight cards and the quality gate measure the same things: decode at one request, prefill, requests at once, long prompts. tools/measure measures each of them once and keeps the result, so a number measured for one of them is never measured again for another.

  • One row list, tools/measure/rows.yaml. A row has a stable id (length.32k, conc.card.c8, prefill.64k, ...), what it measures, its request shape (prompt and its length, tokens out, requests at once, runs or rounds), the views that use it, and the server it runs on. Two rows of a different shape are two rows: the card's 1/2/4/8 at once use 169-246-token prompts, the report's requests at once use ~2k-token prompts with their prefill in the wall time; prefill.32k is one token out, length.32k is 512.
  • One runner. python -m tools.measure run --url URL --rows ... | --view report --out DIR measures only the rows the record store at DIR does not hold for this server's key: the row, the build, the pack, the box and a hash of the server's limit knobs. --force measures a row again. Every request goes to /v1/completions with X-Tandem: raw and every answer's mode is checked; a row of N requests at once is invalid without the server's own count of N decoding together.
  • A record store. One JSON per run under DIR/runs/, and DIR/index.json from the key to the run file. python -m tools.measure merge imports records that exist already (a gate report and its raw results, a report's data.json and the card tool's raw files). An imported record keeps its source, the tool, the date and the file, and the page shows it beside its rows.
  • Three views, none of which measures anything. render report writes the HTML page (benchmark-report.md), for one pack or several packs of one model on one page. render card writes a weight card's numbers in the shape tools/parallel_card.py writes them. render gate --base DIR --head DIR gives the gate's verdict table from two stores, with the gate's own rules; every gate run writes its arms' records to OUT/records/base and OUT/records/head (quality-gate.md). compare --old A --new B sets two builds' rows side by side with a noise band.
  • A plan before the GPU. python -m tools.measure plan --view report --pack P --build B --out DIR lists every row with its record and source, or the hold that will measure it. Rows that need their own fresh server are grouped by server, and the groups are packed into holds of at most 62 minutes. It refuses when a row would be measured twice: a row asked by id that has a record, a warm row whose earlier rows already have records, or code that measures a recorded row beside the one asked.

tools/parallel_card.py, python -m tools.release_report and python -m tools.gate run keep working; their help says where the one method is.

Traps

Each of these cost a wrong number once.

  • A standalone kernel timing is a ranking, never a budget. The same configuration read 0.254 ms in one process and 0.627 ms in another.
  • The first configuration in a fresh process pays for autotuning. It once read a verify of 260.2 ms against 115.5 ms for every later one, and the policy under test "won" by 35 %.
  • A bandwidth-bound engine measured beside another engine measures the other engine. Two engines halve the bandwidth each one gets. Take the board lock.
  • A speculative decoder judged on a short generation hides bugs. One drafter bug read 81 % acceptance at 48 tokens and 24 % at 128.
  • Acceptance with the reasoning block open differs from acceptance with it closed, by up to 72 % on one prompt. Measure in the serving regime.
  • A kernel probe whose working set fits in cache measures the cache. 266 GB/s on a 273 GB/s board is not a real reading.
  • Two runs that agree to six decimals have not agreed. They are the same run: once, a test server had bound its port beside another.
  • A quiet log is not a hung process. Python block-buffers into a pipe, so use python -u.
  • A lane count past the pool's guarantee reads as a slow model. Requests that wait for a lane and for the turn look the same as slow decoding, and a Prometheus row of the decode rate alone cannot tell the two apart. Compare the /health parallel.turn waits with the lane count.
  • A fresh engine can measure better than a used one. The resident conversation and the side slots keep their KV, and a long-lived session has more of it. Restart before comparing configurations.
  • The engine's own turns look like a client's. The preparer and the memory writer run generations over the client's context; count them with qse_prepare_runs_total and qse_memory_writer_runs_total (server/METRICS.md), or measure raw with the conductor off.
  • A first request after a restart is not a baseline. The pack's page cache is dropped at the start, so the first measurement runs on a busy disk.

Measuring Flash-Next

Flash-Next's numbers in these docs come from tools of their own (tools/fn_*.py), each answering one question, and every one ran under the board's lock with one model loaded. Most are single runs of one request, labelled so where they appear; none is a benchmark row yet.

tool the question it answers
fn_server_check.py --compare do 20 request shapes give the same text, tool calls and logprobs with and without speculation; --sampled: do seeded sampled answers
fn_spec_check.py does speculative greedy equal plain greedy in the engine, with the state bit-equal after every run
fn_fit_run.py does a pack hold its largest window: one long request, MemAvailable at 10 Hz, every limit knob the server reports
fn_cache_check.py does a request restored from each cache tier answer as on a cold server
fn_resume_check.py, fn_guest_check.py does a resumed prefill equal a cold one; does a short guest request leave the resident conversation's rows intact
fn_lookup_ab.py one greedy request a class through a server, for an A/B of the drafters
fn_memory_check.py memory after the load, the prefill workspace, the table's resident bytes under each policy
fn_exl3_xcheck.py, fn_refcheck.py our reading of a pack against the converter's; our forward against transformers'

The Flash-Next tools behind the numbers in Qwen3.8-Flash-Next.

Every speculative rate in Qwen3.8-Flash-Next comes with the statement that its texts equalled plain decoding, because a rate whose text changed measures another answer. The runs by context length started a fresh server and report its second round, so the first round's one-time graph captures do not count in them. What Flash-Next has not had yet is the release gate: no row of serve-single-i256-o256-v1, no block A/B and no task suite.

Limits

The published numbers come from one board, one model and one request at a time. Long prompts are measured separately (caches.md). Parallel requests are measured on the weight cards only (above). The quality gates are held-out loss, confident argmax agreement and free generation. No task benchmark suite has been run, so this repository makes no claim about task accuracy.