Architecture¶
A decode step of Qwen3.8-27B reads about 27 GB of weights in the vendor's FP8 format. The DGX Spark reads memory at 273 GB/s on paper and about 240 GB/s in practice. Those two numbers put a floor of about 100 ms under every step, or 10 tok/s, before any code runs. TandemLLM exists to get under that floor while writing the model's own greedy text, with one documented exception at near ties (exactness.md).
It does two things. It cuts the bytes a step reads, from 27 GB to 15 GB. And it makes one step produce about 4.6 tokens: cheap drafters guess, and one pass of the big model checks all the guesses.
The engine serves a second model the same way, Qwen3.8-Flash-Next, a mixture of experts of the same family that reads about 4.23 GB a plain token from turboderp's 3.05 EXL3 pack. The box runs one of the two at a time. Most of this page describes the 27B. The second model says what differs, and Methods written once shows how one server, one set of caches and one speculation method serve both.
The model¶
engine/config.py reads config.json and nothing else. Its text model has 64 decoder layers, hidden size 5120, a dense SiLU MLP of width 17,408 in every layer, and a vocabulary of 248,320 tokens. The model has no mixture of experts. A 0.92 GB vision tower sits in the same checkpoint. The language-model loader skips it by name and engine/vision.py loads it on its own for image input (server.md).
Every fourth layer (3, 7, 11 and so on up to 63) is full attention, 16 layers in all. Gated DeltaNet fills the other 48. It is a recurrent layer that keeps a fixed-size state instead of a growing KV cache.
| layer kind | count | state per sequence |
|---|---|---|
| Gated DeltaNet | 48 | an fp32 state of 48 heads, each 128 by 128 (3.15 MB) and a 4-wide convolution tail (80 KB), per layer |
| full attention | 16 | BF16 keys and values, 4 KV heads of 256, so 4 KB a token per layer |
That split shapes most of the code. Published work on speculation and caching assumes attention, where undoing a token means moving a pointer back by one row. A recurrent state has absorbed every token and has no inverse, and the engine needs exact ways to undo, branch and restore it. speculative-decoding.md shows the undo and the branch, and caches.md shows the restore.
[GDN] [GDN] [GDN] [attention] x 16 = 64 layers: 48 Gated DeltaNet, 16 full attention
The layer pattern of Qwen3.8-27B: three Gated DeltaNet layers, then one full-attention layer, sixteen times.
The byte budget¶
A decode step reads every weight once. This table sums the bytes from the checkpoint's safetensors headers.
| group | bytes a step |
|---|---|
| 48 Gated DeltaNet layers | 18.43 GB |
| 16 attention layers | 5.96 GB |
| the output head, BF16, 248,320 rows by 5,120 | 2.54 GB |
| total, FP8 as shipped | 26.93 GB |
The three MLP projections are 63.6 % of that. Quantising them first, then the rest, moved the budget down in steps, and each step had to pass the quality gate (quantisation.md):
| weight set | bytes a step | floor at 273 GB/s |
|---|---|---|
| FP8 as shipped | 26.93 GB | 98.6 ms |
| NVFP4 MLPs | 19.44 GB | 71.2 ms |
| plus the FP8 (e4m3) head | 18.17 GB | 66.5 ms |
| plus NVFP4 GDN and attention projections | 15.01 GB | 55.0 ms |
State is small beside the weights. The 48 recurrent states are 151 MB together. KV costs 64 KB a token across the 16 attention layers, so 2.1 GB at 32k tokens and about 17 GB at the full 262,144-token context.
| weight set | GB read per decode step | floor at 273 GB/s |
|---|---|---|
| FP8 as shipped | 26.93 | 98.6 ms |
| NVFP4 MLPs | 19.44 | 71.2 ms |
| + FP8 head | 18.17 | 66.5 ms |
| + NVFP4 GDN, attention (served) | 15.01 | 55.0 ms |
Bytes a step for each weight set, and the floor each sets under one step at 273 GB/s (arithmetic). The served set is the last bar. The board reads about 240 GB/s in practice, so the real floors are about 14 % higher.
The decode loop¶
Speed is one fraction:
tok/s = tokens committed a round / (draft ms + verify ms + commit ms)
A round starts from the last committed token. Next, the drafters propose a tree of guesses after it. One forward pass then runs over the whole tree through all 64 layers. The engine reads the model's own choice at every node and keeps the longest path where the model agrees with the guesses. It always keeps one more token, the model's own choice after the last accepted guess, so a round yields at least one token. Then it commits the state of that path and starts again.
On the served profile (the published NVFP4 set, StairCut on) a round on the benchmark row takes 93.4 ms and commits 4.58 tokens, and the row's rate is 49.89 tok/s. The rate is the mean of the 50 requests' own rates (measurement.md), not the quotient of the two means (4.58 tokens in 93.4 ms is 49.0 tok/s). Without speculation, the same box writes 13.69 tok/s at 72.8 ms a token (measured on 28 September on an earlier build, whose path with speculation off is unchanged).
sequenceDiagram
participant C as Client
participant S as Server
participant D as Drafters and router
participant M as Target model
C->>S: POST /v1/chat/completions
S->>M: prefill the prompt (or restore it from a cache)
loop every round, about 93 ms
S->>D: last committed token and context
D->>S: a draft tree (up to 32 nodes)
S->>M: verify the whole tree in one pass
M->>S: the model's own token at every node
S->>M: commit the accepted path (KV rows and recurrent state)
S->>C: stream the committed tokens
end
The second model, Qwen3.8-Flash-Next¶
Flash-Next keeps the 27B's hybrid, Gated DeltaNet layers with an attention layer every fourth, and changes most of what sits around it. engine/models/flashnext/config.py reads its config and refuses any field it does not know by name.
| part | Flash-Next | the 27B |
|---|---|---|
| layers | 48: 36 Gated DeltaNet, 12 attention | 64: 48 Gated DeltaNet, 16 attention |
| MLP | 512 experts a layer, 10 routed and 1 shared run a token | dense, width 17,408 |
| residual stream | 4 streams of 2,560 joined by hyper-connection mixers | one stream of 5,120 |
| attention | sparse past 2,051 tokens: an indexer scores blocks and each row reads the 512 best | full |
| extra input | a hashed n-gram embedding (16 table rows a token) in front of layer 1 | none |
| draft layer | one multi-token-prediction (MTP) layer in the checkpoint | none; DFlash2 drafters beside it |
| state a sequence | 110.3 MiB of sections plus 29,952 bytes a token | about 147 MiB plus 64 KB a token |
| weights served | turboderp's EXL3 packs, 2.05 to 6.05 bits a weight | NVFP4 with an e4m3 head, or EXL3 |
A plain decode token on the 3.05 pack reads about 4.23 GB (arithmetic from the pack's headers, with the mixers as int8). The engine decodes 45.8 tok/s at 1k of context without speculation, about 21.8 ms a token (measured 2026-10-11, one run a class). Its verify rows equal its decode rows bit for bit, so greedy speculation on Flash-Next writes the plain greedy text token for token, with no near-tie exception (exactness.md). The one known difference is where a think budget closes: at the first block boundary past the cap (ENG-235). Through the server on 2026-10-12, one greedy request of up to 512 tokens a class with the MTP drafter and the lookup: prose 51, chat 64, code 83, an edit 132 and a copy 133 tok/s, against 41.5 to 45.7 plain. quantisation.md covers the packs and speculative-decoding.md the drafter.
Methods written once, models plugged in¶
A second model could have meant a second server, a second set of caches and a second StairCut. The engine avoids that with five layers that depend only downward. A method (speculation, a cache tier, the memory governor, the server) reads a model through its adapter and through data files. It never branches on a model's name, never asks hasattr of an engine and never touches a private field. tests/test_load_shared.py registers a third, made-up architecture on the CPU and gets an answer on /v1/chat/completions with no edit under server/.
flowchart TB
subgraph M[methods, written once]
SPEC[StairCut, PriceModel, the MTP drafter, the walk]
CACHE[every cache tier, the KV pool]
SRV[server, memory governor, extension host, setup guide]
end
subgraph A[model adapters, one per architecture]
Q35[qwen3_5: the 27B]
FN[qwen4_exp: Flash-Next]
end
subgraph F[weight formats]
FMT[EXL3, NVFP4, FP8, int8 rows, BF16]
end
subgraph H[hardware]
KREG[kernel registry by op, format and device]
DEV[device profile]
end
subgraph T[topology]
MESH[Mesh, world 1]
end
M --> A --> F --> H --> T
A --> H
Half of it is built. Since 2026-10-12 the adapters, the state, the shared methods and the server's one loader serve every model. Four parts are still design, each with its ticket: format plugins, a kernel registry, device profiles and the mesh.
| layer | what | where | today |
|---|---|---|---|
| methods | StairCut with its price model and calibration, the MTP tree drafter, the walk of a verified block, fingerprint v3 | engine/methods/ |
built (ENG-245, ENG-243); the 27B still cuts with its own StairCut in engine/lenrouter.py until ENG-257 |
| methods | every cache tier over the model's state | engine/cache.py, disk_cache.py, spill.py, kvpool.py |
built (ENG-157, ENG-241) |
| methods | one loader driven by the registry | server/app.py _load |
built (SRV-341) |
| model adapters | the registry, Capabilities, CostModel, ModelState |
engine/models/ |
built for qwen3_5 and qwen4_exp |
| weight formats | the Linear protocol |
engine/linear.py, tools/*_linear.py, engine/models/flashnext/pack.py |
built; a format plugin with read and bind and one checkpoint reader are planned (ENG-253) |
| hardware | KernelImpl keyed by (op, fmt, device), resolved once at load |
none yet: each weight object holds its kernel, and tile tables sit under ops/ per model |
planned (ENG-251) |
| hardware | device profiles | engine/methods/device.py names the device for price files; the Setup Guide's device preset sits in each model.yaml |
devices/<id>.yaml and a probe at start planned (ENG-255) |
| topology | Mesh, Collectives, ShardSpec at world 1 |
none: one process, one GPU | planned (ENG-259), then two boxes (ENG-233) |
Model adapters¶
engine/models/__init__.py keeps a registry of architectures, each registered by its own package's arch.py. An entry names the architectures strings and model_types that select it, its config reader, its engine class, its server builder and its manifest, all by import path, and what the shared code needs to know about the model as data (its tool-call form and reasoning, the files it shares with other families, the kernel options it turns on; adding-a-model.md). architecture(config) returns the entry, or fails naming every registered one.
| name | model | engine | server builder |
|---|---|---|---|
qwen3_5 |
Qwen3.8-27B | engine.models.qwen3_5.adapter.Qwen38Engine |
the server's own, _build_qwen3_5 |
qwen4_exp |
Qwen3.8-Flash-Next | engine.models.flashnext.engine.FlashNextEngine |
engine.models.flashnext.serving.serve |
An adapter is the engine object with a fixed set of verbs. The hot verbs run every round, so they are the engine's own bound methods and never a wrapper: a decode and a verify call the same kernels through the same Python frames as before the layering, and CUDA graphs capture the same launches.
| hot verb | what it does |
|---|---|
forward(ids, start) |
a prompt chunk, a forced phrase or one decode token at the state's length |
forward_tree(block, parents, start) |
a draft block in depth-first order; logits of every row |
forward_block(block, start) |
a chain whose commit waits for rollback_to or the next call |
commit_tree(path), rollback_to(keep) |
keep a verified path; cut a chain back |
picks, hidden_post_norm |
the verify's own greedy tokens; the rows a drafter syncs from |
kv.length, reset() |
where the sequence stands; start a new one in place (the KV rows stay) |
The cold verbs run between requests or around a cache copy. The 27B's are installed on its class by engine/models/qwen3_5/adapter.py, so engine/model.py keeps its bytes and its disk-tier fingerprint.
| cold verb | what it gives |
|---|---|
caps |
Capabilities: caches on or off, side slots, the cache grid (anchor_grid: 1,024 for the 27B, 4,096 for Flash-Next), the shadow reader, joint verify |
cost() |
CostModel: section bytes, view bytes a token, and the prefill workspace (floor, bytes a token, cap) |
cache_state() |
the sequence's state as ModelState (below) |
resume_at(n), settle(), discard_pending(), trim_pending(n) |
continue at n after a restore; land a pending commit; forget an uncommitted verify; cut a pending chain |
manifest() |
the files and tables its stored rows depend on (fingerprint v3, caches.md) |
memory_parts(), resident_files() |
the dashboard's memory split; files it keeps in the page cache on purpose |
precapture(drafter) |
the verify graphs it captures at load |
A model may offer more, and the server checks for each before it uses it: draft_heads() (Flash-Next's MTP layer), side_slot() (a slot over the same weights), forward_tree_multi() (several slots' blocks in one pass), idle_captures() (graphs to capture while the server is idle). Without them the server takes the 27B's path or leaves the feature off.
ModelState: sections and views¶
The caches copy a sequence through eng.cache_state() (engine/models/state.py), and they know no model. A state has two parts:
| part | what | the 27B | Flash-Next |
|---|---|---|---|
| sections | fixed-size tensors by name, copied whole | gdn.S, gdn.conv |
36 GDN states and conv tails, the n-gram convolution and its token context, mtp.h_last: 115,662,864 bytes |
| views | position-indexed buffers with their token axis, sliced by length | K and V (and their e4m3 scales when on) | K, V and the indexer keys of 12 layers, the MTP layer's own: 29,952 bytes a token |
Every tier keeps sections in the state's order and views by length. A signature over names, shapes and dtypes refuses a restore across models. Derived buffers are not stored: Flash-Next's pooled indexer keys are rebuilt from the keys in resume_at. tests/test_cache_any_model.py runs a made-up state, whose sections and views have dtypes and shapes neither model uses, through four tiers with no edit in the cache code.
Weight formats¶
Every weight the engine multiplies by implements Linear (engine/linear.py): a shape, the bytes a use reads, and matmul. The model calls w.matmul(x) and never asks the format. Today's classes: FP8Block, NVFP4Block, FP8Head, BF16Block, EXL3Block (tools/exl3_linear.py), and Flash-Next's Int8Rows for its mixers. Flash-Next's routed experts are not Linear objects: a layer's experts and its shared expert sit in one MoEBank (tools/fn_kernels.py), which the MoE kernels read directly. Each object holds its kernel, so a call costs nothing extra.
The planned format plugin adds two verbs to that: read (one reader for every checkpoint: headers, byte ranges, row and column slices, expert ranges) and bind (set the object's kernel once from the registry below). Until then the 27B reads through engine/loader.py and Flash-Next through engine/models/flashnext/pack.py.
Kernels and devices, then the mesh¶
The planned registry will describe each kernel as a KernelImpl: the op (gemv, moe_experts, attention_rows and so on), the weight format, the row regime, the device features it needs, whether it is row-invariant, and its entry point with its tile table and torch reference. A format will resolve its op once at load, keyed by (op, fmt, device). A device without a kernel will get the reference, and an exact model there will serve plain decode until a row-invariant twin exists, because the load-time self-check gates speculation (kernels.md).
A device profile will hold what the memory governor and the Setup Guide read about a board (memory, reserve, floor, whether page cache counts as available) and what the kernel build reads (bandwidth, compute capability, features). Today the DGX Spark's numbers sit in each model.yaml and in the profiles, and engine/methods/device.py turns the GPU's name into the id price files are keyed by (dgx-spark for the GB10).
The mesh marks where two boxes will split a model. At world 1, the only size built, it is absent from the code path: no collective runs and the graphs hold the same kernels. Its seams are ENG-259's and come before any work across boxes: weights read by slice, a shard tag on each state view, and the mesh in cache keys and price file names.
Stages: one model in two processes¶
A model can be cut between two layers and run as two processes (ENG-307). The front stage holds the embedding and layers [0, k). The tail stage holds layers [k, layers), the head and the sampler, and it is the engine a client talks to. Each round the tail sends a step header: the block's tokens and parents, the length it starts at, and the commit path of the block before it. The front answers with the carry, the block's rows as they leave layer k - 1. Logits never cross. Each side keeps the KV rows and the recurrent sections of its own layers, under the names they have in one process.
| part | where |
|---|---|
| the link: typed frames over one TCP connection, a cap on a frame's size, a deadline on every receive, a 1 s heartbeat, an HMAC challenge handshake that compares both engines' identity blocks | engine/methods/link.py |
the protocol: step header, carry, commit path, epochs, the desync answer, and the StageAdapter interface a family implements |
engine/methods/stages.py |
| a family's stage code, and its tail stage in the server's engine interface | engine/models/<family>/stages.py, tail.py |
| what a stage keeps of its own half of a state under a key: sections, row segments, a folder | engine/methods/stage_store.py |
| the server's part: the flags, the key file, the front's process, the tail's link and its refusals | server/stage.py |
Both families run their torch reference as stages on the CPU, and tests/test_stages.py holds two real processes to the single process at every cut of both tiny test models: logits, sections and KV rows are compared with torch.equal.
lfm2_moe also runs its engine as stages (--stage, server.md). A stage's engine is the engine itself, built over a config whose layer lists hold the stage's own layers, so it loads, keeps and commits only those. Its two passes are the engine's row pass and prompt-chunk pass with the layer loop over [lo, hi); tests/test_lfm2_moe_stage.py compares the loop bodies with the engine's as syntax trees and the rows with the engine's at every cut. The carry is what the engine holds at a layer boundary, the stream and the next layer's norm of it: [rows, 2, hidden] in bf16. Each block runs on the kernels the one process runs it on (a prompt chunk, a decode step, a verify block), eager. The tail checks the cut at load: decode steps against a verify block of the same tokens, bit for bit. A replay after a desync would run decode steps as prompt chunks, on other kernels, so the tail ends that request and drops the sequence instead.
flashnext runs its engine as stages too. The stage's engine is a subclass of the engine built over the same kind of config view. It cuts three passes to [lo, hi): the trunk's layer loop, the row pass and the prompt-chunk pass. tests/test_fn_stage.py compares their statements with the engine's as syntax trees. The staging, verify, commit, the folded commit and the graph classes are the engine's own code, so a stage captures its decode, verify and commit graphs as the one process does. A graph's body ends at the cut: the carry enters it through static buffers and leaves it in the graph's own tensors, and the copy to and from the host runs outside the graph. The carry is the 4 streams and the next site's norm, [rows, 2, 4 x hidden] in bf16, and the streams alone in front of the PLE layer. The tail keeps the draft layer, which reads the streams of the last layer.
Flash-Next's kernels are CUDA, so no kernel of it runs on the CPU. The CPU tests run the real loader, engine, graph classes and serving adapter on a small EXL3 pack in the served layout, with host stand-ins for the launches. There every cut gives the one engine's logits, streams and state bit for bit, eager and with graphs.
On the CPU LFM2 MoE's kernels run under Triton's interpreter.
Both families have run split on one GPU box, the two stages as two processes joined through tools/linkrelay.py. LFM2.5 2.5 bpw cut at layer 20 and Flash-Next 2.05 bpw cut at layers 12 and 24 wrote the single process's tokens on 20 greedy prompts; Flash-Next also on a prompt of 31,991 tokens. A stalled link ended the request with an error after 5 s, and a new link served the same tokens. No two-box speed is measured: on one box both stages share the GPU.
A missed deadline (5 s for a block, 60 s for a prompt chunk) takes the link down, and the request ends with stage_down. A new connection gets a new epoch, and both sides start it from an empty state. A front whose length is not the tail's answers desync; the tail then replays the sequence's tokens. A front whose own pass fails answers stage with the reason and ends the link, so the request ends with that reason and not with a timeout. The link refuses a peer with another key, code, weights, family or layer range, and it has no second transport to fall back to. The handshake proves both ends hold the key; it does not encrypt frames, so the link belongs on a private network segment.
State beyond the running sequence (ENG-423, Flash-Next with --stage-caches on). Each stage names a state by a key: a digest of the blocks that wrote it, in order, with each block's kind, shape and kept path. Both stages compute it from what they ran, and the tail sends it with every header, so a front that holds other rows is found by content. The tail's caches keep the tail's sections and rows with the key as one more section. Reading the sections asks the front to keep its half under that key; putting a state back sends the key to the front. The front keeps a state's sections as a copy and its rows as segments that stay in place until a pass would write over them. A front that does not hold a key is rebuilt from the tail's log of blocks, each sent again as the kind it was, with no carry coming back. A state that neither stage has a log of is refused before any length moves, and the cache that offered it falls back. Slots have a sequence each on both stages, each on a stage engine of its own over the one copy of the weights; a joint round is one request and one carry. tests/test_stage_state.py holds the protocol on a toy family and tests/test_fn_split_state.py holds Flash-Next with the caches as they are: the resident prefix, the end of a generation, a guest, the state store, the disk tier on both sides, a front that starts again, and joint rounds against the one engine's.
The parts¶
flowchart TB
subgraph server[server/]
API[OpenAI API, streaming, tool calls, structured outputs]
LIVE[Live activity, metrics, usage ledger]
DASH[Dashboard reads and the extension host]
end
subgraph engine[engine/]
LOOP[Decode loop and sampling]
METHODS[engine/methods: StairCut, MTP drafter, walk]
ROUTER[the 27B's StairCut, length router, tree router]
DRAFT[DFlash2 drafters and the lookup drafter]
MODELS[engine/models: registry, adapters, state]
MODEL[the 27B: forward, tree verify, commit]
FNX[Flash-Next: forward, verify, commit]
CACHE[Resident prefix, state store, disk tier, spill, suffix store]
end
subgraph kernels[tools/, CUDA and Triton kernels]
W4[NVFP4, FP8 and EXL3 matrix products]
GDN[Gated delta rule for one token, a block, a tree]
FNK[Flash-Next: MoE experts, row steps, prefill]
end
API --> LOOP
LOOP --> ROUTER --> DRAFT
LOOP --> METHODS
LOOP --> MODELS
MODELS --> MODEL
MODELS --> FNX
LOOP --> CACHE
MODEL --> W4
MODEL --> GDN
FNX --> FNK
LOOP --> LIVE
| part | where | page |
|---|---|---|
| checkpoint layout and loading | engine/loader.py, engine/config.py; Flash-Next engine/models/flashnext/pack.py, load.py |
adding-a-model.md |
| the model registry, adapters and state | engine/models/ |
adding-a-model.md |
| the quantiser and its gate | tools/quant_nvfp4.py, tools/quant_head.py, tools/quality_gate.py |
quantisation.md |
| matrix-product kernels | tools/nvfp4_*.py, tools/fp8_linear.py, tools/exl3_linear.py, tools/head_gemv.py; Flash-Next tools/fn_kernels.py |
kernels.md |
| C++ and CUDA sources | tools/csrc/, engine/csrc/, a family's engine/models/<family>/csrc/, read and built by engine/native_build.py |
kernels.md |
| recurrent-layer kernels | tools/gdn_*_kernels.py, engine/gdn.py; Flash-Next tools/fn_seq_kernels.py, tools/fn_prefill_kernels.py |
kernels.md |
| the 27B, verify and commit | engine/model.py, engine/tree.py |
speculative-decoding.md |
| Flash-Next, verify and commit | engine/models/flashnext/engine.py, serving.py, mtp.py |
speculative-decoding.md |
| drafters | engine/drafters/, engine/methods/mtp_drafter.py |
speculative-decoding.md |
| StairCut and the routers | engine/methods/staircut.py; the 27B's engine/lenrouter.py, engine/router.py, engine/prices.py |
speculative-decoding.md |
| sampling and penalties | engine/sample.py, engine/penalty.py |
exactness.md |
| structured outputs | engine/grammar.py |
server.md |
| caches | engine/cache.py, engine/disk_cache.py, engine/spill.py, engine/kvpool.py |
caches.md |
| settings registry | engine/settings.py |
operations.md |
| the server | server/app.py, server/stream.py, server/api/toolcall.py |
server.md |
| live activity, metrics, usage | server/api/live.py, server/api/activity.py, server/api/metrics.py, server/api/ledger.py |
the API contract |
| chat mode: the extension host | server/ext.py |
server.md |
| service scripts and gate | ops/ |
operations.md |
| measurement tools | tools/row3.py, tools/block_ab.py, tools/verify_spec.py; Flash-Next tools/fn_*.py |
measurement.md |
One request, one engine¶
With --parallel 1 the engine holds one sequence at a time; the shipped ops/serve.env runs 8 lanes since 0.15.0 (SRV-707, QSE_PARALLEL=8). Requests wait for the engine lock in a bounded queue (8 by default), and a request that waits longer than the queue timeout gets a 503. The numbers on this page describe that single-request loop. With --parallel N the engine holds N sequences over one copy of the weights and verifies them in one joint pass (parallel-requests.md), as ops/serve.env and the profiles of every family with side slots do, except Ornith's and Flint's, which keep one sequence.