Reference: metrics and the usage ledger¶
Prometheus metrics¶
GET /metrics serves the families below in the Prometheus text format, to anyone while no token is set, else behind the metrics or admin token, or to a request from the machine itself. ops/monitoring/ has a scrape configuration, alert rules with their tests, and a Grafana dashboard generated by tools/grafana_dashboard.py. With the metrics token set, a scrape job needs it as a bearer token:
- job_name: tandemllm
authorization: {type: Bearer, credentials_file: /etc/prometheus/tandem-metrics.token}
static_configs: [{targets: ['gpu-host:8000']}]
The engine's families never count a relayed request. The table lists every family the engine registers with a literal name in the code; families built at scrape time from a table (the cache families by cache label) are listed by their prefix in the help text of the page itself. The extension's families (qse_conductor_*, marked "owner": "ext" in contracts/engine-api-v1/metrics-families.json), the proxy's and the host collector's are in TandemApp's docs.
Prometheus families the engine registers (the extension's are in TandemApp's docs)¶
| Metric | Type, labels | What it counts |
|---|---|---|
qse_requests_total |
counter, finish_reason |
generations that finished, by the reason they finished |
qse_request_success_total |
counter |
generations that finished on the model's own stop token or the caller's token budget |
qse_prompt_tokens_total |
counter |
prompt tokens accepted, including tokens served from a cache |
qse_generation_tokens_total |
counter |
tokens written by the engine |
qse_errors_total |
counter, type |
exceptions that ended a generation, by exception class |
qse_images_left_out_total |
counter |
a conversation's earlier images left out of a request over the image caps (server/images.py) |
qse_time_to_first_token_seconds |
histogram |
arrival of the request to its first token, queue wait included |
qse_time_per_output_token_seconds |
histogram |
gap between two tokens of one generation. |
qse_e2e_request_latency_seconds |
histogram |
arrival of the request to the end of its stream, queue wait included |
qse_spec_decode_num_accepted_tokens_total |
counter |
drafted tokens the target agreed with. The token the target writes itself is not counted here, so accepted over drafted is the draft acceptance rate |
qse_spec_decode_num_draft_tokens_total |
counter |
tokens proposed by the drafter, anchor excluded |
qse_spec_decode_num_drafts_total |
counter |
blocks the drafter proposed |
qse_spec_accept_per_block |
histogram |
tokens committed by one verified block, the target's own token included |
qse_draft_width_chosen_total |
counter, width |
blocks proposed at each width, anchor included. |
qse_verify_seconds |
histogram |
one verify pass over a draft block, as the decode loop paid for it |
qse_draft_seconds |
histogram |
one call to the drafter, lookup and block drafter together |
qse_ext_errors_total |
counter, hook |
extension hooks that failed (raised, a wrong shape, an act out of place); the request went on without them, by hook |
qse_requests_running |
gauge |
generations the engine is working on |
qse_requests_waiting |
gauge |
requests holding a socket and waiting for the engine |
qse_requests_refused_total |
counter, reason |
requests turned away with a Retry-After before the engine saw them |
qse_cache_misses_total |
counter, cache |
prefix lookups that found nothing |
qse_cache_evictions_total |
counter, cache |
entries dropped to stay inside the byte budget |
qse_cache_bytes |
gauge, cache |
bytes held by each cache |
qse_ngram_table_cache_hits_total |
counter |
n-gram table rows served from the hot-row cache |
qse_ngram_table_cache_misses_total |
counter |
n-gram table rows read from disk (O_DIRECT) for a hot-row cache miss |
qse_ngram_table_blocks_read_total |
counter |
4 KiB blocks read from disk for the n-gram table |
qse_ngram_table_cache_bytes |
gauge |
the n-gram table's hot-row cache, in bytes |
qse_ngram_table_file_bytes |
gauge |
the n-gram table's file size, in bytes |
qse_prepare_runs_total |
counter |
summaries the engine prepared ahead of a request |
qse_prepare_served_total |
counter |
prepared summaries a request went on from |
qse_prepare_yielded_total |
counter |
prepare runs that gave the engine to a client request |
qse_prepare_too_long_total |
counter |
prepare runs skipped: the conversation and the summary would not fit the context |
qse_memory_writer_runs_total |
counter |
memory writer turns the engine ran |
qse_memory_writer_notes_total |
counter |
memory notes the writer stored |
qse_memory_bytes |
gauge, part |
the engine's memory against its limits, by part |
qse_memory_shed_total |
counter, level |
caches given back to keep the engine under its ceiling or its peak, by level |
qse_cache_entries |
gauge, cache |
entries held by each cache |
qse_disk_cache_bytes |
gauge, part |
the disk tier: what its entries hold, its budget, the file system's free space and the floor it never crosses, the host staging buffer |
qse_disk_cache_entries |
gauge, kind |
the disk tier's files: anchors (one state each) and chunks (1,024 tokens of rows each) |
qse_disk_cache_lookups_total |
counter |
prefills the disk tier could have shortened by a chunk or more |
qse_disk_cache_hits_total |
counter |
prefills that started from the disk tier |
qse_disk_cache_misses_total |
counter |
lookups the disk tier did not answer |
qse_disk_cache_tokens_restored_total |
counter |
prompt tokens read back from the disk instead of prefilled |
qse_disk_cache_read_bytes_total |
counter |
bytes a restore read |
qse_disk_cache_written_bytes_total |
counter |
bytes the writer wrote |
qse_disk_cache_writes_total |
counter, kind |
files the writer wrote, by kind |
qse_disk_cache_evictions_total |
counter, reason |
entries the disk tier gave up, by reason (budget, floor, ttl, orphan) |
qse_disk_cache_dropped_total |
counter, reason |
writes not done, by reason (window: the next prefill came first; budget, floor, staging, aborted, too_big) |
qse_disk_cache_failed_restores_total |
counter |
restores that failed a check (checksum, layout) and took the cold path |
qse_disk_cache_last_restore_seconds |
gauge |
the last restore's wall time |
qse_cache_collisions_total |
counter, cache |
stored prefixes whose 64-bit hash matched and whose tokens did not |
qse_prefill_tokens_reused_total |
counter |
prompt tokens restored from a snapshot instead of forwarded |
qse_prefill_tokens_forwarded_total |
counter |
prompt tokens run through the 64 layers |
qse_gpu_memory_used_bytes |
gauge |
allocator bytes held by the engine |
qse_gpu_memory_reserved_bytes |
gauge |
bytes the allocator has taken from the driver and not given back |
qse_unified_memory_free_bytes |
gauge |
free bytes in the board's one memory pool |
qse_engine_uptime_seconds |
gauge |
seconds since the server started |
qse_engine_info |
gauge, version, model, drafter, width, tree, nvfp4, fp8_head, max_len, caches |
always 1. The labels are the configuration the numbers above were produced by |
qse_http_requests_total |
counter, route, code |
HTTP responses by route (a fixed set) and status code |
qse_prompt_tokens_cached_total |
counter |
prompt tokens restored from the state store (all of them on a response-cache replay) |
qse_reasoning_tokens_total |
counter |
committed tokens inside the reasoning block |
qse_suffix_store_drafts_total |
counter, outcome |
proposed blocks for which the persistent suffix store matched the context (hit) or did not (miss); counted once a block, never once a lookup |
qse_request_queue_seconds |
histogram |
arrival to the engine lock, per request |
qse_request_prefill_seconds |
histogram |
the engine lock to the first token (template, tokenise, prefill), per request |
qse_request_decode_tokens_per_second |
histogram |
per decoded request: (completion tokens - 1) / (last token - first token), the atlas row's convention, and the predicted_per_second a response carries |
qse_request_prefill_tokens_per_second |
histogram |
per request with a forwarded prompt: forwarded tokens / prefill seconds |
qse_request_prompt_tokens |
histogram |
prompt tokens per request, cached ones included |
qse_request_completion_tokens |
histogram |
completion tokens per request |
qse_suffix_store_tokens |
gauge |
token ids in the persistent suffix store |
qse_state_store_budget_bytes |
gauge |
the state cache's byte budget |
qse_process_resident_memory_bytes |
gauge |
resident set of the server process |
qse_engine_start_time_seconds |
gauge |
when the server started, unix seconds |
qse_log_subscribers |
gauge |
open /v1/dashboard/logs streams |
qse_usage_ledger_rows_total |
counter |
rows the usage ledger has written |
qse_usage_ledger_dropped_total |
counter |
rows the usage ledger lost: a full queue or a failed write |
qse_build_info |
gauge |
always 1. version, the git commit, the code hash row3 records (tools/row3.py::code_hash) and a hash of the effective flags |
The usage ledger¶
server/api/ledger.py writes one SQLite row per request: counts, timings, the finish reason, the cache source, a client id. Never the prompt, the answer, a tool name or argument, a conversation id, the user field, an IP address or a raw token. Refused requests (503, 429) and rejected ones (400) are rows too, with null token fields, so the request count is honest.
It never slows or fails a request. A row is put on a bounded queue (10,000), a writer thread commits in batches at most every second, and a disk error drops that batch, counts it and logs it at most once a minute. Rows older than --usage-retention-days (at least 400) are deleted once a day, and once an ISO week an online backup goes beside the file, the newest 8 kept. The server takes the path from --usage-ledger only, never from the environment, so a benchmark server started with serve.env exported never writes the operator's ledger. Migrations are forward only and in place, and keep every row.
The usage ledger's requests table (schema version 12)¶
| Column | Type, since | What it holds |
|---|---|---|
ts_ms |
INTEGER, v1 |
when the request arrived, ms since the epoch |
request_id |
TEXT, v1 |
the response id |
model |
TEXT, v1 |
the model name the client sent |
client_id |
TEXT, v1 |
k: and 12 hex digits of the bearer token's sha256, or anon |
client_kind |
TEXT, v1 |
read from the User-Agent: open-webui, openai-sdk, opencode, hermes, curl, dashboard, other |
endpoint |
TEXT, v1 |
the route |
stream |
INTEGER, v1 |
1 for a streamed request |
status |
INTEGER, v1 |
the HTTP status |
finish_reason |
TEXT, v1 |
stop, length, tool_calls, timeout, error, abandoned |
prompt_tokens |
INTEGER, v1 |
prompt tokens as the client sent them |
cached_tokens |
INTEGER, v1 |
prompt tokens served from a cache |
completion_tokens |
INTEGER, v1 |
tokens streamed to the client |
reasoning_tokens |
INTEGER, v1 |
visible reasoning tokens |
queue_ms |
REAL, v1 |
time waiting for the engine |
prompt_ms |
REAL, v1 |
prefill time |
ttft_ms |
REAL, v1 |
time to the first token |
decode_ms |
REAL, v1 |
time after the first token |
total_ms |
REAL, v1 |
arrival to the end |
decode_tps |
REAL, v1 |
(completion tokens - 1) / decode time; null for a replay |
prefill_tps |
REAL, v1 |
forwarded prompt tokens a second |
blocks |
INTEGER, v1 |
verify rounds |
draft_tokens |
INTEGER, v1 |
tokens the drafters proposed |
draft_accepted |
INTEGER, v1 |
drafted tokens the model agreed with |
tool_calls |
INTEGER, v1 |
tool calls in the answer |
thinking |
INTEGER, v1 |
1 when the model thought |
cache_source |
TEXT, v1 |
where the prompt resumed: resident, end, store, disk, spill, continuation, prepared, response, none |
max_tokens |
INTEGER, v1 |
the output limit |
error_type |
TEXT, v1 |
the exception class or the refusal |
engine_version |
TEXT, v1 |
the VERSION file |
code_sha |
TEXT, v1 |
a hash over the code folders |
model_served |
TEXT, v2 |
TandemLLM with the conductor on |
profile |
TEXT, v2 |
the conductor profile |
profile_version |
INTEGER, v2 |
its version |
conductor_mode |
TEXT, v2 |
raw, shadow or on |
conductor_acts |
INTEGER, v2 |
the session monitor's executed acts |
conductor_hidden_tokens |
INTEGER, v2 |
tokens the model saw that the client did not send |
conductor_acted |
INTEGER, v2 |
1 when the conductor changed what the model saw or wrote |
lane |
TEXT, v2 |
decode, prefill or self for a side-lane request |
backend |
TEXT, v3 |
the backend id of a relayed request |
backend_kind |
TEXT, v3 |
its kind |
backend_model |
TEXT, v3 |
the id sent upstream |
public_model |
TEXT, v3 |
the id the client sent |
usage_source |
TEXT, v3 |
backend or none |
cache_write_tokens |
INTEGER, v3 |
tokens a provider charged as a cache write |
cost_micros |
INTEGER, v3 |
cost in micro-dollars at the model's price |
proxy_ms |
REAL, v3 |
the proxy's own time |
retries |
INTEGER, v3 |
tries again before the first byte |
fallback_from |
TEXT, v3 |
the backend tried first |
upstream_status |
INTEGER, v3 |
the backend's HTTP status |
user |
TEXT, v3 |
the label of the proxy key |
preemptions |
INTEGER, v4 |
preemptions (vLLM) or retractions (SGLang), from the plugin |
engine_finish |
TEXT, v4 |
the backend engine's own finish reason |
corrupted |
INTEGER, v4 |
vLLM's NaN check |
l2_join |
TEXT, v4 |
exact or approximate join of the plugin's numbers |
wait_own_ms |
REAL, v5 |
ms the request waited behind the engine's own work (the shadow reader, a summary written ahead, the conductor's question) |
parent_request |
TEXT, v6 |
the turn an internal row served (the router's question names the turn it routes) |
slot |
INTEGER, v7 |
the engine's sequence slot with --parallel N (0 the main slot, 1 and up a side slot); null without |
engine_cached |
INTEGER, v8 |
the prompt tokens the engine read from a cache, the hidden notes included (cached_tokens counts the client's among them) |
forwarded_tokens |
INTEGER, v8 |
what the prefill forwarded, hidden tokens included |
wait_mem_ms |
REAL, v8 |
ms the parallel memory admission held the request (inside queue_ms) |
sampling |
TEXT, v9 |
how the request sampled: null greedy, client (its own temperature), regenerate (TandemLLM sampled a regenerate) |
system_fingerprint |
TEXT, v9 |
|
node |
TEXT, v9 |
|
deployment |
TEXT, v9 |
|
upstream_request |
TEXT, v9 |
The conductor's journal shares the file: conductor_events has 53 columns (id, ts_ms, request_id, session_key, kind, task, tier, mode, sp, rule, trigger, score, decision, action, cap, tier1_model, question_id, probabilities, tier1_ms, hidden_text_id, hidden_tokens, tokens_at_cut, position, outcome, next_checker, policy_version, client_kind, model, profile, profile_version, router_version, class_id, confidence, source, kept, stages, classifier_ms, effort, budget, budget_source, thinking, decisions, injected, replayed, shadow, digest, label, label_note, labelled_at, acted, changed, detail, extra), and conductor_texts keeps every hidden text once, by its hash.
A row's speed figures follow the benchmark row's formula, and a replay or a side-lane request has no decode rate. A relayed request has null queue, prefill, blocks, draft counts and cache source unless a plugin reported them, in which case l2_join says how the row was joined. A quick look:
sqlite3 ~/.qwen38-spark-engine/usage/ledger.sqlite3 \
"select datetime(ts_ms/1000,'unixepoch','localtime'), finish_reason, prompt_tokens,
completion_tokens, round(decode_tps,1), cache_source from requests order by id desc limit 10"