Skip to content

Reference: metrics and the usage ledger

Prometheus metrics

GET /metrics serves the families below in the Prometheus text format, to anyone while no token is set, else behind the metrics or admin token, or to a request from the machine itself. ops/monitoring/ has a scrape configuration, alert rules with their tests, and a Grafana dashboard generated by tools/grafana_dashboard.py. With the metrics token set, a scrape job needs it as a bearer token:

- job_name: tandemllm
  authorization: {type: Bearer, credentials_file: /etc/prometheus/tandem-metrics.token}
  static_configs: [{targets: ['gpu-host:8000']}]

The engine's families never count a relayed request. The table lists every family the engine registers with a literal name in the code; families built at scrape time from a table (the cache families by cache label) are listed by their prefix in the help text of the page itself. The extension's families (qse_conductor_*, marked "owner": "ext" in contracts/engine-api-v1/metrics-families.json), the proxy's and the host collector's are in TandemApp's docs.

Prometheus families the engine registers (the extension's are in TandemApp's docs)

Metric Type, labels What it counts
qse_requests_total counter, finish_reason generations that finished, by the reason they finished
qse_request_success_total counter generations that finished on the model's own stop token or the caller's token budget
qse_prompt_tokens_total counter prompt tokens accepted, including tokens served from a cache
qse_generation_tokens_total counter tokens written by the engine
qse_errors_total counter, type exceptions that ended a generation, by exception class
qse_images_left_out_total counter a conversation's earlier images left out of a request over the image caps (server/images.py)
qse_time_to_first_token_seconds histogram arrival of the request to its first token, queue wait included
qse_time_per_output_token_seconds histogram gap between two tokens of one generation.
qse_e2e_request_latency_seconds histogram arrival of the request to the end of its stream, queue wait included
qse_spec_decode_num_accepted_tokens_total counter drafted tokens the target agreed with. The token the target writes itself is not counted here, so accepted over drafted is the draft acceptance rate
qse_spec_decode_num_draft_tokens_total counter tokens proposed by the drafter, anchor excluded
qse_spec_decode_num_drafts_total counter blocks the drafter proposed
qse_spec_accept_per_block histogram tokens committed by one verified block, the target's own token included
qse_draft_width_chosen_total counter, width blocks proposed at each width, anchor included.
qse_verify_seconds histogram one verify pass over a draft block, as the decode loop paid for it
qse_draft_seconds histogram one call to the drafter, lookup and block drafter together
qse_ext_errors_total counter, hook extension hooks that failed (raised, a wrong shape, an act out of place); the request went on without them, by hook
qse_requests_running gauge generations the engine is working on
qse_requests_waiting gauge requests holding a socket and waiting for the engine
qse_requests_refused_total counter, reason requests turned away with a Retry-After before the engine saw them
qse_cache_misses_total counter, cache prefix lookups that found nothing
qse_cache_evictions_total counter, cache entries dropped to stay inside the byte budget
qse_cache_bytes gauge, cache bytes held by each cache
qse_ngram_table_cache_hits_total counter n-gram table rows served from the hot-row cache
qse_ngram_table_cache_misses_total counter n-gram table rows read from disk (O_DIRECT) for a hot-row cache miss
qse_ngram_table_blocks_read_total counter 4 KiB blocks read from disk for the n-gram table
qse_ngram_table_cache_bytes gauge the n-gram table's hot-row cache, in bytes
qse_ngram_table_file_bytes gauge the n-gram table's file size, in bytes
qse_prepare_runs_total counter summaries the engine prepared ahead of a request
qse_prepare_served_total counter prepared summaries a request went on from
qse_prepare_yielded_total counter prepare runs that gave the engine to a client request
qse_prepare_too_long_total counter prepare runs skipped: the conversation and the summary would not fit the context
qse_memory_writer_runs_total counter memory writer turns the engine ran
qse_memory_writer_notes_total counter memory notes the writer stored
qse_memory_bytes gauge, part the engine's memory against its limits, by part
qse_memory_shed_total counter, level caches given back to keep the engine under its ceiling or its peak, by level
qse_cache_entries gauge, cache entries held by each cache
qse_disk_cache_bytes gauge, part the disk tier: what its entries hold, its budget, the file system's free space and the floor it never crosses, the host staging buffer
qse_disk_cache_entries gauge, kind the disk tier's files: anchors (one state each) and chunks (1,024 tokens of rows each)
qse_disk_cache_lookups_total counter prefills the disk tier could have shortened by a chunk or more
qse_disk_cache_hits_total counter prefills that started from the disk tier
qse_disk_cache_misses_total counter lookups the disk tier did not answer
qse_disk_cache_tokens_restored_total counter prompt tokens read back from the disk instead of prefilled
qse_disk_cache_read_bytes_total counter bytes a restore read
qse_disk_cache_written_bytes_total counter bytes the writer wrote
qse_disk_cache_writes_total counter, kind files the writer wrote, by kind
qse_disk_cache_evictions_total counter, reason entries the disk tier gave up, by reason (budget, floor, ttl, orphan)
qse_disk_cache_dropped_total counter, reason writes not done, by reason (window: the next prefill came first; budget, floor, staging, aborted, too_big)
qse_disk_cache_failed_restores_total counter restores that failed a check (checksum, layout) and took the cold path
qse_disk_cache_last_restore_seconds gauge the last restore's wall time
qse_cache_collisions_total counter, cache stored prefixes whose 64-bit hash matched and whose tokens did not
qse_prefill_tokens_reused_total counter prompt tokens restored from a snapshot instead of forwarded
qse_prefill_tokens_forwarded_total counter prompt tokens run through the 64 layers
qse_gpu_memory_used_bytes gauge allocator bytes held by the engine
qse_gpu_memory_reserved_bytes gauge bytes the allocator has taken from the driver and not given back
qse_unified_memory_free_bytes gauge free bytes in the board's one memory pool
qse_engine_uptime_seconds gauge seconds since the server started
qse_engine_info gauge, version, model, drafter, width, tree, nvfp4, fp8_head, max_len, caches always 1. The labels are the configuration the numbers above were produced by
qse_http_requests_total counter, route, code HTTP responses by route (a fixed set) and status code
qse_prompt_tokens_cached_total counter prompt tokens restored from the state store (all of them on a response-cache replay)
qse_reasoning_tokens_total counter committed tokens inside the reasoning block
qse_suffix_store_drafts_total counter, outcome proposed blocks for which the persistent suffix store matched the context (hit) or did not (miss); counted once a block, never once a lookup
qse_request_queue_seconds histogram arrival to the engine lock, per request
qse_request_prefill_seconds histogram the engine lock to the first token (template, tokenise, prefill), per request
qse_request_decode_tokens_per_second histogram per decoded request: (completion tokens - 1) / (last token - first token), the atlas row's convention, and the predicted_per_second a response carries
qse_request_prefill_tokens_per_second histogram per request with a forwarded prompt: forwarded tokens / prefill seconds
qse_request_prompt_tokens histogram prompt tokens per request, cached ones included
qse_request_completion_tokens histogram completion tokens per request
qse_suffix_store_tokens gauge token ids in the persistent suffix store
qse_state_store_budget_bytes gauge the state cache's byte budget
qse_process_resident_memory_bytes gauge resident set of the server process
qse_engine_start_time_seconds gauge when the server started, unix seconds
qse_log_subscribers gauge open /v1/dashboard/logs streams
qse_usage_ledger_rows_total counter rows the usage ledger has written
qse_usage_ledger_dropped_total counter rows the usage ledger lost: a full queue or a failed write
qse_build_info gauge always 1. version, the git commit, the code hash row3 records (tools/row3.py::code_hash) and a hash of the effective flags

The usage ledger

server/api/ledger.py writes one SQLite row per request: counts, timings, the finish reason, the cache source, a client id. Never the prompt, the answer, a tool name or argument, a conversation id, the user field, an IP address or a raw token. Refused requests (503, 429) and rejected ones (400) are rows too, with null token fields, so the request count is honest.

It never slows or fails a request. A row is put on a bounded queue (10,000), a writer thread commits in batches at most every second, and a disk error drops that batch, counts it and logs it at most once a minute. Rows older than --usage-retention-days (at least 400) are deleted once a day, and once an ISO week an online backup goes beside the file, the newest 8 kept. The server takes the path from --usage-ledger only, never from the environment, so a benchmark server started with serve.env exported never writes the operator's ledger. Migrations are forward only and in place, and keep every row.

The usage ledger's requests table (schema version 12)

Column Type, since What it holds
ts_ms INTEGER, v1 when the request arrived, ms since the epoch
request_id TEXT, v1 the response id
model TEXT, v1 the model name the client sent
client_id TEXT, v1 k: and 12 hex digits of the bearer token's sha256, or anon
client_kind TEXT, v1 read from the User-Agent: open-webui, openai-sdk, opencode, hermes, curl, dashboard, other
endpoint TEXT, v1 the route
stream INTEGER, v1 1 for a streamed request
status INTEGER, v1 the HTTP status
finish_reason TEXT, v1 stop, length, tool_calls, timeout, error, abandoned
prompt_tokens INTEGER, v1 prompt tokens as the client sent them
cached_tokens INTEGER, v1 prompt tokens served from a cache
completion_tokens INTEGER, v1 tokens streamed to the client
reasoning_tokens INTEGER, v1 visible reasoning tokens
queue_ms REAL, v1 time waiting for the engine
prompt_ms REAL, v1 prefill time
ttft_ms REAL, v1 time to the first token
decode_ms REAL, v1 time after the first token
total_ms REAL, v1 arrival to the end
decode_tps REAL, v1 (completion tokens - 1) / decode time; null for a replay
prefill_tps REAL, v1 forwarded prompt tokens a second
blocks INTEGER, v1 verify rounds
draft_tokens INTEGER, v1 tokens the drafters proposed
draft_accepted INTEGER, v1 drafted tokens the model agreed with
tool_calls INTEGER, v1 tool calls in the answer
thinking INTEGER, v1 1 when the model thought
cache_source TEXT, v1 where the prompt resumed: resident, end, store, disk, spill, continuation, prepared, response, none
max_tokens INTEGER, v1 the output limit
error_type TEXT, v1 the exception class or the refusal
engine_version TEXT, v1 the VERSION file
code_sha TEXT, v1 a hash over the code folders
model_served TEXT, v2 TandemLLM with the conductor on
profile TEXT, v2 the conductor profile
profile_version INTEGER, v2 its version
conductor_mode TEXT, v2 raw, shadow or on
conductor_acts INTEGER, v2 the session monitor's executed acts
conductor_hidden_tokens INTEGER, v2 tokens the model saw that the client did not send
conductor_acted INTEGER, v2 1 when the conductor changed what the model saw or wrote
lane TEXT, v2 decode, prefill or self for a side-lane request
backend TEXT, v3 the backend id of a relayed request
backend_kind TEXT, v3 its kind
backend_model TEXT, v3 the id sent upstream
public_model TEXT, v3 the id the client sent
usage_source TEXT, v3 backend or none
cache_write_tokens INTEGER, v3 tokens a provider charged as a cache write
cost_micros INTEGER, v3 cost in micro-dollars at the model's price
proxy_ms REAL, v3 the proxy's own time
retries INTEGER, v3 tries again before the first byte
fallback_from TEXT, v3 the backend tried first
upstream_status INTEGER, v3 the backend's HTTP status
user TEXT, v3 the label of the proxy key
preemptions INTEGER, v4 preemptions (vLLM) or retractions (SGLang), from the plugin
engine_finish TEXT, v4 the backend engine's own finish reason
corrupted INTEGER, v4 vLLM's NaN check
l2_join TEXT, v4 exact or approximate join of the plugin's numbers
wait_own_ms REAL, v5 ms the request waited behind the engine's own work (the shadow reader, a summary written ahead, the conductor's question)
parent_request TEXT, v6 the turn an internal row served (the router's question names the turn it routes)
slot INTEGER, v7 the engine's sequence slot with --parallel N (0 the main slot, 1 and up a side slot); null without
engine_cached INTEGER, v8 the prompt tokens the engine read from a cache, the hidden notes included (cached_tokens counts the client's among them)
forwarded_tokens INTEGER, v8 what the prefill forwarded, hidden tokens included
wait_mem_ms REAL, v8 ms the parallel memory admission held the request (inside queue_ms)
sampling TEXT, v9 how the request sampled: null greedy, client (its own temperature), regenerate (TandemLLM sampled a regenerate)
system_fingerprint TEXT, v9
node TEXT, v9
deployment TEXT, v9
upstream_request TEXT, v9

The conductor's journal shares the file: conductor_events has 53 columns (id, ts_ms, request_id, session_key, kind, task, tier, mode, sp, rule, trigger, score, decision, action, cap, tier1_model, question_id, probabilities, tier1_ms, hidden_text_id, hidden_tokens, tokens_at_cut, position, outcome, next_checker, policy_version, client_kind, model, profile, profile_version, router_version, class_id, confidence, source, kept, stages, classifier_ms, effort, budget, budget_source, thinking, decisions, injected, replayed, shadow, digest, label, label_note, labelled_at, acted, changed, detail, extra), and conductor_texts keeps every hidden text once, by its hash.

A row's speed figures follow the benchmark row's formula, and a replay or a side-lane request has no decode rate. A relayed request has null queue, prefill, blocks, draft counts and cache source unless a plugin reported them, in which case l2_join says how the row was joined. A quick look:

sqlite3 ~/.qwen38-spark-engine/usage/ledger.sqlite3 \
  "select datetime(ts_ms/1000,'unixepoch','localtime'), finish_reason, prompt_tokens,
          completion_tokens, round(decode_tps,1), cache_source from requests order by id desc limit 10"