← Architecture map · HTTP & Server

HTTP & Server layer

The thin shell that takes traffic, turns it into work for the engine, streams answers back, and exposes the dashboard and stats endpoints.

Endpoints· Startup· Tokenizer· Streaming (SSE)· Load simulator· Metrics· Dashboard· → Scheduler

HTTP endpoints

All the URLs the server exposes and what each one does.

POST /generate main inference endpoint accepts text, max_tokens, stream, session_id, priority returns SSE stream OR a JSON response tags response with cache-hit token count GET /scheduler/stats live scheduler health queue depth, active rows, admit/reject totals KV pressure, free blocks, active-KV reserved TTFT and TPOT percentiles GET /cache/stats KV cache observability total / used / free blocks, utilization % cumulative hit / miss / eviction counts overall hit rate POST /simulate/start · POST /simulate/stop · GET /simulate/status control the built-in load simulator spins up N virtual users hitting /generate concurrently prompts drawn from weighted buckets, per-user RNG seed jittered think-time between turns to mimic human pacing GET / (and /static/*) serves the dashboard HTML page single-page browser UI for live metrics polls /scheduler/stats and /cache/stats on a timer renders the simulator controls and history charts GET /health liveness probe — process is up always 200, no state touched — cheap for Modal/k8s polls if this fails, the container is dead and traffic should be cut GET /ready readiness probe — model + scheduler are up 503 until lifespan finishes loading, 200 after Modal won't route /generate traffic until this is 200 All endpoints share one ScheduledRequest path /generate is the only endpoint that creates work. Stats endpoints are read-only snapshots. The simulator endpoints internally fire /generate calls — they are not a separate code path.

Startup (the lifespan hook)

Modal-style cold start: every expensive thing happens once, before any request lands.

app boot FastAPI lifespan starts load model + tokenizer auto-detect CUDA / MPS / CPU build KV cache pre-allocate every per-layer block pool start scheduler thread worker loop begins ticking Why everything happens in the lifespan hook Modal-deployable: no lazy first-request init, KV pools live in container memory, cold start is deterministic, keep_warm actually keeps things warm.

Tokenizer

Stateless. Same tokenizer instance shared across every request.

raw user text "summarize this paper..." tokenizer.encode_chat(text) wraps in Gemma chat template ("system" / "user" / "model" turns) returns a list of token IDs token IDs [2, 106, 1645, ... 107] tokenizer.decode(token_ids) turns model output back into text streamed token-by-token to the client template_prefix_len the chat template adds a constant prefix of tokens to every prompt — it's the same N tokens every time. the cache stores blocks for that prefix once and shares them across every session. computed at startup.

Streaming (SSE)

Tokens go to the client as soon as the model produces them. No waiting for the whole response.

/generate?stream=true opens an SSE response scheduler queues request stream queue attached to the row decode produces a token put on the per-request queue event_stream yields data: ... client receives a token, repeat until EOS What rides on each event first event carries TTFT (time-to-first-token in ms) so the client and metrics layer can both record it subsequent events carry the next token (already detokenized to text) final event carries the close marker plus total_ms, tokens_generated, prompt_tokens, cache_hit_tokens non-streaming mode collects all of the above and returns one JSON GenerateResponse instead

Load simulator

A built-in multi-user traffic generator. Lives in simulator_prompts.py; control endpoints are on the server.

KnobWhat it does
num_usersHow many virtual users to run in parallel.
weighted prompt bucketsMix of short, medium, long prompts in realistic ratios.
per-user RNG seedEach user picks prompts from its own deterministic stream so runs are reproducible.
jittered think-timeRandom pause between a user's turns, so requests don't all arrive in lockstep.
session_id per userEach user has a stable session ID so the cache + fairness layers see real session boundaries.
Why this exists: the engine is built for many concurrent users with mixed prompt lengths. Single-request curl tests don't exercise the scheduler, the backpressure gates, or the fairness policy. The simulator does. External Locust testing comes next.

Metrics tracker

metrics.py. Pull-based, lives in process memory, surfaced through /scheduler/stats.

MetricMeaning
TTFTTime-to-first-token. How long the user waits before they see anything.
TPOTTime-per-output-token. The streaming smoothness metric.
throughputTokens per second across the whole engine, plus requests per second.
sliding window60 seconds by default. Old samples are pruned on every snapshot read.
percentilesp50, p95, p99 — what tail latency looks like, not just the average.
Prometheus /metrics (Phase 8, shipped). Pull-based exposition at /metrics (prometheus_metrics.py): counters (requests by outcome, tokens, prefill chunks), histograms (TTFT/TPOT/latency + HTTP duration from the timing middleware), gauges (active batch, queue/prefilling depth, KV pressure — refreshed from the scheduler at scrape time). Completions/rejections/chunks are observed through the existing MetricsTracker, so the scheduler gains no direct Prometheus coupling. Aggregate — no session_id label: session_id is unbounded → a per-session label would explode Prometheus cardinality; per-session detail stays in /scheduler/stats (JSON, pull-on-demand). Only low-cardinality labels (outcome, HTTP method/endpoint/status). Config + dashboard in monitoring/.
Structured logs + timing middleware. LOG_FORMAT=json (default text) switches stdlib logging to one JSON object per line (logging_config.py); logger.info(..., extra={...}) fields merge in. A FastAPI middleware times every HTTP request → the inference_http_request_duration_seconds histogram (labeled by the matched route template, not the raw path) + a structured log line.

Dashboard

src/inference_server/static/index.html. A single-page UI that polls the stats endpoints.

See also: Scheduler · KV Cache · Backend