Inference Server — Architecture Map

Click any block in the diagram below to dive into a detailed page for that subsystem. Last updated 2026-06-14.

Detail pages: HTTP & Server· Scheduler· Backend· KV Cache· Phase status

The big picture

A request walks left to right. The HTTP layer turns text into a job, the scheduler decides who runs when, the backend does the math, and the KV cache makes sure we don't redo work we've already done.

↓ Each colored block is a link — click to open its detail page.

Client browser, curl, load simulator HTTP layer (FastAPI) server.py accepts requests tokenizes prompts streams tokens back (SSE) serves the dashboard /health · /ready for Modal probes click to open → Scheduler scheduler.py decides who runs each tick batches many users together enforces fairness across sessions (FCFS or VTC) protects the system from overload (backpressure) streams long prompts in small chunks queues: pending · prefilling · active click to open → Inference Backend backends/torch_backend.py runs the model handles prefill + decode picks the best hardware (CUDA → MPS → CPU) click to open → Model Gemma 4 E2B / E4B KV Cache (paged, session-aware) kv_cache/cache_manager.py remembers attention state for the prompt prefix so repeated or shared prompts skip the prefill cost Exact-match index whole-prompt fast path Radix tree prefix sharing across sessions Block pool + eviction LRU · SinkLRU · H2O click to open → Tokenizer tokenizer.py turns text ↔ token IDs, applies chat template Metrics tracker metrics.py records TTFT, TPOT, throughput in a 60s window surfaces p50/p95/p99 Load simulator simulator_prompts.py multi-user traffic, weighted prompts, jittered think-time Dashboard static/index.html live metrics, cache stats, scheduler stats Config (env vars) config.py — DEVICE, MODEL_NAME, MAX_BATCH_SIZE, MAX_ACTIVE_KV_TOKENS, PREFILL_CHUNK_SIZE, EVICTION_POLICY, SCHEDULING_POLICY, KV_CACHE_*
HTTP / request path scheduler backend KV cache queues scheduler loop

What each subsystem is responsible for

SubsystemOne-line jobDetail page
HTTP & ServerAccept requests, tokenize, stream tokens back, serve dashboard.arch-server.html
SchedulerDecide which requests run together, when, and at what speed — fairly.arch-scheduler.html
BackendOwn the model. Run prefill and decode forward passes.arch-backend.html
KV CacheRemember attention state so we don't recompute prompt prefixes.arch-cache.html
TokenizerStateless text ↔ tokens with the Gemma chat template.covered in HTTP & Server
Metrics trackerSliding-window TTFT / TPOT / throughput with percentiles.covered in HTTP & Server
Load simulatorBuilt-in multi-user load generator for stress testing.covered in HTTP & Server
DashboardBrowser UI for live cache and scheduler stats.covered in HTTP & Server
ConfigEnv-var-driven settings, no hardcoded paths.covered on every page

How a single request flows through the system

Read this once and the rest of the diagrams will make sense.

  1. Client sends a POST to /generate with the prompt text, max tokens, optional session_id, and optional priority.
  2. Server tokenizes the text using the Gemma chat template and wraps it in a ScheduledRequest object that carries the session ID and arrival time.
  3. Scheduler queues the request. The fairness policy (FCFS or VTC) decides where it lands relative to other waiting requests.
  4. Each tick the scheduler tries to admit pending requests. It only admits if there's room in both the active-KV budget and the cache block pool. If not, the request waits.
  5. If the prompt is long, prefill is split into chunks, one chunk per tick — long prompts no longer freeze every other user's stream.
  6. The cache is checked for any matching prefix. If found, those tokens skip the forward pass entirely.
  7. The backend runs a single batched decode forward pass across every active request, producing one new token per request per tick.
  8. Each new token is streamed to its client via SSE. Metrics track TTFT and TPOT.
  9. When a request hits EOS or max tokens, it leaves the active batch, releases its blocks, and frees its KV reservation.

Phase status

What's done, what's in progress, what's deferred.

PhaseWhat it builtStatus
0 — FoundationProject structure, env-driven config, backend interfacedone
1 — TokenizationChat template + edge-case handlingdone
2 — Generation loopToken-by-token decode, EOS / max-tokens handling, runs in executordone
3 — StreamingSSE endpoint, TTFT measurementdone
4 — Fixed-window batchingReplaced by Phase 6 continuous batchingsuperseded
5 — KV cacheBlock pool, radix tree, exact-match index, three eviction policiesdone H2O wiring deferred
6 — Continuous batchingIteration-level scheduling, immediate slot filldone
6 — Fair schedulingFCFS + Fair (VTC) + priority hooksdone preemption deferred
6 — BackpressureQueue gate + active-KV gate + cache-pool gate, soft-hold and hard-rejectdone
6 — Chunked prefill (alternating)Long prompts split across ticks so they don't block decodein progress
6 — Chunked prefill (mixed-batch, vLLM-style)Needs custom attention pathdeferred — CUDA only
6 — Load simulatorBuilt-in weighted multi-user simulator with per-user RNGdone
6 — Locust load testingExternal load generator, throughput-vs-latency curvesnext
6 — vLLM head-to-head benchmarkSame model, same hardware, measured against vLLMlater
7 — Hardware auto-detectCUDA → MPS → CPU fallback, partialpartial
8 — ObservabilityRolling p50/p95/p99 + Prometheus /metrics (aggregate) + structured JSON logs + timing middleware; Grafana dashboard in monitoring/mostly

Diagram color key — blue = HTTP, purple = scheduler, orange = backend, green = KV cache, pink = queues, yellow dashed = scheduler loop, red dashed = eviction. Maintenance: any new feature or component change must update this file and the relevant detail page (see CLAUDE.md → Documentation Maintenance).