← Architecture map · Scheduler

Scheduler

The engine's heart. Decides which requests are batched together each tick, how to be fair across users, and how to refuse work cleanly when the system would otherwise melt.

Queues· Per-tick loop· Policies· Backpressure· Chunked prefill· Reservation accounting· → Backend· → KV Cache

The three queues

Every request sits in exactly one of these at any time.

_pending requests waiting their turn ordered by the active scheduling policy (FCFS or Fair / VTC) capacity: MAX_QUEUE_SIZE over capacity → HTTP 429 drained by _admit_pending _prefilling prompts being processed in chunks long prompts split into PREFILL_CHUNK_SIZE tokens; one chunk per tick prevents one giant prompt from freezing every other user's stream drained by advance_prefill_chunk _active requests in the decoding batch capacity: MAX_BATCH_SIZE all rows decode together in one forward pass when a row hits EOS or max_tokens it leaves and a pending request can take its slot drained when finished, fed by _prefilling
Why a separate prefilling queue? Prefill is expensive but parallel-friendly across the prompt's tokens. Decode is cheap per token but happens many times. Mixing them in the same logical queue makes the scheduler harder to reason about. Keeping them separate means the chunked-prefill state machine can evolve without disturbing the decode batch.

What happens on a single tick

One iteration of the scheduler's worker thread, in order. The order matters — getting it wrong drops first tokens.

while running: (the worker thread holds the backend lock for the whole tick) ① admit_pending try to start new requests peek at the next pending request ask the cache: how much of this prompt is already cached? check both backpressure gates → if both pass, admit → if not, wait or reject ② advance_prefill_chunk feed long prompts a chunk at a time for each row in _prefilling: push the next N prompt tokens through the model on the final chunk: store the new KV in the cache promote the row → _active ③ evict_finished retire any rows that just ended for each row in _active: if EOS or hit max_tokens: remove row from batched KV release its cache blocks free its KV reservation finish the user's response ④ decode_step one batched forward pass for everyone stack KV from every active row with left-padding to align lengths build attention mask + position IDs run the model once → one new token per row stream each token to its client next tick (immediately, no delay) Order matters: admit must come before evict if a freshly-admitted row's first token were generated in step ④ but step ③ ran first, it would never get streamed (the row wasn't yet "finished" but also wasn't tracked properly). Saved as a project memory; do not reorder these phases without re-deriving correctness.

Scheduling policies

Both implement the same interface. The policy decides who runs next.

FCFS (first-come, first-served) straight FIFO with priority on top sort key: (-priority, arrival_seq) higher-priority requests drain first within a priority tier, oldest goes first cheap, predictable, the right default a heavy user can hog the queue if priorities tie good for: low-concurrency, single-tenant, tests Fair (Virtual-Token-Counter, VTC) per-session fairness using a virtual clock each session has a counter of tokens it has spent sort key: (-priority, counter[sid], arrival_seq) lowest-spend session gets the next slot new sessions inherit the current minimum counter → they neither starve nor get a catch-up burst good for: real multi-user load, the production target
HookWhen it firesWhat both policies do
on_request_arrivedrequest enters _pendinginsert into the policy's ordered structure
peek_nextadmit step inspects headreturn next candidate without removing it
pick_nextadmit step commitsremove and return the head
on_tokens_processedafter each decode stepVTC bumps the counter; FCFS no-op
on_request_finishedrow leaves _activecleanup hook; both no-op for now
Preemption is deliberately deferred. Today's "soft hold" (a fitting request waits for room to open up) is behaviorally equivalent to preemption for the common case. Preemption only becomes load-bearing once strict priority classes or per-tenant SLAs land — see CLAUDE.md.

Backpressure (three gates)

Three independent checks decide whether a pending request can start. All three must pass.

Gate 1 — Queue capacity checked at HTTP entry _pending size vs MAX_QUEUE_SIZE over capacity → reject immediately client gets HTTP 429 with reason protects the server from a flood that would otherwise pile up forever Gate 2 — Active-KV budget checked in admit_pending running sum of (prompt_len + max_tokens) across every active row if adding this request would blow the budget: soft-hold (wait for someone to finish) if the request alone is bigger than the budget: hard-reject (it can never fit) prevents decode-time OOM Gate 3 — Cache block pool checked in admit_pending blocks_needed for prompt vs free blocks not enough free blocks → soft-hold prompt bigger than the whole pool → hard-reject prevents "evict to admit, then immediately re-evict" thrash that wastes work
BehaviorWhat it means
soft-holdrequest stays at the head of _pending; we re-check next tick. Order is preserved so fairness still applies.
hard-rejectrequest can never fit; respond with HTTP 429 + a structured reason instead of letting it hang.
kv_pressureused / total cache blocks. Surfaced in /scheduler/stats.
active_kv_reservedcurrent sum of admitted (prompt_len + max_tokens). Also surfaced.
kv_admit_blockedcounter — how many times the gates blocked an admission this run.

Chunked prefill

Long prompts no longer freeze every other user.

long prompt arrives e.g. 4000 tokens split into chunks PREFILL_CHUNK_SIZE per chunk tick N: feed chunk 1 decode batch still runs in parallel tick N+k: feed last chunk store full KV in cache, promote → active Two versions, only one runs today Version A (alternating, shipped): chunked prefill happens as a separate forward pass alongside the decode batch. Two passes per tick. Works on MPS today. Captures the latency win. Tested. Version B (mixed-batch, vLLM-style, deferred): a single forward pass packs decode rows and one prefill chunk together. Needs a custom attention path (FlashAttention varlen). CUDA-only. Pairs with the vLLM benchmark phase.
Prefill strategy is a config seam (PREFILL_MODE). monolithic (one forward on admit) or chunked (V-A) today; the enum is the forward-compat extension point for mixed_batch (V-B) and disaggregated (P/D) — all reuse the same _prefilling phase + prefill_chunk primitive. Empty config derives the mode from PREFILL_CHUNK_SIZE (back-compat: >0 → chunked); an unimplemented mode fails loud at construction.
The disaggregation seam: _promote_to_decode. Both monolithic admit and the chunked final-chunk path funnel a completed prefill's KV into the decode batch through this one method. Today it's a local splice on the same worker; under P/D disaggregation the KV is produced on a prefill worker and transferred here before the row decodes — so disagg adds a transfer at this single point plus a remote prefill loop, not a scheduler rewrite. The loop is already prefill/decode-decoupled, which is what makes that cheap.

Reservation accounting (the load-bearing invariant)

The rule: _active_kv_reserved is the sum of (prompt_len + max_tokens) across every row in _active. Admit adds. Finish, evict, and any future preemption all subtract. If anything forgets to subtract, the engine starts hard-rejecting requests it could actually serve.
Mask bookkeeping rule: _splice_in sizes the attention-mask cat off the mask's own width (_attention_mask.shape[1]), not backend.kv_length(). For the paged custom backend kv_length = max(per-row len) drops when the longest row is evicted while the mask width doesn't — deriving the mask size from it cat'd mismatched widths and killed the worker (the 2026-06-12 mixed-workload crash). Scheduler-owned tensor state must be self-consistent, not derived from a backend metric whose invariants differ per backend.

See also: Backend for what decode_step_batched and prefill_chunk actually do · KV Cache for how the cache-pool gate is computed.