Projects

Interactive explainer · LLM systems

How an LLM generates a response

Follow one prompt through request routing, prefill, and the decode loop. Watch the KV cache grow as the server streams the answer.

Request trace

Step 01 / 30

Send the request

The client sends “Can shadows outrun light?” to the model API.

Why it matters

The prompt begins as text in an HTTP request.

01 / Serving system

Where is the request?

Request → worker
  1. ClientSends prompt, displays text
  2. API gatewayAuth and transport
  3. SchedulerQueues and routes
  4. GPU workerRuns the model

02 / Model computation

Inside the GPU worker

Prompt

Can shadows outrun light?

Sampled output

No tokens sampled yet.

  1. EmbeddingsToken IDs to vectors
  2. Transformer × NAttention and MLP
  3. LM headVocabulary scores
Inside a Transformer layer

In a typical decoder layer, self-attention uses queries, keys, and values to mix context. A feed-forward network (MLP) transforms the result. Normalization and residual connections help carry the hidden state between layers.

KV cache

0 / 8 positions

Each column represents one token position across the model’s attention layers.

Next-token probabilities

New scores appear when the LM head runs.

Illustrative probabilities derived from example logits.

03 / Delivery

What reaches the client?

Waiting

Client output

Waiting for the first token…

SSE stream text/event-stream

Events appear as text is delivered.

The token IDs, probabilities, timings, and event payloads are illustrative. Systems view shows IDs and latency metrics.

Walkthrough notes

What to watch for

Prefill makes the first prediction

Jump to Prefill and step until “Yes,” appears under sampled output. The model has processed the whole prompt and prepared the cache. The next decode pass uses that first token as its input. Splitwise paper

The cache grows on decode

Watch the cache go from four prompt positions to five when the worker processes “Yes,”. Each pass reuses earlier keys and values and adds the position it just processed. How KV caching works

Sampling and delivery are separate

Compare sampled output with client output. This trace sends each visible token in its own event, then closes with a completion marker. Real servers can buffer several tokens into one event. SSE format

Glossary
Token
A piece of text represented by one vocabulary ID. Its boundaries depend on the tokenizer.
Embedding
A learned vector representation looked up from a token ID before model computation.
Prefill
Processing the input sequence together to produce hidden states and initial attention cache entries.
Decode
Autoregressively processing a new token to predict the next one, reusing cached context.
KV Cache
Stores previously computed attention key and value tensors across layers and heads for reuse during decoding.
Hidden state
A contextual vector produced by Transformer layers for a sequence position.
LM Head
A projection from the final hidden state to one logit per vocabulary token.
Logits
Unnormalized scores for possible next tokens, before a sampling rule is applied.
Softmax
A function that normalizes scores into probabilities. Sampling settings can transform the logits first.
Sampling
The procedure that chooses the next token using the configured score distribution.
SSE
Server-Sent Events carry text events from a server to a client over an open HTTP response.
TTFT
Time to first token measures the wait from sending a request to receiving its first output token.
TPOT
Time per output token here is the average time between delivered output tokens, after the first.
Inter-token latency
The elapsed time between consecutive streamed output tokens, including model and transport time.