Most of the prompt isn't you

The stack is the prompt

Chat UIs lie in a useful way. They show a bubble. The client — a website, an IDE agent, a support copilot — assembles something much larger before any GPU is warm.

On the coding-agent path in the panel above, a 31-token bug report rides under roughly 2,200 tokens of other people's decisions: how the agent is allowed to behave, which functions it may emit, a debug-web skill, repo conventions, LoginForm.tsx, api.ts. The ratio of added-to-typed is the whole game. Silicon has to read all of it.

That is why a bigger context window is not a personality upgrade. It is permission to ship more mass per turn — and prefill time, the bill, and the KV cache all grow with every token you include.

Weights stay put

Training produced a tensor file once. Serving loads replicas of that file onto GPUs and leaves them there. A request is JSON over TLS: token ids, not gigabytes of parameters.

A local 32B in 4-bit is the exception that proves the rule. Those quantized weights really can live in your RAM. They are smaller, slower, and private. The frontier pile does not fit, so most “AI” you use is still the 1970s mainframe pattern with better latency: a terminal on your desk, a brain in a locked room.

The panel will skip the public internet hop if you pick the local model. Prefill and decode still happen. They just happen here.

Two jobs, two bottlenecks

Prefill reads the whole packed prompt in parallel and writes a KV cache. It is compute-bound. It is why a long prompt feels slow before a single word appears.

Decode emits one token at a time, reloading weights and the growing cache from HBM for each step. It is memory-bandwidth-bound. That is why output tokens cost more than input, and why batching helps the operator more than it helps you.

If effort is turned up, a private scratchpad runs first — reasoning tokens that usually bill like output and occupy the cache, then never appear as the answer. Effort is not temperature. Temperature is a sampling knob at decode. Effort is a thinking budget.

A tool call is not the model “doing” anything. It is more JSON. Your runtime reads the disk, appends the result as a new packet, and the cycle starts again. An agent session is many LLM cycles sharing a growing prompt.

The knobs are all mass

Every optimization people argue about is a decision about which tokens enter the window:

• RAG fetches a few passages instead of shipping the library. Wrong chunks, confident wrong answer.
• A knowledge base is the always-on cousin — cheap insurance at a paragraph, expensive noise at an encyclopedia.
• Skills load instruction packs only when the task matches, so the system prompt does not have to cover every job on earth.
• Files are literal bytes. Highest fidelity, highest cost. Attach eighty lines, not the whole repo.
• Compaction summarizes old turns so you can keep talking. It is lossy on purpose. On the long-thread scenario, aggressive compaction drops a timezone you already stated, and the recap comes back in UTC.
• Prefix cache is the quiet one. A stable system prompt plus tools plus skills can reuse their KV from the last turn, billed cheaper, with less prefill. That is why a huge but static agent preamble is expensive on turn one and merely a habit by turn twelve.

None of these replace the context window. They choose what is in it.

The numbers in the panel are a teaching simulation: token counts are character estimates, cloud prices match public xAI rates, KV-cache megabytes are illustrative. The topology is the part to keep.