Cutting an agent's turn from 150k to 10k tokens

The obvious way to make a slow agent faster is to make the hardware faster. The cheaper way is to stop handing it work that did not need to exist.

A turn’s cost is roughly tokens processed × cost per token. Halving the context halves the work whatever the hardware does, and it is the only one of the two factors you fully control. So before touching the runtime I went after the tokens: context per turn came down from roughly 150k to roughly 10k, with no loss in answer quality.

No single change did that. It was a set of small cuts, and this article shows one of them in full — about 35k tokens of tool schemas that almost nothing ever called — because it is the one with a clean measurement and a simple fix. The shape of it repeats everywhere else.

This is the half that came first. The runtime tuning that followed is a separate article, and cutting the context was worth more than all six of its stages on the number a person actually feels.

Everything below runs on an NVIDIA DGX Spark (GB10) serving a 35B sparse MoE through llama.cpp; the machine and the workload are on the About page. What matters for this article is the ratio: a turn reads 9–17 prompt tokens for every token it generates. Expect different absolute numbers on other hardware — the shape is what transfers.

A turn has two waits, and only one of them is visible

Both halves of a turn’s cost move with context length, and they move differently.

Generation slows gradually:

Decode rate against context depth. llama-bench, Qwen3.6-35B-A3B at f16 KV, tg128, -r 2, ±0.50 or better.

Time before the first token does not slow gradually. It runs away, because the whole prompt has to be read before anything can be written, and the rate at which it is read drops as it gets longer:

The same model and settings, prompt-processing rate turned into wall-clock. Measured pp rates: 2,389 t/s at 4k, 2,288 at 16k, 1,997 at 64k, 1,647 at 150k, ±20 or better.

Over the same range decode loses about half its rate. Time to first token goes from under two seconds to more than a minute and a half — better than fifty times worse. One of these is a curve; the other is a cliff.

Five seconds is not folklore here, it is the reporting interval

The agent buys patience by showing its work: its Discord presence updates as it picks a tool, calls it, reads the result back. A person will wait a long time while something is visibly happening, and gives up quickly when nothing is. Cancellations come from silence rather than from slowness.

That is what makes the second curve worse than it looks. Decode is reportable — tokens are arriving, rounds are completing, the presence has something to say and keeps saying it. Prefill is dead air. No token exists yet, no tool has been called, nothing has happened that can be described, and the only honest thing to display is a spinner.

Five seconds of patience buys about eleven thousand tokens of context on this machine. A turn that fits in ten thousand answers before anyone gives up on it. A turn carrying a hundred and fifty thousand has lost the room before it begins — ninety seconds of spinner is not a slow answer, it is an answer nobody stayed for.

That is the whole of the target. Not a token budget chosen for tidiness: the largest prompt that still starts talking inside one reporting interval.

The automated half of the workload has no reporting interval and arrives at the same discipline anyway. Nobody watches an event-driven turn. Somebody is still waiting on the deployment and the automated smoke-test results behind it, and that answer lands at the end of a chain of orchestrated steps rather than at a first token — each step’s prefill compounding into one total that is invisible until it is over.

And those turns share the runtime with the watched ones. A slot occupied by a hundred-and-fifty-thousand-token prompt is a slot the next event queues behind, so an automated turn’s context is charged twice: once to itself, and once to everything waiting on it. Human patience and machine throughput turn out to be two arguments for the same budget.

The clearest example: the tool catalogue

The reduction came from a dozen places, none of them dramatic. The agent’s identity and formatting instructions. What it is allowed to remember, and for how long. The event rules and the signal selectors that decide what an automated turn is even shown. The conversation history policy. Each was worth a few thousand tokens; each took a while to get right; none of them is interesting on its own, and listing them all would be a different and worse article.

The tool catalogue is the one worth showing in full. It is the largest single item, it has a clean measurement, and the fix turned out to be simple — which makes it the best illustration of the thing they all have in common: a cost that is invisible until someone goes looking for it.

What it was not, in any of these cases, was the conversation.

A single real turn, question “what are you?”, measured against the deployed model’s own tokenizer:

95% of that prompt was schemas. The question was four words. That ratio is one prompt on one day, not a constant — but every turn that day paid some version of it.

Four MCP servers had been connected over time, each one individually reasonable:

Two servers were not reachable for direct tokenization; their share is the residual.

Against 137 recorded sessions carrying 603 tool calls across 37 distinct tools — and 348 of those 603 calls were three tools. A hundred and sixteen declared tools were never called once, and every one of them was charged on every round of every turn.

This is the trap, and it is structural rather than careless. Connecting an MCP server is one line of configuration, and the cost lands somewhere nobody is looking — not in a log, not in a latency graph, but spread evenly across every future prompt. The server that cost 18,119 tokens a turn was doing its job correctly.

The fix follows from the numbers: declare a small hot set and put the rest behind a lookup the model can call when it needs something outside it. Replaying the corpus against candidate hot sets says where to stop:

Twenty tools covers nine sessions in ten outright, and costs the tenth about a tenth of a round. That is what runs in production now.

The cache will not save you

The obvious objection is that none of this should matter, because a prompt cache means the prefill is paid once.

It pays within a turn and mostly not across turns. Later rounds of one turn resend a byte-identical prefix and prefill only the newly appended tool result, which is why round one is the expensive one and the rest are not. But automation assembles a different prompt for every event, so there is rarely a prefix to reuse; a chat session stays warm only briefly, and the next question may have nothing to do with the last.

Keeping sessions warm long enough to help means carrying one question’s context into the next, which trades prefill time for the risk of answering the wrong thing. Most turns open cold and pay round one in full.

There is a second reason to care, and it is the one that bites quietly. Prefix matching ends at the first differing byte, so anything that varies per turn and sits near the front of the prompt invalidates everything behind it — and a tool catalogue is exactly the kind of thing that gets rewritten in place.

Lessons learned

Cut the context before tuning the machine. Tuning first means optimising the hardware to process tokens that did not need to exist. The order is not a preference: context reduction is multiplicative with everything the runtime does afterwards, and nothing the runtime does afterwards reduces the token count.

Expect a dozen cuts, not one. The reduction came from half a dozen unrelated places, none of them dramatic, and the largest was about a third of it. Looking for the single big win would have found nothing.

Measure what is in the prompt before assuming it is the conversation. In the prompt measured here it was 95% schemas and 5% conversation, and no instrument on the box reported that. A turn’s token count is visible; its composition is not, and only the composition tells you what to cut.

An MCP server’s cost is not paid where you connect it. It is one configuration line, and it is charged on every round of every turn from then on, whether or not anything calls it. Connecting is cheap and reversible; the bill is neither.

Declare the tools that get used and reach the rest through a lookup. Most of a declared catalogue is never called, and what is called concentrates hard. A small hot set with a lookup behind it covers nearly every session and costs a fraction of a round on the rare miss.

Time to first token is the number a person feels, and it is the one that runs away. Over the same context range decode loses half its rate while the wait before anything appears gets more than fifty times worse. Tokens per second is the figure everyone quotes and the milder of the two.

Pick the budget from the workload, not from the model. Ten thousand tokens is what fits inside one reporting interval on this hardware, and the automated half of the workload lands on the same number for its own reason. Change the machine or the interface and the number moves. The model has nothing to do with it.