AgentLu
← Learn

Efficiency & Optimization

14 min read

Every agent you have built so far works. This chapter is about what it costs - and about the fact that you almost certainly do not know. A pipeline that looks modest can be making thirty calls to answer one question, and nothing in the output will tell you: the answer arrives either way, and the bill arrives later.

Efficiency is not a separate discipline from correctness. It is the same question asked a second time: given that the answer is right, what did it take to produce it? Calls, elapsed time, tokens, and how much of the context was actually used are all measurable, and once they are on screen the obvious waste becomes hard to ignore.

The order of operations matters more than any trick: measure first, then change one thing, then measure again. Optimising what you have not measured is how a pipeline gets faster at answering the wrong question.

Measure before you optimise, and keep quality in the frame

An optimisation is a trade, and a trade needs two numbers. Quality alone tells you the answer is good; cost alone tells you it was cheap. A cheaper approach that answers worse is not an optimisation, it is a different product - so every comparison in this section puts the two side by side, and refuses to show one without the other.

The platform gives you three measurements for free: how many model calls a run made, how many tool calls, and how long it took. Those are facts, not estimates. Token counts are the interesting fourth number, and they are only facts when the model reports them - so when you estimate, say so, in the same place the number appears. An estimate that looks like a measurement is worse than no number at all, because it is trusted.

The habit that follows is short: before changing anything, run the thing and read what it spent. Most of the waste is not subtle - it is a loop that looks up records one at a time, or a prompt with a whole corpus in it - and it becomes obvious the moment the count is on the screen.

Fewer calls: search, do not scan

The most common waste in a tool-using agent is a loop that could be a query. Looking up six students one at a time costs six calls and returns six records; one filtered search costs one call and returns the two that matter. The work is the same, the round trips are not, and the second version is also easier to cache.

Retrieval is the same idea at a larger scale. Sending an entire corpus to the model and letting it find the relevant paragraph works until the corpus grows, at which point it is slow, expensive, and quietly unreliable. Retrieve first, send the shortlist, and the prompt gets smaller while the answer gets better - the same lesson as the Knowledge & Retrieval chapter, arriving from the cost side.

Batching is the third version of it: if the model is asked five nearly identical questions in five calls, one call with five questions in it is usually both cheaper and more consistent. The exception is when the answers need to be independent - a failure in one should not take the others with it - and that is a judgement you make with the numbers in front of you.

Caching, and the one thing that makes it safe

A cache is a promise that the same question has the same answer. That promise is only true until something changes, which is why the interesting part of a cache is not the lookup - it is the key. If the key does not include the version of the thing being cached, the cache cannot tell 'the same answer' from 'the same stale answer'.

So the key carries the version, and a change bumps it. The old entries become unreachable rather than wrong, which is a much better failure mode: nothing has to be invalidated correctly for the answer to be right, and cleaning up is a matter of dropping what is no longer referenced.

The second thing a cache needs is a counter. Hits, misses and invalidations are what turn 'we added a cache' into 'the cache is doing something': in the exercise, one repeated search is one hit and one change is one invalidation, and both are on the card. A cache nobody measures is a cache that quietly stopped working months ago.

Context is a budget, not a container

Every prompt has a size, and every token in it costs money and time - but the cost that bites first is attention. A model given forty records to answer a question about two of them is being asked to ignore thirty-eight, and it will not always succeed. Smaller, better-chosen context is not only cheaper; it is usually more accurate.

A context budget makes that explicit: decide how many records, passages or turns a step may carry, and make the step respect it. When something has to be dropped, drop it deliberately - the oldest turn, the lowest-scoring passage - rather than by letting a truncation happen at the end of a string.

The same discipline applies to the conversation. History is context too, and a chat that carries every turn forever will eventually carry more noise than signal. Keeping the last few turns plus a summary of the rest is a trade you can measure, and the memory exercises are where you build the other half of it.

Comparing two approaches to the register

One question - who is in the netball team, and which tutor group is Maya in - answered twice. The starter only has the expensive version, which is exactly the position most real pipelines are in.

1. Read the baseline before changing it

The starter reads the whole register, looks up every student one at a time, and sends all six records to the model. The card shows one row: seven tool calls and an estimated token count. That row is the thing to beat - not a feeling that it looks slow.

2. Replace the scan with a search

One searchRegister call for the team, one for the named student: two calls instead of seven, and two records in the prompt instead of six. The answer does not change, which is the point: the optimisation is invisible in the output and obvious in the numbers.

3. Cache the repeat, keyed by version

The second search for the same team is a hit, because the key carries the register version as well as the query. The card reports one hit and one miss - and if you had keyed on the query alone, it would have reported the same thing right up until the register changed.

const key = cacheKey({ version: register.version, what: "team", arg: "netball" });
const cached = cacheGet(key);
const roster = cached.hit ? cached.value : await callTool("searchRegister", { team: "netball" });
if (!cached.hit) cacheSet(key, roster);javascript

4. Change something, and read again

Move a student's tutor group, bump the version, invalidate, and search again. The re-read shows the new value - which is the only proof that the cache is not serving a stale answer. This is the step people skip, and it is the one that catches the classic cache bug.

5. Put both rows on one card

Quality, model calls, tool calls, estimated tokens, cache hits: two rows, side by side. The conclusion is not 'the second one is better' in the abstract - it is 'the second one answers the same question with fewer calls and a smaller prompt', which is a sentence you can defend.

6. Be explicit about what is estimated

Calls and elapsed time are measured. The token figures are computed from the length of the text, so the card labels them as estimates. When a hosted model reports real usage, the same row can carry the real number instead - and the label should change with it.

Mistakes worth catching early

What goes wrongWhat it looks likeWhat to do instead
Optimising without measuringA rewrite that feels faster and moves the numbers the wrong way.Read calls and elapsed time before and after, every time.
Cheaper and worseA comparison table with cost columns and no quality column.Measure the answer too. A cheaper wrong answer is not a saving.
A loop that should be a querySix calls to answer one question about six records.Push the filter into the tool and make one call.
Sending the corpusA prompt that grows with the data, and a bill that grows with it.Retrieve a shortlist and send only that.
A cache with no version in the keyAn answer that was right yesterday and is wrong today, with no error anywhere.Key on the version, and prove the invalidation by re-reading.
Estimates dressed as measurementsA token count nobody can reproduce, shown without a label.Label it, or get the number from the provider.

Finding the waste

  • Count the calls first. A run with more tool calls than records returned is usually a loop that should have been a query.
  • Look at the biggest thing in the prompt. If it is data rather than instructions, ask whether the model needed all of it.
  • Check whether the same lookup happens twice in one run. That is a cache waiting to exist.
  • Watch the elapsed time as well as the calls: a single slow call can cost more wall clock than ten fast ones.
  • When a run gets slower after a change, look for a retry loop that now runs more often, or a bigger prompt that was not there before.
  • Compare two configurations on the same input, and change only one thing between them. Otherwise you have a feeling, not a measurement.
  • Keep the quality check in the loop while you optimise. The failure mode of optimisation is not a slower agent, it is a faster one that is quietly wrong.