Memory, long tasks and cost

A worker sends its whole conversation, job description and tool list with every turn. That is what makes it reliable, and also what makes it expensive if you are careless. This chapter covers the levers, in the order to pull them.

1. Prompt caching: the free win

Caching stores the start of your request (job description, tools, earlier turns) so later requests reuse it at about a tenth of the price. The simplest form is one line:

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    cache_control={"type": "ephemeral"},     # cache everything up to the last block
    system=SYSTEM,
    tools=TOOLS,
    messages=messages,
)
print(response.usage.cache_read_input_tokens)   # above zero on the second turn = it works

Caching matches the request byte for byte from the start, in the order tools, then system, then messages. Anything that changes near the top breaks it for everything after:

• No timestamps, request IDs or customer names in the job description.
• The same tools, in the same order, on every request.
• To change instructions mid-conversation, append a system message instead of editing the job description.

If cache_read_input_tokens stays at zero across repeated requests, something near the top is changing.

2. Effort: how hard it thinks

Current models think before they answer. You do not set a thinking budget any more; you set an effort level, from low to max:

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    output_config={"effort": "medium"},   # low | medium | high | xhigh | max
    messages=messages,
)

• low: chat, routing and simple lookups. Fewer tool calls, shorter replies.
• medium: the default on Claude Opus 5.5, and a good start for a support worker.
• high or xhigh: multi-step agent work and coding, where thoroughness pays.
• max: only when being right matters more than time and money.

Measure on real conversations before raising it. Judge the cost per finished job, not per request: a cheap request that needs three retries is not cheap.

3. Long conversations: compaction

A long-running worker eventually fills its context. Compaction lets the API summarise earlier turns automatically when the conversation gets large (by default around 150,000 tokens):

response = client.beta.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    betas=["compact-2026-01-12"],
    context_management={"edits": [{"type": "compact_20260112"}]},
    messages=messages,
)
messages.append({"role": "assistant", "content": response.content})   # the whole content, not just the text

The last line matters: the summary comes back as its own block, and you must keep it in the history. Saving only the text quietly loses it.

A lighter alternative is context editing, which clears old tool results instead of summarising, useful when a worker reads many large documents it no longer needs.

4. Memory between conversations

Compaction works inside one conversation. For things a worker should remember next week (a customer's preferences, what it learned about a recurring problem), use the memory tool. Claude reads and writes files in a /memories folder, and you decide where that folder really lives (your database, a file store):

tools = [{"type": "memory_20250818", "name": "memory"}, ...]

The Python SDK has a helper class for writing the storage side. Never let secrets or card details end up in memory files.

5. Work that does not need an instant answer: batches

Nightly summaries, tagging last month's tickets, drafting replies for staff to review in the morning: send these through the Message Batches API at half the price. Results arrive within a day, in any order, matched by the ID you gave each request.

6. Then, and only then, a smaller model

Once caching and effort are tuned, try the same eval (next chapter) on Sonnet 5.5 or Haiku 5.5. If the scores hold, switch. If one part of the job is simple (sorting messages into categories), give only that part to the smaller model.

A rough cost picture

A support conversation with a two-thousand-word job description, five tools and four turns is typically a few thousand tokens per turn. On Claude Opus 5.5 ($4 per million input tokens, $20 per million output, cached input far cheaper) that is usually a few cents per conversation. Measure your own: the usage field on every response tells you exactly what you were charged for.