Be wary of ephemeral context in AI agents
The cost and quality implications of ephemeral context injection versus Claude Code-style append-only context management.
Suppose you’re working on an AI system that lives in an IDE. In the IDE, the user may have some set of files open, and furthermore, some of those files may have active changes in the buffer. Wouldn’t it feel nice if the agent “just knew” which files the user was looking at? How could that work?
One approach is “ephemeral context” where, along with the user message, you provide some text that’s invisible to the user that tells the model that information.
API call 1
[System prompt]
<open_files>
- app.R- test-app.R (modified)
</open_files>
Could you refactor the handler in the test app?
Sure, but I see you have local changes--do you want to save those first?
API call 2
[System prompt]
Could you refactor the handler in the test app?
Sure, but I see you have local changes--do you want to save those first?
<open_files>
- app.R- test-app.R
</open_files>
Yup, go ahead
I'll read test-app.R and refactor the handler.
When developing AI agents, the idea of ephemeral context can be quite tempting. Ultimately, it seems like it should keep the conversation history shorter and help the agent feel more “integrated” with the context it needs.
In practice, though, agents that lean heavily on ephemeral context are more expensive and lower-quality than agents with append-only conversation history, which use tools to fetch the context they need (and persist those tool outputs in the conversation history). VS Code Copilot Chat used to engineer context using this ephemeral model, but doesn’t anymore. Positron Assistant manages context around context.history directly from the VS Code core chat API, which does strip tool calls and responses. I’ll contrast agents that lean heavily on ephemeral context with Claude Code, which uses an append-only approach. Claude Code is a proxy for Posit Assistant, an agent I’ve spent a good bit of time working on.
Cost
Let’s start on the cost side. In LLM context windows, there are two kinds of input tokens; cache reads (or “cached input”) and what I’ll call normal (or “uncached”) input tokens. Cache reads are input tokens that the model has seen before. Once the model has “read” a conversation history once, if you pass the conversation history again and it hits the cache, it’s as if the model has “read” all of that history already. Broadly, cache reads are less computationally expensive than “normal” input tokens. This difference in computational cost is reflected in many major model providers’ API pricing; cache reads are often priced at a 5x, 10x, or even 100x discount relative to normal input tokens.
| Model | Input | Cache read | Discount |
|---|---|---|---|
| GPT 5.5 | $5.00 / MTok | $0.50 / MTok | 10x |
| Claude Sonnet 4.6 | $3.00 / MTok | $0.30 / MTok | 10x |
| Gemini 3.1 Pro | $2.00 / MTok | $0.20 / MTok | 10x |
| DeepSeek V4 Pro | $1.74 / MTok | $0.0145 / MTok | 120x |
This means that agents with contexts engineered to maximize cache hit rate are more cost-effective.
The most cache-hit-rate optimizing harness is append-only. For example, let’s take a coding agent with a 10,000 token system prompt.1 A user types in some question, perhaps containing 100 tokens. In that first request, 10,000 of the tokens are cache hits, priced at a tenth of the price of normal input tokens. Then, the 100 tokens typed out by the user are ten times more expensive than cache reads. 2 Finally, the model will “respond” with some number of output tokens—let’s say 100—which are typically 5-8x more expensive than input tokens.
Because of those cost differences—5:1:0.1—those 100 output tokens cost 31% of the total cost of the request even though they only accounted for 1% of the tokens in the request. The uncached input tokens, similarly, are 1% of the tokens but 6% of the cost.
Now, in reality, agents are useful because they can do stuff by calling tools. This means that, in response to your 100-token user message, the 100 output tokens likely called a tool. The tool call result will kick off a “user message” automatically with some number of input tokens—say, another 100. In this append-only context, during the second turn, the 100 uncached input tokens from the first turn are now cached. The 100 output tokens from the first turn are now 100 uncached input tokens in the second turn, and the 100 tokens from the tool call result are 100 uncached input tokens as well. What this looks like visually:
Alright, on and on and on. If the user submits 4 messages, and each results in 4 tool calls at 100 tokens in and out, it costs $0.11 total:
Okay, now, I started off yapping about ephemeral context. How would that work?
Let’s say that, by including 1000 tokens of additional context on the messages that the user submits, the agent will only need to make 3 tool calls instead of 4 half of the time; it just knows what file to look at in the first place.3 In that case, the user message is 1100 tokens instead of 100. The request containing that specific user message will be charged for 1000 extra input tokens (and then 1000 extra cache hits in the following tool call/response loop). In Positron Assistant, the “ephemeral” context sticks around until the agent has completed however many tool calls it will make and ultimately responds to the user. Only at that point is the ephemeral context cleared.
There’s also another complicating factor, which is the way that Positron Assistant manages tool call histories. In this system, tool calls and responses do not persist beyond the next user message. For example, I might type some user message. Along with my user message, a bunch of context about my session state is passed along. The agent then might make, say, 4 tool calls. So, the outbound messages look something like:
When the user sends a second message, all of the tool calls and responses from the preceding message, in addition to the automatically-injected ephemeral context, are cleared from the history. So, the next outbound message would look like:
This prevents the conversation length from growing as quickly as it does in a Claude Code style, append-only system.4 At the same time, the agent regularly loses access to information it might need. The practical effect of this is that, if I ask some question, get an answer, and then have a follow-up that requires access to the same information, the agent either has to go and retrieve the same information again or try to “eyeball” a response to the question based on what it said previously. The former is an issue because it means output tokens and uncached input tokens, which cost orders of magnitude more than cached input tokens, are duplicated. The latter is an issue because that eyeballed response is likely a hallucination. It’s a lose-lose.
So, by clearing ephemeral injected context and tool calls + responses when the next user message is submitted, we’re keeping the conversation history shorter. Further, by including this ephemeral, injected context, we might save 0.5 tool calls (+ responses) per user message. At the same time, clearing the tool call history means that the agent will likely need to make more tool calls in response to the next user message to recover context it lost along the way, incurring more output tokens and uncached input tokens. Let’s generously say that the agent will only need to make 0.5 more tool calls per user message, canceling out the win of the injected context. Even then, this shifts the distribution of costs significantly:
In this conversation, with the same number of user messages and presumably same result, the conversation with the ephemeral context-based agent cost $0.12—8.5% more than the append-only agent.
Now, the calculus here really depends on a ton of factors: system prompt length, ephemeral context length, whether the ephemeral context persists through the tool calling loop, the typical number of turns the ephemeral context saves, etc. You can slide some of the numbers here and see how the distributions shift:
#| '!! shinylive warning !!': |
#| shinylive does not work in self-contained HTML documents.
#| Please set `embed-resources: false` in your metadata.
#| label: shinylive-calculator
#| standalone: true
#| viewerHeight: 550
library(shiny)
library(bslib)
compute_append_only_cost <- function(system_prompt, num_messages,
tool_calls_per_msg, tool_result_length,
user_msg_length = 100,
output_length = 100,
cache_discount = 10) {
cache_token_cost <- 3 / cache_discount
tool_calls <- distribute_calls(num_messages, tool_calls_per_msg)
total_cost <- 0
total_input_prev <- 0
is_first <- TRUE
for (m in seq_len(num_messages)) {
turn_inputs <- c(user_msg_length, rep(tool_result_length, tool_calls[m]))
for (turn_input in turn_inputs) {
if (is_first) {
cache <- system_prompt
uncached <- turn_input
is_first <- FALSE
} else {
cache <- total_input_prev
uncached <- output_length + turn_input
}
total_input_prev <- cache + uncached
total_cost <- total_cost +
(cache * cache_token_cost + uncached * 3 + output_length * 15) / 1e6
}
}
total_cost
}
compute_ephemeral_cost <- function(system_prompt, num_messages,
tool_calls_per_msg, tool_result_length,
user_msg_length = 100, output_length = 100,
ephemeral_length = 0,
context_saves_per_msg = 0,
recovery_calls_per_msg = 0,
cache_discount = 10) {
cache_token_cost <- 3 / cache_discount
effective_calls <- tool_calls_per_msg - context_saves_per_msg +
recovery_calls_per_msg
tool_calls <- distribute_calls(num_messages, effective_calls)
total_cost <- 0
total_input_prev <- 0
persisted_input_prev <- 0
for (m in seq_len(num_messages)) {
turn_inputs <- c(user_msg_length, rep(tool_result_length, tool_calls[m]))
for (j in seq_along(turn_inputs)) {
is_user_request <- j == 1
if (m == 1 && is_user_request) {
cache <- system_prompt
uncached <- turn_inputs[j]
} else if (is_user_request) {
cache <- persisted_input_prev
uncached <- output_length + turn_inputs[j]
} else {
cache <- total_input_prev
uncached <- output_length + turn_inputs[j]
}
ephemeral_rate <- if (is_user_request) 3 else cache_token_cost
total_input_prev <- cache + uncached
if (is_user_request) {
persisted_input_prev <- total_input_prev
}
total_cost <- total_cost +
(cache * cache_token_cost + uncached * 3 +
ephemeral_length * ephemeral_rate + output_length * 15) / 1e6
}
}
total_cost
}
distribute_calls <- function(num_messages, calls_per_msg) {
calls_per_msg <- max(calls_per_msg, 0)
floor_calls <- floor(calls_per_msg)
msgs_with_extra_call <- round((calls_per_msg - floor_calls) * num_messages)
floor_calls + as.integer(seq_len(num_messages) <= msgs_with_extra_call)
}
ui <- fluidPage(
tags$head(tags$style(HTML("
.container-fluid { padding-top: 1.5em; }
@import url('https://fonts.googleapis.com/css2?family=Source+Sans+3:wght@300;400;600&display=swap');
@import url('https://fonts.googleapis.com/css2?family=Fraunces:opsz,wght@9..144,200&display=swap');
body { background-color: #fdfdfb; font-family: 'Source Sans 3', sans-serif;
color: #427059; font-weight: 350; }
.form-group label, .control-label { font-weight: 400; color: #427059; }
.checkbox label { font-weight: 350; color: #427059; }
hr { border-color: #DDDDCF; }
.irs--shiny .irs-bar, .irs--shiny .irs-from, .irs--shiny .irs-to,
.irs--shiny .irs-single { background: #5D987B; border-color: #5D987B; }
.irs--shiny .irs-handle { border-color: #5D987B; }
.pct-number { font-family: Fraunces, serif; color: #427059; }
"))),
fluidRow(
column(6,
sliderInput("system_prompt", "System prompt length",
min = 1000, max = 30000, value = 10000, step = 1000,
width = "100%"),
sliderInput("num_messages", "User messages",
min = 1, max = 10, value = 4, step = 1,
width = "100%"),
sliderInput("tool_calls", "Tool calls per user message",
min = 1, max = 10, value = 4, step = 1,
width = "100%"),
sliderInput("tool_result_length", "Tool result length",
min = 50, max = 5000, value = 100, step = 50,
width = "100%")
),
column(6,
div(
style = "text-align:center; padding:2.5em 1em; margin-top:1em; margin-bottom:1em; background:#ffffff; border-radius:6px; box-shadow:0 1px 2px rgba(0,0,0,0.04); display:flex; flex-direction:column; justify-content:center; min-height:320px;",
div(style = "color:#427059; font-size:2rem; font-weight:300; line-height:1.2;",
"Ephemeral context is",
br(),
span(class = "pct-number",
style = "font-size:7rem; font-weight:200; line-height:1;",
textOutput("pct_value", inline = TRUE)),
br(),
textOutput("pct_direction", inline = TRUE)),
div(style = "color:#427059; font-size:1.9rem; font-weight:350; margin-top:1.1em;",
textOutput("dollar_amounts", inline = TRUE))
)
)
),
div(style = "margin-top: 1.5em;"),
fluidRow(
column(3,
sliderInput("cache_discount", "Relative discount for cached tokens",
min = 5, max = 120, value = 10, step = 1,
width = "100%")
),
column(3,
sliderInput("ephemeral_length", HTML("Ephemeral context length<br> "),
min = 0, max = 10000, value = 1000, step = 100,
width = "100%")
),
column(3,
sliderInput("saves", "Tool calls saved per message",
min = 0, max = 2, value = 0.5, step = 0.05,
width = "100%")
),
column(3,
sliderInput("recovery_calls", "Recovery calls per message",
min = 0, max = 2, value = 0.5, step = 0.05,
width = "100%")
)
)
)
server <- function(input, output, session) {
costs <- reactive({
saves <- min(input$saves, input$tool_calls)
recovery_calls <- input$recovery_calls
cost_ao <- compute_append_only_cost(
system_prompt = input$system_prompt,
num_messages = input$num_messages,
tool_calls_per_msg = input$tool_calls,
tool_result_length = input$tool_result_length,
cache_discount = input$cache_discount
)
cost_ephemeral <- compute_ephemeral_cost(
system_prompt = input$system_prompt,
num_messages = input$num_messages,
tool_calls_per_msg = input$tool_calls,
tool_result_length = input$tool_result_length,
ephemeral_length = input$ephemeral_length,
context_saves_per_msg = saves,
recovery_calls_per_msg = recovery_calls,
cache_discount = input$cache_discount
)
list(
append_only = cost_ao,
ephemeral = cost_ephemeral,
pct = (cost_ephemeral / cost_ao - 1) * 100
)
})
output$pct_value <- renderText({
sprintf("%.1f%%", abs(costs()$pct))
})
output$pct_direction <- renderText({
ifelse(costs()$pct > 0, "more expensive", "cheaper")
})
output$dollar_amounts <- renderText({
sprintf(
"Ephemeral: $%.3f | Claude Code: $%.3f",
costs()$ephemeral,
costs()$append_only
)
})
}
shinyApp(ui, server)Quality
Up to this point, I’ve only focused on the costs. Because different kinds of tokens are priced differently, and because ephemeral context results in a distribution of token usage that leans more heavily toward the expensive kind, ephemeral context isn’t really an effective money-saving strategy. Beyond that, though, ephemeral context creates several quality risks:
- Ephemeral context introduces an unintuitive short-term memory model. The agent forgetting its most recent explorations requires the model to “one-shot” complete responses, dissuading follow-up questions and a tighter interaction loop.
- Ephemeral context results in an incoherent conversation history. Removing context results in a conversation history that demonstrates the agent coming up with something out of nothing; the context doesn’t include the evidence that the agent reacted to.
- The agent should be focused on what the user said. Agents are already prone to “losing the thread” when working on tasks. Allotting the vast majority of input tokens to relatively low information-density context worsens this.
Short-term memory
In response to a message, an agent using ephemeral context might fire off a number of tool calls before responding with assistant text. When the user submits a follow-up message, all that the model can see is the previous user messages and assistant responses; it can’t see the evidence that justified its responses.
For one, this is just a weird shape for a thing that a human will interact with. It’s quite difficult to intuit what the agent “remembers” even once you’ve internalized this.
Once you have internalized this, I’ve felt that it discourages steering the agent actively and maintaining a tighter back-and-forth with the model. If I know that the agent will have to go and re-acquire the same information I already paid once when I ask a follow-up, I’d probably prefer that I make sure the agent instead does everything in one pass and answers all of my questions at once. I noticed this in Yihui Xie’s recent post on his interactions with AI coding agents, where he writes “With each request, I often ask Copilot to do several things.” This workflow of waiting for the agent to do everything at once and then trying to understand and critique the whole trajectory is, to me, pretty boring and disempowering. At the least, it does not feel like flow state.
Incoherent histories
Let’s revisit that earlier conversation history:
API call 1
[System prompt]
<open_files>
- app.R- test-app.R (modified)
</open_files>
Could you refactor the handler in the test app?
Sure, but I see you have local changes--do you want to save those first?
API call 2
[System prompt]
Could you refactor the handler in the test app?
Sure, but I see you have local changes--do you want to save those first?
<open_files>
- app.R- test-app.R
</open_files>
Yup, go ahead
I'll read test-app.R and refactor the handler.
In the first thread, the user message includes some automatically-injected, ephemeral <open_files> tags. The agent responds to the user (and, partially, to the information in the tags). Then, in the second message, the user sends a follow-up. This time, the original <open_files> tag is gone, and an updated one is placed on the most recent message.
Reading the conversation history in that second message must be a weird experience for the LLM.5 ‘Why did I say “I see you have local changes?” The user didn’t tell me that.’ This context management results in the agent seeing it has made things up repeatedly over the course of the conversation.6 It’s possible to do this in such a way that the model is still notified that something used to be there–Anthropic addresses this by inserting some placeholder text–but this is not how these systems are designed.
Focus
Even without unrelated context being injected into conversations, agents are already prone to losing sight of the big-picture when working on tasks. Injecting relatively low-information-density context into the conversation reduces response quality and degrades the feeling of “focus.”
Personally, it drives me bonkers when an agent says “I see you have XYZ file open in the editor…” or “Based on the variables in your Python session” as it responds to my question. I provided a collection of context with my question that I’ve intended will steer the agent towards a trajectory I want, and I want the agent to intuit as much as it can about the intent of the context I provided in order to answer my question. The editor is mine. I’m using the editor to do (often unrelated) human things and go down my own rabbit holes. Agents that have this automatic context-injection feel unfocused or distractable compared to Claude Code-style agents.
Conclusion
Rules of thumb:
- Provide agents with tools that allow them to acquire information they need, and persist those tool calls and responses in the history. This should be the main method by which agents acquire context.
- Only inject context when it is brief and high-signal. When you do, keep the injected context in the conversation history or, when removing it, include a placeholder that notifies the agent something used to be there.
Thanks to Winston Chang for the discussion on the ideas in this post.
Footnotes
We’ll assume that, since the agent is multi-user, this system prompt is always a cache hit.↩︎
The final bit of complexity here is the cost to write to the cache. For example, Anthropic offers the guarantee that, if you pay an extra $3.75 / MTok on your input tokens, if you then send the same input tokens in the next 5 minutes, they will be billed as cache reads. Some providers support this and some don’t. To keep things simple, I’ll leave that out here.↩︎
Note that 1000 is much greater than 100, one tool call result’s worth of tokens. This feels about right, as the tool call presumably includes exactly the context that the agent needs. The “auto-injected” context, in contrast, is just throwing a superset of the context one thinks the agent might need and hoping you’ll get lucky. I’ll let you turn the knobs in a bit.↩︎
At the time when this sort of system was designed, typical LLM context lengths were much shorter–often on the order of a few tens of thousands of tokens–so this sort of aggressive compaction was necessary.↩︎
Neural networks are not humans. This is a useful analogy, for these purposes.↩︎
My reflex, developed over the course of building agents and using others’, is that these ghost references lead to greater hallucination rates as the conversation gets longer. To try and make this case, I put together an eval to measure the effects of various context management strategies, but was not able to show a change in hallucination rate in practice. Maybe it’s fine.↩︎