Context Engineering: Why Your AI Agent Forgets, Drifts and Fails on Long Tasks

Here's a bug report we've seen in some form on almost every agent project. "It works great for the first ten minutes, then it starts acting strange."

It asks the user for information they already gave. It re-runs a tool call it ran an hour ago. It quietly drops a constraint from the original request. Nothing crashes. There's no error to grep for. The agent just gets a little worse with every step, like a colleague who's been in back-to-back meetings since morning.

The instinct is to fix the prompt. Sometimes that helps. More often the prompt was never the problem. What the model sees at step 80 is simply not what it saw at step 1, and nobody designed that. That's the territory of context engineering.

What Is Context Engineering?

Context engineering is the practice of deciding what information goes into a language model's context window at each step, in what form and in what order, so the model has what it needs to act correctly and nothing that gets in the way.

The term caught on in 2025, as teams building agents found that clever wording mattered less than curation. Anthropic's engineering team framed the goal well in its September 2025 write-up: find the smallest set of high-signal tokens that maximizes the likelihood of the outcome you want. That's a good test to carry around. Every token in the window should be earning its place.

Key Takeaways

• Prompt engineering is about wording one instruction well. Context engineering is about managing everything the model sees across a whole run: instructions, tools, memory, retrieved documents, and tool results.
• Agents degrade on long tasks mostly because their context accumulates noise, not because the model "gets tired."
• Research on long inputs shows performance can vary with input length and where information sits, even on simple tasks. A bigger window is not the same as better memory.
• Tool results are the most common source of bloat. Trim them at the source.
• Memory needs rules: what gets written, how it's stamped, when it expires, and how it's retrieved.
• Compaction, structured notes, just-in-time retrieval, and sub-agents are the main tools for long-horizon work. Each has a failure mode.
• You can't fix what you can't see. Log the exact context assembled at each step.

Prompt Engineering vs. Context Engineering

Think of prompt engineering as writing a good brief for a contractor. Context engineering is running their whole workday: what's on their desk, which files they can reach, what they wrote down yesterday, and what's been cleared away so they can focus.

	Prompt engineering	Context engineering
Unit of work	One instruction or system prompt	The full input at every step
Main question	"How should I word this?"	"What should the model see right now?"
Time horizon	Single call	Entire multi-step run, across sessions
Inputs managed	Instructions, examples	Instructions, tools, memory, retrieval, tool outputs, history
Typical failure	Misunderstood request	Drift, forgetting, bloat, contradictions
Fixed by	Rewriting text	Designing what gets in, stays, and leaves

They're not rivals. A well-written system prompt is one component of the context. It's just no longer the only one that matters.

What Actually Fills the Window

It helps to see the window as a budget with competing claims. In a typical agent run, these all compete for the same finite space:

+--------------------------------------------------+
| System prompt and rules          (stable)        |
| Tool definitions                 (stable-ish)    |
| Memory and task state            (managed)       |
| Conversation history             (grows)         |
| Retrieved documents              (variable)      |
| Tool results                     (grows fast)    |
| Latest user message / current step               |
+--------------------------------------------------+

In short runs this doesn't matter. In long ones, the bottom half crowds out the top. Usually it's tool results and history that take over, while the instructions that matter most are buried under thousands of tokens of logs.

A useful analogy is a desk. At the start of the day it's clear: your instructions, a notebook, a pen. By mid-afternoon it's covered in printouts you consulted once and never filed. You can still find things, mostly. But you'll miss the sticky note that said "don't email the client before legal signs off."

Why Agents Degrade on Long Tasks

Four things tend to be going on at once.

Bigger windows aren't a free lunch. Chroma's 2025 research on what it called "context rot" tested 18 models and found performance doesn't hold uniformly as inputs get longer, even on simple tasks, and that irrelevant content makes it worse. Earlier academic work ("Lost in the Middle," published in 2024) found models often use information at the start and end of a long input better than information buried in the middle. Models differ and keep improving, so don't treat any specific result as a law. But the direction is consistent enough to design around: more context is not automatically better context.

Noise accumulates. Every failed attempt, redundant tool output, and abandoned line of reasoning stays in the history unless you remove it. The agent then has to reason in the presence of its own clutter.

Goals get diluted. The original request was one message among hundreds. The agent starts optimizing for whatever it was doing most recently, not what it was asked to do.

Summaries lose detail. When history gets compressed, the compression decides what matters. If it guesses wrong, the agent loses information it needs, and it won't know it lost it.

None of this is exotic. It's the same reason a long email thread is harder to act on than a one-page summary.

Failure Examples and Fixes

This is the practical core. Each of these is a pattern we've seen repeatedly, along with what tends to fix it.

1. The agent forgets a constraint from the original request

What happens: The user says at the start, "Only use data from the last 12 months and don't contact customers directly." Forty steps later, the agent pulls three-year-old figures and drafts an outreach email.

Why: The constraint is a few lines at the top of a very long history. Newer content dominates.

Fixes:

• Extract hard constraints into a structured task state (goal, constraints, decisions made, open questions) that is rebuilt and re-presented every step, near the end of the context where it's hard to miss.
• Keep the system prompt short and unambiguous. Long prompts bury their own rules.
• Enforce critical rules in code (validation, permissions), not only in text. If an agent must never email customers, remove the tool.

2. Tool results flood the window

What happens: The agent calls a "get orders" tool and receives 400 records with 30 fields each. It needed three of them. Every later step carries that payload.

Fixes:

• Design tools to return only what's needed: filtered, paginated, with a sensible default limit and the fields the task actually uses.
• Return concise identifiers and let the agent fetch details on demand.
• After a result has been used, replace it with a short stub ("Looked up 400 orders for Account 12; 3 overdue, IDs in notes"). Anthropic's context management guidance lists clearing stale tool results as a simple, low-risk option when results can be re-fetched.

3. Retrieval brings the wrong stuff (or too much)

What happens: A RAG step injects ten chunks, four of which are near-duplicates, two outdated, and one irrelevant but keyword-similar. The agent blends them into a confident wrong answer.

Fixes:

• Retrieve fewer, better chunks. Rerank, deduplicate, and apply metadata filters (date, document type, access level).
• Include source and date with each chunk so the model can weigh conflicts.
• Prefer just-in-time retrieval for large corpora: give the agent search tools and let it pull what it needs when it needs it, rather than preloading everything "just in case."
• Evaluate retrieval separately from generation. If retrieval is wrong, no prompt will rescue the answer.

4. Memory goes stale or contradicts itself

What happens: The agent saved "customer prefers email" in March. In June the customer asked to switch to WhatsApp. Both notes exist. The agent picks whichever it retrieves first.

Fixes:

• Stamp every memory with a timestamp and source.
• Define write rules: what's worth remembering, who or what can write it, and what overrides what.
• Prefer updating or superseding a memory over appending a conflicting one.
• Add expiry for facts that decay, and a review path for important ones.
• Keep user-specific and organization-wide memory separate, with clear access control.

5. The agent drifts away from the goal

What happens: Asked to produce a vendor comparison, the agent spends most of its run perfecting one vendor's pricing table and never gets to the others.

Fixes:

• Maintain an explicit plan and progress file that the agent updates and rereads: what's done, what's next, what's blocked.
• Re-ground periodically by restating the original objective and checking progress against it.
• Put step and budget limits in place so wandering has a ceiling.

6. Compaction drops the one detail that mattered

What happens: At 80% of the window, history is summarized. The summary captures the gist but omits that "the customer's contract renews on the 15th, so don't offer a discount before then." Three steps later, the agent offers a discount.

Fixes:

• Anthropic's guidance on compaction is blunt about the trade: what you keep versus discard is an art, and overly aggressive compaction can lose subtle details whose importance only appears later.
• Tune the summarization prompt on real agent traces, not toy examples. Start by maximizing recall, then trim for precision.
• Maintain a must-keep list (constraints, decisions, IDs, commitments) that is preserved verbatim through compaction.
• Store the full raw log outside the context, so the agent can look something up if the summary falls short.
• Skip compaction if sessions rarely get long. It adds lossiness with no benefit.

7. Too many tools, too much overlap

What happens: The agent has 40 tool definitions. Two of them sound alike. It picks the wrong one a third of the time on edge cases.

Fixes:

• Give each agent or step only the tools it needs.
• Write tool descriptions as if for a new hire: what it does, what it doesn't, when to use it.
• Remove tools nobody uses. Every definition costs tokens on every step.

8. Everything in one context

What happens: One agent researches, drafts, and checks, carrying all the intermediate material through the whole run.

Fixes:

• Hand focused sub-tasks to sub-agents with clean contexts that return a condensed result. This is one of the three long-horizon techniques Anthropic describes, alongside compaction and structured note-taking.
• Be selective. Sub-agents multiply cost and add handoffs. Use them when the intermediate work is large and the result is small.

Memory: Three Layers, Three Jobs

"Memory" gets used for several different things, which causes confusion. We find it helps to separate them.

Layer	Analogy	What it holds	Typical implementation
In-context (working)	What's on your desk	The current step's inputs	The context window itself
Session state (notes)	Shift handover notes	Goal, plan, decisions, open questions	A structured file or object the agent updates
Long-term	The filing cabinet	Durable facts and preferences across sessions	A store with retrieval, timestamps, and rules

Structured note-taking deserves a special mention. Having the agent write progress notes outside the window, then read them back after a context reset, is a simple technique that's remarkably effective for multi-hour tasks. It's the same reason hospitals do formal shift handovers instead of hoping the next nurse remembers.

Retrieval: Librarian, Not Landfill

There are two broad styles, and mature systems use both.

Up-front retrieval pulls relevant material before the model responds. It's fast and suits well-defined questions, like "what's our returns policy?"

Just-in-time retrieval gives the agent search and lookup tools so it can explore, holding lightweight references (file paths, record IDs, queries) and loading details only when needed. It's more flexible for open-ended work, at the cost of extra steps and latency.

The test for either: does the agent receive what it needs and little else? A good librarian hands you the three relevant books. A bad one wheels in the whole archive.

Managing the Window: A Practical Pattern

Here's a sketch of how context assembly can look in code. It's illustrative, not a framework.

def build_context(task_state, history, retrieved, user_msg, budget=60_000):
    parts = [
        SYSTEM_RULES,                                   # short, stable, first
        tools_for(task_state.current_step),             # only what this step needs
        render(task_state),                             # goal, constraints, plan, decisions
        stub_old_results(history, keep_last=3),         # older tool results become one-liners
        top_k(rerank(dedupe(retrieved)), k=4),          # few, ranked, dated, sourced
        user_msg,
    ]
    ctx = assemble(parts)
    if count_tokens(ctx) > budget:
        ctx = compact(ctx, preserve=task_state.must_keep)
    return ctx

A few design notes:

• Put stable content first. Rules and tool definitions that rarely change sit at the front, which also tends to work well with provider-side prompt caching, since caching generally relies on a stable prefix. Check your provider's current behavior.
• Put critical, current information near the end. Task state and the latest step benefit from recency.
• Budget explicitly. Decide how many tokens each component can use, instead of letting history take whatever's left.
• Make it testable. Context assembly is a function. You can unit test it.

Debugging Context Problems

Most teams debug the model's output. Fewer inspect the model's input.

1. Log the full assembled context for every step (with sensitive data handled appropriately). When behavior goes wrong, read what the model actually saw.
2. Diff contexts between a good step and a bad one. What entered? What dropped out? What grew?
3. Replay the failure with a trimmed or restructured context to confirm the cause.
4. Build long-run evaluations, not just short ones. A test that passes at 5 steps and fails at 50 is telling you something.
5. Track context size over a run. A steadily climbing line is a warning.
6. Test in your real language mix. If your users switch between Arabic and English, mixed-script content can behave differently in retrieval and token counts, so check it specifically.

Common Mistakes

Assuming a bigger window solves it. Capacity and attention are different things. More room often just means more clutter.

Treating context as append-only. History should be curated: pruned, stubbed, summarized, or moved out.

Putting rules only in the prompt. Anything critical should also be enforced in code.

Writing memory with no rules. Unbounded, unstamped memory turns into a junk drawer that contradicts itself.

Over-compacting. Aggressive summarization looks efficient until it drops a key detail.

Dumping retrieval results in bulk. Quantity isn't quality.

Skipping long-horizon tests. Short demos hide every problem in this article.

Context Engineering Checklist

Design

• Define what the agent must always know (goal, constraints, current state)
• Separate stable content from changing content
• Set a token budget per component

Tools and results

• Return only needed fields, with pagination and limits
• Stub or clear stale tool results
• Limit tools to what each step needs

Retrieval

• Rerank, deduplicate, and filter by metadata
• Attach source and date to retrieved content
• Evaluate retrieval separately from generation

Memory

• Stamp memories with time and source
• Define write, update, and expiry rules
• Keep a structured task-state or notes file

Long-horizon

• Tune compaction on real traces
• Keep a must-keep list
• Store the raw log outside the context
• Use sub-agents for large, self-contained subtasks

Observability

• Log assembled context per step
• Run long-task evaluations
• Track context size over time

Frequently Asked Questions

What is context engineering?

It's the practice of designing what a language model sees at each step: instructions, tools, memory, retrieved documents, and tool results. The aim is to give it the information it needs to act correctly while keeping out noise that causes errors.

How is context engineering different from prompt engineering?

Prompt engineering focuses on wording a single instruction. Context engineering manages the entire input across a multi-step run, including what to retrieve, remember, trim, and summarize as the task progresses.

Why do AI agents forget things during long tasks?

Usually because their context accumulates noise. Early instructions get buried under tool outputs and history, and research on long inputs suggests models don't use all positions in a long context equally well. Summarization can also drop details.

What is context rot?

It's the term used for performance degrading as input length grows, particularly when irrelevant content is present. Chroma's 2025 research evaluated 18 models and found non-uniform degradation even on simple tasks. Specific behavior varies by model and changes as models improve.

What is context compaction?

Compaction means summarizing a conversation that's nearing the context limit and continuing with the summary. It extends how long an agent can work, but it's lossy, so the summarization needs tuning and critical details should be preserved explicitly.

Is a larger context window the solution?

Not by itself. Larger windows help, but longer contexts can still dilute attention and add noise. Curating what goes in is usually more effective than adding capacity.

How is RAG related to context engineering?

RAG is one component of it. Retrieval decides what documents enter the context. Context engineering also covers how many, in what format, alongside what memory and tool results, and when to remove them.

How do I test for context problems?

Run long-task evaluations, not just short ones. Log the assembled context per step, compare good and bad runs, and track context size over time. Problems that appear only at step 50 won't show up in a five-step demo.

The Real Takeaway

When an agent fails on a long task, the question isn't only "is the model smart enough?" It's "what did it see when it made that decision?"

Often the answer is: too much, in the wrong order, with the important part buried. That's fixable, and the fixes are mostly unglamorous: trim tool outputs, structure the task state, retrieve less but better, write memory with rules, and compact carefully.

The trade-off is engineering effort. Context management adds code, evaluation, and observability that a quick prototype doesn't need. But it's usually the difference between an agent that impresses in a demo and one that's still reliable at step 200.

If you're building agents that need to hold up over long, real-world tasks, we're happy to review your setup. At O2Devs we design and build production agentic systems, including the memory, retrieval, and context layers that keep them reliable. Get in touch and we'll go through it with you.