Why an AI chat forgets earlier messages

An assistant that "forgets" hasn't lost anything in the way a person does. Usually the earlier message has fallen out of what the model can see, or it's still there but being used poorly.

Here are the main causes, as the makers describe them.

The chat is re-read every turn

A chat feels like a continuing conversation. Under the hood, a model works on one request at a time. Anthropic's documentation says each turn's input holds all the previous conversation history plus the current user message.¹

In other words, the model is handed the whole transcript again each time, and that transcript has to fit the context window.²

Each reply the model writes becomes part of the history for the next turn, so the transcript grows by both sides.¹

What happens when the transcript gets too long

Something has to be cut, shortened or refused. Makers handle it in different ways.

• Rolling window. Anthropic notes that chat interfaces such as claude.ai can manage the window on a rolling "first in, first out" basis. Plainly, the oldest material goes first.¹
• Summarising. Anthropic also describes compaction, which summarises earlier parts of the conversation on the server so it can continue past the limit.¹ It's a beta feature in Anthropic's API for certain models, so it isn't necessarily what a given chat app does.¹ A summary is shorter than the original, so some detail is likely gone.
• Trimming by the builder. OpenAI's page tells people building with its tools to remove unnecessary or repeated context, or to condense text before sending it.²

I can't tell you which of these your particular app uses. Apps differ, and the makers' pages describe options, not a rule for every product. Check the help pages of the tool your business uses.

It can go wrong before the window is full

Here's the part that surprises people. Anthropic's documentation warns that long contexts lose accuracy and recall, and it has a name for the effect: context rot.¹

That means a very long chat may handle early details less well even while they're technically still inside the window.

What you can do about it

None of these is a guarantee, but they follow from how the makers describe the system.

1. Restate the key facts. If something matters, say it again in a recent message.
2. Start a fresh chat for a new task. Old, unrelated text only takes up room. This follows the advice on removing unnecessary context.²
3. Paste a short summary at the top of a new chat. Write it yourself, and leave out customer details unless your workplace policy allows them.
4. Keep documents lean. Send the pages that matter, not the whole file.
5. Check your tool's own help pages. They say how that product handles long chats.

A quick test

If the assistant loses track of a detail, put the detail back in and carry on, or begin again with a short summary. Asking the assistant to repeat what it knows isn't a reliable test of what it can see, so don't treat its answer as proof either way.

Just don't read the miss as the assistant being careless. It's working from a limited view that you can shape.

Wondering how large that view can be? See what is a token and how many words fit in a token budget.