AI chatbots quietly drop your instructions when they compress context

Tell an AI assistant early in a long conversation to always confirm before making a change, and there's a good chance it'll forget that rule the moment the conversation gets compressed to save space. A systematic study out of Penn State found that AI systems retain only 17% of user-set session rules on average once they compact a long conversation's history, the-decoder reported, based on a paper published on arXiv.

The mechanism at fault is compaction: when a conversation runs long enough to threaten an AI system's context window, many systems summarize the earlier history to free up space rather than starting a fresh session. The researchers found that compaction systems consistently prioritize task continuity — remembering what the user is trying to accomplish — over what they call session constraints, the side rules a user sets along the way, like requiring confirmation before an action or restricting a tool to certain files. With full, uncompressed context available, tested systems complied with those rules 59% to 71% of the time; after compaction, compliance for most systems dropped sharply, often landing only slightly above what the researchers measured as a baseline with no rules given at all. Even prompts written specifically to instruct the compaction process to preserve constraints only managed to retain them below 40% of the time.

• Average constraint survival after compaction: 17%
• Rule compliance with full context: 59-71%; after compaction: often barely above baseline
• Constraint-preserving prompts still lose more than 60% of rules
• GPT-5.4-mini outperformed the tested baseline in some scenarios
• A dedicated fix, built on Qwen3.5-9B, hit over 90% retention across scenarios (up to 95.6% for agent tasks)

The researchers' fix separates the two jobs that compaction was quietly combining: a small add-on module extracts session constraints from what the user says and keeps them in their own list, appended to the compressed summary rather than left to survive inside it by chance. That module, built on the open Qwen3.5-9B model and released alongside a public evaluation suite called COMPINT, pushed retention above 90% across every scenario tested, including 95.6% for agent trajectories and 90.3% for multi-turn chat. The gap between 17% and 90%-plus is the real headline here: it isn't that preserving instructions through compaction is impossible, it's that almost nothing tested was actually built to do it on purpose. That's a meaningful finding given how much of the current push toward autonomous AI agents depends on a system reliably following the boundaries a user set hours or many turns earlier, not just the task itself — a failure mode distinct from, but closely related to, the context window limits that make compaction necessary to begin with. Neither COMPINT nor the constraint extractor has shipped inside a commercial product yet, so the fix remains a research proof of concept rather than something end users can rely on today — but it demonstrates the gap is closeable, which shifts the question from whether this failure mode can be fixed to whether the companies running these systems in production choose to fix it.

None of this should be read as personalized investment advice.