The context problem killed our Slack bot for 2 months. Here’s what finally fixed it.

Posted here a while back about building AI agents for internal Slack ops and the stuff that broke first. Got way more replies than I expected, and a few people asked specifically about the context handling part how do you get an agent to follow a Slack thread that jumps around, references old messages, and has people talking past each other?

That problem ate up most of our engineering time for about two months. Here’s what we actually landed on after a lot of dead ends.

The thing that didn’t work: just dumping more thread history into the context window. We tried last 20 messages, last 50, full thread, full channel for the day. More context made it worse, not better. The agent started picking up on irrelevant tangents and confidently answering questions that weren’t asked.

What actually worked was a small summarizer step before the main agent ever sees the conversation. We run a cheap, fast model over the thread first and ask it one thing: what is the user’s current question, and what earlier context is relevant to answering it? That summary becomes the input to the main agent. Everything else gets dropped.

This sounds obvious in hindsight but it took us forever to get there because we kept trying to make one agent smart enough to do both jobs. Splitting it fixed accuracy and cut our token costs by more than half.

The other thing that helped: we stopped trying to handle every edge case in the prompt. Instead the agent has an explicit I’m not sure what you’re asking, can you rephrase? path. Used to feel like a failure. Now I think it’s the single biggest reliability win we shipped. Users don’t mind being asked a clarifying question. They mind getting a confident wrong answer.

Still working on a few things multi user threads where two people are asking different questions at once is still rough, and anything that references a screenshot or attachment we basically can’t handle yet.

For anyone in the middle of this: the fix was almost never a better prompt. It was usually a smaller, more focused agent doing one thing well. Curious if anyone else landed somewhere different on context handling especially for messy chat-based inputs.