Genuine question to this community.
Every chatbot platform I’ve looked at measures response time, deflection rate, CSAT. None of them measure: was the answer actually correct?
The bot could be confidently wrong 40% of the time and the dashboard would show “great performance” because tickets were “resolved.”
I’ve been building a support agent that does this — tracks accuracy per topic category based on real human feedback. When the bot has been wrong too often on a specific topic → automatic handoff instead of guessing.
Live demo if anyone wants to try it: https://7611-2a00-6020-479c-a300-fd00-39bb-6853-6da8.ngrok-free.app
Try asking a shipping question (bot answers) vs. a medical question (bot hands off). That’s the whole idea — answer only what you actually know.
Would love to hear from people running chatbots in production: how do you currently catch when your bot is wrong? Manual log review? Customer complaints? Or do you just not know?
submitted by /u/Suspicious_Welcome12
[link] [comments]