so about 8 months ago I launched a chatbot for a Discord community I run and also as a widget on our website. the idea was simple, train it on our docs and let it answer the repetitive questions instead of me spending half my day on support.
first version was embarassing. the bot would give these confident, well-written answers that were just… wrong. like it would mix up information from different docs or just make stuff up when it didn’t have a good match. users started screenshotting the bad answers and posting them in the server which was fun.
the thing I got wrong was assuming that just uploading documents would be enough. turns out the hard part isn’t generating the answer, its finding the right information to generate FROM. most chatbot tools (and I tried a few, Chatbase, a custom GPT thing) do pretty basic matching and call it a day. the accuracy was always hit or miss.
I ended up spending a few months reworking how the bot actually finds and connects relevant information from the knowledge base. took a completley different approach to how docs get processed and indexed. the accuracy went from “please don’t use this” to “actually useful for straightforward questions.” still not perfect, response time is kinda slow (10-15 seconds) and you have to manually rebuild the KB when docs change which is annoying.
the other thing that helped a lot was building a system where the bot learns from moderator answers automatically. so when a mod corrects something or answers a question the bot missed, that gets captured and the bot uses it next time. that one feature probably improved answer quality more than anything else I did on the technical side.
anyway the thing is called BestChatBot (bestchatbot.io) if anyone wants to poke at it. free tier is pretty limited but enough to test. curious if anyone else has gone through this cycle of “this is garbage” to “ok this actually works” with a chatbot project. feels like nobody talks about how bad v1 always is
submitted by /u/cryptoviksant
[link] [comments]