kagglebot feedback

Hello guys, i had this idea of building a local RAG model specialized in kaggle competitions, the purpose is to make the bot answer questions about any competition based on given context (contexts are gathered and stored in faiss vectorstore), which can help the user to save time from reading all those discussion threads and notebooks.

with 8gb RAM and cpu only, i managed to use Mistral-7b-Q4_K_M.gguf model, but the flow in increadibly slow, i feel like i am waiting forever to get an answer (about two minutes, sometimes longer).

i’ve put the codebase on github, feel free to check https://github.com/kaoutaar/KaggleBOT

do you think it’s a good sideproject to put on my resume, i need your feedbacks please.

and how can i improve the latency?