What would you ask the Machine Learning model?

General Artificial Intelligence is still out of reach, but we already managed to produce a nice amount of Machine Learning models. They are widely adopted in many areas of our lives. For this reason, understanding decisions of blackbox models becomes critical and we are seeing increasing interest and number of tools in the field of eXplainable AI (XAI).

Don’t forget about human!

Yes, we should keep in mind that we explain models to humans. For that reason we build explanations which are visual or textual —it is more readable and informative than the raw numbers. But how do we know what should be explained?

At the moment we are given just a set of static tools. We need to make the process interactive. Imagine yourself trying to understand the decision made by a human expert. What would you do? Talk to them and ask questions! We need to apply the same pattern for predictive models. If we want to understand what people want to know we need to let them ask questions. Only having these questions we can think of the ways of answering them.

Screenshot from the example conversation

Human, please meet an ML model, ML model this is human

We build a conversational system (xaibot — chatbot for XAI) with a human on one end and the ML model on the other. This xaibot was named dr Ant (after doctorant — PhD student). It allows talking about Random Forest model predicting survival on Titanic. You can see an example conversation below.

Example conversation with drAnt

DrAnt understands and responds to several groups of queries:

  • Supplying information about the user (passenger), e.g. specifying age or gender. Alternatively, you can start as one of the main characters from the movie Titanic.
  • Inference — asking about the probability of survival.
  • Dialogue support queries — listing variables, restarting etc.

Trending Bot Articles:

1. The Messenger Rules for European Facebook Pages Are Changing. Here’s What You Need to Know

2. This Is Why Chatbot Business Are Dying

3. Facebook acquires Kustomer: an end for chatbots businesses?

4. The Five P’s of successful chatbots

Xaibot uses visual explanations from DALEX toolset. It answers what-if questions with Ceteris Paribus profiles and feature contributions queries with iBreakDown.

Rose and Jack — main characters of the movie Titanic. Source: https://vignette.wikia.nocookie.net

So, what people ask about?

They say Gentlemen do not read each other’s mail. I’m very sorry drAnt — we make an exception this time. We shared xaibot in this post and collected over 1000 human-model conversations.

There are certain repeating patterns in user queries. Here are the most frequent:

  • why — general explanation queries, such as ”why”,
    explain it to me”, ”how was that derived/calculated”.
  • what-if — alternative scenario queries. Frequent exam-
    ples: what if I’m older, what if I travelled in the 1st
  • what do you know about me
  • data-related questions — e.g. feature histogram, distribution, dataset size
  • local feature importance  How does age influence my
    survival, What makes me more likely to survive
  • global feature importance  How does age influence sur-
    vival across all passengers
  • how to improve — actionable queries for maximizing
    the prediction, e.g. what should I do to survive, how
    can I increase my chances

Full results of the analysis might be found in this paper:

What Would You Ask the Machine Learning Model? Identification of User Needs for Model Explanations Based on Human-Model Conversations

It’s just a beginning of a human-model interaction

We see people engaging in the conversation with the Machine Learning model. It lets them understand more than a single decision and model metrics such as accuracy. And for researchers in the field of eXplainable AI this method provides insight into human needs to be addressed with the explanation methods.

People say talking to plants helps them grow. Talking to Machine Learning models makes those blackboxes more transparent and that is a very desirable trait.

Screenshot from the end of the example conversation

PS drAnt has its own phone number! Yes, it is possible to call the Machine Learning model! Not the most practical thing in this particular case though.

PS2 Let’s not disclose this number. We have already read drAnt’s messages. After all, even the bot deserves some privacy.

Don’t forget to give us your 👏 !

What would you ask the Machine Learning model? was originally published in Chatbots Life on Medium, where people are continuing the conversation by highlighting and responding to this story.