Analyse Twitter Streaming data with Node.js & free NLP API

Analyse Tweets from Twitter Streaming with Node.js & free NLP API

Intro

According to WordStream article from 04.2020 everyday Twitter users send circa 500 million tweets. In this huge sea of information, it is hard to manually find and analyse tweets about your brand or product. Fortunately Twitter API gives us a possibility to follow a number of keywords automatically.

In this article I would like to present you how to easily create a Node.js application to follow provided keywords on Twitter in real time. Next, we will analyse them using a free NLP tool from ThadeusAI (fyi: you could use any other AI). Our selected keyword will be Cyberpunk. Eventually we will be able to build a simple bot for gathering tweets about Cyberpunk 2077, analyse them with a taught NLP and enable auto answering.

Twitter App Creation & Streaming Setup

Let’s start by creating a Twitter app. Go to Twitter Developer Portal and create a new app. If you didn’t have developer access before, firstly you have to fill a form about your use cases and Twitter staff has to accept your developer application. Twitter approval usually works pretty fast, so it should not be a big problem.

When you create a new app, generate:
a) access token & secret,
b) API key & secret.
and save them somewhere.

For Twitter streaming connection we will use a NPM package called “twitter”. Let’s move to testing Twitter streaming.

require('dotenv').config();
   const Twitter = require('twitter');
   const client = new Twitter({
   consumer_key: process.env.TWITTER_CONSUMER_KEY,
   consumer_secret: process.env.TWITTER_CONSUMER_SECRET,           
   access_token_key: process.env.TWITTER_ACCESS_TOKEN_KEY,
   access_token_secret: process.env.TWITTER_ACCESS_TOKEN_SECRET
});

const stream = client.stream('statuses/filter', {track: 'cyberpunk', tweet_mode: 'extended', language: 'en'});

stream.on('data', (data) => {
   if (!data.retweeted_status) {
      const tweetText = data?.extended_tweet?.full_text || data.text;
      console.log(tweetText);
   }
});

stream.on('error', (error) => {
   throw error;
});

So what has happened here:

I am loading Twitter credentials from .env file with “dotenv” package
I am initialising “Twitter” package with my previously generated app credentials
I am starting a stream with options to track tweets with “cyberpunk” keyword only, using an extended mode (by default longer tweets are cut) and filtering tweets to only english
If a new tweet has been posted, a “data” listener will run.
By checking “retweeted_status” we are ignoring re-tweets.

Trending Bot Articles:

1. The Messenger Rules for European Facebook Pages Are Changing. Here’s What You Need to Know

2. This Is Why Chatbot Business Are Dying

3. Facebook acquires Kustomer: an end for chatbots businesses?

4. The Five P’s of successful chatbots

Feel free to run it and test on your keyword. My “Cyberpunk” stream worked like that:

NLP Model Setup & usage of Thadeus NPM package

First step is behind us, now let’s create an account in Thadeus AI platform and teach our NLP model.

For now let us pause the development part in order to prepare and teach our model. Please let me quickly explain how ThadeusAi works.

Go to Thadeus AI, create an account and login in a dashboard. After logging into the dashboard, you have to create a “Workspace” — it’s a single NLP model. In your workspace management page, you create intents with at least 5 examples. You can think about intents like they were message categories. For example, you can create an intent “love” and provide many tweets (in our case) which are saying that Cyberpunk 2077 is a great game. Hopefully the below screenshots of my trained model will help you to imagine how it works.

And examples used in my intent:

Of course, the more examples in intents, the better the model works, so 5 is an absolute minimum. After a new intent is added, remember to re-train your AI. Note that you can also use “Test Talk” to test your model anytime.

You can also add examples with Thadeus API. For instance, you can create intents examples from a CSV file instead of manually adding them in the dashboard.

What is more, the team from ThadeusAI also prepared an NPM package so it’s really simple to integrate with your Node.js application.

Once you have your model prepared and taught, let’s move back to our application and add intent recognition with our trained model.

const { Thadeus } = require('thadeus');

const thadeus = new Thadeus({
  apiSecret: process.env.THADEUS_API_SECRET,
  apiKey: process.env.THADEUS_API_KEY,
  workspaceId: process.env.THADEUS_WORKSPACE_ID,
});

Initialise the Thadeus client with your workspace credentials and id (generated in ThadeusAI dashboard).

const stream = client.stream('statuses/filter', {track: 'cyberpunk', tweet_mode: 'extended', language: 'en'});

stream.on('data', async (data) => {
  if (!data.retweeted_status) {
    const tweetText = data?.extended_tweet?.full_text || data.text;
    console.log('tweet', tweetText);
    const predictions = await thadeus.predictIntents(tweetText);
    console.log('predictions', predictions);
    console.log('n===============n');
  }
});

I have updated our previous code with the predict intents method. Now every tweet will be sent to our NLP model and it will return predictions about intents. ThadeusAI also offers entity recognition, but for now it only works with predefined ones for every workspace. Some time in the near future, Thadeus is likely to add custom entities support which should be working similarly to intents.

What comes next? How to handle ingested data?

Now it’s up to you how you would like to handle this data. In this tutorial I will add one more functionality to have a real Twitter bot.

I want to auto-answer every tweet with a “refund” intent which has probability over 70%. (CDProjektRed, do not worry, I will send you the code later 😀 )

Updated “data” listener with answering on tweet:

stream.on('data', async (data) => {
  if (!data.retweeted_status) {
    const tweetText = data?.extended_tweet?.full_text || data.text;
    const predictions = await thadeus.predictIntents(tweetText);
    if (predictions.some(({ intent, probability }) => 
intent === 'Refund' && probability * 100 > 70 )) {
        await client.post( 'statuses/update', {
          status: 'Hi, we are sorry to hear about your problems. To get a refund, please send an email to refunds@refunds.com',
          in_reply_to_status_id: data.id,
        })
    }
  }
});

That’s all. We have created a Twitter bot for CDProjektRed for handling users issues automatically.

ThadeusAI is still in development. The team is currently working on custom entities and sentiment analysis. Please leave a comment/note if you found this article interesting, if so, once Thadeus adds any new features I will prepare a quick guide how to utilise them.

Feel free to ask if you have any problems with this code or if you have any questions about Thadeus.

Don’t forget to give us your 👏 !

<a href="https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href">https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href</a>

Analyse Twitter Streaming data with Node.js & free NLP API was originally published in Chatbots Life on Medium, where people are continuing the conversation by highlighting and responding to this story.