ProgrammingAuto Populating Twitch Username Logging [NodeJS]
Posted:

ProgrammingAuto Populating Twitch Username Logging [NodeJS]Posted:

Github
  • New Member
Status: Offline
Joined: Aug 29, 2021
Posts: 11
Reputation Power: 0
Status: Offline
Joined: Aug 29, 2021
Posts: 11
Reputation Power: 0
Hello and welcome TheTechGame Community

Brief Description
In this tutorial I will show you how I created a NodeJS application to scrape specified source Twitch channels for chatting username's and then use that list to keep re-populating our bot.

Requirements

Make sure you are logged into your second twitch account you will be using for the both when you get your OAuth token.

Getting Your Work Environment Organized

  • Create A New Folder
  • Open Command Prompt


Type the following to navigate to the project folder.

cd <PATH TO FOLDER>


It will look something like this.
C:\Users\Github\Desktop\Self Populating Twitch Username Scraper


Once you have navigated your command prompt to the project directory we are going to now need to install our required packages to allow our application to function so type each of the lines below and hit enter.


npm install node.js
npm install tmi.js
npm install fs
npm install line-reader


Once that is complete we can move onto the next step.

Jumping Into The Code
Let's start by adding our imports like so.
const tmi = require('tmi.js');
const fs = require('fs');
const lineReader = require('line-reader');


Now we need to create an empty array to store our channels.
var UpdatedChannels = [];


Time to declare our bot function, and to call it at the beginning of the document, Then after the function we will set a timeout. This entire piece will look something like this.

TwitchScraper();

function TwitchScraper() {
// code will go here.
}

setTimeout(TwitchScraper, 300000);


The timeout is set to 5 minutes, This is used to tell the bot to clear and restart with the newly added channels it was scraping. It is in milliseconds, Change it as you wish.

CHECKPOINT
So far you should have this in your editor, This is simply just a basic review to make sure we are on the same page.
https://i.gyazo.com/bfa43a4173c6c05c8151485423a1d03a.png

Continuing Where We Left Off, Adding The Function To Scrape
We will clear the console and create our needed files if they don't exist.

console.clear();
if (!fs.existsSync("./data/")) { fs.mkdirSync("data"); }
if (!fs.existsSync("./data/source.txt")) { fs.writeFileSync("./data/source.txt", ""); }
if (!fs.existsSync("./data/users.txt")) { fs.writeFileSync("./data/users.txt", ""); }


Next we will be adding a reset for our updated channels, the reader to read each line of our source file, declare the client, and connect.

Make sure to add your secondary twitch username and OAuth token under `identity`.

UpdatedChannels = [];

  lineReader.eachLine('./data/source.txt', function(user) {
    UpdatedChannels.push(user);
  });

  const client = new tmi.Client({
    options: { debug: true },
    connection: {
      secure: true,
      reconnect: true
    },
    identity: {
      username: '<TWITCH USERNAME>',
      password: '<TWITCH OAUTH>'
    },
    channels: UpdatedChannels
  });
  client.connect();


Now we can move onto handling our messages, This is how we will scrape the usernames.

  client.on('message', (channel, tags, message, self) => {
      const thisUsername = tags.username.replace('@', '');
      const thisChannel = channel.replace('#', '');
      ThreadChannel(thisUsername, thisChannel);
      function ThreadChannel(user, channel) {
        fs.appendFile("./data/users.txt", "Username: " + user + "\nProfile: https://twitch.tv/" + user + "\nSource: " + channel + "\n\n", (err) => {
          if (err) { console.log(err);return; }
          Users = fs.readFileSync("./data/users.txt", "utf-8");
          Source = fs.readFileSync("./data/source.txt", "utf-8");
          if (!Source.includes(channel)) {
            fs.appendFile("./data/source.txt", channel + "\n", (err) => {
              if (err) { console.log(err);return; }
            });
          }
          if (!Source.includes(user)) {
            fs.appendFile("./data/source.txt", user + "\n", (err) => {
              if (err) { console.log(err);return; }
            });
          }
          if (!Users.includes(user)) {
            fs.appendFile("./data/users.txt", user + "\n", (err) => {
              if (err) { console.log(err);return; }
            });
          }
        });
      }
  });


Entire Code Overview
So now all of your code should look something like this.

// ------------------------------------------------------------
// --- Created By TTG Github
// ------------------------------------------------------------
// --- Author: TTG Github
// --- Website: https://thetechgame.com/Github
// --- Version: 1.0.0
// ------------------------------------------------------------

// IMPORTS
const tmi = require('tmi.js');
const fs = require('fs');
const lineReader = require('line-reader');

// CHANNEL COLLECTION
var UpdatedChannels = [];

// INITIATE THE BOT
TwitchScraper();

// FUNCTION OF SCRAPER + LOG
function TwitchScraper() {

  // CLEAR THE CONSOLE
  console.clear();

  // IF LOG FILES DO NOT EXIST CREATE THEM
  if (!fs.existsSync("./data/")) { fs.mkdirSync("data"); }
  if (!fs.existsSync("./data/source.txt")) { fs.writeFileSync("./data/source.txt", ""); }
  if (!fs.existsSync("./data/users.txt")) { fs.writeFileSync("./data/users.txt", ""); }

  // RESET CHANNELS TO READ THE NEW LIST
  UpdatedChannels = [];

  // READ EACH LINE, PUSH TO CHANNEL ARRAY
  lineReader.eachLine('./data/source.txt', function(user) {
    UpdatedChannels.push(user);
  });

  // CONNECT TO TWITCH CLIENT WITH EACH CHANNEL IN ARRAY
  const client = new tmi.Client({
    options: { debug: true },
    connection: {
      secure: true,
      reconnect: true
    },
    identity: {
      username: '<TWITCH USERNAME>',
      password: '<TWITCH OAUTH>'
    },
    channels: UpdatedChannels
  });

  // CONNECT CLIENT
  client.connect();

  // DETECT MESSAGES
  client.on('message', (channel, tags, message, self) => {

      // USERNAME OF USER SPEAKING, SOURCE CHANNEL THEY ARE SPEAKING ON
      // MIGHT BE A BETTER WAY TO DO THIS BUT ITS OKAY
      const thisUsername = tags.username.replace('@', '');
      const thisChannel = channel.replace('#', '');

      ThreadChannel(thisUsername, thisChannel);

      function ThreadChannel(user, channel) {

        // ADD TO THE LIST OF LOGS FOR YOU
        fs.appendFile("./data/users.txt", "Username: " + user + "\nProfile: https://twitch.tv/" + user + "\nSource: " + channel + "\n\n", (err) => {
          if (err) { console.log(err);return; }

          // CHECK IF USERNAME EXISTS IN LOG FILES
          Users = fs.readFileSync("./data/users.txt", "utf-8");
          Source = fs.readFileSync("./data/source.txt", "utf-8");

          // ADDS CHANNEL TO SOURCE LIST IF IT ISN'T THERE YET
          if (!Source.includes(channel)) {
            fs.appendFile("./data/source.txt", channel + "\n", (err) => {
              if (err) { console.log(err);return; }
            });
          }

          // WILL ADD SCRAPED USER TO LIST TO CREATE A REVOLVING DOOR
          if (!Source.includes(user)) {
            fs.appendFile("./data/source.txt", user + "\n", (err) => {
              if (err) { console.log(err);return; }
            });
          }

          // WILL POPULATE A LOG FILE FOR YOU IF THEY DON'T EXIST IN IT
          if (!Users.includes(user)) {
            fs.appendFile("./data/users.txt", user + "\n", (err) => {
              if (err) { console.log(err);return; }
            });
          }
        });
      }
  });
}

// RESTARTS + LOADS NEWLY SCRAPED CHANNELS
setTimeout(TwitchScraper, 300000);


Now let's give it a try, First though let's create a launch file just take make it easier in the future. Create a new file in that project folder and name it something like `start.bat` and add this code.

@echo off
node twitch.js
pause


Now when you double click that it will auto launch.

Running The Application
First simply find a popular stream and add it to your `source.txt` file, Run the application and let it work.
https://i.gyazo.com/35e6c9f21fa82860b21b12c899d9c286.png

Now run the bot, It will populate the data files, Here is a preview of it working.
https://i.gyazo.com/85bc9d485793c48b67924112008e0c96.png

Here is a preview of the `users.txt` file log it generates and appends to.

Username: jungst3d
Profile: https://twitch.tv/jungst3d
Source: simcopter1

Username: z1rra
Profile: https://twitch.tv/z1rra
Source: simcopter1

Username: caillend
Profile: https://twitch.tv/caillend
Source: simcopter1

Username: human13_
Profile: https://twitch.tv/human13_
Source: simcopter1

Username: dormuth
Profile: https://twitch.tv/dormuth
Source: simcopter1

Username: lordoverflow
Profile: https://twitch.tv/lordoverflow
Source: simcopter1

Username: agentneverblink
Profile: https://twitch.tv/agentneverblink
Source: simcopter1

Username: arizonaball
Profile: https://twitch.tv/arizonaball
Source: simcopter1

Username: gamesto0
Profile: https://twitch.tv/gamesto0
Source: simcopter1

Username: marrowalk
Profile: https://twitch.tv/marrowalk
Source: simcopter1

Username: darnaux
Profile: https://twitch.tv/darnaux
Source: simcopter1


Summary of Usage
Once the initial source channels are added it then scrapes their channels for the usernames chatting, then adding them to a source file which is then processed into an array, connected by the client and then rinsing and repeating with them.

Troubleshooting
If you get this issue:
https://i.gyazo.com/fb06c548989780ed511fdcdffa49c184.png
Don't be worried as it means your OAuth token has expired, Just get a new one.

If your application doesn't run due to an error, Make sure nodejs is installed, If you are having other issues just post a screenshot and I will help.




Created by Github

The following 1 user thanked Github for this useful post:

Chris (09-05-2021)
Users browsing this topic: None
Jump to:


RECENT POSTS

HOT TOPICS