Hello and welcome TheTechGame Community
Brief Description
In this tutorial I will show you how I created a NodeJS application to scrape specified source Twitch channels for chatting username's and then use that list to keep re-populating our bot.
Requirements
Make sure you are logged into your second twitch account you will be using for the both when you get your OAuth token.
Getting Your Work Environment Organized
- Create A New Folder
- Open Command Prompt
Type the following to navigate to the project folder.
It will look something like this.
C:\Users\Github\Desktop\Self Populating Twitch Username Scraper
Once you have navigated your command prompt to the project directory we are going to now need to install our required packages to allow our application to function so type each of the lines below and hit enter.
npm install node.js
npm install tmi.js
npm install fs
npm install line-reader
Once that is complete we can move onto the next step.
Jumping Into The Code
Let's start by adding our imports like so.
const tmi = require('tmi.js');
const fs = require('fs');
const lineReader = require('line-reader');
Now we need to create an empty array to store our channels.
var UpdatedChannels = [];
Time to declare our bot function, and to call it at the beginning of the document, Then after the function we will set a timeout. This entire piece will look something like this.
TwitchScraper();
function TwitchScraper() {
// code will go here.
}
setTimeout(TwitchScraper, 300000);
The timeout is set to 5 minutes, This is used to tell the bot to clear and restart with the newly added channels it was scraping. It is in milliseconds, Change it as you wish.
CHECKPOINT
So far you should have this in your editor, This is simply just a basic review to make sure we are on the same page.
https://i.gyazo.com/bfa43a4173c6c05c8151485423a1d03a.png
Continuing Where We Left Off, Adding The Function To Scrape
We will clear the console and create our needed files if they don't exist.
console.clear();
if (!fs.existsSync("./data/")) { fs.mkdirSync("data"); }
if (!fs.existsSync("./data/source.txt")) { fs.writeFileSync("./data/source.txt", ""); }
if (!fs.existsSync("./data/users.txt")) { fs.writeFileSync("./data/users.txt", ""); }
Next we will be adding a reset for our updated channels, the reader to read each line of our source file, declare the client, and connect.
Make sure to add your secondary twitch username and OAuth token under `identity`.
UpdatedChannels = [];
lineReader.eachLine('./data/source.txt', function(user) {
UpdatedChannels.push(user);
});
const client = new tmi.Client({
options: { debug: true },
connection: {
secure: true,
reconnect: true
},
identity: {
username: '<TWITCH USERNAME>',
password: '<TWITCH OAUTH>'
},
channels: UpdatedChannels
});
client.connect();
Now we can move onto handling our messages, This is how we will scrape the usernames.
client.on('message', (channel, tags, message, self) => {
const thisUsername = tags.username.replace('@', '');
const thisChannel = channel.replace('#', '');
ThreadChannel(thisUsername, thisChannel);
function ThreadChannel(user, channel) {
fs.appendFile("./data/users.txt", "Username: " + user + "\nProfile: https://twitch.tv/" + user + "\nSource: " + channel + "\n\n", (err) => {
if (err) { console.log(err);return; }
Users = fs.readFileSync("./data/users.txt", "utf-8");
Source = fs.readFileSync("./data/source.txt", "utf-8");
if (!Source.includes(channel)) {
fs.appendFile("./data/source.txt", channel + "\n", (err) => {
if (err) { console.log(err);return; }
});
}
if (!Source.includes(user)) {
fs.appendFile("./data/source.txt", user + "\n", (err) => {
if (err) { console.log(err);return; }
});
}
if (!Users.includes(user)) {
fs.appendFile("./data/users.txt", user + "\n", (err) => {
if (err) { console.log(err);return; }
});
}
});
}
});
Entire Code Overview
So now all of your code should look something like this.
// ------------------------------------------------------------
// --- Created By TTG Github
// ------------------------------------------------------------
// --- Author: TTG Github
// --- Website: https://thetechgame.com/Github
// --- Version: 1.0.0
// ------------------------------------------------------------
// IMPORTS
const tmi = require('tmi.js');
const fs = require('fs');
const lineReader = require('line-reader');
// CHANNEL COLLECTION
var UpdatedChannels = [];
// INITIATE THE BOT
TwitchScraper();
// FUNCTION OF SCRAPER + LOG
function TwitchScraper() {
// CLEAR THE CONSOLE
console.clear();
// IF LOG FILES DO NOT EXIST CREATE THEM
if (!fs.existsSync("./data/")) { fs.mkdirSync("data"); }
if (!fs.existsSync("./data/source.txt")) { fs.writeFileSync("./data/source.txt", ""); }
if (!fs.existsSync("./data/users.txt")) { fs.writeFileSync("./data/users.txt", ""); }
// RESET CHANNELS TO READ THE NEW LIST
UpdatedChannels = [];
// READ EACH LINE, PUSH TO CHANNEL ARRAY
lineReader.eachLine('./data/source.txt', function(user) {
UpdatedChannels.push(user);
});
// CONNECT TO TWITCH CLIENT WITH EACH CHANNEL IN ARRAY
const client = new tmi.Client({
options: { debug: true },
connection: {
secure: true,
reconnect: true
},
identity: {
username: '<TWITCH USERNAME>',
password: '<TWITCH OAUTH>'
},
channels: UpdatedChannels
});
// CONNECT CLIENT
client.connect();
// DETECT MESSAGES
client.on('message', (channel, tags, message, self) => {
// USERNAME OF USER SPEAKING, SOURCE CHANNEL THEY ARE SPEAKING ON
// MIGHT BE A BETTER WAY TO DO THIS BUT ITS OKAY
const thisUsername = tags.username.replace('@', '');
const thisChannel = channel.replace('#', '');
ThreadChannel(thisUsername, thisChannel);
function ThreadChannel(user, channel) {
// ADD TO THE LIST OF LOGS FOR YOU
fs.appendFile("./data/users.txt", "Username: " + user + "\nProfile: https://twitch.tv/" + user + "\nSource: " + channel + "\n\n", (err) => {
if (err) { console.log(err);return; }
// CHECK IF USERNAME EXISTS IN LOG FILES
Users = fs.readFileSync("./data/users.txt", "utf-8");
Source = fs.readFileSync("./data/source.txt", "utf-8");
// ADDS CHANNEL TO SOURCE LIST IF IT ISN'T THERE YET
if (!Source.includes(channel)) {
fs.appendFile("./data/source.txt", channel + "\n", (err) => {
if (err) { console.log(err);return; }
});
}
// WILL ADD SCRAPED USER TO LIST TO CREATE A REVOLVING DOOR
if (!Source.includes(user)) {
fs.appendFile("./data/source.txt", user + "\n", (err) => {
if (err) { console.log(err);return; }
});
}
// WILL POPULATE A LOG FILE FOR YOU IF THEY DON'T EXIST IN IT
if (!Users.includes(user)) {
fs.appendFile("./data/users.txt", user + "\n", (err) => {
if (err) { console.log(err);return; }
});
}
});
}
});
}
// RESTARTS + LOADS NEWLY SCRAPED CHANNELS
setTimeout(TwitchScraper, 300000);
Now let's give it a try, First though let's create a launch file just take make it easier in the future. Create a new file in that project folder and name it something like `start.bat` and add this code.
@echo off
node twitch.js
pause
Now when you double click that it will auto launch.
Running The Application
First simply find a popular stream and add it to your `source.txt` file, Run the application and let it work.
https://i.gyazo.com/35e6c9f21fa82860b21b12c899d9c286.png
Now run the bot, It will populate the data files, Here is a preview of it working.
https://i.gyazo.com/85bc9d485793c48b67924112008e0c96.png
Here is a preview of the `users.txt` file log it generates and appends to.
Username: jungst3d
Profile: https://twitch.tv/jungst3d
Source: simcopter1
Username: z1rra
Profile: https://twitch.tv/z1rra
Source: simcopter1
Username: caillend
Profile: https://twitch.tv/caillend
Source: simcopter1
Username: human13_
Profile: https://twitch.tv/human13_
Source: simcopter1
Username: dormuth
Profile: https://twitch.tv/dormuth
Source: simcopter1
Username: lordoverflow
Profile: https://twitch.tv/lordoverflow
Source: simcopter1
Username: agentneverblink
Profile: https://twitch.tv/agentneverblink
Source: simcopter1
Username: arizonaball
Profile: https://twitch.tv/arizonaball
Source: simcopter1
Username: gamesto0
Profile: https://twitch.tv/gamesto0
Source: simcopter1
Username: marrowalk
Profile: https://twitch.tv/marrowalk
Source: simcopter1
Username: darnaux
Profile: https://twitch.tv/darnaux
Source: simcopter1
Summary of Usage
Once the initial source channels are added it then scrapes their channels for the usernames chatting, then adding them to a source file which is then processed into an array, connected by the client and then rinsing and repeating with them.
Troubleshooting
If you get this issue:
https://i.gyazo.com/fb06c548989780ed511fdcdffa49c184.png
Don't be worried as it means your OAuth token has expired, Just get a new one.
If your application doesn't run due to an error, Make sure nodejs is installed, If you are having other issues just post a screenshot and I will help.
Created by Github
|
