Twitter Scraper (Web API)

Useful tool to scrape information from twitter users and tweets.

*In order to use this scraper you need an active twitter account. Scraper will sign in and use session to scrape data

*A {username}.json file with AUTH cookies will be saved in your system TMP folder

Features

• You can use it as module or in terminal
• Scraper followers, tweets, favorites from a twitter username(public profile only)
• Scrape retweeted by, liked by, comments from a tweet id
• Results can be saved in to a CSV and JSON files

Followers List Tweets List Comments List Liked By List and more

Limits:

• retweeted by - 97 inputes (web api limit).
• liked by - 100 inputes (web api limit).
• followers, comments, tweets, favorites - unlimited

Possible errors

• you can receive "rate limit" error, in that case wait and increase timeout

Installation

Install from NPM

$ npm i -g twitter-buddy

Install from YARN

$ yarn global add twitter-buddy

USAGE

In Terminal

$ twitter-buddy --help

Usage: twitter-buddy <command> [options]

Commands:
  twitter-buddy retweetedBy [id]   users who retweeted specific tweet
  twitter-buddy likedBy [id]       users who liked specific tweet
  twitter-buddy conversation [id]  tweet comments
  twitter-buddy followers [id]     user followers
  twitter-buddy tweets [id]        user tweets
  twitter-buddy favorites [id]     user favorites/likes

Options:
  --help, -h            help                                           [boolean]
  --version             Show version number                            [boolean]
  --login-username, -u  Your twitter username             [string] [default: ""]
  --login-password, -p  Your twitter password             [string] [default: ""]
  --number, -n          Number of inputs to scrape                 [default: 20]
  --csv                 Save to a CSV file?            [boolean] [default: true]
  --json                Save to a JSON file?          [boolean] [default: false]
  --timeout             If the "rate limit" error received or emtpy result then
                        you can try to set {timeout}. Timeout is in
                        milliseconds: 1000 = 1 second               [default: 0]
Examples:
  twitter-buddy followers TWITTER_USERNAME -u TWITTER_USERNAME -p
  TWITTER_PASSWORD -n 100
  
  twitter-buddy retweetedBy TWEET_ID -u TWITTER_USERNAME -p TWITTER_PASSWORD
  -n 100

Example 1 Scrape 2000 followers from twitter profile 'realDonaldTrump' and save everything to a CSV file

$ twitter-buddy followers realDonaldTrump -u TWITTER_USERNAME -p TWITTER_PASSWORD -n 2000

CSV file saved: /{FOLDER_PATH}/followers_1571390379072_realDonaldTrump.csv

Example 2 Scrape 1000 tweets from twitter profile 'realDonaldTrump'.

$ twitter-buddy tweets realDonaldTrump -u TWITTER_USERNAME -p TWITTER_PASSWORD -n 1000

CSV file saved: /{FOLDER_PATH}/tweets_1571390379072_realDonaldTrump.csv

Example 3 Scrape 100 users who retweeted this tweet 'https://twitter.com/realDonaldTrump/status/1183796209942695938'.

$ twitter-buddy retweetedBy 1183796209942695938 -u TWITTER_USERNAME -p TWITTER_PASSWORD -n 100

CSV file saved: /{FOLDER_PATH}/retweetedBy_1571390379072_1183796209942695938.csv

Module

Options

{
    // Twitter username
    username: {string},
    
    // Twitter password
    password: {string},
    
    //If {true} then event emitter will be enabled, you won't be able to use promise
    event: {boolean},
    
    //Timeout between requests, is usefull if the 'rate limit' error was received
    timeout: {int}
    
    //Https Proxy, for example 127.0.0.1:8000
    proxy: {string}
    
}

Promise

const twitterScraper = require('twitter-buddy');

(async() => {
    try{
        let bot = new twitterScraper({
            username:'YOUR_TWITTER_USERNAME', 
            password:'YOUR_TWITTER_PASSWORD',
            //event: {boolean} Enable event emitter
            //timeout: {int} Timeout between requests
            //proxy: {string} Https proxy, ex: 127.0.0.1:8080
        });
        
        // Scrape 1000 followers from twitter profile 'realDonaldTrump'
        let followers = await bot._input({ 
            type: 'scrape',
            action: 'followers',
            input: 'realDonaldTrump',
            number: 1000,
         });
         
         // Scrape 500 tweets from twitter profile 'realDonaldTrump'
        let tweets = await bot._input({ 
            type: 'scrape',
            action: 'tweets',
            input: 'realDonaldTrump',
            number: 500,
        });
        
        // Scrape 500 favorites from twitter profile 'realDonaldTrump'
        let favorites = await bot._input({ 
            type: 'scrape',
            action: 'favorites',
            input: 'realDonaldTrump',
            number: 500,
        });
        
        // Scrape 300 comments from tweet https://twitter.com/realDonaldTrump/status/1183796209942695938'
        let favorites = await bot._input({ 
            type: 'scrape',
            action: 'conversation',
            input: '1183796209942695938',
            number: 300,
        });
        
        // Scrape 100 users who retweeted tweet https://twitter.com/realDonaldTrump/status/1183796209942695938'
        let favorites = await bot._input({ 
            type: 'scrape',
            action: 'retweetedBy',
            input: '1183796209942695938',
            number: 300,
        });
        
        // Scrape 100 users who liked tweet https://twitter.com/realDonaldTrump/status/1183796209942695938'
        let favorites = await bot._input({ 
            type: 'scrape',
            action: 'likedBy',
            input: '1183796209942695938',
            number: 300,
        });
    }catch(error){
        console.log(error);
    }
})()

Event

const twitterScraper = require('twitter-buddy');

let bot = new twitterScraper({
    username:'YOUR_TWITTER_USERNAME', 
    password:'YOUR_TWITTER_PASSWORD',
    event: true, // Enable event emitter
    //timeout: {int} Timeout between requests
    //proxy: {string} Https proxy, ex: 127.0.0.1:8080
 });


bot._input({ 
    type: 'scrape',
    action: 'followers',
    input: 'realDonaldTrump',
    number: 1000,
});

bot.on('data', data => {
    // data
})

bot.on('error', error => {
    // error
})

License

MIT

Free Software