npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

twitter2mongodb-cli

v1.1.5

Published

Command line tool for extracting Twitter data to MongoDB databases

Downloads

20

Readme

twitter2mongodb-cli

Richard Wen
[email protected]

Command line tool for extracting Twitter data to MongoDB databases

npm version Build Status npm GitHub license Donarbox Donate PayPal Donate Twitter

Install

  1. Install Node.js
  2. Install twitter2mongodb-cli via npm
npm install -g twitter2mongodb-cli

For the latest developer version, see Developer Install.

Usage

Get help:

twitter2mongodb --help

Open documentation in web browser:

twitter2mongodb doc twitter2mongodb
twitter2mongodb doc twitter
twitter2mongodb doc mongodb

See twitter2mongodb for programmatic usage.

Environment File

An environment file .env is used to store Twitter API credentials and MongoDB details.

Step 1. Set the default config for the .env file:

  • Every twitter2mongodb command will now use the designated .env file
twitter2mongodb config set env path/to/.env

Step 2. Set Twitter API credentials

twitter2mongodb env set TWITTER_CONSUMER_KEY ***
twitter2mongodb env set TWITTER_CONSUMER_SECRET ***
twitter2mongodb env set TWITTER_ACCESS_TOKEN_KEY ***
twitter2mongodb env set TWITTER_ACCESS_TOKEN_SECRET ***

Step 3. Set MongoDB connection

twitter2mongodb env set MONGODB_CONNECTION mongodb://localhost:27017

REST API

The REST API obtains Twitter data in batches using search queries.

Step 1. Setup default twitter options:

  1. Set Twitter REST method (one of get, post, delete or stream)
  2. Set Twitter path
  3. Set Twitter parameters for path
twitter2mongodb config set twitter.method get
twitter2mongodb config set twitter.path search/tweets
twitter2mongodb config set twitter.params "{\"q\":\"twitter\"}"

Step 2. Setup default MongoDB options:

  1. Set database to store streamed Twitter data
  2. Set collection to store streamed Twitter data
  3. Set MongoDB query method for streamed Twitter data
  4. Set jsonata filter before inserting
twitter2mongodb config set mongodb.database twitter2mongodb_database
twitter2mongodb config set mongodb.collection twitter_data
twitter2mongodb config set mongodb.method insertMany
twitter2mongodb config set jsonata statuses

Step 3. Extract Twitter data into MongoDB collection given setup options:

twitter2mongodb > log.csv

Stream API

The Stream API obtains Twitter data in real-time using tracking filters.

Step 1. Setup default twitter options:

  1. Set Twitter stream method
  2. Set Twitter path
  3. Set Twitter stream parameters
twitter2mongodb config set twitter.method stream
twitter2mongodb config set twitter.path statuses/filter
twitter2mongodb config set twitter.params "{\"track\":\"twitter\"}"

Step 2. Setup default MongoDB options:

  1. Set database to store streamed Twitter data
  2. Set collection to store streamed Twitter data
  3. Set MongoDB query method for streamed Twitter data
twitter2mongodb config set mongodb.database twitter2mongodb_database
twitter2mongodb config set mongodb.collection twitter_data
twitter2mongodb config set mongodb.method insertOne

Step 3a. Stream Twitter data into MongoDB collection given setup options:

twitter2mongodb > log.csv

Step 3b. Stream Twitter data into a MongoDB collection as a service:

  1. Save a node runnable script of the current options
  2. Install pm2 (npm install pm2 -g)
  3. Use pm2 to run the saved script as a service
twitter2mongodb save path/to/script.js
pm2 start path/to/script.js
pm2 save

Logs

The logs are in the following Comma-Separated Values (CSV) format:

  • time_iso8601: Time and date in ISO 8601 format
  • status: Status of the log
  • message: Relevant messages
  • json: JSON object containing relevant debugging information

time_iso8601 | status | message | json --- | --- | --- | --- ... | ... | ... | ...

Contributions

  1. Reports for issues and suggestions can be made using the issue submission interface.
  2. Code contributions are submitted via pull requests

See CONTRIBUTING.md for more details.

Developer Notes

Developer Install

Install the latest developer version with npm from github:

npm install git+https://github.com/rrwen/twitter2mongodb-cli

Install from git cloned source:

  1. Ensure git is installed
  2. Clone into current path
  3. Install via npm
git clone https://github.com/rrwen/twitter2mongodb-cli
cd twitter2mongodb-cli
npm install

Tests

  1. Clone into current path git clone https://github.com/rrwen/twitter2mongodb-cli
  2. Enter into folder cd twitter2mongodb-cli
  3. Ensure devDependencies are installed and available
  4. Run tests with a .env file (see tests/README.md)
  5. Results are saved to tests/log with each file corresponding to a version tested
npm install
npm test

Upload to Github

  1. Ensure git is installed
  2. Inside the twitter2mongodb-cli folder, add all files and commit changes
  3. Push to github
git add .
git commit -a -m "Generic update"
git push

Upload to npm

  1. Update the version in package.json
  2. Run tests and check for OK status
  3. Login to npm
  4. Publish to npm
npm test
npm login
npm publish

Implementation

The module twitter2mongodb-cli uses the following npm packages for its implementation:

npm | Purpose --- | --- path | Handle file and directory paths fs | Read and write config file envfile | Parse and write env files dotenv | Load environmental variables from a file yargs | Command line builder and parser yargs-command-config | Command for managing config files yargs-command-env | Command for managing env files twitter2mongodb | Extracts Twitter data to MongoDB opn | Open online browser documentation mongodb | Send queries to MongoDB database parse-mongo-url | Parse MongoDB urls

  path     <-- Handle file and dir paths
   |
   fs      <-- Read and write config file
   |
 envfile   <-- parse and write env file
   |
 dotenv    <-- load env file
   |
 yargs
   |--- yargs-command-config   <-- manage config
   |--- yargs-command-env      <-- manage env
   |--- twitter2mongodb        <-- default command
   |--- opn                    <-- doc
   |--- mongodb                <-- query
   |--- parse-mongo-url        <-- parse MongoDB url for info