Reads files from XML Team and puts them onto Thoonk job queues

Reads the file and uses xml-stream to isolate the <sports-metadata/> node. This tells us on which type of queue the file should be placed for downstream processing.

Requirements

Assumes REDIS is already running.

How to Use It

Install globally with npm install -g ots-xt-q. There are two ways to use this script.

1. Invoked with command line arguments:

ots-xt-q --file path/to/file.xml

2. Pipe in a list of files:

find path/to/files/ -type f | ots-xt-q

The inotify configuration with incrontab will invoke the script via option 1. The only time you might be piping in a list of files is if you are batch processing a bunch of files. In this case, you're likely reprocessing them.

Job Queue Types

Since we're interested in parallelizing the processing of jobs, there are two main queue types.

1. general: files that come in less frequently.
2. progressive: real-time data files received during games.

Queues are separated by sport. XML Team uses the concept of leagues to identify a sport at the most granular level. For example, MLB Baseball has a league code of l.mlb.com while NFL football has a league code of l.nfl.com. The job queuer will therefore place files onto the following types of queues:

• l.mlb.com:general
• l.mlb.com:progressive
• l.nfl.com:general
• l.nfl.com:progressive

This will allow us to prioritize and process each queue as we see fit downstream.

Assumptions

1. You are reading XML Team files that may or may not contain a <sports-metadata> node
2. REDIS is running and you can connect to it.

CLI Options

• --file: the relative or absolute path to the file of interest
• --logs: relative or absolute path to logs. Assumes ./logs if none specified.
• --host: REDIS host. Default localhost
• --port: REDIS port. Default 6379
• --db: REDIS db index. Default 0