pcaptorabbit

With this program you can monitor a folder for new pcap files and send the data to a RabbitMQ. Each pcap packet will become a single message in Rabbit RabbitMQ. The message is a json object with all the information about the source of the pcap packet.

Requirements:

• NodeJS (v20.0.0 or higher).
• Capture file must be in pcap or pcap-ng (data in 'Enhanced Packet Block' blocks) format.

Installation:

This programm also exists on the online NPM repository. Install from there with 'npm install @todle/pcaptorabbit -g'. This will install de program 'pcaptorabbit' globally.

To install the checkedout git repository version use the command 'npm install -g' will install the program 'pcaptorabbit' globally.

Usage

Usage: pcaptorabbit [options]
Show this help message
    --help
Only process tipcp packets and split them over multiple queues based on least significant ip address byte.
    --splitIp
Only process tcp packets and split them over multiple queues based on ip-port combinations.
    --splitTcp
Only ouput packets to/from specified port(s) (comma separated list of port numbers)
    --portfilter <value>
Number of queues to write to Quename will be '<quename>_<#>'. (Default: 1).
    --queueCount <value>
Path to file when it exists the processing of files is paused at the next file. Default not used.
    --pauseFile <value>
RabbitMq queue (Default: 10). Use -1 for indefinitly
    --amqpReconnectTries <value>
RabbitMq queue (Default: 20 seconds)
    --amqpReconnectTimeout <value>
RabbitMq queue (Default: pcap)
    --amqpQueue <value>
Lawfull intercept ID (Default: pcap)
    --amqpRoutingKey <value>
RabbitMQ exchange (Default: pcap)
    --amqpExchange <value>
URI to amqp (RabbitMQ) host (Default: amqp://localhost)
    --amqpHost <value>
Path to folder to move pcap files to after an error
    -e, --errorpath <value>
Path to folder to move pcap files to after reading
    -o, --outpath <value>
Path to pcap files to read
    -i, --inpath <value>
Regular expression for filename(s) to find. Default: '.*'.
    -p, --pattern <value>
When processing multiple files (--inpath or --fileList) do not move file to specified out path (--outpath).
    --doNotMoveAfterProcessing
File containing list of files to process. One file per line.
    --fileList <value>
Read only specified file
    -f, --filename <value>
Show extra debug info
    -d, --debug

For examle read all existing and new pcap files below "/data/in".

pcaptorabbit -i "/data/in/" -p ".*\\.pcap$" -o "/data/out/" -e "/data/error/"

The "-p" ("--pattern") argument is a javascript regular expression. You must escape backslashes once if you want them to be used as a regular espression backslash and twice if you want it to be a backslash character in you regular expression. So ".*\\.pcap$" becomes ".*\.pcap$" for your javascript regular expression and ".*\\\.pcap$" becomes ".*\\.pcap$".

The data is send packet as a protocol buffer. For the layout see the 'pcaptorabbit.proto' file.

IP or TCP packet can be split over multiple RabbitMQ queues to spread the load. For this use the "--queueCount", "--splitTcp" or "--splitIp" options. For TCP all packets of the same session (tuple address) are send to the same queue and for IP the least significat byte of the address is used to determine the queue.

When "--splitTcp" option is used yo can also use the "--portfilter" option to oly send TCP packets where the source or destination port is in the port filter list. Limiting the amount of data send to RabbitMQ.

When "--queueCount" is 1 then data is send to the specified queue name without the count index "<queuename>". When this value is bigger than 1 then queue's are created with thename "<queuename>_<#>".