folderMonitor

Dependencies:

• NodeJS v16 or higher.
• inotifywatcher (On Ubuntu use apt install inotify-tools)

Description:

With this program you can monitor folder and their subfolders for new files. When new files are created or moved into the specified folder path it will execute the specified program.

Usage:

Usage: node ./dist/foldermonitor.js [options]
Show this help message
    --help
Start processing immediately instead if first finding files already in search paths.
    -i, --immediate
Job idle timeout value to scan for new files missed by inotifywatch when no jobs are running for this ammount of time in seconds. (Default: 10)
    -t, --timeout <value>
Show extra debugging info
    -d, --debug
Path to json config file
    -c, --config <value> (required)

Configuration:

This program needs a configuration file in the json format. It should be an array of job objects.

Run the programs as: foldermonitor -c config.json

Following is the main format:

[
    {
        <job object>
    },
    {
        <job object>
    },
    .....
]

The <job object> has the following properties.

• id: string. Label for this job. Will be used in the stdout logging.
• path: string. Path to monitor for folders.
• recursive: boolean. Also check for new files in sub folders below path.
• regex?: string. Regular expression. Only files for which the basename matches this regular expression are used for executing the program.
• program: string. Full path to executable to run when a new file event is triggered.
• arguments? Array<string>. Array of string values. Will be used as argument when program is executed.
  • The following strings in the arguments will be replaced when seen:
    • %FILENAME% will be replaced with the fullpath of the file which triggered the execution of the program.
    • %BASENAME% will be replaced with the basename of the file which triggered the execution of the program.
    • %DIRNAME% will be replaced with the dirname of the file which triggered the execution of the program.
    • %PATH% will be replaced with the path property of the job object.
• cwd?: string. Directory to change to before executing program.
• timeout?: number. When specified is the max time in seconds the program is allowed to run.
• maxRunners?: number. Defines how many numbers of the same program may run at the same time.
• cpuQuota?: string. The maximum amount of CPU time a job can get on a CPU core. Takes a percentage string and is passed directly to systemd-run.
• memoryMax?: string. The maximum amount of memory the job is allowed to use. If it exceeds this threshold it is killed. Takes a byte value suffixed by K, M, G or T. See systemd.resource-control(5) for more details on the exact syntax.

Example of a config file:

[
    {
        "id": "Test 1 Only one instance at a time",
        "path": "/data/test1/in",
        "recursive": true,
        "regex": ".*\\.pcap$",
        "maxRunners": 1,
        "program": "/data/scripts/process_pcap.sh",
        "arguments": [
            "-i", "%FILENAME%",
            "-o", "/data/test1/out/%BASENAME%",
            "-e", "/data/test1/error/%BASENAME%"
        ]
    },
    {
        "id": "Test 2 Max 4 instances at a time",
        "path": "/data/test2/in",
        "recursive": true,
        "maxRunners": 4,
        "program": "/data/scripts/test2.sh",
        "arguments": [
            "%FILENAME%"
        ]
    }
]

Timeout Option "--timeout":

Due to a known bug in inotifywatch it can sometimes mis new files when a file is created or moved into a folder which was created just before the file.

This timeout will force the program to scan for "missed" files when for the specified timeout time a job was not started. This interval is per configured job.

some notes about process accounting

Internally, folderMonitor heavily uses systemd features to ensure that a stray job takes down the entire system. The process is as follows:

• the folder monitor defines a separate slice unit with restricted CPU, Memory and Task limits. All jobs are launched into this slice
• each job that the folder monitor creates is launched via systemd-run into its own scope unit with limited resources. This scope is also configured to clean the entire scope of processes when stopping, so forking jobs should be mindful that they cannot fork and expect their children to hang around indefinitely