npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

openhim-mediator-file-queue

v3.0.1

Published

An async file queue for OpenHIM mediators

Downloads

31

Readme

File queue

To install, execute npm install -g openhim-mediator-file-queue

Once installed you will have to manually edit the config files in the location where global npm modules are installed.

  • With NVM this is usually: ~/.nvm/versions/node/v4.1.0/lib/node_modules/openhim-mediator-file-queue/config
  • With a standard node install this is usually: /usr/local/lib/node_modules/openhim-mediator-file-queue/config

You may add a production.json file to the config folder to override or add any config values.

For development purposes, in order to communicate to the OpenHIM core, since it uses self-signed certificates, execute:

  NODE_TLS_REJECT_UNAUTHORIZED=0 npm start

See this visual example of how this work as an OpenHIM mediator. Note the curl commands return instantly but the mediator knows to tell the OpenHIM that these are still processing and updates them after they complete asynchronously.

Alt text

How it works

The file queue simply handles incoming requests, writing the files to a directory on the filesystem, and then processes the queue, sending the files to a configured endpoint. If the file is successfully sent, then the queued file is deleted from the filesystem, otherwise it is moved to an error directory.

Multiple "endpoints" can be configured. Each endpoint handles incoming requests for a specific URL, queues them, and then sends them to another configured URL. An endpoint has a "worker" which is responsible for reading the files from the queue and processing them. Workers can process multiple files in parallel as configured (by default 2 at a time). Workers can be paused/unpaused or repopulated via a RESTlike endpoint. Pausing a worker will stop it from processing files from the queue, but the endpoint will continue accepting requests and writing the files to the queue. Repopulating a worker will cause it to refresh its queue from the files on the filesystem. This is useful when manually adding files to or removing files from the queue.

To pause a queue: ./scripts/pause.sh <worker-name>

To resume a queue: ./scripts/resume.sh <worker-name>

To repopulate a queue: mv ./error/<worker-name>/* ./queue/<worker-name>/ ./scripts/repopulate.sh <worker-name>

Each "endpoint" must have a matching channel and route registered in OpenHIM, for receiving requests and forwarding them to the file queue mediator. When the mediator starts up a new channel will automatically be created/updated based off the settings for the upstream server (where the files will be forwarded to). Note the the channel will be updated each time the endpoint settings are updated.

Please note that the following manual steps might be required to get the File Queue to function correctly:

  • If the mediator is running on the same server as the OpenHIM server, then you may need to update the route host to the IP address of that server for OpenHIM, instead of using localhost.
  • The user role file-queue has been created by default for all channels. This role needs to be added to the user account which will be used to push new files onto the queue.

Here is an example config for a channel:

  {
    "name": "File Queue",
    "urlPattern": "^/test$",
    "status": "enabled",
    "routes": [
      {
        "name": "File queue",
        "host": "localhost",
        "path": "/test",
        "port": "4002",
        "secured": false,
        "primary": true,
        "type": "http"
      }
    ],
    "authType": "private",
    "allow": [
      "file-queue"
    ]
  }

Endpoint config

An array of endpoints should be configured in the config file. An endpoint can have the following properties:

  • name (required) - The name of the endpoint which is used for setting up the RESTlike routes for the worker.
  • path (required) - The path to use for handling incoming requests. A '*' may be used to match any number of characters in a path. E.g. 'this/is/a/test/*'. Note that this path must match the path for the route accepting these requests.
  • url (required) - The URL to send the files to when processing them from the queue.
  • paused - Whether or not the endpoint's worker should be paused by default. This must be a boolean value. Defaults to false.
  • parallel - The number of files that the worker should process in parallel. Defaults to 2.
  • updateTx - Whether or not to attempt to update the OpenHIM transaction once the message has been processed. Defaults to false.
  • forwardMetadata - Whether or not to use the url, headers and HTTP method of the original request when it is forwarded. The path of the original request will be appended to the url property. Defaults to false.

Here is an example config file:

{
  "port": 4002,
  "log_level": "info",
  "endpoints": [
    {
      "name": "test",
      "path": "/test",
      "url": "http://localhost:9999",
      "updateTx": true,
      "forwardMetadata": true
    }
  ]
}