npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@datagica/datanote-api-job-balancer

v0.0.3

Published

Datanote API Job Balancer

Downloads

5

Readme

Datanote API job balancer

Job control server for the Datanote API.

Architecture

TLDR

The job control server is in charge of collecting user requests, transform them into jobs, and serve jobs to job runner instances. The user requests stay on hold using Promises.

The job runner instances process the requests, then submit the results back to the master which finalizes the pending request promises.

Key components

There are four key components in the API:

datanote-api-job-balancer

The single entry point to the API connecting user requests with runner instances. The current implementation is designed to have only one instance of the balancer running, as all the data is stored in the memory of the instance, a few Kb (up to a few Mb for large books).

In the future if we want to scale to multiple balancers we will need to store data in a shared database such as Redis, or use an intermediary between the api and workers, like a message queue.

datanote-api-job-runner

A job runner instance that connects to the balancer using websockets, to get jobs to do in the background. Sends the results back to the balancer server once completed.

datanote-api-engine

The heart of Datanote, a big bundle of all modules and datasets used to analyze documents. It also exposes a toolkit of utility functions. Used by the workers.

datanote-api-client

Client library used to perform requests on the Datanote API using websockets. This is the module used by the Datanote desktop app.

Deployment

The Datanote API is currently hosted on Now.sh on a Pro account.

Deployment of the job balancer

cd datanote-api-job-balancer
npm run deploy
now alias <NEW_INSTANCE> beta-api.datanote.io
now scale beta-api.datanote.io 1

Note: for the moment, there can be only one instance running on the beta-api

Deployment of the job runners

cd datanote-api-job-runner
npm run deploy

At this point, you have to understand that shuting down old workers will stop any pending query on it, which is not good for the quality of service.

The simple solution to this is to increment the version of workers and send it to the master. The master will only give jobs to workers of version >= current.

So, wait for a few minutes, then you can finally scale the new version:

now alias <NEW_INSTANCE> beta-api-worker.datanote.io

(this will kill all old instances except for one, which will be frozen a bit later automatically)

Scaling the API

For the moment we have to keep a single master instance, but we can scale the number of workers this way:

now scale beta-api-worker.datanote.io <SIZE>

For testing, it is best to use a size like 3. For heavy loads, we could use something like 10 or 20.

TODO

  • support multiple formats:
    • datanote: a custom, low-level format supported by Datanote
    • json: basic list of entities
    • gexf: GEXF graph
    • csv: CSV graph (for Neo4J) https://neo4j.com/developer/guide-import-csv/

List of features

Domains

The following domains are recognized by the API, but you can also define your own (see the Custom fields paragraph).

detective:

  • address
  • email
  • event
  • evidence
  • family
  • phone
  • position
  • protagonist
  • weapon

lifesciences:

  • muscle
  • animal
  • artery
  • bacteria
  • cell
  • disease
  • drug
  • healthRisk
  • institution
  • nerve
  • protagonist
  • location
  • protein
  • symptom
  • virus

marketing:

  • businessEvent
  • institution
  • position
  • protagonist
  • social
  • product

engineer:

  • engineeringElement
  • engineeringEvent
  • healthRisk
  • institution
  • protagonist

workplace:

  • address
  • diploma
  • email
  • family
  • institution
  • interest
  • language
  • phone
  • position
  • protagonist
  • skill
  • social

fiction:

  • character
  • location
  • weapon
  • event
  • evidence

arts:

  • email
  • phone
  • address
  • compensation
  • protagonist
  • artist
  • interest

generic:

  • businessEvent
  • engineeringEvent
  • intelligenceEvent
  • protagonist
  • location

Custom fields

Optional url parameters:

  • locale: en, fr (example: ?locale=en, &locale=fr..)
  • fields: values to keep (example: fields=id,label, &fields=label,links,target..)
  • domain: PoliceReport, see source for more (example: ?domain=PoliceReport..)
  • types: bacteria, address, event, see source for more
    • format: graphson, gdf, gexf (example: ?format=gdf..)

Note: since domain cannot be used at the same time as types, types will have priority and domain will have no effect.

Domains and entity types

Current extraction model (you can change this, if your edit engine.js):

{
  PoliceReport: [
    'email',
    'phone',
    'location',
    'evidence',
    'event',
    'protagonist',
    'position',
    'weapon',
  ],
  generic: [
    'protagonist',
  ]
}

Legacy REST API Usage

Examples use httpie with jq, but you can also use curl or something else.

The content-type is optional, it can help the app if there is an encoding issue with magic number.

Example with curl

curl -X POST "http://localhost:3000?locale=en&types=animal&format=gdf" -d "THE HIPPO KILLS THE DOLPHIN"
curl -X POST "http://localhost:3000?locale=en&types=protagonist,weapon&format=gdf" -d "James bond buys an ak-47"
curl -X POST "http://localhost:3000" --data-binary "@tests/fixtures/police_en.txt"
curl -X POST "http://localhost:3000?locale=en&types=protagonist,virus" -d "James Bond has caught the terrorist carrying H5N1"

Example with httpie and jq

https POST "http://localhost:3000?locale=en&types=virus" body="the monkey died of ebola" | jq
https POST "http://localhost:3000" body="James Bond" | jq
https POST "http://localhost:3000" body="James Bond" | jq
https POST "http://localhost:3000?&fields=label,links,link,target&locale=en" body="James Bond"  | jq
https POST "http://localhost:3000?locale=en" body="James Bond"  | jq
https POST "http://localhost:3000?&fields=label,links,link,target" body="James Bond"  | jq
https POST "http://localhost:3000?locale=en&types=protagonist,virus" body="James Bond has caught the terrorist carrying H5N1" | jq

Longer example

https POST "http://localhost:3000?fields=link,links,target,properties,ngram,begin,end,label,gender,number,firstname,lastname&locale=en" body="James Bond buys an AK-47"

output:

{
  "type": "record",
  "label": {},
  "properties": {},
  "links": [
    {
      "link": {
        "type": "link",
        "label": "Mentions"
      },
      "properties": {
        "ngram": "James Bond",
        "begin": 0,
        "end": 10
      },
      "target": {
        "properties": {
          "firstname": "james",
          "lastname": "bond",
          "gender": [
            "m"
          ]
        },
        "links": [
          {
            "link": {
              "type": "link",
              "label": "Type"
            },
            "properties": {},
            "target": {
              "type": "entity",
              "label": "Protagonist"
            }
          },
          {
            "link": {
              "type": "purchase",
              "label": "Purchase"
            },
            "properties": {},
            "target": {
              "properties": {
                "number": "singular",
                "gender": "neutral"
              },
              "links": [
                {
                  "link": {
                    "type": "link",
                    "label": "Type"
                  },
                  "properties": {},
                  "target": {
                    "type": "entity",
                    "label": "Generic"
                  }
                }
              ],
              "label": "AK-47",
              "type": "entity"
            }
          }
        ],
        "label": "James BOND",
        "type": "entity"
      }
    },
    {
      "link": {
        "type": "link",
        "label": "Mentions"
      },
      "properties": {
        "begin": 19,
        "end": 24,
        "ngram": "AK-47"
      },
      "target": {
        "properties": {
          "number": "singular",
          "gender": "neutral"
        },
        "links": [
          {
            "link": {
              "type": "link",
              "label": "Type"
            },
            "properties": {},
            "target": {
              "type": "entity",
              "label": "Generic"
            }
          }
        ],
        "label": "AK-47",
        "type": "entity"
      }
    }
  ]
}```

### Medical example

```bash
https POST "http://localhost:3000?locale=en&types=virus" body="H5N1" | jq
{
  "type": "record",
  "id": "record:undefined__undefined",
  "date": "2017-07-11T22:27:51.438Z",
  "label": {},
  "indexed": "H5N1",
  "properties": {},
  "links": [
    {
      "link": {
        "type": "link",
        "id": "link:mention",
        "label": "Mentions",
        "description": "Mention in a document",
        "aliases": [
          "mentioned in",
          "has a mention",
          "is mentioned",
          "are mentioned"
        ]
      },
      "properties": {
        "ngram": "H5N1",
        "score": 1,
        "sentence": 1,
        "word": 0,
        "begin": 0,
        "end": 4
      },
      "target": {
        "properties": {
          "category": "species"
        },
        "links": [
          {
            "link": {
              "type": "link",
              "id": "link:instanceof",
              "label": "Type",
              "plural": "Types",
              "description": "Of type",
              "aliases": [
                "of type"
              ]
            },
            "properties": {},
            "target": {
              "type": "entity",
              "id": "entity:virus",
              "label": "Virus",
              "plural": "Viruses",
              "description": "Virus",
              "aliases": [
                "virus",
                "viruses"
              ]
            }
          }
        ],
        "id": "entity:virus__influenza-a-virus-h5n1",
        "label": "Influenza A (H5N1)",
        "description": "Influenza A virus (subtype H5N1)",
        "aliases": [
          "H5N1",
          "H5N1 flu",
          "Influenza A H5N1",
          "Influenza A (H5N1)",
          "Influenza A subtype H5N1",
          "Influenza A (subtype H5N1)",
          "Influenza A (H5N1 subtype)"
        ],
        "type": "entity"
      }
    }
  ]
}

GDF

curl -X POST "http://localhost:3000?locale=en&types=animal,virus&format=gdf" -d "the monkey has ebola"
nodedef>id VARCHAR,label VARCHAR
entity:animal__monkey,Monkey
entity:virus__ebolavirus,Ebolavirus
edgedef>id VARCHAR,source VARCHAR,target VARCHAR

Graphson

curl -X POST "http://localhost:3000?locale=en&types=animal,virus&format=graphson" -d "the monkey has ebola"
{
  "graph": {
    "mode": "NORMAL",
    "vertices": [
      {
        "_id": "entity:animal__monkey",
        "name": "Monkey",
        "_type": "vertex"
      },
      {
        "_id": "entity:virus__ebolavirus",
        "name": "Ebola",
        "_type": "vertex"
      }
    ],
    "edges": []
  }
}