categorize-metadata

A quick-and-dirty module for calculating the mean (average) keyword density of a set of metadata fields against a category dictionary you pass in. Returns an array of categories and their mean keyword density when it is equal to or above the configurable keyword density threshold, ordered from highest to lowest.

Usage and options documented below. Check the /test directory for more details.

NPM Install

npm install categorize-metadata --save

Options

See config/index.js. Any of the config'd fields can be overridden using the options argument. The following is the default config object:

{
  /* keywordDensityThreshold:
   * the desired minimum keyword density to qualify the metadata for a category
   * note that the default is pretty low, you can increase it to raise the bar
   */
  keywordDensityThreshold: 0.05,

  /* maxCategories:
   * the max number of categories to return, in order of keyword density
   * by default, there is no limit. set to integer to override default.
   */
  maxCategories: undefined

  /* fields:
   * the metadata fields to evaluate together, plus their relative weighting.
   * the defaults below assume you are evaluating a web page, but you can
   * easily pass in different fields/weights to evaluate any kind of metadata.
   * The default weight of each field is 1, but you can tweak the weight below
   * to favor one field over another since the keyword density will be
   * multiplied by the weight at the end. For example, you could set the `url`
   * weight to 1.2 and the others to 0.8 to favor urls when calculating the
   * overall mean keyword density.
   */
  fields: {
    'url': { weight: 1 },
    'title': { weight: 1 },
    'image': { weight: 1 },
    'description': { weight: 1 },
    'keywords': { weight: 1 },
    'og:type': { weight: 1 }
  },

  /* stopwords
   * stopwords are the throw-away words that are excluded from keyword density
   * measure ('a', 'the', 'is', etc). the default stopwords here are in
   * english from the npm `stopwords` module but can be overridden:
   * https://www.npmjs.com/package/stopwords
   */
  stopwords: require('stopwords').english,

  /* metadata.decode
   * by default, the decode function assumes the metadata fields are not
   * encoded in any way. if you want to pass in encoded metadata fields, you
   * can set a decoding function here
   */
  metadata: {
    decode: function (value) { return value }
  },

  /* debugOutput
   * set to true for cli output
   */
  debugOutput: false
}

Usage Inside your program:

Example usage below. For more details, look at /test/basic.js.

const categorize = require('categorize-metadata')

const metadata = {
  url: 'http://example.com/foo',
  title: 'My Example Webpage Title',
  image: 'http://image.foo.com/foo.jpg',
  description: 'lorem ipsum foo bar'
}

const categoryDictionary = {
  ClimateChange: {
    keywords: [ 'arctic', 'carbon', 'climate' ]
  },
  Environment: {
    keywords: [ 'aquifer', 'poison', 'pollution', 'river']
  }
}

const options = {
  metadata: {
    decode: function (str) {
      return decodeURIComponent(str).replace(/['*]/g, unescape)
    }
  },
  debugOutput: true
}

categorize(metadata, categoryDictionary, options,
  function (err, categories) {
    if (err) return console.log(err)
    console.log(categories)
    // do more stuff here!
  })

Returns

Array of categories whose mean keyword density is above the configured keywordDensityThreshold. Ordered by meanKeywordDensity:

[ { category: 'Environment', meanKeywordDensity: 0.2171 },
  { category: 'ClimateChange', meanKeywordDensity: 0.0765 } ]