huba-gramophone
v0.0.4
Published
extracts most frequently used keywords and phrases from text
Downloads
5
Readme
Gramophone
Extracts most frequently used keywords and phrases from text. It excludes common stop words. It can be configured to extract arbitary length phrases (ngrams) rather than just keywords.
// Get the top two most commonly used phrases of two more words in the Stream Handbook
request('https://github.com/substack/stream-handbook/blob/master/readme.markdown')
.pipe(gramophone.stream({ngrams: 2, html: true, limit: 2}))
.on('data', console.error.bind(console));
Would write out:
readable stream
writable stream
API
- gramophone.extract()
- gramophone.stream()
- gramophone.transformStream()
gramophone.extract(text[, options])
Synchronously extracts keywords from the text. By default it returns any keyword phrases that occur more than once. It also removes any common English words. It returns the results reverse ordered by frequency i.e. the first result is the most common phrase.
keyword.extract('beep beep and foo bar and beep beep and beep beep and foo bar')
Returns ['beep beep', 'foo bar']
.
Option: score
Returns each keyword as an object where term
is the keyword and tf
is the
number of times the phrase was used i.e. the term frequency. Off by default.
keyword.extract('beep beep and foo bar and beep beep and beep beep and foo bar', {score: true})
Returns [{term: 'beep beep', tf: 3}, {term: 'foo bar', tf: 2}]
.
Option: limit
Returns the top N results. The default is to not limit the results.
keyword.extract('beep beep and foo bar and beep beep and beep beep and foo bar', {limit: 1})
Returns ['beep beep']
.
Option: flatten
Returns all occurrences of the ngram. Useful for passing data to Natural's TF-IDF function. Note: the original order is not maintained. Off by default.
keyword.extract('beep beep and foo bar and beep beep and beep beep and foo bar', {flatten: true})
Returns ['beep beep', 'beep beep', 'beep beep', 'foo bar', 'foo bar']
.
Option: html
Extracts the keywords from html text elements. The default is false.
keyword.extract('<strong>beep</strong>, <strong>beep</strong> and <strong>foo</strong>', {html: true})
Returns ['beep', 'foo']
.
Option: min
Only returns results with greater than or equal to N occurences. The default value is 2.
keyword.extract('beep and beep and beep and foo and foo', {min: 3})
Returns ['beep']
.
Option: ngrams
If ngrams is a number (N), only look for phrases with N words. If ngrams is a list ([N1, N2]), only look for the phrases with N1 or N2 words etc.. The defualt is too look for [1, 2, 3] word ngrams.
keyword.extract('beep and beep and beep bop boop and foo and foo bar', {ngrams: [2, 3]})
Returns ['beep bop boop', 'foo bar']
.
Option: stopWords
Add extra stopWords to be used in addition to the English set.
keyword.extract('foo et bar et foo et bar et foo', {stopWords: ['et']})
Returns ['foo', 'bar']
.
Option: startWords
Any words in this list are whitelisted even if they are a stop word.
keyword.extract('foo and bar with foo and bar', {startWords: ['and']})
Returns ['foo and bar']
Option: stem
Apply stemming before extracting keywords. The returned keyword will be the most frequently used word.
keyword.extract('fooing and foo and fooing', {stem: true})
Returns ['fooing']
Option: cutoff
Allows you to specify the cutoff for determining whether to include a phrase that is a component of another phrase. E.g. should "node" and "runs" be extracted as keywords as well as "node runs".
A component phrase is filtered based on the following formula:
phrase freq. / component phrase freq. >= 1 - cutoff
E.g., let's say you have some text that includes the phrase "node runs" 20 times,
"node" 40 times and "runs" 22 times. If the cutoff was 0.5 (the default),
"node" would be included as 20 / 40 >= 1 - 0.5
. However, "runs" would not
be returned as a keyword as 20 / 22 < 1 - 0.5
.
Wow. I could probably make this more intuitive. Open to suggestions.
Option: alternativeTokenizer
Gramophone uses Natural to tokenize and extract ngrams. By default, natural uses
its WordTokenizer which splits words using /\W+/
as a separator regular
expression.
This causes words like "London's" to be split into two tokens rather than one. It also breaks up accented words, using the accented character as the separator.
Setting alternativeTokenizer
to true
uses a more generous regular expression
that respects these characters.
keyword.extract('Lörem Ipsüm Lörem Ipsüm.', {alternativeTokenizer: true})
gramophone.stream([options])
Returns a through stream that reads in the text stream and emits keywords
based on the options passed. It uses the same options as extract
. Note: this
stream behaves like a sink and will buffer the stream completely before emitting
keywords.
See first example.
gramophone.transformStream([options])
Returns a through stream that reads in the stream and emits keywords for each
data read. By default, it assumes that each data read in a string. Alternatively
the stream can read and write to objects. To read
the text from an object property, specify the from
option. If you want to
write the keywords back to the object, also specify the to
option.
var stream = gramophone.transformStream({from: 'text', to: 'keywords'});
stream.write({ text: 'foo and bar and foo'});
stream.end();
Emits the data: { text: 'foo and bar and foo', keywords: [foo] }
.
Related projects
- node-alchemy: a cloud based keyword extraction service.
- natural: a fantastic natural language processing library for node.js. Checkout Tf-Idf if you're looking to extract keywords based on their relative frequency to other documents. If people are interested, I might add Tf-Idf support to gramophone.
Licence & copyright
gramophone is Copyright (c) 2012 B.J. Rossiter.
gramophone is licensed under the MIT licence. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE file for more details.