@iarna/word-count

Count words, with Unicode! Uses Unicode 11.0.0 character classes for improved clarity of implementation.

const wordCount = require('@iarna/word-count')
console.log(wordCount("how many words is this?")) // 5

Specifically, we consider a word a run of 1 or more characters in these sets:

• General_Category/Letter
• General_Category/Letter_Number
• General_Category/Connector_Punctuation
• General_Category/Number
• Word_Break/MidNum
• Word_Break/MidNumLet
• Word_Break/Single_Quote
• Soft hyphens (\u00AD)
• A singular hyphen

Multiple hyphens in a row, that is -- and --- are considered word separators as they're often used as stand-ins for endashes – and emdashes — respectively.

The tests make it pretty clear what it's doing:

  t.is(wordCount('This is a test'), 4, 'plain text')
  t.is(wordCount('now with 23 a number'), 5, 'integer')
  t.is(wordCount('now with 23.17'), 3, 'decimal')
  t.is(wordCount("emoji 😍😍 do not count"), 4, 'emoji')
  t.is(wordCount("possessive's are one word"), 4, 'possessive')
  t.is(wordCount('possessive’s are one word'), 4, 'possessive unicode')
  t.is(wordCount('some "quoted text" does not impact'), 6, 'quotes')
  t.is(wordCount("also 'single quotes' are ok"), 5, 'single quotes')
  t.is(wordCount("don't do contractions"), 3, 'contractions count as a single word')
  t.is(wordCount('hyphenated words-are considered whole'), 4, 'hyphenated words')
  t.is(wordCount('underbars are_too just one'), 4, 'underbars')
  t.is(wordCount('n-dash ranges 1–3 are NOT'), 6, 'en-dash')
  t.is(wordCount('m-dash connected—bits also are not'), 6, 'em-dash')

Many more naive implementations match just \w but that only get's you (some) English and even then things like possessives and, depending on how you look at, contractions get over counted.

To the best of my knowledge this should successfully count words in any language that uses word-separators. Counting words in languages without word-separators is rather harder and the heuristics are language specific.

If you happen to give this a run of, say, Chinese characters, it will consider each group outside of punctuation to be a word, massively under counting. So yeah, use a language specific counter:

For instance, for Chinese there's nseg an implementation of MMSEG.

PRIOR ART

• word-count — A word counter that matches \w plus some ranges of CJK characters. CJK characters are each counted as one-word-per-character.
• wordcount — A fork of the origin of word-count, it adds ranges for Cyrillic.
• split-string-words — not a word counter per say, but you could count the words it returns. It looks for \S while allowing double quoted strings.
• wordcount-stream — A stream based word counter that matches using \w. *
• string_utils — Splits words on the space character.
• wordcounter — Counts words using iso-8859-1, that is, some English and European characters.
• string-stats — Uses a list it documents as "Latin, Greek, Coptic, Cyrillic, Armenian, Hebrew, Syriac, Arabic" though it's ad-hoc so I don't know if its complete or if that's all it has.
• word-counter — Splits on \s. Has a run-time dependency on coffee-script.

FOR FUTURE REFERENCE

Unicode has a report relating to word boundaries that may be useful. It's focused on determining boundaries for things like double-clicking and spell check, but it's definitately a related function.