Character primitives used by Yozora tokenizers: code-point constants, character predicates, source-position tracking, entity references, and Unicode case folding.

Install

npm install --save @yozora/character

Usage

import {
  createNodePointGenerator,
  foldCase,
  isPunctuationCharacter,
} from '@yozora/character'

const points = [...createNodePointGenerator('Hello, 世界')]
const identifier = foldCase('Yozora')
const commaIsPunctuation = isPunctuationCharacter(','.codePointAt(0)!)

Key exports include:

• AsciiCodePoint, Unicode category enums, and VirtualCodePoint
• ASCII/Unicode whitespace, punctuation, control, digit, and letter predicates
• createNodePointGenerator and helpers for reading text from node-point intervals
• named and numeric entity-reference parsing
• foldCase, stripChineseCharacters, and tightenChineseCharacters

Related

• homepage