npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

sentence-splitter

v5.0.0

Published

split {japanese, english} text into sentences.

Downloads

274,225

Readme

sentence-splitter

Split {Japanese, English} text into sentences.

What is sentence?

This library split next text into 3 sentences.

We are talking about pens.
He said "This is a pen. I like it".
I could relate to that statement.

Result is:

Sentence Image

You can check actual AST in online playground.

Second sentence includes "This is a pen. I like it", but this library can not split it into new sentence. The reason is "..." and 「...」 text is ambiguous as a sentence or a proper noun. Also, HTML does not have suitable semantics for conversation.

As a result, The second line will be one sentence, but sentence-splitter add a contexts info to the sentence node.

{
    "type": "Sentence",
    "children": [
      {
        "type": "Str",
        "value": "He said \"This is a pen. I like it\""
      },
      ...
    ],
    "contexts": [
        {
            "type": "PairMark",
            "pairMark": {
                "key": "double quote",
                "start": "\"",
                "end": "\""
            },
            "range": [
                8,
                33
            ],
            ...
        ]
    ]
}

Probably, textlint rule should handle the "..." and 「...」 text after parsing sentences by sentence-splitter.

Installation

npm install sentence-splitter

Usage

export interface SeparatorParserOptions {
    /**
     * Recognize each characters as separator
     * Example [".", "!", "?"]
     */
    separatorCharacters?: string[]
}

export interface AbbrMarkerOptions {
    language?: Language;
}

export interface splitOptions {
    /**
     * Separator & AbbrMarker options
     */
    SeparatorParser?: SeparatorParserOptions;
    AbbrMarker?: AbbrMarkerOptions;
}

/**
 * split `text` into Sentence nodes.
 * This function return array of Sentence nodes.
 */
export declare function split(text: string, options?: splitOptions): TxtParentNodeWithSentenceNode["children"];

/**
 * Convert Paragraph Node to Paragraph Node that includes Sentence Node.
 * Paragraph Node is defined in textlint's TxtAST.
 * See https://github.com/textlint/textlint/blob/master/docs/txtnode.md
 */
export declare function splitAST(paragraphNode: TxtParentNode, options?: splitOptions): TxtParentNodeWithSentenceNode;

See also TxtAST.

Example

Node

This node is based on TxtAST.

Node's type

  • Str: Str node has value. It is same as TxtAST's Str node.
  • Sentence: Sentence Node has Str, WhiteSpace, or Punctuation nodes as children
  • WhiteSpace: WhiteSpace Node has \n.
  • Punctuation: Punctuation Node has .,

Get these SentenceSplitterSyntax constants value from the module:

import { SentenceSplitterSyntax } from "sentence-splitter";

console.log(SentenceSplitterSyntax.Sentence);// "Sentence"

Node's interface

export type SentencePairMarkContext = {
  type: "PairMark";
  range: readonly [startIndex: number, endIndex: number];
  loc: {
    start: {
      line: number;
      column: number;
    };
    end: {
      line: number;
      column: number;
    };
  };
};
export type TxtSentenceNode = Omit<TxtParentNode, "type"> & {
    readonly type: "Sentence";
    readonly contexts?: TxtPairMarkNode[];
};
export type TxtWhiteSpaceNode = Omit<TxtTextNode, "type"> & {
    readonly type: "WhiteSpace";
};
export type TxtPunctuationNode = Omit<TxtTextNode, "type"> & {
    readonly type: "Punctuation";
};

Fore more details, Please see TxtAST.

Node layout

Node layout image.

This is 1st sentence. This is 2nd sentence.

<Sentence>
    <Str />                      |This is 1st sentence|
    <Punctuation />              |.|
</Sentence>
<WhiteSpace />                   | |
<Sentence>
    <Str />                      |This is 2nd sentence|
    <Punctuation />              |.|
</Sentence>

Note: This library will not split Str into Str and WhiteSpace(tokenize) Because, Tokenize need to implement language specific context.

For textlint rule

You can use splitAST for textlint rule. splitAST function can preserve original AST's position unlike split function.

import { splitAST, SentenceSplitterSyntax } from "sentence-splitter";

export default function(context, options = {}) {
    const { Syntax, RuleError, report, getSource } = context;
    return {
        [Syntax.Paragraph](node) {
            const parsedNode = splitAST(node);
            const sentenceNodes = parsedNode.children.filter(childNode => childNode.type === SentenceSplitterSyntax.Sentence);
            console.log(sentenceNodes); // => Sentence nodes
        }
    }
}

Examples

Reference

This library use "Golden Rule" test of pragmatic_segmenter for testing.

Related Libraries

Tests

Run tests:

npm test

Create input.json from _input.md

npm run createInputJson

Update snapshots(output.json):

npm run updateSnapshot

Adding snapshot testcase

  1. Create test/fixtures/<test-case-name>/ directory
  2. Put test/fixtures/<test-case-name>/_input.md with testing content
  3. Run npm run updateSnapshot
  4. Check the test/fixtures/<test-case-name>/output.json
  5. If it is ok, commit it

Contributing

  1. Fork it!
  2. Create your feature branch: git checkout -b my-new-feature
  3. Commit your changes: git commit -am 'Add some feature'
  4. Push to the branch: git push origin my-new-feature
  5. Submit a pull request :D

License

MIT