sentence-splitter

Split {Japanese, English} text into sentences.

What is sentence?

This library split next text into 3 sentences.

We are talking about pens.
He said "This is a pen. I like it".
I could relate to that statement.

Result is:

You can check actual AST in online playground.

• https://sentence-splitter.netlify.app/#We%20are%20talking%20about%20pens.%0AHe%20said%20%22This%20is%20a%20pen.%20I%20like%20it%22.%0AI%20could%20relate%20to%20that%20statement.

Second sentence includes "This is a pen. I like it", but this library can not split it into new sentence. The reason is "..." and 「...」 text is ambiguous as a sentence or a proper noun. Also, HTML does not have suitable semantics for conversation.

• html - Most semantic way to markup a conversation (or interview)? - Stack Overflow

As a result, The second line will be one sentence, but sentence-splitter add a contexts info to the sentence node.

1{
2    "type": "Sentence",
3    "children": [
4      {
5        "type": "Str",
6        "value": "He said \"This is a pen. I like it\""
7      },
8      ...
9    ],
10    "contexts": [
11        {
12            "type": "PairMark",
13            "pairMark": {
14                "key": "double quote",
15                "start": "\"",
16                "end": "\""
17            },
18            "range": [
19                8,
20                33
21            ],
22            ...
23        ]
24    ]
25}

• Example: https://sentence-splitter.netlify.app/#He%20said%20%22This%20is%20a%20pen.%20I%20like%20it%22.

Probably, textlint rule should handle the "..." and 「...」 text after parsing sentences by sentence-splitter.

• Issue: Nesting Sentences Support · Issue #27 · textlint-rule/sentence-splitter
• Related PR
  • https://github.com/textlint-ja/textlint-rule-no-doubled-joshi/pull/47
  • https://github.com/textlint-ja/textlint-rule-no-doubled-conjunctive-particle-ga/pull/27
  • https://github.com/textlint-ja/textlint-rule-max-ten/pull/24

Installation

npm install sentence-splitter

Usage

1export interface SeparatorParserOptions {
2    /**
3     * Recognize each characters as separator
4     * Example [".", "!", "?"]
5     */
6    separatorCharacters?: string[]
7}
8
9export interface AbbrMarkerOptions {
10    language?: Language;
11}
12
13export interface splitOptions {
14    /**
15     * Separator & AbbrMarker options
16     */
17    SeparatorParser?: SeparatorParserOptions;
18    AbbrMarker?: AbbrMarkerOptions;
19}
20
21/**
22 * split `text` into Sentence nodes.
23 * This function return array of Sentence nodes.
24 */
25export declare function split(text: string, options?: splitOptions): TxtParentNodeWithSentenceNode["children"];
26
27/**
28 * Convert Paragraph Node to Paragraph Node that includes Sentence Node.
29 * Paragraph Node is defined in textlint's TxtAST.
30 * See https://github.com/textlint/textlint/blob/master/docs/txtnode.md
31 */
32export declare function splitAST(paragraphNode: TxtParentNode, options?: splitOptions): TxtParentNodeWithSentenceNode;

See also TxtAST.

Example

• Online playground: https://sentence-splitter.netlify.app/

Node

This node is based on TxtAST.

Node's type

• Str: Str node has value. It is same as TxtAST's Str node.
• Sentence: Sentence Node has Str, WhiteSpace, or Punctuation nodes as children
• WhiteSpace: WhiteSpace Node has \n.
• Punctuation: Punctuation Node has ., 。

Get these SentenceSplitterSyntax constants value from the module:

1import { SentenceSplitterSyntax } from "sentence-splitter";
2
3console.log(SentenceSplitterSyntax.Sentence);// "Sentence"

Node's interface

1export type SentencePairMarkContext = {
2  type: "PairMark";
3  range: readonly [startIndex: number, endIndex: number];
4  loc: {
5    start: {
6      line: number;
7      column: number;
8    };
9    end: {
10      line: number;
11      column: number;
12    };
13  };
14};
15export type TxtSentenceNode = Omit<TxtParentNode, "type"> & {
16    readonly type: "Sentence";
17    readonly contexts?: TxtPairMarkNode[];
18};
19export type TxtWhiteSpaceNode = Omit<TxtTextNode, "type"> & {
20    readonly type: "WhiteSpace";
21};
22export type TxtPunctuationNode = Omit<TxtTextNode, "type"> & {
23    readonly type: "Punctuation";
24};

Fore more details, Please see TxtAST.

Node layout

Node layout image.

• Example: https://sentence-splitter.netlify.app/#This%20is%201st%20sentence.%20This%20is%202nd%20sentence.

This is 1st sentence. This is 2nd sentence.

<Sentence>
    <Str />                      |This is 1st sentence|
    <Punctuation />              |.|
</Sentence>
<WhiteSpace />                   | |
<Sentence>
    <Str />                      |This is 2nd sentence|
    <Punctuation />              |.|
</Sentence>

Note: This library will not split Str into Str and WhiteSpace(tokenize) Because, Tokenize need to implement language specific context.

For textlint rule

You can use splitAST for textlint rule. splitAST function can preserve original AST's position unlike split function.

1import { splitAST, SentenceSplitterSyntax } from "sentence-splitter";
2
3export default function(context, options = {}) {
4    const { Syntax, RuleError, report, getSource } = context;
5    return {
6        [Syntax.Paragraph](node) {
7            const parsedNode = splitAST(node);
8            const sentenceNodes = parsedNode.children.filter(childNode => childNode.type === SentenceSplitterSyntax.Sentence);
9            console.log(sentenceNodes); // => Sentence nodes
10        }
11    }
12}

Examples

• textlint-ja/textlint-rule-max-ten: textlint rule that limit maxinum ten(、) count of sentence.

Reference

This library use "Golden Rule" test of pragmatic_segmenter for testing.

• diasks2/pragmatic_segmenter: Pragmatic Segmenter is a rule-based sentence boundary detection gem that works out-of-the-box across many languages.

Related Libraries

• textlint-util-to-string
• and https://github.com/textlint/textlint/wiki/Collection-of-textlint-rule#rule-libraries

Tests

Run tests:

npm test

Create input.json from _input.md

npm run createInputJson

Update snapshots(output.json):

npm run updateSnapshot

Adding snapshot testcase

1. Create test/fixtures/<test-case-name>/ directory
2. Put test/fixtures/<test-case-name>/_input.md with testing content
3. Run npm run updateSnapshot
4. Check the test/fixtures/<test-case-name>/output.json
5. If it is ok, commit it

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request :D

License

MIT