Function simpleTokenizer

  • simpleTokenizer(value: string): string[]

    Simple tokenizer for breaking text into lowercase terms.

    • Converts the string to lowercase using toLocaleLowerCase()
    • Splits on any sequence of non-word characters or underscores
    • Filters out any empty strings from the result

    Parameters

    • value: string

      The text to tokenize

    Returns string[]

    An array of lowercase tokens

    Example

    simpleTokenizer("Hello, World!");
    // → ["hello", "world"]

Settings

Member Visibility