Cheshire3 Objects: Tokenizer

Description

A Tokenizer takes a string of language and processes it to produce an ordered list of tokens.

Implementations

The following implementations are pre-configured and ready to use.
They may be used out of the box in configurations for Indexes, Workflows etc.

API

Module: cheshire3.tokenizer
Classes:

Methods:

FunctionParametersReturnsDescription
__init__config, parent 
process_stringsession, datalistProcess a raw string to produce an ordered list of tokens
process_hashsession, datalistProcess the text of each item in the hash to produce an ordered list of tokens for that hash entry.

Sub-Packages