A TokenizerProvider provides a Tokenizer for a given language.
More...
#include <zorba/tokenizer.h>
A TokenizerProvider provides a Tokenizer for a given language.
Definition at line 284 of file tokenizer.h.
virtual zorba::TokenizerProvider::~TokenizerProvider |
( |
| ) |
|
|
virtual |
Creates a new Tokenizer.
- Parameters
-
lang | The language of the text that the tokenizer will tokenize. |
state | The State to use. If null , t is not set. |
t | If not null , set to point to a Tokenizer for lang. |
- Returns
- Returns
true
only if this provider can provide a tokenizer for lang.
The documentation for this class was generated from the following file: