A Callback is called once per token. More...
#include <zorba/tokenizer.h>
Public Types | |
typedef Tokenizer::size_type | size_type |
Public Member Functions | |
virtual void | item (Item const &item, bool entering) |
This member-function is called whenever an item that is being tokenized is entered or exited. | |
virtual void | token (char const *utf8_s, size_type utf8_len, locale::iso639_1::type lang, size_type token_no, size_type sent_no, size_type para_no, Item const *item=0)=0 |
This member-function is called once per token. | |
virtual | ~Callback () |
A Callback is called once per token.
This is only internally by Zorba. You do not need to derive from this class.
Definition at line 74 of file tokenizer.h.
Definition at line 76 of file tokenizer.h.
|
virtual |
This member-function is called whenever an item that is being tokenized is entered or exited.
item | The item being entered or exited. |
entering | If true , the item is being entered; if false , the item is being exited. |
|
pure virtual |
This member-function is called once per token.
utf8_s | The UTF-8 token string. It is not null-terminated. |
utf8_len | The number of bytes in the token string. |
lang | The language of the token. |
token_no | The token number. Token numbers start at 0. |
sent_no | The sentence number. Sentence numbers start at 1. |
para_no | The paragraph number. Paragraph numbers start at 1. |
item | The Item this token is from, if any. |