Class Tokenizer

Simple stream-oriented parser for efficient basic recognition of incoming data.

class Tokenizer( [seps], [options], [tokLen], [source] )

more...

Summary

initSimple stream-oriented parser for efficient basic recognition of incoming data.
next()Advances the tokenizer up to the next token.
nextToken()Returns the next token from the tokenizer
parse()Changes or set the source data for this tokenizer.
rewind()Resets the status of the tokenizer.
token()Get the current token.

Detailed description

Simple stream-oriented parser for efficient basic recognition of incoming data.

The tokenizer class is meant to provide simple and efficiente logic to parse incoming data (mainly, incoming from string).

The source can also be set at a second time with the Tokenizer.parse method. seps defaults to " " if not given.

Init Block

Simple stream-oriented parser for efficient basic recognition of incoming data.

init Tokenizer( [seps], [options], [tokLen], [source] )

sepsA string representing the separators.
optionsTokenization options.
tokLenMaximum length of returned tokens.
sourceThe string to be tokenized, or a stream to be read for tokens.

The tokenizer class is meant to provide simple and efficiente logic to parse incoming data (mainly, incoming from string).

The source can also be set at a second time with the Tokenizer.parse method. seps defaults to " " if not given.

Methods

next()

Advances the tokenizer up to the next token.

Tokenizer.next( )

Returns:True if a new token is now available, false otherwise.
Raises:
IoErroron errors on the underlying stream.
CodeErrorif called on an unprepared Tokenizer.

Contrarily to iterators, it is necessary to call this method at least once before Tokenizer.token is available.

For example:

 t = Tokenizer( source|"A string to be tokenized" )
while t.next()
   > "Token: ", t.token()
end

nextToken()

Returns the next token from the tokenizer

Tokenizer.nextToken( )

Returns:A string or nil at the end of the tokenization.
Raises:
IoErroron errors on the underlying stream.
CodeErrorif called on an unprepared Tokenizer.

This method is actually a combination of Tokenizer.next followed by Tokenizer.token.

Sample usage:

 t = Tokenizer( source|"A string to be tokenized" )
while (token = t.nextToken()) != nil
   > "Token: ", token
end

Note: When looping, remember to check the value of the returned token against nil, as empty strings can be legally returned multiple times, and they are considered false in logic checks.

parse()

Changes or set the source data for this tokenizer.

Tokenizer.parse( source )

sourceA string or a stream to be used as a source for the tokenizer.

rewind()

Resets the status of the tokenizer.

Tokenizer.rewind( )

Raises:
IoErrorif the tokenizer is tokenizing a non-rewindable stream.

token()

Get the current token.

Tokenizer.token( )

Returns:True if a new token is now available, false otherwise.
Raises:
IoErroron errors on the underlying stream.
CodeErrorif called on an unprepared Tokenizer, or before next().

Contrarily to iterators, it is necessary to call this Tokenizer.next at least once before calling this method.


Made with faldoc 2.1.0