protected static class JsonFormat.Tokenizer extends Object
String
.
The Java standard library provides many classes that you might think would be useful for implementing this, but aren't. For example:
java.io.StreamTokenizer
: This almost does what we want -- or, at least, something
that would get us close to what we want -- except for one fatal flaw: It automatically
un-escapes strings using Java escape sequences, which do not include all the escape sequences
we need to support (e.g. '\x').
java.util.Scanner
: This seems like a great way at least to parse regular
expressions out of a stream (so we wouldn't have to load the entire input into a single
string before parsing). Sadly, Scanner
requires that tokens be delimited with some
delimiter. Thus, although the text "foo:" should parse to two tokens ("foo" and ":"), Scanner
would recognize it only as a single token. Furthermore, Scanner
provides no
way to inspect the contents of delimiters, making it impossible to keep track of line and
column numbers.
Luckily, Java's regular expression support does manage to be useful to us. (Barely: We need
Matcher.usePattern()
, which is new in Java 1.5.) So, we can use that, at least.
Unfortunately, this implies that we need to have the entire input in one contiguous string.
Constructor and Description |
---|
Tokenizer(CharSequence text)
Construct a tokenizer that parses tokens from the given text.
|
Modifier and Type | Method and Description |
---|---|
boolean |
atEnd()
Are we at the end of the input?
|
void |
consume(String token)
If the next token exactly matches
token , consume it. |
boolean |
consumeBoolean()
If the next token is a boolean, consume it and return its value.
|
com.google.protobuf.ByteString |
consumeByteString()
If the next token is a string, consume it, unescape it as a
ByteString , and return it. |
double |
consumeDouble()
If the next token is a double, consume it and return its value.
|
float |
consumeFloat()
If the next token is a float, consume it and return its value.
|
String |
consumeIdentifier()
If the next token is an identifier, consume it and return its value.
|
int |
consumeInt32()
If the next token is a 32-bit signed integer, consume it and return its value.
|
long |
consumeInt64()
If the next token is a 64-bit signed integer, consume it and return its value.
|
String |
consumeString()
If the next token is a string, consume it and return its (unescaped) value.
|
int |
consumeUInt32()
If the next token is a 32-bit unsigned integer, consume it and return its value.
|
long |
consumeUInt64()
If the next token is a 64-bit unsigned integer, consume it and return its value.
|
String |
currentToken() |
boolean |
lookingAtBoolean()
Returns
true if the next token is a boolean (true/false), but does not consume it. |
boolean |
lookingAtFloat()
Returns
true if the next token is an float, but does not consume it. |
boolean |
lookingAtInteger()
Returns
true if the next token is an integer, but does not consume it. |
void |
nextToken()
Advance to the next token.
|
JsonFormat.ParseException |
parseException(String description)
Returns a
JsonFormat.ParseException with the current line and column numbers in the
description, suitable for throwing. |
JsonFormat.ParseException |
parseExceptionPreviousToken(String description)
Returns a
JsonFormat.ParseException with the line and column numbers of the previous token
in the description, suitable for throwing. |
boolean |
tryConsume(String token)
If the next token exactly matches
token , consume it and return true . |
public Tokenizer(CharSequence text)
public boolean atEnd()
public void nextToken()
public boolean tryConsume(String token)
token
, consume it and return true
.
Otherwise, return false
without doing anything.public void consume(String token) throws JsonFormat.ParseException
token
, consume it. Otherwise, throw a
JsonFormat.ParseException
.JsonFormat.ParseException
public boolean lookingAtFloat()
true
if the next token is an float, but does not consume it.public boolean lookingAtInteger()
true
if the next token is an integer, but does not consume it.public boolean lookingAtBoolean()
true
if the next token is a boolean (true/false), but does not consume it.public String currentToken()
public String consumeIdentifier() throws JsonFormat.ParseException
JsonFormat.ParseException
.JsonFormat.ParseException
public int consumeInt32() throws JsonFormat.ParseException
JsonFormat.ParseException
.JsonFormat.ParseException
public int consumeUInt32() throws JsonFormat.ParseException
JsonFormat.ParseException
.JsonFormat.ParseException
public long consumeInt64() throws JsonFormat.ParseException
JsonFormat.ParseException
.JsonFormat.ParseException
public long consumeUInt64() throws JsonFormat.ParseException
JsonFormat.ParseException
.JsonFormat.ParseException
public double consumeDouble() throws JsonFormat.ParseException
JsonFormat.ParseException
.JsonFormat.ParseException
public float consumeFloat() throws JsonFormat.ParseException
JsonFormat.ParseException
.JsonFormat.ParseException
public boolean consumeBoolean() throws JsonFormat.ParseException
JsonFormat.ParseException
.JsonFormat.ParseException
public String consumeString() throws JsonFormat.ParseException
JsonFormat.ParseException
.JsonFormat.ParseException
public com.google.protobuf.ByteString consumeByteString() throws JsonFormat.ParseException
ByteString
, and return it. Otherwise, throw a
JsonFormat.ParseException
.JsonFormat.ParseException
public JsonFormat.ParseException parseException(String description)
JsonFormat.ParseException
with the current line and column numbers in the
description, suitable for throwing.public JsonFormat.ParseException parseExceptionPreviousToken(String description)
JsonFormat.ParseException
with the line and column numbers of the previous token
in the description, suitable for throwing.Copyright © 2017. All rights reserved.