public final class TokenizerModel extends BaseModel
TokenizerModel
is the model used
by a learnable Tokenizer
.TokenizerME
artifactMap, FACTORY_NAME, MANIFEST_ENTRY, toolFactory, TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY
Constructor and Description |
---|
TokenizerModel(AbstractModel tokenizerModel,
Map<String,String> manifestInfoEntries,
TokenizerFactory tokenizerFactory)
Initializes the current instance.
|
TokenizerModel(File modelFile) |
TokenizerModel(InputStream in)
Initializes the current instance.
|
TokenizerModel(String language,
AbstractModel tokenizerMaxentModel,
boolean useAlphaNumericOptimization)
Deprecated.
Use
TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory)
instead and pass in a TokenizerFactory . |
TokenizerModel(String language,
AbstractModel tokenizerMaxentModel,
boolean useAlphaNumericOptimization,
Map<String,String> manifestInfoEntries)
Deprecated.
Use
TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory)
instead and pass in a TokenizerFactory . |
TokenizerModel(String language,
AbstractModel tokenizerMaxentModel,
Dictionary abbreviations,
boolean useAlphaNumericOptimization,
Map<String,String> manifestInfoEntries)
Deprecated.
Use
TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory)
instead and pass in a TokenizerFactory . |
TokenizerModel(URL modelURL) |
Modifier and Type | Method and Description |
---|---|
Dictionary |
getAbbreviations() |
protected Class<? extends BaseToolFactory> |
getDefaultFactory()
Sub-classes should override this method if their module has a default
BaseToolFactory sub-class.
|
TokenizerFactory |
getFactory() |
AbstractModel |
getMaxentModel() |
static void |
main(String[] args) |
boolean |
useAlphaNumericOptimization() |
protected void |
validateArtifactMap()
Validates the parsed artifacts.
|
checkArtifactMap, createArtifactSerializers, createArtifactSerializers, getArtifact, getArtifactSerializer, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, setManifestProperty
public TokenizerModel(AbstractModel tokenizerModel, Map<String,String> manifestInfoEntries, TokenizerFactory tokenizerFactory)
tokenizerModel
- the modelmanifestInfoEntries
- the manifesttokenizerFactory
- the factorypublic TokenizerModel(String language, AbstractModel tokenizerMaxentModel, Dictionary abbreviations, boolean useAlphaNumericOptimization, Map<String,String> manifestInfoEntries)
TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory)
instead and pass in a TokenizerFactory
.tokenizerMaxentModel
- useAlphaNumericOptimization
- public TokenizerModel(String language, AbstractModel tokenizerMaxentModel, boolean useAlphaNumericOptimization, Map<String,String> manifestInfoEntries)
TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory)
instead and pass in a TokenizerFactory
.language
- tokenizerMaxentModel
- useAlphaNumericOptimization
- manifestInfoEntries
- public TokenizerModel(String language, AbstractModel tokenizerMaxentModel, boolean useAlphaNumericOptimization)
TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory)
instead and pass in a TokenizerFactory
.language
- tokenizerMaxentModel
- useAlphaNumericOptimization
- public TokenizerModel(InputStream in) throws IOException, InvalidFormatException
in
- IOException
InvalidFormatException
public TokenizerModel(File modelFile) throws IOException, InvalidFormatException
IOException
InvalidFormatException
public TokenizerModel(URL modelURL) throws IOException, InvalidFormatException
IOException
InvalidFormatException
protected void validateArtifactMap() throws InvalidFormatException
BaseModel
InvalidFormatException
.
Note:
Subclasses should generally invoke super.validateArtifactMap at the beginning
of this method.validateArtifactMap
in class BaseModel
InvalidFormatException
public TokenizerFactory getFactory()
protected Class<? extends BaseToolFactory> getDefaultFactory()
BaseModel
getDefaultFactory
in class BaseModel
BaseToolFactory
for the module, or null if none.public AbstractModel getMaxentModel()
public Dictionary getAbbreviations()
public boolean useAlphaNumericOptimization()
public static void main(String[] args) throws IOException
IOException
Copyright © 2017 The Apache Software Foundation. All rights reserved.