Class SmartChineseAnalyzer
- java.lang.Object
-
- org.apache.lucene.analysis.Analyzer
-
- org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public final class SmartChineseAnalyzer extends Analyzer
SmartChineseAnalyzer is an analyzer for Chinese or mixed Chinese-English text. The analyzer uses probabilistic knowledge to find the optimal word segmentation for Simplified Chinese text. The text is first broken into sentences, then each sentence is segmented into words.
Segmentation is based upon the Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.
This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.
The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!
- Since:
- 3.1
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
SmartChineseAnalyzer.DefaultSetHolder
Atomically loads the DEFAULT_STOP_SET in a lazy fashion once the outer class accesses the static final set the first time.;-
Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer
Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents
-
-
Field Summary
Fields Modifier and Type Field Description private static java.lang.String
DEFAULT_STOPWORD_FILE
private static java.lang.String
STOPWORD_FILE_COMMENT
private CharArraySet
stopWords
-
Fields inherited from class org.apache.lucene.analysis.Analyzer
GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY
-
-
Constructor Summary
Constructors Constructor Description SmartChineseAnalyzer()
Create a new SmartChineseAnalyzer, using the default stopword list.SmartChineseAnalyzer(boolean useDefaultStopWords)
Create a new SmartChineseAnalyzer, optionally using the default stopword list.SmartChineseAnalyzer(CharArraySet stopWords)
Create a new SmartChineseAnalyzer, using the providedSet
of stopwords.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description Analyzer.TokenStreamComponents
createComponents(java.lang.String fieldName)
Creates a newAnalyzer.TokenStreamComponents
instance for this analyzer.static CharArraySet
getDefaultStopSet()
Returns an unmodifiable instance of the default stop-words set.protected TokenStream
normalize(java.lang.String fieldName, TokenStream in)
Wrap the givenTokenStream
in order to apply normalization filters.-
Methods inherited from class org.apache.lucene.analysis.Analyzer
attributeFactory, close, getOffsetGap, getPositionIncrementGap, getReuseStrategy, getVersion, initReader, initReaderForNormalization, normalize, setVersion, tokenStream, tokenStream
-
-
-
-
Field Detail
-
stopWords
private final CharArraySet stopWords
-
DEFAULT_STOPWORD_FILE
private static final java.lang.String DEFAULT_STOPWORD_FILE
- See Also:
- Constant Field Values
-
STOPWORD_FILE_COMMENT
private static final java.lang.String STOPWORD_FILE_COMMENT
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
SmartChineseAnalyzer
public SmartChineseAnalyzer()
Create a new SmartChineseAnalyzer, using the default stopword list.
-
SmartChineseAnalyzer
public SmartChineseAnalyzer(boolean useDefaultStopWords)
Create a new SmartChineseAnalyzer, optionally using the default stopword list.
The included default stopword list is simply a list of punctuation. If you do not use this list, punctuation will not be removed from the text!
- Parameters:
useDefaultStopWords
- true to use the default stopword list.
-
SmartChineseAnalyzer
public SmartChineseAnalyzer(CharArraySet stopWords)
Create a new SmartChineseAnalyzer, using the provided
Set
of stopwords.Note: the set should include punctuation, unless you want to index punctuation!
- Parameters:
stopWords
-Set
of stopwords to use.
-
-
Method Detail
-
getDefaultStopSet
public static CharArraySet getDefaultStopSet()
Returns an unmodifiable instance of the default stop-words set.- Returns:
- an unmodifiable instance of the default stop-words set.
-
createComponents
public Analyzer.TokenStreamComponents createComponents(java.lang.String fieldName)
Description copied from class:Analyzer
Creates a newAnalyzer.TokenStreamComponents
instance for this analyzer.- Specified by:
createComponents
in classAnalyzer
- Parameters:
fieldName
- the name of the fields content passed to theAnalyzer.TokenStreamComponents
sink as a reader- Returns:
- the
Analyzer.TokenStreamComponents
for this analyzer.
-
normalize
protected TokenStream normalize(java.lang.String fieldName, TokenStream in)
Description copied from class:Analyzer
Wrap the givenTokenStream
in order to apply normalization filters. The default implementation returns theTokenStream
as-is. This is used byAnalyzer.normalize(String, String)
.
-
-