Package org.apache.lucene.classification
Class BM25NBClassifier
java.lang.Object
org.apache.lucene.classification.BM25NBClassifier
- All Implemented Interfaces:
Classifier<BytesRef>
A classifier approximating naive bayes classifier by using pure queries on BM25.
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final Analyzer
Analyzer
to be used for tokenizing unseen input textprivate final String
name of the field to be used as a class / category outputprivate final IndexReader
IndexReader
used to access theClassifier
's indexprivate final IndexSearcher
IndexSearcher
to run searches on the index for retrieving frequenciesprivate final Query
Query
used to eventually filter the document set to be used to classifyprivate final String[]
names of the fields to be used as input text -
Constructor Summary
ConstructorsConstructorDescriptionBM25NBClassifier
(IndexReader indexReader, Analyzer analyzer, Query query, String classFieldName, String... textFieldNames) Creates a new NaiveBayes classifier. -
Method Summary
Modifier and TypeMethodDescriptionassignClass
(String inputDocument) Assign a class (with score) to the given text Stringprivate List<ClassificationResult<BytesRef>>
assignClassNormalizedList
(String inputDocument) Calculate probabilities for all classes for a given input textprivate double
calculateLogLikelihood
(String[] tokens, Term term) private double
calculateLogPrior
(Term term) getClasses
(String text) Get all the classes (sorted by score, descending) assigned to the given text String.getClasses
(String text, int max) Get the firstmax
classes (sorted by score, descending) assigned to the given text String.private double
getTermProbForClass
(Term classTerm, String... words) private ArrayList<ClassificationResult<BytesRef>>
normClassificationResults
(List<ClassificationResult<BytesRef>> assignedClasses) Normalize the classification results based on the max score availableprivate String[]
tokenize aString
on this classifier's text fields and analyzer
-
Field Details
-
indexReader
IndexReader
used to access theClassifier
's index -
textFieldNames
names of the fields to be used as input text -
classFieldName
name of the field to be used as a class / category output -
analyzer
Analyzer
to be used for tokenizing unseen input text -
indexSearcher
IndexSearcher
to run searches on the index for retrieving frequencies -
query
Query
used to eventually filter the document set to be used to classify
-
-
Constructor Details
-
BM25NBClassifier
public BM25NBClassifier(IndexReader indexReader, Analyzer analyzer, Query query, String classFieldName, String... textFieldNames) Creates a new NaiveBayes classifier.- Parameters:
indexReader
- the reader on the index to be used for classificationanalyzer
- anAnalyzer
used to analyze unseen textquery
- aQuery
to eventually filter the docs used for training the classifier, ornull
if all the indexed docs should be usedclassFieldName
- the name of the field used as the output for the classifier NOTE: must not be heavely analyzed as the returned class will be a token indexed for this fieldtextFieldNames
- the name of the fields used as the inputs for the classifier, NO boosting supported per field
-
-
Method Details
-
assignClass
Description copied from interface:Classifier
Assign a class (with score) to the given text String- Specified by:
assignClass
in interfaceClassifier<BytesRef>
- Parameters:
inputDocument
- a String containing text to be classified- Returns:
- a
ClassificationResult
holding assigned class of typeT
and score - Throws:
IOException
- If there is a low-level I/O error.
-
getClasses
Description copied from interface:Classifier
Get all the classes (sorted by score, descending) assigned to the given text String.- Specified by:
getClasses
in interfaceClassifier<BytesRef>
- Parameters:
text
- a String containing text to be classified- Returns:
- the whole list of
ClassificationResult
, the classes and scores. Returnsnull
if the classifier can't make lists. - Throws:
IOException
- If there is a low-level I/O error.
-
getClasses
Description copied from interface:Classifier
Get the firstmax
classes (sorted by score, descending) assigned to the given text String.- Specified by:
getClasses
in interfaceClassifier<BytesRef>
- Parameters:
text
- a String containing text to be classifiedmax
- the number of return list elements- Returns:
- the whole list of
ClassificationResult
, the classes and scores. Cut for "max" number of elements. Returnsnull
if the classifier can't make lists. - Throws:
IOException
- If there is a low-level I/O error.
-
assignClassNormalizedList
private List<ClassificationResult<BytesRef>> assignClassNormalizedList(String inputDocument) throws IOException Calculate probabilities for all classes for a given input text- Parameters:
inputDocument
- the input text as aString
- Returns:
- a
List
ofClassificationResult
, one for each existing class - Throws:
IOException
- if assigning probabilities fails
-
normClassificationResults
private ArrayList<ClassificationResult<BytesRef>> normClassificationResults(List<ClassificationResult<BytesRef>> assignedClasses) Normalize the classification results based on the max score available- Parameters:
assignedClasses
- the list of assigned classes- Returns:
- the normalized results
-
tokenize
tokenize aString
on this classifier's text fields and analyzer- Parameters:
text
- theString
representing an input text (to be classified)- Returns:
- a
String
array of the resulting tokens - Throws:
IOException
- if tokenization fails
-
calculateLogLikelihood
- Throws:
IOException
-
getTermProbForClass
- Throws:
IOException
-
calculateLogPrior
- Throws:
IOException
-