com.exalead.indexing.analysis.v10

Class LanguageDetector

java.lang.Object

com.exalead.indexing.analysis.v10.DocumentProcessor

com.exalead.indexing.analysis.v10.MultiContextDocumentProcessor

com.exalead.indexing.analysis.v10.LanguageDetector

All Implemented Interfaces:

com.exalead.util.Checkable, java.io.Serializable

public class LanguageDetector
extends MultiContextDocumentProcessor
implements com.exalead.util.Checkable, java.io.Serializable

Language detection is performed using the text of all the DocumentChunks associated with the specified input ContextNames for which language was not already detected or specified.
The whole text of all these DocumentChunks is taken into account by a statistical algorithm that detects the language.
This language is then set as the language for all specified chunks. For example, the language attribute of a DocumentChunk is used by semantic processing.
Language is represented by its iso639-1 code: fr, en.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.exalead.indexing.analysis.v10.MultiContextDocumentProcessor

MultiContextDocumentProcessor.InputContexts, MultiContextDocumentProcessor.Transformer<T>

Nested classes/interfaces inherited from class com.exalead.indexing.analysis.v10.DocumentProcessor

DocumentProcessor.FromDataModel

Field Summary

Fields

Modifier and Type	Field and Description
static boolean	DEFAULT_EXCLUDE
static int	DEFAULT_LANGUAGES_TO_KEEP
static int	DEFAULT_MIN_LANG_PERCENTAGE
java.lang.String	defaultLanguage
boolean	exclude
java.lang.String	languageContext
java.lang.String	languagesToDetect
int	languagesToKeep
int	minLangPercentage
java.lang.String	outputContext

Fields inherited from class com.exalead.indexing.analysis.v10.MultiContextDocumentProcessor

inputContexts

Fields inherited from class com.exalead.indexing.analysis.v10.DocumentProcessor

acceptCondition, dataModelClass, dataModelProperty, dataModelState, DEFAULT_DISABLED, disabled, fromDataModel, name

Constructor Summary

Constructors

Constructor and Description
LanguageDetector()
LanguageDetector(LanguageDetector o) Copy constructor

Method Summary

Modifier and Type	Method and Description
<T> T	accept(DocumentProcessor.Transformer<T> transformer, T[] t)
<T> T	accept(MultiContextDocumentProcessor.Transformer<T> transformer, T[] t)
void	check(boolean deep, java.lang.String errorContext) Checks this LanguageDetector.
static LanguageDetector	fromString(java.lang.String s) String representation of this LanguageDetector.
java.lang.String	getDefaultLanguage() If not null, 'defaultLanguage' will be used as the default language when automatic detection fails.
java.lang.String	getLanguageContext() If this is not null and if there is a DocumentChunk with a ContextName matching 'languageContext': no automatic detection will be performed, the language specified will be used as the language of the DocumentChunks associated with the ContextNames specified as input.
java.lang.String	getLanguagesToDetect() If not null, restrict the language detector to a set of languages. If you only have a small set of languages to detect, you can restrict language detector to this set to improve precision.
int	getLanguagesToKeep() Keeps the n most represented languages in the document.
int	getMinLangPercentage() Minimum ratio ([0-100]) of language to be detected (0 = always keeps a detected language)
java.lang.String	getOutputContext() ContextName of the DocumentChunk to create. It will contain the language detected in the processed DocumentChunks as defined in ISO 639-1.
boolean	isExclude() If true, "inputContexts" is an exclude list instead of an include list. Language detection is then performed on all DocumentChunks except those whose ContextName appears in 'inputContexts'.
LanguageDetector	makeCopy() Creates and returns a deep copy of this LanguageDetector.
static LanguageDetector	readFrom(java.io.InputStream is) Read this LanguageDetector from an XML fragment.
void	setDefaultLanguage(java.lang.String defaultLanguage) If not null, 'defaultLanguage' will be used as the default language when automatic detection fails.
void	setExclude(boolean exclude) If true, "inputContexts" is an exclude list instead of an include list. Language detection is then performed on all DocumentChunks except those whose ContextName appears in 'inputContexts'.
void	setLanguageContext(java.lang.String languageContext) If this is not null and if there is a DocumentChunk with a ContextName matching 'languageContext': no automatic detection will be performed, the language specified will be used as the language of the DocumentChunks associated with the ContextNames specified as input.
void	setLanguagesToDetect(java.lang.String languagesToDetect) If not null, restrict the language detector to a set of languages. If you only have a small set of languages to detect, you can restrict language detector to this set to improve precision.
void	setLanguagesToKeep(int languagesToKeep) Keeps the n most represented languages in the document.
void	setMinLangPercentage(int minLangPercentage) Minimum ratio ([0-100]) of language to be detected (0 = always keeps a detected language)
void	setOutputContext(java.lang.String outputContext) ContextName of the DocumentChunk to create. It will contain the language detected in the processed DocumentChunks as defined in ISO 639-1.
java.lang.String	toString() String representation of this LanguageDetector.
LanguageDetector	withAcceptCondition(AcceptCondition acceptCondition)
LanguageDetector	withDataModelClass(java.lang.String dataModelClass)
LanguageDetector	withDataModelProperty(java.lang.String dataModelProperty)
LanguageDetector	withDataModelState(java.lang.String dataModelState)
LanguageDetector	withDefaultLanguage(java.lang.String defaultLanguage)
LanguageDetector	withDisabled(boolean disabled)
LanguageDetector	withDisabled(java.lang.Boolean disabled)
LanguageDetector	withExclude(boolean exclude)
LanguageDetector	withExclude(java.lang.Boolean exclude)
LanguageDetector	withFromDataModel(DocumentProcessor fromDataModel)
LanguageDetector	withInputContexts(java.util.Collection<StringValue> __values)
LanguageDetector	withInputContexts(StringValue... __values) The processor will only be applied to DocumentChunks with a ContextName specified in this list.
LanguageDetector	withLanguageContext(java.lang.String languageContext)
LanguageDetector	withLanguagesToDetect(java.lang.String languagesToDetect)
LanguageDetector	withLanguagesToKeep(int languagesToKeep)
LanguageDetector	withLanguagesToKeep(java.lang.Integer languagesToKeep)
LanguageDetector	withMinLangPercentage(int minLangPercentage)
LanguageDetector	withMinLangPercentage(java.lang.Integer minLangPercentage)
LanguageDetector	withName(java.lang.String name)
LanguageDetector	withOutputContext(java.lang.String outputContext)
void	writeTo(java.io.OutputStream os) Write this LanguageDetector as an XML fragment

Methods inherited from class com.exalead.indexing.analysis.v10.MultiContextDocumentProcessor

getInputContexts, setInputContexts, withInputContexts

Methods inherited from class com.exalead.indexing.analysis.v10.DocumentProcessor

getAcceptCondition, getDataModelClass, getDataModelProperty, getDataModelState, getFromDataModel, getName, isDisabled, setAcceptCondition, setDataModelClass, setDataModelProperty, setDataModelState, setDisabled, setFromDataModel, setName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

languageContext

public java.lang.String languageContext

languagesToDetect

public java.lang.String languagesToDetect

defaultLanguage

public java.lang.String defaultLanguage

exclude

public boolean exclude

DEFAULT_EXCLUDE

public static final boolean DEFAULT_EXCLUDE

See Also:

Constant Field Values

outputContext

public java.lang.String outputContext

minLangPercentage

public int minLangPercentage

DEFAULT_MIN_LANG_PERCENTAGE

public static final int DEFAULT_MIN_LANG_PERCENTAGE

See Also:

Constant Field Values

languagesToKeep

public int languagesToKeep

DEFAULT_LANGUAGES_TO_KEEP

public static final int DEFAULT_LANGUAGES_TO_KEEP

See Also:

Constant Field Values

Constructor Detail

LanguageDetector

public LanguageDetector()

LanguageDetector

public LanguageDetector(LanguageDetector o)

Copy constructor

Method Detail

withInputContexts

public LanguageDetector withInputContexts(StringValue... __values)

The processor will only be applied to DocumentChunks with a ContextName specified in this list.

Overrides:

withInputContexts in class MultiContextDocumentProcessor

withInputContexts

public LanguageDetector withInputContexts(java.util.Collection<StringValue> __values)

Overrides:

withInputContexts in class MultiContextDocumentProcessor

withAcceptCondition

public LanguageDetector withAcceptCondition(AcceptCondition acceptCondition)

Overrides:

withAcceptCondition in class MultiContextDocumentProcessor

withName

public LanguageDetector withName(java.lang.String name)

Overrides:

withName in class MultiContextDocumentProcessor

withDataModelState

public LanguageDetector withDataModelState(java.lang.String dataModelState)

Overrides:

withDataModelState in class MultiContextDocumentProcessor

withFromDataModel

public LanguageDetector withFromDataModel(DocumentProcessor fromDataModel)

Overrides:

withFromDataModel in class MultiContextDocumentProcessor

withDataModelClass

public LanguageDetector withDataModelClass(java.lang.String dataModelClass)

Overrides:

withDataModelClass in class MultiContextDocumentProcessor

withDataModelProperty

public LanguageDetector withDataModelProperty(java.lang.String dataModelProperty)

Overrides:

withDataModelProperty in class MultiContextDocumentProcessor

withDisabled

public LanguageDetector withDisabled(boolean disabled)

Overrides:

withDisabled in class MultiContextDocumentProcessor

withDisabled

public LanguageDetector withDisabled(java.lang.Boolean disabled)

Overrides:

withDisabled in class MultiContextDocumentProcessor

setLanguageContext

public void setLanguageContext(java.lang.String languageContext)

If this is not null and if there is a DocumentChunk with a ContextName matching 'languageContext':

• no automatic detection will be performed,
• the language specified will be used as the language of the DocumentChunks associated with the ContextNames specified as input.

getLanguageContext

public java.lang.String getLanguageContext()

If this is not null and if there is a DocumentChunk with a ContextName matching 'languageContext':

• no automatic detection will be performed,
• the language specified will be used as the language of the DocumentChunks associated with the ContextNames specified as input.

withLanguageContext

public LanguageDetector withLanguageContext(java.lang.String languageContext)

setLanguagesToDetect

public void setLanguagesToDetect(java.lang.String languagesToDetect)

If not null, restrict the language detector to a set of languages.
If you only have a small set of languages to detect, you can restrict language detector to this set to improve precision.
List is comma-separated, ex: "en,fr"

getLanguagesToDetect

public java.lang.String getLanguagesToDetect()

If not null, restrict the language detector to a set of languages.
If you only have a small set of languages to detect, you can restrict language detector to this set to improve precision.
List is comma-separated, ex: "en,fr"

withLanguagesToDetect

public LanguageDetector withLanguagesToDetect(java.lang.String languagesToDetect)

setDefaultLanguage

public void setDefaultLanguage(java.lang.String defaultLanguage)

If not null, 'defaultLanguage' will be used as the default language when automatic detection fails.

getDefaultLanguage

public java.lang.String getDefaultLanguage()

If not null, 'defaultLanguage' will be used as the default language when automatic detection fails.

withDefaultLanguage

public LanguageDetector withDefaultLanguage(java.lang.String defaultLanguage)

setExclude

public void setExclude(boolean exclude)

If true, "inputContexts" is an exclude list instead of an include list.
Language detection is then performed on all DocumentChunks except those whose ContextName appears in 'inputContexts'.

isExclude

public boolean isExclude()

If true, "inputContexts" is an exclude list instead of an include list.
Language detection is then performed on all DocumentChunks except those whose ContextName appears in 'inputContexts'.

withExclude

public LanguageDetector withExclude(boolean exclude)

withExclude

public LanguageDetector withExclude(java.lang.Boolean exclude)

setOutputContext

public void setOutputContext(java.lang.String outputContext)

ContextName of the DocumentChunk to create.
It will contain the language detected in the processed DocumentChunks as defined in ISO 639-1.

getOutputContext

public java.lang.String getOutputContext()

ContextName of the DocumentChunk to create.
It will contain the language detected in the processed DocumentChunks as defined in ISO 639-1.

withOutputContext

public LanguageDetector withOutputContext(java.lang.String outputContext)

setMinLangPercentage

public void setMinLangPercentage(int minLangPercentage)

Minimum ratio ([0-100]) of language to be detected (0 = always keeps a detected language)

getMinLangPercentage

public int getMinLangPercentage()

Minimum ratio ([0-100]) of language to be detected (0 = always keeps a detected language)

withMinLangPercentage

public LanguageDetector withMinLangPercentage(int minLangPercentage)

withMinLangPercentage

public LanguageDetector withMinLangPercentage(java.lang.Integer minLangPercentage)

setLanguagesToKeep

public void setLanguagesToKeep(int languagesToKeep)

Keeps the n most represented languages in the document. A value of 0 lets the minLangPercentage select the languages.

getLanguagesToKeep

public int getLanguagesToKeep()

Keeps the n most represented languages in the document. A value of 0 lets the minLangPercentage select the languages.

withLanguagesToKeep

public LanguageDetector withLanguagesToKeep(int languagesToKeep)

withLanguagesToKeep

public LanguageDetector withLanguagesToKeep(java.lang.Integer languagesToKeep)

makeCopy

public LanguageDetector makeCopy()

Creates and returns a deep copy of this LanguageDetector.

Overrides:

makeCopy in class MultiContextDocumentProcessor

readFrom

public static LanguageDetector readFrom(java.io.InputStream is)
                                 throws javax.xml.bind.JAXBException

Read this LanguageDetector from an XML fragment.

Throws:

javax.xml.bind.JAXBException

writeTo

public void writeTo(java.io.OutputStream os)
             throws javax.xml.bind.JAXBException,
                    java.io.IOException

Write this LanguageDetector as an XML fragment

Overrides:

writeTo in class MultiContextDocumentProcessor

Throws:

javax.xml.bind.JAXBException

java.io.IOException

fromString

public static LanguageDetector fromString(java.lang.String s)
                                   throws javax.xml.bind.JAXBException,
                                          java.io.UnsupportedEncodingException

String representation of this LanguageDetector.

Throws:

javax.xml.bind.JAXBException

java.io.UnsupportedEncodingException

toString

public java.lang.String toString()

String representation of this LanguageDetector.

Overrides:

toString in class MultiContextDocumentProcessor

check

public void check(boolean deep,
                  java.lang.String errorContext)
           throws com.exalead.util.TypedException

Checks this LanguageDetector.

Specified by:

check in interface com.exalead.util.Checkable

Overrides:

check in class MultiContextDocumentProcessor

Throws:

com.exalead.util.TypedException

accept

public <T> T accept(MultiContextDocumentProcessor.Transformer<T> transformer,
                    T[] t)
             throws com.exalead.util.TypedException

Specified by:

accept in class MultiContextDocumentProcessor

Throws:

com.exalead.util.TypedException

accept

public <T> T accept(DocumentProcessor.Transformer<T> transformer,
                    T[] t)
             throws com.exalead.util.TypedException

Specified by:

accept in class DocumentProcessor

Throws:

com.exalead.util.TypedException