com.exalead.linguistic.v10

Class TokenizationConfig

java.lang.Object

com.exalead.linguistic.v10.TokenizationConfig

All Implemented Interfaces:

com.exalead.util.Checkable, java.io.Serializable

public class TokenizationConfig
extends java.lang.Object
implements com.exalead.util.Checkable, java.io.Serializable

How to tokenize documents, ie split the input strings in tokens. Tokens are usually words.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
FormIndexingConfig	formIndexingConfig
java.lang.String	name
NormalizerConfig	normalizerConfig
java.util.List<Tokenizer>	tokenizer

Constructor Summary

Constructors

Constructor and Description
TokenizationConfig()
TokenizationConfig(TokenizationConfig o) Copy constructor

Method Summary

Modifier and Type	Method and Description
void	check(boolean deep, java.lang.String errorContext) Checks this TokenizationConfig.
static TokenizationConfig	fromString(java.lang.String s) String representation of this TokenizationConfig.
FormIndexingConfig	getFormIndexingConfig() How to index different word forms (exact, normalized, lemmatized) for each language.
java.lang.String	getName() The tokenization configuration's name.
NormalizerConfig	getNormalizerConfig() How to normalize words.
java.util.List<Tokenizer>	getTokenizer() List of enabled tokenizers.
TokenizationConfig	makeCopy() Creates and returns a deep copy of this TokenizationConfig.
static TokenizationConfig	readFrom(java.io.InputStream is) Read this TokenizationConfig from an XML fragment.
void	setFormIndexingConfig(FormIndexingConfig formIndexingConfig) How to index different word forms (exact, normalized, lemmatized) for each language.
void	setName(java.lang.String name) The tokenization configuration's name.
void	setNormalizerConfig(NormalizerConfig normalizerConfig) How to normalize words.
java.lang.String	toString() String representation of this TokenizationConfig.
TokenizationConfig	withFormIndexingConfig(FormIndexingConfig formIndexingConfig)
TokenizationConfig	withName(java.lang.String name)
TokenizationConfig	withNormalizerConfig(NormalizerConfig normalizerConfig)
TokenizationConfig	withTokenizer(java.util.Collection<Tokenizer> __values)
TokenizationConfig	withTokenizer(Tokenizer... __values) List of enabled tokenizers.
void	writeTo(java.io.OutputStream os) Write this TokenizationConfig as an XML fragment

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

tokenizer

public java.util.List<Tokenizer> tokenizer

formIndexingConfig

public FormIndexingConfig formIndexingConfig

normalizerConfig

public NormalizerConfig normalizerConfig

name

public java.lang.String name

Constructor Detail

TokenizationConfig

public TokenizationConfig()

TokenizationConfig

public TokenizationConfig(TokenizationConfig o)

Copy constructor

Method Detail

getTokenizer

public java.util.List<Tokenizer> getTokenizer()

List of enabled tokenizers. Tokenizer choice depends on the document's language.

withTokenizer

public TokenizationConfig withTokenizer(Tokenizer... __values)

List of enabled tokenizers. Tokenizer choice depends on the document's language.

withTokenizer

public TokenizationConfig withTokenizer(java.util.Collection<Tokenizer> __values)

setFormIndexingConfig

public void setFormIndexingConfig(FormIndexingConfig formIndexingConfig)

How to index different word forms (exact, normalized, lemmatized) for each language.

getFormIndexingConfig

public FormIndexingConfig getFormIndexingConfig()

How to index different word forms (exact, normalized, lemmatized) for each language.

withFormIndexingConfig

public TokenizationConfig withFormIndexingConfig(FormIndexingConfig formIndexingConfig)

setNormalizerConfig

public void setNormalizerConfig(NormalizerConfig normalizerConfig)

How to normalize words.

getNormalizerConfig

public NormalizerConfig getNormalizerConfig()

How to normalize words.

withNormalizerConfig

public TokenizationConfig withNormalizerConfig(NormalizerConfig normalizerConfig)

setName

public void setName(java.lang.String name)

The tokenization configuration's name.

getName

public java.lang.String getName()

The tokenization configuration's name.

withName

public TokenizationConfig withName(java.lang.String name)

makeCopy

public TokenizationConfig makeCopy()

Creates and returns a deep copy of this TokenizationConfig.

readFrom

public static TokenizationConfig readFrom(java.io.InputStream is)
                                   throws javax.xml.bind.JAXBException

Read this TokenizationConfig from an XML fragment.

Throws:

javax.xml.bind.JAXBException

writeTo

public void writeTo(java.io.OutputStream os)
             throws javax.xml.bind.JAXBException,
                    java.io.IOException

Write this TokenizationConfig as an XML fragment

Throws:

javax.xml.bind.JAXBException

java.io.IOException

fromString

public static TokenizationConfig fromString(java.lang.String s)
                                     throws javax.xml.bind.JAXBException,
                                            java.io.UnsupportedEncodingException

String representation of this TokenizationConfig.

Throws:

javax.xml.bind.JAXBException

java.io.UnsupportedEncodingException

toString

public java.lang.String toString()

String representation of this TokenizationConfig.

Overrides:

toString in class java.lang.Object

check

public void check(boolean deep,
                  java.lang.String errorContext)
           throws com.exalead.util.TypedException

Checks this TokenizationConfig.

Specified by:

check in interface com.exalead.util.Checkable

Throws:

com.exalead.util.TypedException