com.exalead.linguistic.v10

Class StandardTokenizer

java.lang.Object

com.exalead.linguistic.v10.Tokenizer

com.exalead.linguistic.v10.StandardTokenizer

All Implemented Interfaces:

com.exalead.util.Checkable, java.io.Serializable

public class StandardTokenizer
extends Tokenizer
implements com.exalead.util.Checkable, java.io.Serializable

Sets a specific configuration for the standard tokenizer. If this object is not
in the Semantic processors list, standard tokenization is used.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StandardTokenizer.CharOverrides
static class	StandardTokenizer.PatternOverrides

Field Summary

Fields

Modifier and Type	Field and Description
protected StandardTokenizer.CharOverrides	charOverrides
boolean	concatAlphaNum
boolean	concatNumAlpha
static boolean	DEFAULT_CONCAT_ALPHA_NUM
static boolean	DEFAULT_CONCAT_NUM_ALPHA
java.lang.String	language
protected StandardTokenizer.PatternOverrides	patternOverrides
java.util.List<TokenizerPlugin>	tokenizerPlugin

Constructor Summary

Constructors

Constructor and Description
StandardTokenizer()
StandardTokenizer(StandardTokenizer o) Copy constructor

Method Summary

Modifier and Type	Method and Description
void	check(boolean deep, java.lang.String errorContext) Checks this StandardTokenizer.
static StandardTokenizer	fromString(java.lang.String s) String representation of this StandardTokenizer.
StandardTokenizer.CharOverrides	getCharOverrides()
java.lang.String	getLanguage() The language handled by this tokenizer.
StandardTokenizer.PatternOverrides	getPatternOverrides()
java.util.List<TokenizerPlugin>	getTokenizerPlugin()
boolean	isConcatAlphaNum() Concat alpha and num chars
boolean	isConcatNumAlpha() Concat num and alpha chars
StandardTokenizer	makeCopy() Creates and returns a deep copy of this StandardTokenizer.
static StandardTokenizer	readFrom(java.io.InputStream is) Read this StandardTokenizer from an XML fragment.
void	setCharOverrides(StandardTokenizer.CharOverrides __value)
void	setConcatAlphaNum(boolean concatAlphaNum) Concat alpha and num chars
void	setConcatNumAlpha(boolean concatNumAlpha) Concat num and alpha chars
void	setLanguage(java.lang.String language) The language handled by this tokenizer.
void	setPatternOverrides(StandardTokenizer.PatternOverrides __value)
java.lang.String	toString() String representation of this StandardTokenizer.
StandardTokenizer	withCharOverrides(java.util.Collection<StandardTokenizerOverride> __values)
StandardTokenizer	withCharOverrides(StandardTokenizer.CharOverrides __value)
StandardTokenizer	withCharOverrides(StandardTokenizerOverride... __values)
StandardTokenizer	withConcatAlphaNum(boolean concatAlphaNum)
StandardTokenizer	withConcatAlphaNum(java.lang.Boolean concatAlphaNum)
StandardTokenizer	withConcatNumAlpha(boolean concatNumAlpha)
StandardTokenizer	withConcatNumAlpha(java.lang.Boolean concatNumAlpha)
StandardTokenizer	withLanguage(java.lang.String language)
StandardTokenizer	withPatternOverrides(java.util.Collection<StandardTokenizerOverride> __values)
StandardTokenizer	withPatternOverrides(StandardTokenizer.PatternOverrides __value)
StandardTokenizer	withPatternOverrides(StandardTokenizerOverride... __values)
StandardTokenizer	withTokenizerPlugin(java.util.Collection<TokenizerPlugin> __values)
StandardTokenizer	withTokenizerPlugin(TokenizerPlugin... __values)
void	writeTo(java.io.OutputStream os) Write this StandardTokenizer as an XML fragment

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

tokenizerPlugin

public java.util.List<TokenizerPlugin> tokenizerPlugin

charOverrides

protected StandardTokenizer.CharOverrides charOverrides

patternOverrides

protected StandardTokenizer.PatternOverrides patternOverrides

language

public java.lang.String language

concatAlphaNum

public boolean concatAlphaNum

DEFAULT_CONCAT_ALPHA_NUM

public static final boolean DEFAULT_CONCAT_ALPHA_NUM

See Also:

Constant Field Values

concatNumAlpha

public boolean concatNumAlpha

DEFAULT_CONCAT_NUM_ALPHA

public static final boolean DEFAULT_CONCAT_NUM_ALPHA

See Also:

Constant Field Values

Constructor Detail

StandardTokenizer

public StandardTokenizer()

StandardTokenizer

public StandardTokenizer(StandardTokenizer o)

Copy constructor

Method Detail

getTokenizerPlugin

public java.util.List<TokenizerPlugin> getTokenizerPlugin()

withTokenizerPlugin

public StandardTokenizer withTokenizerPlugin(TokenizerPlugin... __values)

withTokenizerPlugin

public StandardTokenizer withTokenizerPlugin(java.util.Collection<TokenizerPlugin> __values)

getCharOverrides

public StandardTokenizer.CharOverrides getCharOverrides()

setCharOverrides

public void setCharOverrides(StandardTokenizer.CharOverrides __value)

withCharOverrides

public StandardTokenizer withCharOverrides(StandardTokenizerOverride... __values)

withCharOverrides

public StandardTokenizer withCharOverrides(java.util.Collection<StandardTokenizerOverride> __values)

withCharOverrides

public StandardTokenizer withCharOverrides(StandardTokenizer.CharOverrides __value)

getPatternOverrides

public StandardTokenizer.PatternOverrides getPatternOverrides()

setPatternOverrides

public void setPatternOverrides(StandardTokenizer.PatternOverrides __value)

withPatternOverrides

public StandardTokenizer withPatternOverrides(StandardTokenizerOverride... __values)

withPatternOverrides

public StandardTokenizer withPatternOverrides(java.util.Collection<StandardTokenizerOverride> __values)

withPatternOverrides

public StandardTokenizer withPatternOverrides(StandardTokenizer.PatternOverrides __value)

setLanguage

public void setLanguage(java.lang.String language)

The language handled by this tokenizer. This can be null for all unhandled languages.

getLanguage

public java.lang.String getLanguage()

The language handled by this tokenizer. This can be null for all unhandled languages.

withLanguage

public StandardTokenizer withLanguage(java.lang.String language)

setConcatAlphaNum

public void setConcatAlphaNum(boolean concatAlphaNum)

Concat alpha and num chars

isConcatAlphaNum

public boolean isConcatAlphaNum()

Concat alpha and num chars

withConcatAlphaNum

public StandardTokenizer withConcatAlphaNum(boolean concatAlphaNum)

withConcatAlphaNum

public StandardTokenizer withConcatAlphaNum(java.lang.Boolean concatAlphaNum)

setConcatNumAlpha

public void setConcatNumAlpha(boolean concatNumAlpha)

Concat num and alpha chars

isConcatNumAlpha

public boolean isConcatNumAlpha()

Concat num and alpha chars

withConcatNumAlpha

public StandardTokenizer withConcatNumAlpha(boolean concatNumAlpha)

withConcatNumAlpha

public StandardTokenizer withConcatNumAlpha(java.lang.Boolean concatNumAlpha)

makeCopy

public StandardTokenizer makeCopy()

Creates and returns a deep copy of this StandardTokenizer.

Overrides:

makeCopy in class Tokenizer

readFrom

public static StandardTokenizer readFrom(java.io.InputStream is)
                                  throws javax.xml.bind.JAXBException

Read this StandardTokenizer from an XML fragment.

Throws:

javax.xml.bind.JAXBException

writeTo

public void writeTo(java.io.OutputStream os)
             throws javax.xml.bind.JAXBException,
                    java.io.IOException

Write this StandardTokenizer as an XML fragment

Overrides:

writeTo in class Tokenizer

Throws:

javax.xml.bind.JAXBException

java.io.IOException

fromString

public static StandardTokenizer fromString(java.lang.String s)
                                    throws javax.xml.bind.JAXBException,
                                           java.io.UnsupportedEncodingException

String representation of this StandardTokenizer.

Throws:

javax.xml.bind.JAXBException

java.io.UnsupportedEncodingException

toString

public java.lang.String toString()

String representation of this StandardTokenizer.

Overrides:

toString in class Tokenizer

check

public void check(boolean deep,
                  java.lang.String errorContext)
           throws com.exalead.util.TypedException

Checks this StandardTokenizer.

Specified by:

check in interface com.exalead.util.Checkable

Overrides:

check in class Tokenizer

Throws:

com.exalead.util.TypedException