com.exalead.mot.components

Class GermanTokenizer

java.lang.Object

com.exalead.gni.GNIObject

com.exalead.mot.core.Processor

com.exalead.mot.components.SubTokenizer

com.exalead.mot.components.GermanTokenizer

public class GermanTokenizer
extends SubTokenizer

German tokenizer

Field Summary

Fields

Modifier and Type	Field and Description
java.lang.String	splitPolicy The split policy, one of { "splitToken", "addToken", "keepToken", "keepToken2", "keepToken3" }
boolean	stickTokens Should the token be sticked
boolean	useCustomNormalizationTID Should we use custom normalization TID

Fields inherited from class com.exalead.mot.components.SubTokenizer

normalizer, resource

Fields inherited from class com.exalead.mot.core.Processor

fields, name

Constructor Summary

Constructors

Constructor and Description
GermanTokenizer(java.lang.String name, java.lang.String resource, Normalizer normalizer, java.lang.String splitPolicy, boolean useCustomNormalizationTID, boolean stickTokens, java.lang.String fields) Initialize a new German tokenizer

Method Summary

Modifier and Type	Method and Description
protected void	finalize()
static java.lang.String	getOriginalTokenTag()
void	init(java.lang.String name, java.lang.String[] fields) Initialize the processor
protected void	init(java.lang.String name, java.lang.String resource, Normalizer normalizer, java.lang.String splitPolicy, boolean useCustomNormalizationTID, boolean stickTokens)

Methods inherited from class com.exalead.mot.components.SubTokenizer

getConcatLowercaseTag, getConcatNormalizeTag, getLowercaseTag, getNormalizeTag

Methods inherited from class com.exalead.mot.core.Processor

checkResource, destroy, getName, init

Methods inherited from class com.exalead.gni.GNIObject

printInternalCppPointer

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

splitPolicy

public java.lang.String splitPolicy

The split policy, one of { "splitToken", "addToken", "keepToken", "keepToken2", "keepToken3" }

useCustomNormalizationTID

public boolean useCustomNormalizationTID

Should we use custom normalization TID

stickTokens

public boolean stickTokens

Should the token be sticked

Constructor Detail

GermanTokenizer

public GermanTokenizer(java.lang.String name,
                       java.lang.String resource,
                       Normalizer normalizer,
                       java.lang.String splitPolicy,
                       boolean useCustomNormalizationTID,
                       boolean stickTokens,
                       java.lang.String fields)

Initialize a new German tokenizer

Parameters:

name - Its name

resource - The associated resource name

normalizerResource - The normalizer resource name

splitPolicy - The split policy

useCustomNormalizationTID - Should we use custom normalization TID

stickTokens - Should the tokens be sticked

fields - The list of fields on which it's active

Method Detail

init

public void init(java.lang.String name,
                 java.lang.String[] fields)

Initialize the processor

Specified by:

init in class Processor

Parameters:

name - Its name

fields - The list of fields on which it's active

init

protected void init(java.lang.String name,
                    java.lang.String resource,
                    Normalizer normalizer,
                    java.lang.String splitPolicy,
                    boolean useCustomNormalizationTID,
                    boolean stickTokens)

finalize

protected void finalize()

Overrides:

finalize in class java.lang.Object

getOriginalTokenTag

public static java.lang.String getOriginalTokenTag()