com.exalead.mot.components

Class TokenizerResource

java.lang.Object

com.exalead.gni.GNIObject

com.exalead.mot.core.Resource

com.exalead.mot.components.TokenizerResource

public class TokenizerResource
extends Resource

Tokenizer resource

Field Summary

Fields

Modifier and Type	Field and Description
TokenizationConfig	config

Fields inherited from class com.exalead.mot.core.Resource

lang

Constructor Summary

Constructors

Constructor and Description
TokenizerResource(java.lang.String name)
TokenizerResource(java.lang.String name, java.lang.String path)
TokenizerResource(java.lang.String name, java.lang.String path, int lang)

Method Summary

Modifier and Type	Method and Description
protected java.lang.String	_getParametersDigest()
void	addTokenizationException(java.lang.String pattern, int kind) Add a tokenization exception
protected void	alloc(java.lang.String name, java.lang.String path)
protected void	allocate() Function you should override to perform allocation of the underlying data.
protected void	free() Function you should override to perform deallocation of the underlying data.
int	getTokenKind(java.lang.String token) Stand-alone use of token kind computing
void	overwriteCharKind(char ch, int kind) Add a character kind exception
boolean	overwriteUnicodePropertyKind(java.lang.String property, int kind) Add a kind exception for a whole set of character defined by a unicode property

Methods inherited from class com.exalead.mot.core.Resource

equals, finalize, getExpandedPath, getExpandedPath, getExpandedPath, getExpandedPathIfExists, getExpandedPathIfExists, getIdentifier, getLang, getLangName, getName, hashCode, hasPath, register, register, release, release

Methods inherited from class com.exalead.gni.GNIObject

printInternalCppPointer

Methods inherited from class java.lang.Object

clone, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

public TokenizationConfig config

Constructor Detail

TokenizerResource

public TokenizerResource(java.lang.String name,
                         java.lang.String path,
                         int lang)

TokenizerResource

public TokenizerResource(java.lang.String name,
                         java.lang.String path)

TokenizerResource

public TokenizerResource(java.lang.String name)

Method Detail

_getParametersDigest

protected java.lang.String _getParametersDigest()

Overrides:

_getParametersDigest in class Resource

allocate

protected void allocate()
                 throws ResourceNotFoundException,
                        ResourceAllocationFailedException

Function you should override to perform allocation of the underlying data.

Specified by:

allocate in class Resource

Throws:

ResourceNotFoundException

ResourceAllocationFailedException

alloc

protected void alloc(java.lang.String name,
                     java.lang.String path)

free

protected void free()

Function you should override to perform deallocation of the underlying data.

Specified by:

free in class Resource

addTokenizationException

public void addTokenizationException(java.lang.String pattern,
                                     int kind)

Add a tokenization exception

Parameters:

pattern - A regular expression matching the exception

kind - The token kind of the exception

overwriteCharKind

public void overwriteCharKind(char ch,
                              int kind)

Add a character kind exception

Parameters:

The - character the kind of which must be overriden

kind - The token kind of the character

overwriteUnicodePropertyKind

public boolean overwriteUnicodePropertyKind(java.lang.String property,
                                            int kind)

Add a kind exception for a whole set of character defined by a unicode property

Parameters:

property - The unicode property defining the character set The supported unicode property names are those of the regular expression syntax that can be found at https://www.regular-expressions.info/unicode.html#category

kind - The token kind of the character

getTokenKind

public int getTokenKind(java.lang.String token)

Stand-alone use of token kind computing