com.exalead.pdoc.analysis

Class JavaCustomTokenizer

java.lang.Object

com.exalead.pdoc.analysis.CustomDocumentProcessor

com.exalead.pdoc.analysis.JavaCustomTokenizer

All Implemented Interfaces:

CVComponent, DocumentProcessor

public abstract class JavaCustomTokenizer
extends CustomDocumentProcessor

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	JavaCustomTokenizer.InvalidAnnotationException Exception thrown when a malformed annotation is pushed to the semantic pipe.
static class	JavaCustomTokenizer.InvalidTokenException Exception thrown when a malformed token is pushed to the semantic pipe.

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.HashSet<java.lang.String>	contexts
protected java.util.HashSet<java.lang.Integer>	languages

Constructor Summary

Constructors

Constructor and Description
JavaCustomTokenizer(CVComponentConfig config)

Method Summary

Modifier and Type	Method and Description
protected void	addDocumentAnnotation(Annotation annotation) Attach an annotation to the currently processed document after checking its validity.
abstract java.lang.String[]	declareAnnotations() Called at initialization to retrieve the annotation tags that are planned to be produced during tokenization.
abstract void	endDocument() Called when there is no more input to process in the current document.
Referential	getReferential()
void	init(DocumentProcessingContext ctx) Called at start of analysis job
protected Annotation	newAnnotation(java.lang.String tag, java.lang.String displayForm, int nbTokens) Allocate a new annotation with the provided tag, value and length.
abstract void	newDocument() Called when a new document is about to get processed.
protected AnnotatedToken	newToken(java.lang.String form) Allocate a new token of the provided form.
void	process(DocumentProcessingContext dContext, ProcessableDocument doc) Process a document.
abstract void	processChunk(java.lang.String text, int language, java.lang.String context) Called when a new input chunk is to be processed.
protected void	pushToken(AnnotatedToken token) Send a token to the output stream.
void	release() Called at end of analysis job
void	setInputContexts(java.util.Collection<java.lang.String> c)

Methods inherited from class com.exalead.pdoc.analysis.CustomDocumentProcessor

getCondition, setCondition

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

languages

protected java.util.HashSet<java.lang.Integer> languages

contexts

protected java.util.HashSet<java.lang.String> contexts

Constructor Detail

JavaCustomTokenizer

public JavaCustomTokenizer(CVComponentConfig config)
                    throws java.lang.Exception

Throws:

java.lang.Exception

Method Detail

newDocument

public abstract void newDocument()

Called when a new document is about to get processed.

endDocument

public abstract void endDocument()

Called when there is no more input to process in the current document. This is the last chance to attach annotations to the document if needed.

processChunk

public abstract void processChunk(java.lang.String text,
                                  int language,
                                  java.lang.String context)
                           throws java.lang.Exception

Called when a new input chunk is to be processed. The processor must: - produce tokens from the text using newToken() and newAnnotation() - send the said tokens to the semantic pipe with pushToken()

Parameters:

text - the chunk text

language - the chunk language

context - the chunk context

Throws:

JavaCustomTokenizer.InvalidTokenException

java.lang.Exception

See Also:

newToken(), newAnnotation(), pushToken()

declareAnnotations

public abstract java.lang.String[] declareAnnotations()

Called at initialization to retrieve the annotation tags that are planned to be produced during tokenization.

Returns:

the list of all annotation tags needed or null if none

pushToken

protected void pushToken(AnnotatedToken token)
                  throws JavaCustomTokenizer.InvalidTokenException

Send a token to the output stream. - validity of the token is checked - the token is added to the output buffer - if needed, the output buffer is flushed - the token is recycled In all cases, the token and its annotations are not usable anymore after the call.

Parameters:

token - A token allocated through a call to newToken()

Throws:

JavaCustomTokenizer.InvalidTokenException

See Also:

newToken(), newAnnotation()

newToken

protected AnnotatedToken newToken(java.lang.String form)
                           throws JavaCustomTokenizer.InvalidTokenException

Allocate a new token of the provided form. The token is either created or recycled from a previous use. The token type and language are deduced from the form and the chunk input language (they can be overridden though).

Parameters:

form - the new token form

Returns:

a fresh or recycled token

Throws:

JavaCustomTokenizer.InvalidTokenException

newAnnotation

protected Annotation newAnnotation(java.lang.String tag,
                                   java.lang.String displayForm,
                                   int nbTokens)
                            throws JavaCustomTokenizer.InvalidAnnotationException

Allocate a new annotation with the provided tag, value and length. The annotation is either created or recycled from a previous use.

Parameters:

tag - the new annotation tag

displayForm - the new annotation value

nbTokens - the new annotation length

Returns:

a fresh or recycled annotation

Throws:

JavaCustomTokenizer.InvalidAnnotationException

addDocumentAnnotation

protected void addDocumentAnnotation(Annotation annotation)
                              throws JavaCustomTokenizer.InvalidAnnotationException

Attach an annotation to the currently processed document after checking its validity.

Parameters:

annotation - the annotation to attach

Throws:

JavaCustomTokenizer.InvalidAnnotationException

See Also:

newAnnotation()

process

public final void process(DocumentProcessingContext dContext,
                          ProcessableDocument doc)
                   throws java.lang.Exception

Description copied from interface: DocumentProcessor

Process a document.

Parameters:

dContext - Context to perform various operations related to processing

doc - The document being processed.

Throws:

java.lang.Exception

setInputContexts

public final void setInputContexts(java.util.Collection<java.lang.String> c)

getReferential

public final Referential getReferential()

init

public final void init(DocumentProcessingContext ctx)

Description copied from class: CustomDocumentProcessor

Called at start of analysis job

Specified by:

init in interface DocumentProcessor

Overrides:

init in class CustomDocumentProcessor

release

public final void release()

Description copied from class: CustomDocumentProcessor

Called at end of analysis job

Specified by:

release in interface DocumentProcessor

Overrides:

release in class CustomDocumentProcessor