• LinguisticFactory.buildPipe(List<Tokenizer> tokenizers, NormalizerConfig norm, List<SemanticProcessor> processors, int version)
Use the methods signatures with additional ResourcesContext arguments to share resources between multiple pipes. For example, LinguisticFactory.buildPipe(ResourcesContext ctx, String motConfigPath)
Once created, you must initialize the pipe using its init() method to ensure the loading of the resources. To free resources when the pipe is no longer used, call the release() method.
Create a MOTPipe in Java
Below is a code snippet that initializes the pipe named _pipe to do simple tokenization of text:
List<Tokenizer> tokenizers = new ArrayList<Tokenizer>(); tokenizers.add(new StandardTokenizer()); MOTPipe pipe = LinguisticFactory.buildPipe(tokenizers, new NormalizerConfig(), new ArrayList<SemanticProcessor>()); System.out.println("Loading pipe..."); pipe.init(); System.out.println("Pipe loaded.");
Create a MOTPipe with an XML Configuration File
Creating a MOTPipe programmatically can be tedious task when its number of processors increases. It is more convenient to rely on an XML configuration file. For example:
To process the input text, declare the processing of a new document using the newDocument() method. It must be followed after processing by a call to endDocument(). These two calls can trigger some specific tasks from processors, for example, adding annotations directly at the document level.
The following snippet processes the provided string content. The language is unspecified (Language.XX), as only tokenization is done.
When you run it, you obtain an output similar to the following.
Loading pipe... Pipe loaded. Enter a sentence to parse: hello world Token[hello] kind[ALPHA] lng[xx] offset[0] Annotation[hello] tag[LOWERCASE] nbTokens[1] Annotation[hello] tag[NORMALIZE] nbTokens[1] Token[ ] kind[SEP_SPACE] lng[xx] offset[5] Token[world] kind[ALPHA] lng[xx] offset[6] Annotation[world] tag[LOWERCASE] nbTokens[1] Annotation[world] tag[NORMALIZE] nbTokens[1]
Perform Language Detection and Word Lemmatization
Language Detection with Java
The MOTPipe.process() method accepts language as a parameter. The tokenizer uses this language later for all the tokens it creates.
Let us redo the previous example using Language.EN instead of Language.XX. Note how the lng attribute of the tokens has changed from lng[xx] to lng[en].
Processors may support all, one or a subset of languages. A lemmatizer, for example, is dedicated to tokens in the language of its resource only.
Recommendation: Perform a detection when the language of the content is unknown, or when it may contain several languages. To do so, use the Language Detector processor.
The following snippet illustrates the creation of such a processor and its declaration in a pipe.
List<Tokenizer> tokenizers = new ArrayList<Tokenizer>(); tokenizers.add(new StandardTokenizer()); LanguageDetector languageDetector = new LanguageDetector(); languageDetector.setName("languageDetector"); List<SemanticProcessor> processors = new ArrayList<SemanticProcessor>(); processors.add(languageDetector); pipe = LinguisticFactory.buildPipe(tokenizers, new NormalizerConfig(), processors);
With this initialization, running the program detects the language of tokens. The processor replaces the language used in the call to MOTPipe.process() with the one it detects.
Note: A sufficient number of words is required to correctly perform the detection. In a long text containing different languages, the processor gives each token its most probable language.
Language Detection with XML Configuration File
The following configuration file illustrates the creation of the processor and its declaration in a pipe.
Both tokens and documents can receive annotations. You can retrieve document annotations with a call to MOTPipe.getDocumentAnnotations() after the call to MOTPipe.endDocument() (processor usually uses this trigger to add their annotations to the document).
The Language Detector processor adds an annotation for each detected language in the document with the following snippet:
Enter a sentence to parse: have a good day Token[have] kind[ALPHA] lng[en] offset[0] Annotation[have] tag[LOWERCASE] nbTokens[1] Annotation[have] tag[NORMALIZE] nbTokens[1] Token[ ] kind[SEP_SPACE] lng[en] offset[4] Token[a] kind[ALPHA] lng[en] offset[5] Annotation[a] tag[LOWERCASE] nbTokens[1] Annotation[a] tag[NORMALIZE] nbTokens[1] Token[ ] kind[SEP_SPACE] lng[en] offset[6] Token[good] kind[ALPHA] lng[en] offset[7] Annotation[good] tag[LOWERCASE] nbTokens[1] Annotation[good] tag[NORMALIZE] nbTokens[1] Token[ ] kind[SEP_SPACE] lng[en] offset[11] Token[day] kind[ALPHA] lng[en] offset[12] Annotation[day] tag[LOWERCASE] nbTokens[1] Annotation[day] tag[NORMALIZE] nbTokens[1] Document Annotation[en] tag[language] nbTokens[0]
Lemmatization with Java
Lemmatization is a common linguistic task that consists in identifying the lemma of each word using a language dictionary.
To enable lemmatization in both English and French, we need to add two dedicated processors. To do so, complete the pipe creation as shown in Language Detection with Java by the following snippet:
Lemmatizer englishLemmatizer = new Lemmatizer(); englishLemmatizer.setName("englishLemmatizer"); englishLemmatizer.setLanguage("en"); processors.add(englishLemmatizer); Lemmatizer frenchLemmatizer = new Lemmatizer(); frenchLemmatizer.setName("frenchLemmatizer"); frenchLemmatizer.setLanguage("fr"); processors.add(frenchLemmatizer);
The lemmatizers then enrich the tokens generated. The result looks like the output below:
Here a new annotation is added for the token books. It corresponds to an array containing the following information: [lowercase masculine singular, normalized masculine singular, genre, number, lowercase singular, normalized singular, category]. You can access it in a more convenient way using the Lemmatizer.deserialize() method, which returns a LemmaInformation instance that provides accessors.
You can add several annotations if there is any ambiguity. For example, use a Part of Speech Tagger processor next to disambiguate.
You can see that the Standard Tokenizer accepts specific overrides:
• For characters: the : character is considered as an alphabetical character and not punctuation. For example, a:b is only one token.
• For patterns: any sequence of characters that match the specified regexp are considered as a whole token. For example, R&D that matches the first pattern.
You can also configure the Normalizer with normalization exceptions. Here, the forms thé, Thé, or THÉ are all normalized to thé and not the.
Extract Named Entities
You can extract more complex information such as Named Entities.
Note: Dependencies that are not declared in the configuration are implicitly included during initialization.
A dedicated processor is available for this task: the NamedEntitiesMatcher. For more information, see the CloudView Configuration Reference and "Named Entities Matcher" in the Exalead CloudView Configuration Guide.
To use it, configure it in the XML configuration file as follows:
Required dependencies are automatically added during initialization (RelatedTerms and recursively PartOfSpeechTagger). You do not need to include them explicitly in the configuration.
The NamedEntitiesMatcher processor adds named entities annotations on the tokens.
Note: Annotations are added only on the first tokens composing the entities.
The nbTokens property indicates the scope of the annotation. The result looks like the output below:
• relatedTerm = noun phrase in a lemmatized + normalized form
• relatedTermDisplay = noun phrase full-text form
• sub = internal indicator that you can ignore
• exalead.nlp.firstnames = internal indicator (presence of a first name)
• exalead.people = internal indicator (entity present in the Exalead persons dictionary built from wikipedia/freebase/dbpedia)
• NE and NE.famouspeople = result of named entity detection under two different forms based on the specified prefix (that is, "NE" in our example)
The last annotation is the most useful. It contains the canonical form of the entity as well as a tag specifying its type. You can rely on the tag prefix (defined in the configuration file) to locate named entities annotations. When possible, the content of the annotation is a canonical form of the entity that is specified in a thesaurus (for example, "United States" and "U.S." are normalized to "USA"), or inferred with linguistic rules.
Use Case
This use case describes how to extract vehicle registration plates.
For such a task, a simple RulesMatcher processor is adequate. This processor can match patterns expressed by rules against a token stream. Rules are described in a dedicated XML configuration file. For the syntax description, see "Rules Matcher (rule-based)" in the Exalead CloudView Configuration Guide.
Example: the following sample shows how to extract French vehicle registration plates.
<TRules xmlns="exa:com.exalead.mot.components.transducer"> <!-- SIV (e.g. AA-229-AA) --> <TRule priority="0"> <MatchAnnotation kind="NE.plates.SIV"/> <Seq> <Or> <TokenRegexp value="[A-Za-z]{2}"/> <Word value="W" level="exact"/> <!-- to match temporary plates starting with only one W (e.g. W-001-AA) --> </Or> <Opt> <Word value="-" level="exact"/> </Opt> <TokenRegexp value="[1-9]{3}"/> <Opt> <Word value="-" level="exact"/> </Opt> <TokenRegexp value="[A-Za-z]{2}"/> </Seq> </TRule>
In this example, we used the Exalead resource protocol. It implies that the resource is relative to the NGRESOURCEPATH directory (see Java Project Requirements). The file protocol is also supported and you can use it to specify an absolute path. The result looks like the output below.