Since tokenization is a required processing step for all searchable alphanumeric text, it is set up automatically as part of the Exalead CloudView installation. This setup is known as the default tokenization config, tok0.

A tokenization configuration specifies which tokenizers to use when Exalead CloudView analyzes incoming documents at index-time. It also specifies how to tokenize queries at search-time.

Cloudview tokenizes many languages natively. This is the Standard support.

This tokenizer is set up by default when you install Exalead CloudView. It breaks down text into tokens whenever it encounters a space or a punctuation mark. It recognizes all known punctuation marks for all languages.

You can have either:

When Exalead CloudView tokenizes text, it examines each character and checks the character override rules for any special processing instructions for this particular character.

WhenExalead CloudView tokenizes text, it searches for any patterns between the current location to the end of the document, based on the pattern override rules. For each matching pattern, it applies the corresponding processing instruction to this particular pattern.

Specify patterns using PERL 5 regular expressions.

Selecting these options for German, Norwegian, or Dutch means once the tokenizer has produced a token during indexing, it checks if the token is a compound word. If it is, it adds annotations for each part of the compound word. This way, searching for one part of the compound word can match the whole word.

Fore more details on decompounding, see About Decompounding.

Selecting these options ensures words that include numbers are processed as a single token. For example:

Normal separator rules for tokenization still apply: if there is a separator between numbers and other letters, the numbers and letters are processed as two separate tokens.

You can activate the transliteration option in the Exalead CloudView XML configuration (it is not available in the Administration Console), for transliterating Unicode Latin Extended B characters. Several characters are converted to their closest Latin equivalent to facilitate query typing. This is useful when you want to match documents in another charset from the one you use for searching.

For example, you may want to search for words containing "Ł" using the closest Latin character "L" even though it does not match phonetically.

As for now, the supported transliterations are the following:

This tokenizer is set up by default. You can have only one Japanese tokenizer per tokenization config.

The 4 alphabets used in Japanese texts (kanji, hiragana, katakana, romaji) are implemented by this tokenizer.

The text normalization process for Japanese is more complex than for European languages:

This processor supports the following transcriptions:

The default configuration uses only tokens and their hiragana transcriptions. You can use any processor in the semantic pipe to exploit other transcriptions produced for linguistic purposes. To annotate text with parts of speech to flag nouns, verbs, and adjectives, select the option Add morphology in Linguistics > Tokenizations > your tokenization > Japanese.

Since tokenizing japanese text is a difficult task, the processor may produce different outputs for the same input. Some of them may contain errors.

For example, in the following cases, queries may fail and skip documents (thus increasing silence):

To maximize recall and avoid errors, a character-based over-indexing is enabled by default. This over-indexing reduces search silence by trying to match character sequences independently of tokenization, in addition to the tokenized query. However, to avoid an unreasonable amount of noise in top search results, this character-based query expansion has a much lower contribution to the document score than the tokenized part.

If too many results are displayed, you can disable this recall by changing the value of the favor option from recall to precision in the Japanese tokenizer configuration (in <DATADIR>/config/Linguistic.xml).

If you have purchased Extended Languages, you can remove this tokenizer and instead set up a Basis Tech tokenizer for Japanese tokenization.

The main differences are:

This tokenizer is specified by default. You can only have one Chinese tokenizer per tokenization config.

It includes the option to annotate the text with simplified Chinese transliterations.

The Chinese tokenizer retrieves first (and with a higher score) the sets of ideograms that correspond to real words. It also tokenizes ideogram by ideogram to support arbitrary ideogram combination, but this is less relevant in terms of meaning.

You can define a Basis Tech tokenizer for a specific language only. Basis Tech tokenizers generally offer richer semantic processing for Asian, Middle-eastern, and African languages than the native Exalead CloudView tokenizers.

Create one Basis Tech tokenizer for each language to be tokenized.

Before enabling Extended Languages, keep in mind the following:

With Exalead CloudView’s native tokenization, we only index the original words. At query time this adds an OR expansion for lemmatization.

For example: q=alsacien is expanded to q=alsacien OR alsacienne OR alsaciens OR alsaciennes.

When using BasisTech to tokenize a language, you must index the lemmatized form of the word in addition to the original form. Indeed, Basis Tech does not include the resource that associates a form to all its expansions.

A tokenization configuration specifies which tokenizers to use when Exalead CloudView analyzes incoming documents at index-time. It also specifies how to tokenize queries at search-time.

By default, Exalead CloudView uses tok0 as the tokenization configuration for converting text into tokens. However, if you create additional tokenization configs, you must specify them explicitly in Data Model > Semantic Types and Data Processing.

You may want to create a new tokenization config to define:

Specify a tokenization config for a specific index mapping or semantic type, when you know that a certain meta contains characters that need to be interpreted differently than other alphanumeric metas. Typical examples are identifiers like user IDs, model numbers and product codes.

When you specify a new tokenization config for document analysis at index-time, you must also specify this same tokenization config for interpreting queries at search-time.

You can specify a default tokenization config for a search logic, as well as an "exception" tokenization config for specific prefix handlers. For more information, see Specify a Tokenization Configuration for Prefix Handlers.

Indexing is both morphological and ngram-based. This gives Exalead CloudView a kind of fallback mechanism. For example, "cjk" (that is, Chinese Japanese Korean) in Linguistics > Tokenizations > Advanced > Form indexing triggers the indexing of isolated characters but we still index words made of several characters output by tokenizers. This way, in case of different tokenizations between indexing and querying for the same text (either because of a different context or because of a tokenization error), we can still find documents containing the sequence of characters.

A tokenization config that is defined for a specific:

For example:

To achieve this, Exalead CloudView splits compound words into at most two components using a dictionary. Three issues may occur during the process:

In these cases, the user can enrich the dictionary with his own rules by defining a list of words which should not be decompounded and a list of explicit decompoundings taking precedence over the Exalead CloudView-provided resource.

The user dictionary is a UTF-8 text file (user.txt) in the directory KITDIR/resource/all-arch/subtokenizer/ID where KITDIR is the root of CloudView unzipped kit directory and ID is one of following language identifiers: de (german), nl (dutch), no (norwegian).

The file format must be as follows (lines failing to match this format are ignored):

Note that words are matched case-insensitively but accents matter ("Kuchen" is not "Küchen").

When you specify a new tokenization config for document analysis at index-time, you must also specify the same tokenization config for interpreting queries at search-time.

After editing the user dictionary:

If issues should appear, turn the logging level of the indexingserver to debug and restart it. Search for [subtokenizer] in the logs to filter relevant lines.