The major changes introduced in this release concern the Consolidation Server. This component is now mature and has been thoroughly tested. APIs provided by the Consolidation server will be supported for further releases. If needed, the next releases will provide the migration tools required to migrate existing configurations. Some major enhancements have been added to the Consolidation Server (listed in the features below), to reach the GA status. For more details, see the Consolidation Server Guide.
Features
• Pushing documents inside the Consolidation Server can be done independently of the aggregation process. The "Transformation" and "Aggregation" steps are now paralleled and concomitant. Each of them benefits from a dedicated trigger configuration. These new mechanisms greatly improve the overall performance of the indexing process and allow connectors to push documents without timeout errors. Those two steps have their own dedicated triggering configuration.
• The previous impact detection mechanism was based on a static analysis of the rules set within aggregation processors. It was not possible to build dynamic match expression (based on metas for instance) and to have the associated impact detection. This limitation is no longer present. Any matching expression, either static or dynamic, used inside an aggregation processor, is computed at run time and stored for impact detection operations (#24145).
• Transformation processors and Aggregation processors are now supporting Java language (#23583). It is possible to define a processing function inside a standard Java Plugin and to declare it in a processor.
• Processors have now an associated operation context: create/update or delete. Combined with the new ordering capabilities of processors inside aggregation or transformation pipes, it is then possible to finely tune the way operations are processed for each context case.
• Processors being ordered, it is possible to simplify the way document could be discarded or promoted to the next processing step. By default, any document that is pushed inside a transformation or aggregation pipe is automatically yielded to the next step. If new documents are created inside a processor, a yield operation is still mandatory to transfer them to the next processing step. If you want to remove a document from the processing pipe, a new discard function is provided. Discarding a document in a processor will immediately interrupt the processing of that object (#23955).
• When creating links between documents coming from various sources, it is often needed to create a virtual document hosting the common business identifier. Each source document can keep its own identifier and link itself to this intermediary document. The life cycle of this particular document is now managed automatically. Any document created in a transform processor using createDocument() is a managed document and has a dedicated life cycle. At the end of a transformation step, if there is no more vertices attached to a managed document, a delete operation on this node is added to the transformation pipe. The managed document will then be deleted and this delete operation could be used in the aggregation processor if needed (#23623).
Enhancements
• The consolidation introspection console has been improved to provide a better visualization of the graph (#23336)
• When exporting the object graph, it is possible to set the maximum number of arcs per node to limit its width (#23308)
• Matching rules expression can be protected if using special characters (for instance "node['start/My-Type-Name/end'|other]") (#24002)
• For hot backup procedure, freezing operation is taken into account by consolidation server components (#23958 & #23896)
• Existing probes for consolidation server provide more relevant information (#23907)
• Force aggregation command is available directly in the Administration Console. Launching a full forced aggregation operation re-inits also dynamic impact detection rules (#24266)
• The forwarding rules are supporting document type hierarchy (#24221)
• An Helper (PushAPITransformationHelpers) is provided for connector programmer to build directives on a document that will be executed during the transformation step (#24336).
Known limitations
• Fetch operation on a consolidation server does not work with document that are not present inside the consolidation storage. If documents are created during the aggregation step, they haven't any associated storage and thus will not be available for fetch.
• Fetch operation on child document created during the transformation step is not working. This will be fixed in the next release.
• DSL processor should not be mixed with other processor (Java/Groovy). In case of DSL processor is needed in Transformation or Aggregation step, we recommend using only a single processor containing the DSL code.
What about the SDC Connector?
• The SDC connector is now in maintenance mode only (#23581).
Other Improvements
Connectors
Connector Server
• It is now possible to define a fixed port instead of an arbitrary port for fetching operation on the connector server. This setting is done on the connectors server role (#23705). For more details, see the 'Configure HTTPS' section in CloudView Connectors Guide > About Connectors > Deploying Standard Connectors
CSV Connector
• In a CSV file, if there is no value for a column on a line, instead of pushing a meta with a string value "", the meta is not sent. If a default value is set on the data model property, it is then correctly used (#16712).
• A default value is provided for the escaping character (#21835).
• Documents pushed by a CSV connector have now by default the following metas: file name, file folder path and line number (#22029)
• When building a custom URI for a CSV document line, it is now possible to add a custom separator between column values (#23799)
Platform
Admin UI
• The current limitation on size when uploading plugins has been raised from 10MB to 100MB (#23566).
• When enabling Related Terms on a search logic, all existing dictionaries for RT are now suggested (#23632).
• Opening states are now displayed in the status of a slice (replica detail section) (#20888).
Indexing
• When using a prefix rule for mapping a meta into a dynamic property, it is possible to remove the prefix string value from the stored meta name (#23975)
Search
• Proximity boost for query containing stop words is computed in a better way (#21183). Matching "the days of the week" with the query "days of the week" or "days week" will result in the same score for proximity.
• Search performances on numerical field and date/datetime field have been greatly improved (#20760).
• When using multiple disjunctive facets for refinement, the number of queries executed by the search server has been reduced (#24043)
• When dealing with unknown prefix handler, it is possible to consider the UQL node as a false expression (#24082). For more details, see missingPrefixBehaviour option in Cloudview Configuration Reference Guide > Search > Search Logics > User Query Language.
• When sorting on an invalid field, it is now possible to ignore this invalid field in the sort instead of throwing an error (#24188). For more details, see discardInvalidSort option in Cloudview Configuration Reference Guide > Search > Search Logics > Sort & Relevance > SortConfig.
• When dealing with an invalid field in use_logic_hit_metas command, it is now possible to ignore this invalid field instead of throwing an error (#24251). For more details, see option allow_unknown_hit_meta in Search API Parameters Reference Guide.
• A new operator "BOR" is available in the user query language. It does not take into account the ranking properties while processing the query and thus is faster than the OR operator. For more details, see BOR operator in Technical Reference Guide > Configuring Search Queries > User Query Language.
• Expansion limit when targeting a field with "Enable pattern search" option can be set with "Max pattern matches" option on the prefix handler configuration (#24029). Default expansion limit is 50000.
Suggest
• Provide more variety in suggest results when using a suggest dispatcher (#23478). For more details, see the Boost variety option in Technical Reference Guide > Configuring Search Queries > Adding Search Suggestions > Configure a suggest dispatcher.
Semantics
• It is possible to declare a resource of type Annotation manager in the Administration console (#20254).
• A new NE.creditcard semantic resource has been added for Credit Card number detection (#13949). For more details, see the Technical Reference Guide > Appendix: Using Semantic Processors > Information Extraction Processors > Named Entities Matcher.
• A new NE.ip semantic resource has been added for IP addresses detection (#13948). For more details, see the Technical Reference Guide > Appendix: Using Semantic Processors > Information Extraction Processors > Named Entities Matcher.
• New language detector on query for Arabic and Cyrillic based on Unicode range (#23159).
Mashup
• Browser is allowed to keep thumbnail in cache for a week like any other resources (#24107).
• CloudView Taglibs for Mashup UI are shipped in mashup-ui-core.jar (#23313). They will be automatically taken into account in an Eclipse Web project having dependencies on Mashup-UI tags.
• FeedTrigger classes have init() and uninit() methods like Feed class (#24066). Those methods are called each time configuration is applied. It can be useful to reset static variables in custom implementations.
• The name of messages properties files used inside mashup components is now strictly compliant with local standard (#24248). For instance, it is possible to define the following name: messages_zh.properties, messages_zh_CH.properties or messages_zh_TW.properties.
Alerting
For this release, the realtime alerting module benefits from various enhancement when dealing with a large number of alerts, large document and robustness.
• For real time alerting, when a document triggers a given alert, the identifier of the alert is provided in the logs (#24157).
• FastRules builder now correctly handles wildcards, regexps and numerals that could have been set in an alert query (#22523).
• Query arguments for creating an alert using the Management API do not required to be URL encode (#23796).
Storage Service
• When targeting an external storage, passwords set in StorageService.xml configuration file can be cypher like any other password (#24323).
• The storage service supports SQL Server as external database (#24325).
Tools
• It is possible to launch other processes before starting the product (see --preStartHookProcess command line argument). For example, this can be useful to mount a drive at service startup (#23990). For more details, see the 'How to mount drive at service startup' section in CloudView Connectors Guide > Files Connector.
• Dr CloudView provides more relevant messages on suggest topics (#18447).
Licensing
• For a named user license, a token is consumed after a 30 days period starting from the user's first login. Instead of using the last access date for each user to calculate that period, the starting point will be the oldest access date (over the last 30 days) (#23950).
Upgrade notes
• Consolidation: You cannot migrate automatically your consolidation configuration from R2015x.SP2 to R2015x.SP3. Copy your configuration settings (transformation processors, aggregation processors, and consolidation options) and recreate the configuration in R2015x.SP3 entirely.
Thing that might break things:
• Alerting: due to URL encoding enhancement (#23796), if alerts have been set using MAMI, query should not be URL encoding anymore.
• MashupBuilder: due to enhancement on message properties files naming (#24248), it is not possible anymore to have message properties file having only the country in the name (messages_FR.properties). The county code is now mandatory (messages_fr_FR.properties or messages_fr.properties only).
Major fixed issues
Ticket
Title
Existing since
Component
#23876
ResultFeed not serializable
V6R2014x
360/Mashup UI & API
#23920
Real time alerting : Memory consumption crashes the gateway
V6R2014x.SP4
Alerting
#23830
JDBC connector not scaling on 'IncrementalDeleteQuery', leading to OOM crash
V6R2014x.SP4
Connectors / Connector JDBC
#24124
SDC connector: "Abort Scan", then "Scan": indexing is not resumed
V6R2015x.SP2
Connectors / Structured Data Consolidation
#23974
Searchserver OutOfMemoryError when displaying the thumbnail of big XLS file
V6R2014 GA
Convert
#24228
Corrupted tar may lead to read overflow
V6R2014x
Core
#24303
Archive with filenames embedding non-Latin characters replaced by '?'
V6R2014x.SP4
Core
#23567
Refine is not take into account with real time alerting
V6R2014x.SP4
Documentation
#24156
Replace comment in SchedulingConfig by proper xml comment
V6R2015x.SP2
Documentation
#23644
Under Nessus vulnerability scan, hostagent takes 100% CPU forever and needs complete CV reboot
Prehistoric defect
Generic
#23842
System CPU on index slices cause huge response times
V6R2013x.SP3
Generic
#23720
Indexing server falling into contention mode with high system CPU usage
V6R2014x.SP4
Indexing
#23678
RT migrator does not migrate builders on slaves
V6R2015x.SP2
Migration
#23916
Some explicitly named metas are not in CSV output when streaming=true
V6R2013x
Search
#24049
OutOfMemoryError on searchserver after 2 hours or 6500 queries which ever comes last
V6R2015x.SP2
Search
#24077
java classes leak in searchserver
V6R2015x
Search
#24132
Possible deadlock of search server
V6R2015x.SP2
Search
#23937
RelatedTerm extractor crash
V6R2015x.SP1
Semantic
#23047
We must have document frequencies in the related-terms dictionary
V6R2015x.SP1
Semantic
#23742
Some NE are missing when using BasisTechTokenizerCompatibility/BT
V6R2015x.SP2
Semantic
#24296
Strange output when activating related-terms and BasisTech
V6R2015x.SP2
Semantic
Other fixed issues
Ticket
Title
Existing since
Component
#21778
Mashup plugin replication does not work on multi-front environments.
V6R2014x.SP2
360/Mashup Builder
#23708
Can't delete plugin mashup in master slave instance due to missing config check
V6R2014x.SP4
360/Mashup Builder
#24034
Cancel your last change popup wrong behavior in mashup builder
V6R2015x.SP2
360/Mashup Builder
#24220
Pack resources in a single file and strange display
