Timezone can be defined for any date time transformation (#23128). A new parameter is available on the Search API to set a query timezone. This parameter can be used to adjust any date time value defined in CloudView to the required timezone.
Dedicated new functionalities are available for this purpose:
• #adjust_timezone to recompute a date time value according to the selected timezone,
• Existing time features sensitive to timezone have been extended to support the adjustment (yesterday_tz, month_ago_tz, days_ago_tz, ... )
For more information, see "Specifying a Timezone for date time metas" in the Technical Reference Guide.
Performance
• Item Store / Search: Synthesis performance has been improved when using the new attribute structure. The R2016x configuration is set to format="ItemOrientedStore" for the IndexSchema AttributeGroupStore. Expected performance is similar to the one you got using the runType="RAMColumnDense" parameter in R2015x.
• Item Store / Indexing: Enhancements on Push Server combined with new Item Store structure lead to a better throughput and a better average latency (we observed an average gain factor of 30% on our standard benches).
Removed / Changed features
• Metadata Compaction: Using the Metadata Compaction Push API filter is no longer recommended in CloudView R2016x. Several improvements have been done at the Push Server level to properly handle use cases where Metadata Compaction was required to get better performance. Typically when high multi-threaded throughput was required for small documents containing more than 20 numerical metas or with very small alphanumeric values.
A setup with metadata compaction should always have a higher CPU consumption than a setup without. This was however not always the case in R2015x and prior versions. To save network bandwidth, we recommend using infrastructure solution, like HTTP tunneling with compression.
Known issues
• In this release, the spell check feature (accessible in Search Logics > Query Expansion) has been reviewed to provide better and more accurate suggestions. A few issues are however still present and will be fixed in next releases:
o spellcheck suggestions are displayed even for soundslike prefix handler,
o some suggestion limits are not completely respected (score, minimum number of hits),
o the Calculate No. of hits for each suggestion option might remove valid suggestions.
This release also contains various enhancements, please read the Other Improvements section carefully.
Other Improvements
Connectors
• JDBC connector: BLOB objects in an SQLite storage are correctly retrieved (#22255). For more information, see the "To retrieve BLOBs from a database" procedure in the "CloudView Connectors Guide > JDBC Database Connector > Configuring the JDBC Database Connector > Define the fields" section.
• Crawler connector: When crawling a site and pushing its documents, the original page information is pushed with the document itself in the "referer" meta (#25380).
Consolidation server
The Consolidation Server Introspection UI has been enhanced to provide:
• A clearer view of objects in the graph and how they are connected with one another. When selecting a node in the Introspection UI, the list of arcs linked to it and the ownership of each arc is displayed in the Node details panel (#24991). The Node details also indicate if the node is of managed type (#25527).
• A tool to test any matching rule on a node to identify which graph elements are impacted (#23146). With this new tool, you can also check for any node which are the impacted nodes, according to the existing detected rules, when a change occurs on that particular element.
• Simplified node queries: URI are fully editable and it is also possible to filter nodes by type (#24858)
Some useful administration operations are available:
• It is possible to force a full compact of the Consolidation Server storage directly within the Monitoring Console (#25494)
For more information, see "To check if the Consolidation Storage Compact works properly" in the Consolidation Server Guide.
• JRDS probes for consolidation monitoring have been slightly reviewed to present information in a better way (#25590)
Core
• When having multiple CloudView instances in a cluster, it is useful to be able to switch off the configuration replication on a slave instance without shutting the slave down (#25020). Use the following MAMI operation:
For more information, see the "CloudView Installation & Administration Guide > Maintaining your Installation > Administration tools > Get started with command-line interfaces > cvinit > Advanced operations" section.
• Gateway is less verbose when a slave instance is stopped (#25360)
• The #has_category function accepts a string as path parameter and thus supports all characters accepted for a category path (#25513) #has_category("Top/people/mail/marcel@proust", categories)
Old syntax is kept for compatibility reason but is limited to the following characters: [a-zA-Z0-9_]
• Supported ciphers for communication are restricted by default to trusted ones only (#25154). As for now, in the default CloudView configuration, only TLS is enabled for HTTPS connections. If SSL is required, restrictions on supported protocols can be edited in the DeploymentInternal.xml file. For more information, see the "CloudView Installation & Administration Guide > Securing CloudView > Configure SSL Cipher suites" section.
Administration Console
• Administrator password is less vulnerable to brute force attack (#25081)
• The ranking level can be chosen in the property setting of the data model if "Searchable without prefix" option is selected (#25587). It is no longer necessary to customize the mapping configuration if you want to have a better ranking in the default text field. For more information, see the "Technical Reference Guide > Configuring CloudView with the Data Model > Using Properties to Configure Document Metas > Indexing options for index fields and facets" section.
• The Data Model UI has been reviewed to simplify the use of dynamic fields for class properties (#22747). The UI now displays only the relevant option, depending on the technical fields used to store the property. When adding a new property, you can chose to store it in a dedicated index field or in a shared dynamic field. Using a shared dynamic field reduces file operations when indexing documents.
Some administration functions have been added to the Administration Console.
• The Clear command is available for each replica (#25282).
• Full Compact operation is available for each build group (#25425).
• Once a custom document processor is added to the analysis pipeline, the name of the java class is added to the processor description for better readability (#15893)
Indexing
• When a commit is triggered, if no task is present, no work is started anymore (#25605). This lowers the number of new generations created for index components and reduces the overall load on the system.
• Compaction of index structures has been improved. Unused terms in alphanumeric fields are cleaned (#25351), unused values in value fields also (#25370).
• For document processing in the semantic pipeline, a timeout can be set to be protected against huge documents that might block the indexing (default value is set to "no limit") (#25259).
Search
• When defining multiple sorts within a Search logic, if no sort options were specified at query time, all sorts were applied. If a large number of sorts were defined, this could have a bad impact on performance. Active sort(s) can now be specified on a search logic (#25401).
• Having multiple Template Prefix Handler defined in a search logic does not impact performance anymore (#25377).
• UnitsOfMeasurement prefix handler is already reporting incompatible units (not related to the magnitude set on the property). It also reports unknown units too (#25339). Support for all numbers pattern has been extended and conversions are performed only if really needed (#25337).
• Support of double bytes spaces and characters (duplicate alphabets in Unicode) for UQL parser (#23334).
• Related Terms synthesis performance has been improved. On a standard corpus, the estimated improvement is around 40% for processing (#25482).
• The languages set or detected when resolving a query are available for reporting by using the column parameter "expansion_languages" (#24882).
• Semantic Query Analysis has a new unused word policy value "as-is". This policy triggers the rewriting of the matching parts only, leaving the rest of the query unchanged (#22001)
Semantics
• When testing a semantic resource in the Business Console, you can now display consistent annotations only, by hiding intermediate annotations. This unclutters the Annotations panel view (#22999).
Mashup UI & Builder
• Fetching a document is no longer limited by the size of the security context (POST Request) (#25345)
• The roundingMode parameter has been added to the numberFormat MEL function. It selects a rounding mode depending on the locale by default. It is yet possible to specify a rounding mode if necessary. See Java rounding mode to know which rounding modes are supported. (#25634) The numberFormat MEL function has the following parameters: ${number:format(value, minimumIntDigit, maximumIntDigit, minimumFractionDigit, maximumFractionDigit, isGroupingUsed, roundingMode)} For example, you can specify ${number:format(0.25, 1, 4, 1, 1, true, "HALF_UP")} to round up 0.25 to 0.3 (and not 0.2) .
MAMI & Tools
• Usage statistics for the ItemStore component are available in cvconsole (#25385)
cvdebug index dump-attribute-group-headers
• MAMI operations targeting a slice can also target all slices in a single operation (#25385).
• Product safety improved when using cvconsole commands (#20584).
Monitoring
• New Network probes are available for Linux (#25014)
• PAPI Thread usage is available in the system probes of a build group (Services > Exalead > Indexing > bg name > System) (#25260)
OS Support
CloudView now officially supports Suse 12 (#24714) and RedHat 7.2.
Upgrade notes
The CloudView R2016x configuration is fully compatible with CloudView R2016x R1.
Major fixed issues
Ticket
Title
Existing since
Component
#25511
Missing cloudview-java-plugin.jar in mashup-ui-sdk classpath
V6R2015x.SP4
360/Plugins
#25437
JRDS graph on compacter - slots count shows activity even when there are no imports
V6R2016x
Admin UI
#25767
Time shift for scheduled events when edited with UI in a different timezone
Prehistoric defect
Admin UI
#22228
The status of the "always-on" Crawler-connector is "idle" instead of "working"
V6R2015x
Connectors Framework Java
#25408
ConnectorRunner should trigger indexing job when clearing a connector
Prehistoric defect
Connectors Framework Java
#25454
Java processor instance MUST NOT be shared across aggregation threads
V6R2015x.SP3
Consolidation
#25452
Compact slot is not working correctly after a massive delete
V6R2015x.SP4
Consolidation
#25465
Extra aggregation orders should be flushed before exceeding heap capacity
V6R2015x.SP4
Consolidation
#25469
Payload MUST NOT be strongly referenced
V6R2015x.SP1
Consolidation
#25515
Impossible to change a port once used by a Consolidation Server
V6R2015x.SP2
Consolidation
#25534
Impacted documents are pushed twice to build group in a single consolidation job
V6R2015x.SP4
Consolidation
#25584
PDF indexation issue: Internal UTF8 error
Prehistoric defect
Convert
#25728
MAMI FullCompactIndex operation compacts only slice 0 when no slice is given
Prehistoric defect
Core
#25585
Crawler connector is not working with Consolidation Server
V6R2015x.SP4
Crawler
#19470
Gateway stuck with lot of thread in use
V6R2013x.SP3
Generic
#25733
Global dictionary status may remain "building" when a process dies during publish
V6R2012x
Generic
#25391
Very slow imports on some big indexing jobs
V6R2014 GA
Indexing
#25298
Unit of measurement does not support negative measures
V6R2016x
Indexing
#25349
indexing process takes too long with BasisTech+NEmatcher on very big documents
V6R2014x.SP4
Indexing
#25356
Unit of measurement property does not like °C at push
V6R2016x
Indexing
#25397
Indexing vertical scalability with homogeneous corpus
V6R2016x
Indexing
#25493
Big logs in Indexing server could lead to OOM crash
V6R2014 GA
Indexing
#25625
Number of documents in index may be negative or lower than reality when discard doc proc is used
V6R2016x
Indexing
#25796
Error at postinstall on a machine with no vcredist
V6R2015x.SP4
Installer
#25572
SQA: Encoding error at offset 24 - beware of optional parts which may be unmatched
V6R2015x.SP4
Search
#21293
Timeouted query keeps running in slice using timeout=0 & absolutetimeout!=0
V6R2014x.SP1
Search
#25319
Unable to perform queries when related terms are on slave instance
V6R2015x.SP1
Search
#25367
Unexpected failures while querying measures with decimal numbers
V6R2016x
Search
#25450
Wrong Proximity boost when using Prefix handler on multi-target index fields
V6R2015x.SP4
Search
#25478
RelatedTerms facet is empty when max.categories is set to 0
V6R2015x.SP1
Search
#25551
Fatal error for Search server on slice timeout
Prehistoric defect
Search
#25568
Surprising division by zero when using custom ranking formula
V6R2013x
Search
#25612
Possible searchServer crash after slice timeout
Prehistoric defect
Search
#25640
PHP SearchClient does not handle multiple security tokens
Prehistoric defect
Search
#25610
bug retrievable: annotation created by an annotationmanager and value facet
V6R2016x
Semantic
#25384
Revamp BasisTech suggested languages in UI
V6R2016x
Semantic
Other fixed issues
Ticket
Title
Existing since
Component
#23760
MEL: elseif in a foreach throws a ParseError
Prehistoric defect
360/Mashup UI & API
#25053
Horizontal Facet widget does not handle exclude correctly
Prehistoric defect
360/Mashup UI & API
#25113
CustomContextPathFilter crash with '%' as parameter value
V6R2015x.SP3
360/Mashup UI & API
#23999
Display bug with Arabic language
V6R2014x.SP4
360/Mashup UI & API
#24868
Adding a checkbox to uncheck all nodes at once
V6R2015x.SP3
Admin-ui
#24941
can't display part details in data processing test tab when meta with same name removed
V6R2014x
Admin-ui
#25184
Inconsistent result in the Test UI of the analysis pipeline
V6R2015x.SP3
Admin-ui
#25058
Admin UI check syntax helper is broken
V6R2014x.SP2
Connector Logs
#25019
Documentation doesn't instruct to detach the slave during maintenance operation
V6R2015x.SP3
Documentation
#25056
Wrong documentation about uri blacklisting
V6R2015x.SP3
Documentation
#25230
SearchAPI Param: xbin, ybin - size of the bin in the X and Y directions
V6R2015x.SP4
Documentation
#24559
PointInteger and PointDouble should be public
Prehistoric defect
Generic
#25036
Sometimes, deployment status tag convert has loop crashing but is wrong
V6R2014x.SP4
Generic
#25479
Slice crash with message: "Fatal inconsistency: two threads in manager critical section"
V6R2015x.SP4
Generic
#25600
NRT Indexing: builder faster than index
V6R2015x.SP4
Generic
#24011
Crash indexing server when Field[keyword] is mapped but was not declared
V6R2014x.SP4
Indexing
#25067
In Analysis.xml, a ReplaceValues definition with duplicate keys crashes indexingserver
V6R2015x.SP3
Indexing
#17147
Clear Build Group can leave replica index in inconsistent state
Prehistoric defect
Mercury/Indexing
#25186
blockSize is not supported error during migration
V6R2015x.SP3
Migration
#24835
Scoring issues in Russian
V6R2014x.SP4
Search
#25195
Do not prefix the prop with the class name when using a dynamic prop in the non-default class
V6R2015x.SP4
Search
#25103
Inconsistent results when using min parameter of SortBy in similar documents computing