Creating Collaborative Widgets Using Storage Service
When using a collaborative widget, you actually enrich your document with data corresponding to the user’s inputs. This data can be stored using the storage service.
The storage service is an untyped key/value storage that is accessed through a HTTP REST interface using simple commands such as get or put.
The storage service main intent is to provide persistence for Mashup UI applications that stretches beyond HTTP cookies and flat file storage. It is not a general-purpose database, and does not support SQL or any other query language.
Two client wrapper implementations exist for Javascript client use (asynchronous) and Java client use (synchronous).
Using the storage, you can:
• Get arbitrary chunks of data (JSON strings, images, Java objects, etc.) attached to your Mashup UI application.
• Delete, replace and append metas and categories to Exalead CloudView documents.
• Store private data per user.
Note: Keep in mind that data consistency/operation atomicity is not supported (not suitable for sensitive data).
Limitations
The storage does not:
• Support any kind of typing - All data that is put in the storage is treated as binary blobs, which means that they are not treated at all.
• Support transactions or ACID.
• Guarantee data consistency.
By default the storage works with an SQLite engine and is therefore limited in terms of:
• concurrent accesses, as each write action blocks the whole database,
• scalability, when the number of entries is really big.
However, you can configure its JDBC connection to work with compatible databases (SQL Server, MySQL or Oracle) if you need the storage to be more scalable.
<!-- Tables are prefixed, useful if you are using the same database for many different applications --> <Parameter value="cv360" name="tablePrefix"/>
<!-- Internal read/write synchronization, disable for performance gain. Tries to avoid potential file-system-level locking problems that can can occur if any of the following applies: 1. You are using SQLite over a networked file-system 2. You are using SQLite under Mac OS/X 3. You are using an old version of SQLite If none of the above applies you can safely disable this option.See http:// www.sqlite.org/lockingv3.html for more information on this issue --> <Parameter value="true" name="synchronizedWrites"/>
<!-- The database connections are pooled, to avoid reconnecting every time. When using a non-SQLite RDBMS, You can tweak (increase) the settings below for a performance gain.poolMaxActive: Max. num opened connections - Keep low for --> <Parameter value="1" name="poolInitialSize"/> <Parameter value="3" name="poolMaxActive"/> <Parameter value="2" name="poolMaxIdle"/>
Important: After each configuration change, you need to run the <DATADIR>/bin/cvcmd.sh applyConfig script and restart your Exalead CloudView instance.
Storage type scopes
Document scope
The DOCUMENT scope is used for attaching pairs (metas) to Exalead CloudView documents. Pairs stored on documents can optionally be indexed and pushed back into the index.
The DOCUMENT scope saves the doc ID, the build group and the source of the document, to link the data to the document. This is how it works:
• When a document X is pushed in the storage, a pair A is added to the storage and linked to document X.
• A repushFromCache request is triggered on document X.
• The document travels through the analysis pipeline, the StorageServiceDocumentProcessor queries the storage to retrieve all the pairs linked to document X.
• Each document pair is added as a meta within document X. For example, the processor attaches meta A to document X (if a meta A already exists it is replaced, otherwise it is created). Multi-valued pairs are pushed as multi-valued metas.
User scope
The USER scope allows a user to store private data. If user X stores the key A in his/her user scope, it is not accessible to any other user. This is good for keeping per-user state in applications (shopping carts, preferences, etc.).
The USER scope data is not meant to be indexed by Exalead CloudView as it contains private user data.
Shared scope
The SHARED scope is a general-purpose scope which is shared across Mashup UI applications. It allows you to store data that you do not want to link to a document (see DOCUMENT scope).
The SHARED scope data is not meant to be indexed by Exalead CloudView.
Common operations
The common operations supported by the REST protocol are described in the following table.
Note: Both the Javascript and the Java clients have been designed to help you execute these operations.
Operation
Description
AGGREGATE
Returns aggregated values applied to the key.
The possible operations are COUNT, MIN, MAX, AVG, SUM.
The value must be a number except for COUNT which works with everything (numbers, alphanumerical strings, etc.)
GET
Gets all values associated with a single key.
GET_MANY
Gets many keys and all their corresponding values simultaneously.
ENUMERATE
Prefixed get.
Provides a prefix and gets all pairs matching that prefix.
For example, an enumerate query ‘france_paris_’ would get all values associated with ‘france_paris_chatelet, france_paris_montmartre, etc.’
SET
Adds a key/value pair to the storage. If the key exists, it is replaced or appended (whether it is single or multi-valued).
SET_MANY
Sets many values for a given key and replaces existing ones by the new ones. See also PUT_MANY.
PUT
Adds a key/value pair to the storage. If the key exists, an error is returned and the value is left untouched.
PUT_MANY
Adds several values to a given key and keeps existing ones.
DELETE
Deletes a single key with its associated values
How clients communicate with the Storage Service
There are two ways to communicate with the storage service:
The Java client communicates directly with the service at: http://<HOSTNAME>:<BASEPORT+10>/storage-service
The Javascript client communicates with the service through the storage proxy at: http://<HOSTNAME>:<BASEPORT>/mashup-ui/storage
The Javascript client is meant to communicate through the proxy for the 3 following reasons:
• The XHR requests issued by the Javascript client is subject to the cross-domain restrictions that apply to all XHR requests. Therefore, a proxy on the same port/domain is necessary.
• The proxy has rudimentary XSRF protection (X-Requested-With header checking) for the Javascript client's calls, preventing a user X to make changes to the state of user Y's data using XSRF.
• The USER scope. The proxy automatically appends the login name of the user who is currently logged in to the outgoing requests. When communicating directly with the storage service, the user needs to supply the user token manually.
Javascript client use
By default, the Javascript client is included in the Mashup UI. Your custom widget must reference it in its widget.xml file as shown below.
// Instantiates a new JS storage client var storage = new StorageClient(storageType, url, options)
The parameters of the StorageClient class are described in the following table.
Parameter
Description
storageType
The value can be user, shared or document depending on the selected scope.
url
The URL of the storage proxy:
http://<HOSTNAME>:<BASEPORT>/mashup-ui/storage
Note that the JS client is only capable of communicating with the proxy, never directly with the storage-service (because of the cross-domain restriction of XHR requests).
If you are in the context of a Mashup UI page, the url parameter is optional. It will be discovered by the storage client automatically.
options
Optional object that accepts any/all of the following properties:
• timeout: Timeout of the XHR requests in milliseconds (ms) before the error callback is invoked.
• defaultErrorCallback: function(httpStatusCode, XmlHttpRequestObj, storageErrorEnum, storageErrorEnumDesc): Overrides the default error handler that is invoked on every failed request.
• defaultSuccessCallback: function(items, XmlHttpRequestObj): Overrides the default success handler that is invoked on every successful request.
All requests are asynchronous. You always need a callback function to read the output from the client's calls:
// This will not work // The alert is executed before the response is ever received var storage = new StorageClient('shared'); var value = storage.get('myStorageKey'); alert(value); // undefined
// This will work better: A callback function is invoked // after the client has received the response storage.get('myStorageKey', function(items) { alert(items[0].value); // prints the first item in the collection });
If the storage service client is instantiated with USER or SHARED, all non-destructive calls (get, getMany, enumerate) take a key parameter, a success and an error callback.
// Gets all the possible permutations for the docObject and key parameters var docObject1 = { docBuildGroup : "docBuildGroup", docSource : "docSource", docUrl : "docUri" };
For the USER and SHARED scopes, destructive calls look like this:
// singleKey is a key targeting a single element (Ex: 'mySingleKey')
// tries to put singleKey -> value in the storage, call errorCallback if // key already exists for this scope storage.put(singleKey, value, successCallback, errorCallback);
// puts singleKey -> value in the storage. If the key already exists, // its value is overwritten. storage.set(singleKey, value, successCallback, errorCallback);
// removes the pair with key singleKey from the storage storage.del(singleKey, successCallback, errorCallback);
// multiKey is targeting [1:M] values
// appends another value to the bundle pointed to by multiKey storage.put(multiKey, value, successCallback, errorCallback); storage.set(multiKey, value, successCallback, errorCallback);
// adds several values to the key and keeps existing ones storage.putMany(multiKey,[value1,value2, ...], successCallback, errorCallback); // sets several values to the key and replaces existing ones storage.setMany(multiKey,[value1,value2, ...], successCallback, errorCallback);
// deletes all values stored for multiKey storage.del(multiKey, successCallback, errorCallback);
// multiKeyElement is targeting 1 value in a multi valued context // The only way to get a 'multiKeyElement' is to get all the pairs for a multiKey, // and then to use one of the keys in the response
storage.set(multiKeyElement, value, successCallback, errorCallback); // updates the existing value storage.del(multiKeyElement, successCallback, errorCallback); // deletes one value
For the DOCUMENT scope, destructive calls require:
// adds several values to the key and keeps existing ones storage.putMany(documentBuildGroup, documentSource, documentId,multiKey,[value1,value2, ...], successCallback, errorCallback); // sets several values to the key and replaces existing ones storage.setMany(documentBuildGroup, documentSource, documentId, multiKey,[value1,value2, ...], successCallback, errorCallback);
// deletes all pairs stored for a given document: storage.del(documentBuildGroup, documentSource, documentId, successCallback, errorCallback);
// ... the multiKey and multiKeyElement examples are like above, but with // documentBuildGroup, documentSource, documentId prepended the other parameters
For real examples, go to your <DATADIR>/webapps/mashup-ui/WEB-INF/jsp/widgets/ directory, and look at the source code of the following widgets:
• savedQueries: which uses multi-valued keys with USER or SHARED storage
• todoList: which uses single-valued keys with USER or SHARED storage
• starRating: which uses single-valued keys with DOCUMENT storage.
These widgets make extensive use of the storage service.
Java client use
To use the Java client, make sure that the 360-storage-client.jar is included in your Eclipse project.
All the Java client's calls are synchronous, meaning that they are blocked until a response is received from the server.
The Java client is made to communicate directly with the storage service at:
http://<HOSTNAME>:<BASEPORT+10>/storage-service
The Java client provides very destructive methods (clear operations) therefore it should be used with caution.
StorageClient client = new StorageClient(Constants.STORAGE_SERVICE_URL, Constants.MASHUPUI_APPID);
client.scratch(); // Scratches everything in the Storage client.scratchForApplication("default"); // Scratches everything for application 'default'. Other applications’ states are preserved client.scratchForWidget("default", "starRating"); // Scratches all states for the 'starRating' widget of application 'default'
//For Cloudview Documents: use a document descriptor(buildgroup,source,docurl) DocumentDescriptor descriptor = new DocumentDescriptor("bg0", "docSource", "doc1");
dclient.put(descriptor , "helloWorldKey" "Hello World DocumentStorage!".getBytes("UTF-8")); // puts value if key does not already exist
List<byte[]> myValues = new ArrayList<byte[]>(); myValues.add("Hello World DocumentStorage!".getBytes("UTF-8")); myValues.add("Good bye!".getBytes("UTF-8"));
dclient.putMany(descriptor , "helloWorldKey[]", myValues); //puts multiple values to the key and keeps existing ones dclient.setMany(descriptor , "helloWorldKey[]", myValues); //sets multiple values to the key and replaces existing values
Entry entry = dclient.get(descriptor , "helloWorldKey"); dclient.set(descriptor , "helloWorldKeyDuplicate", entry.getValue()); // replaces whatever value (if any) that was previously set in helloWorldKeyDuplicate
DocumentDescriptor descriptor = new DocumentDescriptor("buildGroup", "source", "document1"); String[] bagKey = new String[] {"testbagkey[]"}; StorageAggregationType[] aggrs = new StorageAggregationType[] { StorageAggregationType.COUNT, StorageAggregationType.AVG, StorageAggregationType.MAX, StorageAggregationType.MIN, StorageAggregationType.SUM };
On some social networks, a user's profile can be awarded with one or several predefined badges. Badges can be represented as a string of text, and each badge can be either ‘on’ or ‘off’. In the example below, a badge is ‘on’ if the text string is stored for a particular user, and ‘off’ if it does not exist.
The class attaches one or several predefined text strings to a particular Exalead CloudView document with a multi-valued key ending with ‘[]’. No other values than the predefined ones are allowed to exist on the key.
public class BadgeManager { private final String badgeDatabaseKey; private final ImmutableSet<String> availableBadges; private final DocumentStorage db; // availableBadges are a list of the predefined text-strings that you want to use, // for example ['cool', 'humid', 'warm', 'really-warm'] // badgeTag is the multi-valued key to store on in the storage, for example "perceivedTemperature[]" public BadgeManager(String[] availableBadges, String badgeTag) { this.availableBadges = ImmutableSet.of(availableBadges); this.badgeDatabaseKey = badgeTag; // Instantiate the Java Storage Client given the URL to the Storage Service // (Normally http://CVHOST:[BASEPORT+10]/storage-service) and the Mashup Application ID. // Instead of .getDocumentStorage() we could have used .getUserStorage() or .getSharedStorage() this.db = new StorageClient(Constants.STORAGE_SERVICE_URL, Constants.MASHUPUI_APPID).getDocumentStorage(); } public Set<String> getAvailableBadges() { return this.availableBadges; } public void addBadge(DocumentDescriptor document, String badge) throws Exception { if (!hasBadgeEntry(componentId, badge)) { // To put a pair on a document we supply the Document Source and the document id // The document source is the source connector's name, the document id is the URI of the document // All values are stored as byte blobs, so the string needs to be converted into a byte[] before submitting db.put(document, badgeDatabaseKey, badge.getBytes("UTF-8")); } } public void removeBadge(DocumentDescriptor document, String badge) throws Exception { if (hasBadgeEntry(document, badge)) { Entry toRemove = getBadgeEntry(document, badge); // To delete a single entry from a multi-valued meta an argument "unique" is needed. // Unique is an extra key that identifies a single value in a multi-valued pair deleteByUniqueKey(document, toRemove.getKey().getKey(), toRemove.getKey().getUnique()); } } public Set<String> getAllBadgesFor(DocumentDescriptor document) throws Exception { Set<String> badges = Sets.newHashSet(); for (Entry e : getBadgesFromStorage(document)) { badges.add(new String(e.getValue(), "UTF-8")); } return badges; } public void removeAllFor(DocumentDescriptor document) throws StorageClientException, Exception { // If no unique argument is provided, all of the pair's values are deleted db.delete(document, badgeDatabaseKey); } private boolean hasBadgeEntry(String documentId, String badge) throws Exception { return getBadgeEntry(document, badge) != null; }
private Entry getBadgeEntry(DocumentDescriptor document, String badge) throws Exception { for (Entry b : getBadgesFromStorage(document)) { if (new String(b.getValue(), "UTF-8").equals(badge)) { return b; } } return null; } private List<Entry> getBadgesFromStorage(DocumentDescriptor document) throws Exception { // Returns all the values associated with the document 'docId' and the key 'badgeDatabaseKey' return db.get(document, badgeDatabaseKey); } }