Implementing the Connector

package com.exalead.papi.framework.connectors.example.filesystem;

import com.exalead.papi.framework.connectors.ConnectorConfig;
import com.exalead.config.bean.IsMandatory;
import com.exalead.config.bean.PropertyLabel;

import java.io.File;
/**
* The configuration class for the filesystem connector.
*/
public class DemoFileSystemConnectorConfig extends ConnectorConfig {
private File startingFolder = null;
@IsMandatory(false)
@PropertyLabel("My starting folder")
public void setStartingFolder(final File startingFolder) {
this.startingFolder = startingFolder;
}
public File getStartingFolder() {
return startingFolder;
}
/**
* Optional method listing all option names as they will be displayed in the UI
**/
public static String[] getMethods() {
return new String[] { "StartingFolder" };
}
}

...import com.exalead.config.security.Crypter;

public class MyDataBaseConf extends ConnectorConfig{
private String password;

// setter to define Password property
@BeeKeyValueType("encrypted")
public void setPassword(final String password) {
this.password = Crypter.getInstance().decrypt(password);
}
...

public class BasicFilesystemConnector extends Connector {
private final BasicFilesystemConnectorConfig config;
public BasicFilesystemConnector(final BasicFilesystemConnectorConfig config)
throws Exception {
super(config);
this.config = config;
}

Method	Description
public void scan(PushAPI papi, String mode, Object modeConfig) throws Exception;	This method is used to scan/synchronize all documents. The provided PushAPI object is to be used for synchronization operations. The optional mode parameter, and its optional configuration object, can be used in specialized scan cases defined with the @ConnectorCapabilities annotation. You can also set the @ConnectorCapabilities annotation to get a continuous scan. See Implement a continuous scan.
public Document fetch(String uri) throws Exception	This method is used to retrieve a document from the source, for example, to create a thumbnail or when a user clicks on a search result. uri is the URI of the indexed document.
public MetaContainer getDocumentSecurityTokens(String uri);	This method is used to retrieve the security tokens of a document. This method is not used during indexing but when Exalead CloudView needs to retrieve security tokens at real time. uri is the URI of the indexed document. When you push documents, you must add a security meta: document.addMeta("security", "mytoken");

/**
* Called by the framework when scan abort is requested.
* This method must not change the state of the connector, only wake up some calls
* which might be blocking.
* The scan will only be considered as aborted when the scan() method has returned.
*/
@Override
public void onScanAborted() throws Exception;

/**
* Called by the framework when scan suspend is requested.
* This method must not change the state of the connector, only wake up some calls
* which might be blocking.
* It is then the responsibility of the connector to set the status of the connector
* to SUSPENDED when effectively taken into account.
*/
@Override
public void onScanSuspended() throws Exception;

/**
* Called by the framework when scan resume is requested.
* This method must not change the state of the connector, only wake up some calls
* which might be blocking.
* It is then the responsibility of the connector to set the status of the connector
* to WORKING when effectively taken into account.
*/
@Override
public void onScanResumed() throws Exception;

/**
* Checks the current connector status and, if an abort command was
* sent, throws an ConnectorAbortingException exception.
*
* This function is a helper which can be called by connector. It is not
* called by the framework.
**/
@Override
public void checkAbortingOperation() throws ConnectorAbortingException;

Scan modes describe how connectors index documents. By default, a connector has one scan mode called full which starts when you click the Scan button in the Administration Console. Additional scan modes can be developed and provided with the connector. They are started when you select CONNECTOR NAME > Operation > More actions and click their corresponding Run buttons.

A scan-based indexing exits when the scan is done. The connector starts, scans and indexes documents in the scan() method, and quits. This method is either called periodically to index new documents automatically, or manually by clicking the Scan button.

A permanent-work indexing (also called continuous scan) does not exit when a first scan is done. The connector loops forever in the scan() method, indexing new documents permanently. This method is started automatically by Exalead CloudView just after the connector's initialization, so that when Exalead CloudView starts, your connector starts running immediately. When a connector is set to permanent-work mode, we recommend implementing an "abort" command, to let users click the Abort scan button, when they want the connector to terminate its job and exit.

@ConnectorCapabilities(
scanModes = {
@ScanModeDefinition(
name = "full",
workflow = ConnectorCapabilities.Workflow.PERMANENT_WORK)
}
)public class ContinuousScanConnector extends Connector implements CVComponent {
public ContinuousScanConnector(ContinuousScanConfig config) throws Exception {
super(config);
}

@Override
public void scan(final PushAPI papi, final String scanMode, final Object scanModeConfig)
throws Exception {

while( true ) {
try {
// do the job, index documents
// ...
// check for an abort
if( getStatus(scanMode) == ConnectorStatus.ABORTING ) {
// log the abort and quit
break;
}
}
catch(final Exception e) {
// log the exception
// handle the problem
}
finally {
// clean everything
// be ready for a next run
}
}
}
}

For example, let’s say that your connector is scheduled to launch full scan operations on a regular basis (for example, every 5 minutes) and that you want to run another scan operation once a week to update the index regarding what was deleted in the source content. The two scan operations may conflict at a given time and the second operation may not even be triggered.

You can write a config check class to validate the configuration of your connector. This class must implement the CVComponentConfigCheck interface and override the check() method. In this method, you should check whether all parameters of your configuration contain valid values.

If a parameter contains an invalid value, you should throw a ConfigurationException with a message describing the problem. The check() method is either called when you click the Check config button in the connector Configuration tab, or when you click the Apply button. Throwing a ConfigurationException will stop the validation process and you can't apply an invalid configuration.

public class MyConnectorConfigCheck implements CVComponentConfigCheck<MyConnectorConfig> {
@Override
public void check(final MyConnectorConfig config, final boolean useNow) throws ConfigurationException,
Exception {

// use the MyParam getter
final int myParam = config.getMyParam();

// let's check the parameter value
if( myParam < 0) {
final ConfigurationException e = new ConfigurationException("Invalid MyParam value (must be >= 0)");
e.setConfigKey("MyParam"); throw e;
}
}
}

When a connector runs many things can happen. Knowing which exceptions where met by your connector is necessary to fix issues. This is why it is better to avoid creating your own loggers using either the standard java logger or the Log4J Logger.getLogger().

We recommend using the Connector.getLogger() method, or even better, the Connector.getLogger(String suffix) method. These methods create loggers with the connector instance name. This allows you to differentiate multiple instances of the same connector. Use the method with the suffix argument when you have multiple classes.

public class MyConnectorClass extends Connector implements CVComponent {
Logger thisClassLogger = getLogger();
MyConnectorSubClass sub = new MyConnectorSubClass(getLogger("MyConnectorSubClass"));
// ...
}
public class MyConnectorSubClass {
public MyConnectorSubClass(Logger logger) {
// ...
}
}

When performing tasks within other tasks, you can use com.exalead.log4ng.Log4NGContext which will append pushed contexts before each logging message in the same order they were pushed. This context stack is different for each thread.

Log4NGContext.push("State1");
getLogger().error("FIRST LOGGER");
Log4NGContext.push("State2");
getLogger("myLogger").error("SecondLogger");
Log4NGContext.pop();

Thread t = new Thread(new Runnable() {

@Override
public void run() {
Log4NGContext.push("Thread");
getLogger("myLogger2").error("ThrirdLogger");
}
});
t.start();
t.join();
getLogger().error("FIRST LOGGER AGAIN");

Log4NGContext.pop();
getLogger().error("FIRST LOGGER ONE LAST TIME");