com.exalead.papi.framework.connectors

Class GenericMultiThreadedConnector<SKey extends Key>

java.lang.Object

com.exalead.papi.framework.connectors.Connector

com.exalead.papi.framework.connectors.GenericMultiThreadedConnector<SKey>

All Implemented Interfaces:

CVComponent, ConnectorStatusExceptionInterface, ConnectorInterface

Direct Known Subclasses:

GenericContainerMultiThreadedConnector, MultiThreadedConnector

public abstract class GenericMultiThreadedConnector<SKey extends Key>
extends Connector

Generic class of connectors with multithreaded enumeration. The following methods must be implemented:

• getKeyFromUri() to transform a generic Document URI into a "Key" (@see Key)
• buildSourceEnumerator() to get a folder enumerator (allowing to enumerate document or folder Keys of a folder), given a foler Key
• buildSourceAdapter() to get a source adapter (allowing to get data and metadata of a Key), given a document Key or a folder Key

The following method can be overridden:

• buildStartKeyFromConf() to get the root folder Keys, given the configuration root paths (the default is to consider root paths as regular folder URI)

A multithreaded connector extending this class handles opaque "Key" objects (@see Key), which are basically transformed Document's URI, with additional information (is the given URI a folder, what is the parent, etc.) The root folder paths are transformed into Key at startup, and added to the list of folders/documents to be crawled. The Key class can be extended, to store connector-specific information regarding the URI to be processed. Key.getUri() can be used to get the original URI, and getKeyFromUri() (in this class) can be used to build a Key, given an URI. The initial "root" Keys to be fetched are read from the configuration (taken as Document URI, and transformed into Key by getKeyFromUri()) The buildSourceEnumerator() function returns an object used to enumerate folder keys (including the initial root Keys) through the "enumerates()" function. The buildSourceAdapter() function returns an object used to access to the Document (@see Document), and various metadata (security metadata, stamp ..)

See Also:

class

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	GenericMultiThreadedConnector.EnumeratesFolderDeletionHelperClient

Nested classes/interfaces inherited from class com.exalead.papi.framework.connectors.Connector

Connector.ConnectorNameThreadFactory

Field Summary

Fields

Modifier and Type	Field and Description
protected EnumerableOperationBuilder<SKey>	addDocumentOperationBuilderForFirstAdd The OperationBuilder used to create DeleteOperation during Index Enumeration
protected EnumerableOperationBuilder<SKey>	addDocumentOperationBuilderForNotModified
protected EnumerableOperationBuilder<SKey>	addDocumentOperationBuilderForUpdate
protected static java.lang.String	DEFAULT_LAST_UPDATE_CHECKPOINT_BEGIN
protected static java.lang.String	DEFAULT_LAST_UPDATE_CHECKPOINT_END
protected EnumerableOperationBuilder<SKey>	deletedDocumentOperationBuilder
protected boolean	doDelete
protected OperationPipelinesManager<SKey>	enumeratedPipelinesManager
protected boolean	isUpdate
protected java.lang.String	last_update_checkpoint_begin
protected java.lang.String	last_update_checkpoint_end
protected boolean	neverSendDeletes
protected com.exalead.concurrent.ConcurrentProcessingQuota	processingQuota
protected boolean	pushFolder
SourceAdapterManager<SKey>	sourceAdapterManager
SourceEnumerator<SKey>	sourceEnumerator
protected java.util.List<SKey>	startKeys
protected boolean	syncEnumeratedError
SynchronizerCallbackContainer<SKey>	synchronizerCallbackContainer
protected java.util.concurrent.ThreadFactory	threadFactory

Fields inherited from class com.exalead.papi.framework.connectors.Connector

config, DEFAULT_SCAN_MODE, papi

Constructor Summary

Constructors

Constructor and Description
GenericMultiThreadedConnector(GenericMultiThreadedConnectorConfig config)
GenericMultiThreadedConnector(PushAPI papi, GenericMultiThreadedConnectorConfig config) Deprecated.

Method Summary

Modifier and Type	Method and Description
void	addDocumentOperationToEnumeratedOperationPipeline(DocumentOperation<SKey> op) Add a document operation to the pipeline manager
void	addFolderOperationToEnumeratedOperationPipeline(FolderOperation<SKey> op) Add a folder operation to the pipeline manager
protected void	addRootKeyInPipelineManager(SKey root, EnumerableOperationBuilder<SKey> opBuilder, EnumerableOperationBuilder<SKey> opBuilderUpdate, EnumerableOperationBuilder<SKey> operationBuilderNotModified) Add root key to operation (called by initializePipelinesManager())
void	addSynchronizerCallback(SynchronizerCallback<SKey> cb) Add a new SynchronizerCallback which will be used during synchronization, allowing to execute specific operations
protected EnumerableOperationBuilder<SKey>	buildAddDocumentOperationBuilderForFirstSync()
protected EnumerableOperationBuilder<SKey>	buildAddDocumentOperationBuilderForUpdate() Get AddDocumentOperationBuilder
protected DeletedDocumentOperationBuilder<SKey>	buildDeletedDocumentOperationBuilder()
FolderDeletionHelper	buildDeletionHelper(FolderDeletionHelper.FolderDeletionHelperClient client, java.lang.String rootPathUri) Returns this connector's deletion helper implementation provided this connector is in update mode
protected abstract SourceAdapter<SKey>	buildSourceAdapter() This method return a SourceAdapter implementation
protected abstract SourceEnumerator<SKey>	buildSourceEnumerator() This method return a SourceEnumerator implementation
protected java.util.List<SpecializedSourceAdapter<SKey>>	buildSpecializedSourceAdapters() Get SpecializedSourceAdapters
protected SKey	buildStartKeyFromConf(java.lang.String configFolder, boolean isFolder) Call to transform Folder Name (from RootFolders) to FolderKey Override this function if the Document URI and root folder URI do not share the same logic.
void	callBeforeDestruction() Override this method to do some task before the destruction of the connector
Document	fetch(java.lang.String uri) This method is used to retrieve document from the source when CloudView product needs it.
protected java.util.Iterator<UpdateItem>	getActionToUpdate() Override default operations executed for an update (that is, adding root keys to the list of operations to be executed) Default implementation is to return null (ie.
GenericMultiThreadedConnectorConfig	getConnectorConfig() Return the connector's configuration.
PushAPI	getCurrentPAPI()
java.util.concurrent.ThreadFactory	getDefaultThreadFactory()
MetaContainer	getDocumentSecurityTokens(java.lang.String uri) This method is used to retrieve security tokens of a document.
abstract SKey	getKeyFromUri(java.lang.String uri) Get a Key, given the document URI.
int	getNbThreadsPerEnumeratedDocumentOperationPipeline() Number of threads processing documents inside the pipeline manager
int	getNbThreadsPerEnumeratedFolderOperationPipeline() Number of threads processing folders inside the pipeline manager
protected long	handleRootDeletes() Add root keys to operation
void	init() Initialization method called by connector framework after construction
protected void	initializePipelinesManager(boolean forUpdate) Add root keys to operation
boolean	isdoDelete() Do deletes for document that do not exist anymore
boolean	isPushFolder() Should folders be pushed as regular documents too ?
boolean	isSynchronizeSourceEnumeratorsError() Has en an error occurred during synchronization ?
boolean	isUpdate() The current synchronization job is an update of a previous synchronization job
protected void	processActionToUpdate(java.util.Iterator<UpdateItem> items) Execute a list of UpdateItem Called if getActionToUpdate() returned a non-null list
com.exalead.concurrent.ConcurrentProcessingQuotaWorker	registerDocInProgress(Document doc) Register a document in processing, for quota handling.
com.exalead.concurrent.ConcurrentProcessingQuotaWorker	registerWorkInProgress(long size) Register a worker in processing, for quota handling.
void	removeSynchronizerCallback(SynchronizerCallback<SKey> cb) Remove a SynchronizerCallback previously added by a call to addSynchronizerCallback()
void	scan(PushAPI papi, java.lang.String mode, java.lang.Object modeConfig) This method is used to push (or synchronize) all documents from the source to CloudView product.
void	setCurrentPAPI(PushAPI newPAPI)
void	setNbThreadsPerEnumeratedDocumentOperationPipeline(int nb) Set the number of threads processing documents inside the pipeline manager
void	setNbThreadsPerEnumeratedFolderOperationPipeline(int nb) Set the number of threads processing folders inside the pipeline manager
void	setNbThreadsPerEnumeratedOperationPipeline(int nb) Set the number of threads processing documents or folders inside the pipeline manager
void	setSynchronizeSourceEnumeratorsError(boolean value) Called when an error occurred during synchronization
void	terminateThreads() Terminate all threads used by the pipeline manager

Methods inherited from class com.exalead.papi.framework.connectors.Connector

checkAbortingOperation, checkAbortingOperation, checkDataSourceAvailability, checkSecurityTokens, fetch, getConnectorName, getEndpointBuilder, getLogger, getLogger, getScanModes, getState, getState, getStatus, getStatus, getStatusException, init, onScanAborted, onScanAborted, onScanResumed, onScanResumed, onScanSuspended, onScanSuspended, scratch, selfAbortScan, selfAbortScan, synchronizeDocuments

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

threadFactory

protected java.util.concurrent.ThreadFactory threadFactory

DEFAULT_LAST_UPDATE_CHECKPOINT_BEGIN

protected static final java.lang.String DEFAULT_LAST_UPDATE_CHECKPOINT_BEGIN

See Also:

Constant Field Values

DEFAULT_LAST_UPDATE_CHECKPOINT_END

protected static final java.lang.String DEFAULT_LAST_UPDATE_CHECKPOINT_END

See Also:

Constant Field Values

syncEnumeratedError

protected volatile boolean syncEnumeratedError

pushFolder

protected volatile boolean pushFolder

neverSendDeletes

protected final boolean neverSendDeletes

enumeratedPipelinesManager

protected OperationPipelinesManager<SKey extends Key> enumeratedPipelinesManager

doDelete

protected volatile boolean doDelete

isUpdate

protected volatile boolean isUpdate

startKeys

protected java.util.List<SKey extends Key> startKeys

last_update_checkpoint_begin

protected java.lang.String last_update_checkpoint_begin

last_update_checkpoint_end

protected java.lang.String last_update_checkpoint_end

processingQuota

protected com.exalead.concurrent.ConcurrentProcessingQuota processingQuota

sourceEnumerator

public SourceEnumerator<SKey extends Key> sourceEnumerator

sourceAdapterManager

public SourceAdapterManager<SKey extends Key> sourceAdapterManager

synchronizerCallbackContainer

public SynchronizerCallbackContainer<SKey extends Key> synchronizerCallbackContainer

addDocumentOperationBuilderForFirstAdd

protected EnumerableOperationBuilder<SKey extends Key> addDocumentOperationBuilderForFirstAdd

The OperationBuilder used to create DeleteOperation during Index Enumeration

addDocumentOperationBuilderForUpdate

protected EnumerableOperationBuilder<SKey extends Key> addDocumentOperationBuilderForUpdate

addDocumentOperationBuilderForNotModified

protected EnumerableOperationBuilder<SKey extends Key> addDocumentOperationBuilderForNotModified

deletedDocumentOperationBuilder

protected EnumerableOperationBuilder<SKey extends Key> deletedDocumentOperationBuilder

Constructor Detail

GenericMultiThreadedConnector

@Deprecated
public GenericMultiThreadedConnector(PushAPI papi,
                                                 GenericMultiThreadedConnectorConfig config)
                                          throws java.lang.Exception

Deprecated.

Throws:

java.lang.Exception

GenericMultiThreadedConnector

public GenericMultiThreadedConnector(GenericMultiThreadedConnectorConfig config)
                              throws java.lang.Exception

Throws:

java.lang.Exception

Method Detail

init

public void init()
          throws java.lang.Exception

Description copied from class: Connector

Initialization method called by connector framework after construction

Overrides:

init in class Connector

Throws:

java.lang.Exception

getConnectorConfig

public GenericMultiThreadedConnectorConfig getConnectorConfig()

Description copied from class: Connector

Return the connector's configuration. The returned object can be casted to the connector's configuration class.

Overrides:

getConnectorConfig in class Connector

getDefaultThreadFactory

public java.util.concurrent.ThreadFactory getDefaultThreadFactory()

Returns:

The default thread factory of this connector

buildSourceAdapter

protected abstract SourceAdapter<SKey> buildSourceAdapter()
                                                   throws java.lang.Exception

This method return a SourceAdapter implementation

Returns:

SourceAdapter implementation

Throws:

java.lang.Exception

buildSourceEnumerator

protected abstract SourceEnumerator<SKey> buildSourceEnumerator()
                                                         throws java.lang.Exception

This method return a SourceEnumerator implementation

Returns:

SourceEnumerator implementation

Throws:

java.lang.Exception

buildStartKeyFromConf

protected SKey buildStartKeyFromConf(java.lang.String configFolder,
                                     boolean isFolder)
                              throws java.lang.Exception

Call to transform Folder Name (from RootFolders) to FolderKey Override this function if the Document URI and root folder URI do not share the same logic. Warning : can be called with null parameter when the RootFolders is empty (to manage default root path)

Parameters:

configFolder - The configuration root folder

Returns:

root folder key The default implementation uses getKeyFromUri(), and calls setFolder()

Throws:

java.lang.Exception

buildAddDocumentOperationBuilderForUpdate

protected EnumerableOperationBuilder<SKey> buildAddDocumentOperationBuilderForUpdate()

Get AddDocumentOperationBuilder

Returns:

buildAddDocumentOperationBuilderForFirstSync

protected EnumerableOperationBuilder<SKey> buildAddDocumentOperationBuilderForFirstSync()

buildDeletedDocumentOperationBuilder

protected DeletedDocumentOperationBuilder<SKey> buildDeletedDocumentOperationBuilder()

buildDeletionHelper

public FolderDeletionHelper buildDeletionHelper(FolderDeletionHelper.FolderDeletionHelperClient client,
                                                java.lang.String rootPathUri)
                                         throws PushAPIException

Returns this connector's deletion helper implementation provided this connector is in update mode

Parameters:

client - handles the actual deletes of the documents.

rootPathUri - the folder root key

Returns:

A valid deletion helper, or if isUpdate() null

Throws:

PushAPIException

getKeyFromUri

public abstract SKey getKeyFromUri(java.lang.String uri)
                            throws java.io.IOException

Get a Key, given the document URI.

Throws:

java.io.IOException

buildSpecializedSourceAdapters

protected java.util.List<SpecializedSourceAdapter<SKey>> buildSpecializedSourceAdapters()

Get SpecializedSourceAdapters

Returns:

the list of SpecializedSourceAdapter

isPushFolder

public boolean isPushFolder()

Should folders be pushed as regular documents too ?

setSynchronizeSourceEnumeratorsError

public void setSynchronizeSourceEnumeratorsError(boolean value)

Called when an error occurred during synchronization

isSynchronizeSourceEnumeratorsError

public boolean isSynchronizeSourceEnumeratorsError()

Has en an error occurred during synchronization ?

isdoDelete

public boolean isdoDelete()

Do deletes for document that do not exist anymore

isUpdate

public boolean isUpdate()

The current synchronization job is an update of a previous synchronization job

addSynchronizerCallback

public void addSynchronizerCallback(SynchronizerCallback<SKey> cb)

Add a new SynchronizerCallback which will be used during synchronization, allowing to execute specific operations

See Also:

SynchronizerCallback

removeSynchronizerCallback

public void removeSynchronizerCallback(SynchronizerCallback<SKey> cb)

Remove a SynchronizerCallback previously added by a call to addSynchronizerCallback()

getActionToUpdate

protected java.util.Iterator<UpdateItem> getActionToUpdate()

Override default operations executed for an update (that is, adding root keys to the list of operations to be executed) Default implementation is to return null (ie. update root folder keys by default)

processActionToUpdate

protected void processActionToUpdate(java.util.Iterator<UpdateItem> items)
                              throws java.lang.Exception

Execute a list of UpdateItem Called if getActionToUpdate() returned a non-null list

Throws:

java.lang.Exception

scan

public void scan(PushAPI papi,
                 java.lang.String mode,
                 java.lang.Object modeConfig)
          throws java.lang.Exception

Description copied from class: Connector

This method is used to push (or synchronize) all documents from the source to CloudView product.

Overrides:

scan in class Connector

Parameters:

papi - PushAPI used to dialog with CloudView

mode - Type of scan to perform. Is always Connector.DEFAULT_SCAN_MODE unless the connector declares additional scan modes with ConnectorCapabilities annotation.

modeConfig - Optional configuration given to the scanMode, according to ScanModeDefinition annotation.

Throws:

java.lang.Exception - Upon error

fetch

public Document fetch(java.lang.String uri)
               throws java.lang.Exception

Description copied from class: Connector

This method is used to retrieve document from the source when CloudView product needs it.

Overrides:

fetch in class Connector

Parameters:

uri - document identifier to retrieve

Returns:

requested document

Throws:

java.lang.Exception

getDocumentSecurityTokens

public MetaContainer getDocumentSecurityTokens(java.lang.String uri)
                                        throws java.io.IOException

Description copied from class: Connector

This method is used to retrieve security tokens of a document. This method is only used for fetch, in the default implementation of "checkSecurityTokens".

Overrides:

getDocumentSecurityTokens in class Connector

Parameters:

uri - Uri of document

Returns:

Security tokens of this document. The default implementation returns a "Everybody" grant ACL, allowing to access all documents.

Throws:

java.io.IOException

handleRootDeletes

protected long handleRootDeletes()
                          throws PushAPIException

Add root keys to operation

Returns:

the number of deletes issued, either document, or root path

Throws:

PushAPIException

initializePipelinesManager

protected void initializePipelinesManager(boolean forUpdate)
                                   throws java.lang.Exception

Add root keys to operation

Throws:

java.lang.Exception

addRootKeyInPipelineManager

protected void addRootKeyInPipelineManager(SKey root,
                                           EnumerableOperationBuilder<SKey> opBuilder,
                                           EnumerableOperationBuilder<SKey> opBuilderUpdate,
                                           EnumerableOperationBuilder<SKey> operationBuilderNotModified)
                                    throws java.lang.Exception

Add root key to operation (called by initializePipelinesManager())

Throws:

java.lang.Exception

terminateThreads

public void terminateThreads()
                      throws java.lang.Exception

Terminate all threads used by the pipeline manager

Throws:

java.lang.Exception

callBeforeDestruction

public void callBeforeDestruction()
                           throws java.lang.Exception

Description copied from class: Connector

Override this method to do some task before the destruction of the connector

Overrides:

callBeforeDestruction in class Connector

Throws:

java.lang.Exception

addFolderOperationToEnumeratedOperationPipeline

public void addFolderOperationToEnumeratedOperationPipeline(FolderOperation<SKey> op)
                                                     throws java.lang.Exception

Add a folder operation to the pipeline manager

Throws:

java.lang.Exception

addDocumentOperationToEnumeratedOperationPipeline

public void addDocumentOperationToEnumeratedOperationPipeline(DocumentOperation<SKey> op)
                                                       throws java.lang.Exception

Add a document operation to the pipeline manager

Throws:

java.lang.Exception

getNbThreadsPerEnumeratedFolderOperationPipeline

public int getNbThreadsPerEnumeratedFolderOperationPipeline()

Number of threads processing folders inside the pipeline manager

getNbThreadsPerEnumeratedDocumentOperationPipeline

public int getNbThreadsPerEnumeratedDocumentOperationPipeline()

Number of threads processing documents inside the pipeline manager

setNbThreadsPerEnumeratedFolderOperationPipeline

public void setNbThreadsPerEnumeratedFolderOperationPipeline(int nb)

Set the number of threads processing folders inside the pipeline manager

setNbThreadsPerEnumeratedDocumentOperationPipeline

public void setNbThreadsPerEnumeratedDocumentOperationPipeline(int nb)

Set the number of threads processing documents inside the pipeline manager

setNbThreadsPerEnumeratedOperationPipeline

public void setNbThreadsPerEnumeratedOperationPipeline(int nb)

Set the number of threads processing documents or folders inside the pipeline manager

registerWorkInProgress

public com.exalead.concurrent.ConcurrentProcessingQuotaWorker registerWorkInProgress(long size)

Register a worker in processing, for quota handling. You must call the release() member of the returned object upon completion, even if exception occurred within the code, such as in: ConcurrentProcessingQuotaWorker work = conn.registerWorkInProgress(size); try { .. code } finally { work.release(); }

registerDocInProgress

public com.exalead.concurrent.ConcurrentProcessingQuotaWorker registerDocInProgress(Document doc)

Register a document in processing, for quota handling. You must call the release() member of the returned object upon completion, even if exception occurred within the code, such as in: ConcurrentProcessingQuotaWorker work = conn.registerDocInProgress(doc); try { .. code } finally { work.release(); }

getCurrentPAPI

public PushAPI getCurrentPAPI()

setCurrentPAPI

public void setCurrentPAPI(PushAPI newPAPI)