com.exalead.mercury.mami.crawl.v21

Class ICrawler

java.lang.Object

com.exalead.mercury.mami.crawl.v21.ICrawler

All Implemented Interfaces:

com.exalead.util.Checkable, java.io.Serializable, java.lang.Cloneable

Direct Known Subclasses:

Crawler, FeedFetcher

public abstract class ICrawler
extends java.lang.Object
implements com.exalead.util.Checkable, java.io.Serializable, Cloneable

No documentation for this element.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ICrawler.MimeTypes
static class	ICrawler.PushAPIFilter
static class	ICrawler.SessionIdBlacklist

Field Summary

Fields

Modifier and Type	Field and Description
boolean	aggressive
boolean	archiveDocuments
java.lang.String	buildGroup
java.lang.String	connectorServer
java.lang.String	crawlerServer
CrawlSchedulerConfig	crawlSchedulerConfig
boolean	crawlSitemaps
CustomCrawlConfig	customCrawlConfig
java.lang.String	dataModel
static boolean	DEFAULT_AGGRESSIVE
static boolean	DEFAULT_ARCHIVE_DOCUMENTS
static boolean	DEFAULT_CRAWL_SITEMAPS
static boolean	DEFAULT_DEFAULT_ACCEPT
static boolean	DEFAULT_DEFAULT_FOLLOW
static boolean	DEFAULT_DEFAULT_FOLLOW_ROOTS
static boolean	DEFAULT_DEFAULT_INDEX
static boolean	DEFAULT_DISABLE_CONDITIONAL_GET
static boolean	DEFAULT_ENABLE_CONSOLIDATION
static boolean	DEFAULT_ENABLE_CONVERT_PROCESSOR
static boolean	DEFAULT_ENABLE_SIMPLE_SITE_COLLAPSING
static boolean	DEFAULT_IGNORE_ROBOTS_TXT
static java.lang.String	DEFAULT_MIME_TYPES_MODE
static boolean	DEFAULT_NEAR_DUPLICATE_DETECTOR
static int	DEFAULT_NTHREADS
static boolean	DEFAULT_PATTERNS_DETECTOR
static int	DEFAULT_SIMPLE_SITE_COLLAPSING_DEPTH
static boolean	DEFAULT_SMART_REFRESH
static int	DEFAULT_SMART_REFRESH_MAX_AGE_S
static int	DEFAULT_SMART_REFRESH_MIN_AGE_S
static boolean	DEFAULT_STORE_TEXT_ONLY
static int	DEFAULT_THROTTLE_TIME_M_S
boolean	defaultAccept
boolean	defaultFollow
boolean	defaultFollowRoots
boolean	defaultIndex
boolean	disableConditionalGet
java.lang.String	documentsType
boolean	enableConsolidation
boolean	enableConvertProcessor
boolean	enableSimpleSiteCollapsing
java.lang.String	fetcher
boolean	ignoreRobotsTxt
protected ICrawler.MimeTypes	mimeTypes
java.lang.String	mimeTypesMode
java.lang.String	name
boolean	nearDuplicateDetector
int	nthreads
boolean	patternsDetector
protected ICrawler.PushAPIFilter	PushAPIFilter
java.util.List<Rules>	rules
protected ICrawler.SessionIdBlacklist	sessionIdBlacklist
int	simpleSiteCollapsingDepth
boolean	smartRefresh
int	smartRefreshMaxAgeS
int	smartRefreshMinAgeS
boolean	storeTextOnly
int	throttleTimeMS
UrlTesterData	urlTesterData

Constructor Summary

Constructors

Constructor and Description
ICrawler()
ICrawler(ICrawler o) Copy constructor

Method Summary

Modifier and Type	Method and Description
void	check(boolean deep, java.lang.String errorContext) Checks this ICrawler.
abstract ICrawler	clone()
static ICrawler	fromString(java.lang.String s) String representation of this ICrawler.
java.lang.String	getBuildGroup() Target build group.
java.lang.String	getConnectorServer() Connector server hosting the indexing part of this crawler.
java.lang.String	getCrawlerServer() Crawler server hosting this crawler.
CrawlSchedulerConfig	getCrawlSchedulerConfig()
CustomCrawlConfig	getCustomCrawlConfig()
java.lang.String	getDataModel() The default data model for documents indexed by this crawler.
java.lang.String	getDocumentsType() The type of documents pushed by this connector. The type of documents must match one of the types declared in your CloudView license file.
java.lang.String	getFetcher() Which fetcher to use.
ICrawler.MimeTypes	getMimeTypes()
java.lang.String	getMimeTypesMode() Mime types white/black list
java.lang.String	getName() The crawler name.
int	getNthreads() The number of crawl threads which must be strictly positive.
ICrawler.PushAPIFilter	getPushAPIFilter()
java.util.List<Rules>	getRules()
ICrawler.SessionIdBlacklist	getSessionIdBlacklist()
int	getSimpleSiteCollapsingDepth() How many path segments to use to generate the site collapsing ID.
int	getSmartRefreshMaxAgeS() Age in seconds at which we force the refresh of old urls.
int	getSmartRefreshMinAgeS() Age in seconds at which we may refresh old urls.
int	getThrottleTimeMS() In the case of non-aggressive crawl, this defines the sleep interval between requests to the same host.
UrlTesterData	getUrlTesterData()
boolean	isAggressive() Whether to enable aggressive crawl, that never sleeps between two requests to the same host.
boolean	isArchiveDocuments() When enabled, deleted documents are not deleted, but kept with their deletion date.
boolean	isCrawlSitemaps() Whether to crawl sitemaps.
boolean	isDefaultAccept() Whether to crawl a url by default when it matches no other accept rule.
boolean	isDefaultFollow() Whether to follow by default when a url matches no follow rule.
boolean	isDefaultFollowRoots() Whether to automatically follow root urls
boolean	isDefaultIndex() Whether to index by default when a url matches no index rule.
boolean	isDisableConditionalGet() Whether to always fetch documents, even if the server tells it has not changed.
boolean	isEnableConsolidation() Define if we use a standard PAPI or a consolidation PAPI.
boolean	isEnableConvertProcessor() Whether to enable remoteconvert-based processor for links extracting in binary documents.
boolean	isEnableSimpleSiteCollapsing() Whether to generate a site ID suitable for document collapsing.
boolean	isIgnoreRobotsTxt() Whether to ignore robots.txt rules.
boolean	isNearDuplicateDetector() Whether to enable the near-duplicate content detector.
boolean	isPatternsDetector() Whether to enable patterns detection in pages.
boolean	isSmartRefresh() Whether to crawl a fraction of refreshed urls.
boolean	isStoreTextOnly() Whether to store original binary documents, or only converted text.
ICrawler	makeCopy()
static ICrawler	readFrom(java.io.InputStream is) Read this ICrawler from an XML fragment.
void	setAggressive(boolean aggressive) Whether to enable aggressive crawl, that never sleeps between two requests to the same host.
void	setArchiveDocuments(boolean archiveDocuments) When enabled, deleted documents are not deleted, but kept with their deletion date.
void	setBuildGroup(java.lang.String buildGroup) Target build group.
void	setConnectorServer(java.lang.String connectorServer) Connector server hosting the indexing part of this crawler.
void	setCrawlerServer(java.lang.String crawlerServer) Crawler server hosting this crawler.
void	setCrawlSchedulerConfig(CrawlSchedulerConfig crawlSchedulerConfig)
void	setCrawlSitemaps(boolean crawlSitemaps) Whether to crawl sitemaps.
void	setCustomCrawlConfig(CustomCrawlConfig customCrawlConfig)
void	setDataModel(java.lang.String dataModel) The default data model for documents indexed by this crawler.
void	setDefaultAccept(boolean defaultAccept) Whether to crawl a url by default when it matches no other accept rule.
void	setDefaultFollow(boolean defaultFollow) Whether to follow by default when a url matches no follow rule.
void	setDefaultFollowRoots(boolean defaultFollowRoots) Whether to automatically follow root urls
void	setDefaultIndex(boolean defaultIndex) Whether to index by default when a url matches no index rule.
void	setDisableConditionalGet(boolean disableConditionalGet) Whether to always fetch documents, even if the server tells it has not changed.
void	setDocumentsType(java.lang.String documentsType) The type of documents pushed by this connector. The type of documents must match one of the types declared in your CloudView license file.
void	setEnableConsolidation(boolean enableConsolidation) Define if we use a standard PAPI or a consolidation PAPI.
void	setEnableConvertProcessor(boolean enableConvertProcessor) Whether to enable remoteconvert-based processor for links extracting in binary documents.
void	setEnableSimpleSiteCollapsing(boolean enableSimpleSiteCollapsing) Whether to generate a site ID suitable for document collapsing.
void	setFetcher(java.lang.String fetcher) Which fetcher to use.
void	setIgnoreRobotsTxt(boolean ignoreRobotsTxt) Whether to ignore robots.txt rules.
void	setMimeTypes(ICrawler.MimeTypes __value)
void	setMimeTypesMode(java.lang.String mimeTypesMode) Mime types white/black list
void	setName(java.lang.String name) The crawler name.
void	setNearDuplicateDetector(boolean nearDuplicateDetector) Whether to enable the near-duplicate content detector.
void	setNthreads(int nthreads) The number of crawl threads which must be strictly positive.
void	setPatternsDetector(boolean patternsDetector) Whether to enable patterns detection in pages.
void	setPushAPIFilter(ICrawler.PushAPIFilter __value)
void	setSessionIdBlacklist(ICrawler.SessionIdBlacklist __value)
void	setSimpleSiteCollapsingDepth(int simpleSiteCollapsingDepth) How many path segments to use to generate the site collapsing ID.
void	setSmartRefresh(boolean smartRefresh) Whether to crawl a fraction of refreshed urls.
void	setSmartRefreshMaxAgeS(int smartRefreshMaxAgeS) Age in seconds at which we force the refresh of old urls.
void	setSmartRefreshMinAgeS(int smartRefreshMinAgeS) Age in seconds at which we may refresh old urls.
void	setStoreTextOnly(boolean storeTextOnly) Whether to store original binary documents, or only converted text.
void	setThrottleTimeMS(int throttleTimeMS) In the case of non-aggressive crawl, this defines the sleep interval between requests to the same host.
void	setUrlTesterData(UrlTesterData urlTesterData)
java.lang.String	toString() String representation of this ICrawler.
ICrawler	withAggressive(boolean aggressive)
ICrawler	withAggressive(java.lang.Boolean aggressive)
ICrawler	withArchiveDocuments(boolean archiveDocuments)
ICrawler	withArchiveDocuments(java.lang.Boolean archiveDocuments)
ICrawler	withBuildGroup(java.lang.String buildGroup)
ICrawler	withConnectorServer(java.lang.String connectorServer)
ICrawler	withCrawlerServer(java.lang.String crawlerServer)
ICrawler	withCrawlSchedulerConfig(CrawlSchedulerConfig crawlSchedulerConfig)
ICrawler	withCrawlSitemaps(boolean crawlSitemaps)
ICrawler	withCrawlSitemaps(java.lang.Boolean crawlSitemaps)
ICrawler	withCustomCrawlConfig(CustomCrawlConfig customCrawlConfig)
ICrawler	withDataModel(java.lang.String dataModel)
ICrawler	withDefaultAccept(boolean defaultAccept)
ICrawler	withDefaultAccept(java.lang.Boolean defaultAccept)
ICrawler	withDefaultFollow(boolean defaultFollow)
ICrawler	withDefaultFollow(java.lang.Boolean defaultFollow)
ICrawler	withDefaultFollowRoots(boolean defaultFollowRoots)
ICrawler	withDefaultFollowRoots(java.lang.Boolean defaultFollowRoots)
ICrawler	withDefaultIndex(boolean defaultIndex)
ICrawler	withDefaultIndex(java.lang.Boolean defaultIndex)
ICrawler	withDisableConditionalGet(boolean disableConditionalGet)
ICrawler	withDisableConditionalGet(java.lang.Boolean disableConditionalGet)
ICrawler	withDocumentsType(java.lang.String documentsType)
ICrawler	withEnableConsolidation(boolean enableConsolidation)
ICrawler	withEnableConsolidation(java.lang.Boolean enableConsolidation)
ICrawler	withEnableConvertProcessor(boolean enableConvertProcessor)
ICrawler	withEnableConvertProcessor(java.lang.Boolean enableConvertProcessor)
ICrawler	withEnableSimpleSiteCollapsing(boolean enableSimpleSiteCollapsing)
ICrawler	withEnableSimpleSiteCollapsing(java.lang.Boolean enableSimpleSiteCollapsing)
ICrawler	withFetcher(java.lang.String fetcher)
ICrawler	withIgnoreRobotsTxt(boolean ignoreRobotsTxt)
ICrawler	withIgnoreRobotsTxt(java.lang.Boolean ignoreRobotsTxt)
ICrawler	withMimeTypes(java.util.Collection<StringConstantValue> __values)
ICrawler	withMimeTypes(ICrawler.MimeTypes __value)
ICrawler	withMimeTypes(StringConstantValue... __values)
ICrawler	withMimeTypesMode(java.lang.String mimeTypesMode)
ICrawler	withName(java.lang.String name)
ICrawler	withNearDuplicateDetector(boolean nearDuplicateDetector)
ICrawler	withNearDuplicateDetector(java.lang.Boolean nearDuplicateDetector)
ICrawler	withNthreads(int nthreads)
ICrawler	withNthreads(java.lang.Integer nthreads)
ICrawler	withPatternsDetector(boolean patternsDetector)
ICrawler	withPatternsDetector(java.lang.Boolean patternsDetector)
ICrawler	withPushAPIFilter(java.util.Collection<KeyValue> __values)
ICrawler	withPushAPIFilter(ICrawler.PushAPIFilter __value)
ICrawler	withPushAPIFilter(KeyValue... __values)
ICrawler	withRules(java.util.Collection<Rules> __values)
ICrawler	withRules(Rules... __values)
ICrawler	withSessionIdBlacklist(java.util.Collection<StringConstantValue> __values)
ICrawler	withSessionIdBlacklist(ICrawler.SessionIdBlacklist __value)
ICrawler	withSessionIdBlacklist(StringConstantValue... __values)
ICrawler	withSimpleSiteCollapsingDepth(int simpleSiteCollapsingDepth)
ICrawler	withSimpleSiteCollapsingDepth(java.lang.Integer simpleSiteCollapsingDepth)
ICrawler	withSmartRefresh(boolean smartRefresh)
ICrawler	withSmartRefresh(java.lang.Boolean smartRefresh)
ICrawler	withSmartRefreshMaxAgeS(int smartRefreshMaxAgeS)
ICrawler	withSmartRefreshMaxAgeS(java.lang.Integer smartRefreshMaxAgeS)
ICrawler	withSmartRefreshMinAgeS(int smartRefreshMinAgeS)
ICrawler	withSmartRefreshMinAgeS(java.lang.Integer smartRefreshMinAgeS)
ICrawler	withStoreTextOnly(boolean storeTextOnly)
ICrawler	withStoreTextOnly(java.lang.Boolean storeTextOnly)
ICrawler	withThrottleTimeMS(int throttleTimeMS)
ICrawler	withThrottleTimeMS(java.lang.Integer throttleTimeMS)
ICrawler	withUrlTesterData(UrlTesterData urlTesterData)
void	writeTo(java.io.OutputStream os)

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

rules

public java.util.List<Rules> rules

crawlSchedulerConfig

public CrawlSchedulerConfig crawlSchedulerConfig

customCrawlConfig

public CustomCrawlConfig customCrawlConfig

name

public java.lang.String name

documentsType

public java.lang.String documentsType

fetcher

public java.lang.String fetcher

crawlerServer

public java.lang.String crawlerServer

connectorServer

public java.lang.String connectorServer

buildGroup

public java.lang.String buildGroup

dataModel

public java.lang.String dataModel

storeTextOnly

public boolean storeTextOnly

DEFAULT_STORE_TEXT_ONLY

public static final boolean DEFAULT_STORE_TEXT_ONLY

See Also:

Constant Field Values

nthreads

public int nthreads

DEFAULT_NTHREADS

public static final int DEFAULT_NTHREADS

See Also:

Constant Field Values

aggressive

public boolean aggressive

DEFAULT_AGGRESSIVE

public static final boolean DEFAULT_AGGRESSIVE

See Also:

Constant Field Values

throttleTimeMS

public int throttleTimeMS

DEFAULT_THROTTLE_TIME_M_S

public static final int DEFAULT_THROTTLE_TIME_M_S

See Also:

Constant Field Values

ignoreRobotsTxt

public boolean ignoreRobotsTxt

DEFAULT_IGNORE_ROBOTS_TXT

public static final boolean DEFAULT_IGNORE_ROBOTS_TXT

See Also:

Constant Field Values

enableConvertProcessor

public boolean enableConvertProcessor

DEFAULT_ENABLE_CONVERT_PROCESSOR

public static final boolean DEFAULT_ENABLE_CONVERT_PROCESSOR

See Also:

Constant Field Values

nearDuplicateDetector

public boolean nearDuplicateDetector

DEFAULT_NEAR_DUPLICATE_DETECTOR

public static final boolean DEFAULT_NEAR_DUPLICATE_DETECTOR

See Also:

Constant Field Values

patternsDetector

public boolean patternsDetector

DEFAULT_PATTERNS_DETECTOR

public static final boolean DEFAULT_PATTERNS_DETECTOR

See Also:

Constant Field Values

crawlSitemaps

public boolean crawlSitemaps

DEFAULT_CRAWL_SITEMAPS

public static final boolean DEFAULT_CRAWL_SITEMAPS

See Also:

Constant Field Values

disableConditionalGet

public boolean disableConditionalGet

DEFAULT_DISABLE_CONDITIONAL_GET

public static final boolean DEFAULT_DISABLE_CONDITIONAL_GET

See Also:

Constant Field Values

defaultAccept

public boolean defaultAccept

DEFAULT_DEFAULT_ACCEPT

public static final boolean DEFAULT_DEFAULT_ACCEPT

See Also:

Constant Field Values

defaultIndex

public boolean defaultIndex

DEFAULT_DEFAULT_INDEX

public static final boolean DEFAULT_DEFAULT_INDEX

See Also:

Constant Field Values

defaultFollow

public boolean defaultFollow

DEFAULT_DEFAULT_FOLLOW

public static final boolean DEFAULT_DEFAULT_FOLLOW

See Also:

Constant Field Values

defaultFollowRoots

public boolean defaultFollowRoots

DEFAULT_DEFAULT_FOLLOW_ROOTS

public static final boolean DEFAULT_DEFAULT_FOLLOW_ROOTS

See Also:

Constant Field Values

enableSimpleSiteCollapsing

public boolean enableSimpleSiteCollapsing

DEFAULT_ENABLE_SIMPLE_SITE_COLLAPSING

public static final boolean DEFAULT_ENABLE_SIMPLE_SITE_COLLAPSING

See Also:

Constant Field Values

simpleSiteCollapsingDepth

public int simpleSiteCollapsingDepth

DEFAULT_SIMPLE_SITE_COLLAPSING_DEPTH

public static final int DEFAULT_SIMPLE_SITE_COLLAPSING_DEPTH

See Also:

Constant Field Values

mimeTypesMode

public java.lang.String mimeTypesMode

DEFAULT_MIME_TYPES_MODE

public static final java.lang.String DEFAULT_MIME_TYPES_MODE

See Also:

Constant Field Values

mimeTypes

protected ICrawler.MimeTypes mimeTypes

sessionIdBlacklist

protected ICrawler.SessionIdBlacklist sessionIdBlacklist

smartRefresh

public boolean smartRefresh

DEFAULT_SMART_REFRESH

public static final boolean DEFAULT_SMART_REFRESH

See Also:

Constant Field Values

smartRefreshMinAgeS

public int smartRefreshMinAgeS

DEFAULT_SMART_REFRESH_MIN_AGE_S

public static final int DEFAULT_SMART_REFRESH_MIN_AGE_S

See Also:

Constant Field Values

smartRefreshMaxAgeS

public int smartRefreshMaxAgeS

DEFAULT_SMART_REFRESH_MAX_AGE_S

public static final int DEFAULT_SMART_REFRESH_MAX_AGE_S

See Also:

Constant Field Values

archiveDocuments

public boolean archiveDocuments

DEFAULT_ARCHIVE_DOCUMENTS

public static final boolean DEFAULT_ARCHIVE_DOCUMENTS

See Also:

Constant Field Values

PushAPIFilter

protected ICrawler.PushAPIFilter PushAPIFilter

urlTesterData

public UrlTesterData urlTesterData

enableConsolidation

public boolean enableConsolidation

DEFAULT_ENABLE_CONSOLIDATION

public static final boolean DEFAULT_ENABLE_CONSOLIDATION

See Also:

Constant Field Values

Constructor Detail

ICrawler

public ICrawler()

ICrawler

public ICrawler(ICrawler o)

Copy constructor

Method Detail

getRules

public java.util.List<Rules> getRules()

withRules

public ICrawler withRules(Rules... __values)

withRules

public ICrawler withRules(java.util.Collection<Rules> __values)

setCrawlSchedulerConfig

public void setCrawlSchedulerConfig(CrawlSchedulerConfig crawlSchedulerConfig)

getCrawlSchedulerConfig

public CrawlSchedulerConfig getCrawlSchedulerConfig()

withCrawlSchedulerConfig

public ICrawler withCrawlSchedulerConfig(CrawlSchedulerConfig crawlSchedulerConfig)

setCustomCrawlConfig

public void setCustomCrawlConfig(CustomCrawlConfig customCrawlConfig)

getCustomCrawlConfig

public CustomCrawlConfig getCustomCrawlConfig()

withCustomCrawlConfig

public ICrawler withCustomCrawlConfig(CustomCrawlConfig customCrawlConfig)

setName

public void setName(java.lang.String name)

The crawler name. It must be unique across all crawlers.

getName

public java.lang.String getName()

The crawler name. It must be unique across all crawlers.

withName

public ICrawler withName(java.lang.String name)

setDocumentsType

public void setDocumentsType(java.lang.String documentsType)

The type of documents pushed by this connector.
The type of documents must match one of the types declared in your CloudView license file.

getDocumentsType

public java.lang.String getDocumentsType()

The type of documents pushed by this connector.
The type of documents must match one of the types declared in your CloudView license file.

withDocumentsType

public ICrawler withDocumentsType(java.lang.String documentsType)

setFetcher

public void setFetcher(java.lang.String fetcher)

Which fetcher to use.

getFetcher

public java.lang.String getFetcher()

Which fetcher to use.

withFetcher

public ICrawler withFetcher(java.lang.String fetcher)

setCrawlerServer

public void setCrawlerServer(java.lang.String crawlerServer)

Crawler server hosting this crawler. See Deployment configuration.

getCrawlerServer

public java.lang.String getCrawlerServer()

Crawler server hosting this crawler. See Deployment configuration.

withCrawlerServer

public ICrawler withCrawlerServer(java.lang.String crawlerServer)

setConnectorServer

public void setConnectorServer(java.lang.String connectorServer)

Connector server hosting the indexing part of this crawler. See Deployment configuration.

getConnectorServer

public java.lang.String getConnectorServer()

Connector server hosting the indexing part of this crawler. See Deployment configuration.

withConnectorServer

public ICrawler withConnectorServer(java.lang.String connectorServer)

setBuildGroup

public void setBuildGroup(java.lang.String buildGroup)

Target build group.

getBuildGroup

public java.lang.String getBuildGroup()

Target build group.

withBuildGroup

public ICrawler withBuildGroup(java.lang.String buildGroup)

setDataModel

public void setDataModel(java.lang.String dataModel)

The default data model for documents indexed by this crawler.

getDataModel

public java.lang.String getDataModel()

The default data model for documents indexed by this crawler.

withDataModel

public ICrawler withDataModel(java.lang.String dataModel)

setStoreTextOnly

public void setStoreTextOnly(boolean storeTextOnly)

Whether to store original binary documents, or only converted text.

isStoreTextOnly

public boolean isStoreTextOnly()

Whether to store original binary documents, or only converted text.

withStoreTextOnly

public ICrawler withStoreTextOnly(boolean storeTextOnly)

withStoreTextOnly

public ICrawler withStoreTextOnly(java.lang.Boolean storeTextOnly)

setNthreads

public void setNthreads(int nthreads)

The number of crawl threads which must be strictly positive.

getNthreads

public int getNthreads()

The number of crawl threads which must be strictly positive.

withNthreads

public ICrawler withNthreads(int nthreads)

withNthreads

public ICrawler withNthreads(java.lang.Integer nthreads)

setAggressive

public void setAggressive(boolean aggressive)

Whether to enable aggressive crawl, that never sleeps between two requests to the same host.

isAggressive

public boolean isAggressive()

Whether to enable aggressive crawl, that never sleeps between two requests to the same host.

withAggressive

public ICrawler withAggressive(boolean aggressive)

withAggressive

public ICrawler withAggressive(java.lang.Boolean aggressive)

setThrottleTimeMS

public void setThrottleTimeMS(int throttleTimeMS)

In the case of non-aggressive crawl, this defines the sleep interval between requests to the same host.

getThrottleTimeMS

public int getThrottleTimeMS()

In the case of non-aggressive crawl, this defines the sleep interval between requests to the same host.

withThrottleTimeMS

public ICrawler withThrottleTimeMS(int throttleTimeMS)

withThrottleTimeMS

public ICrawler withThrottleTimeMS(java.lang.Integer throttleTimeMS)

setIgnoreRobotsTxt

public void setIgnoreRobotsTxt(boolean ignoreRobotsTxt)

Whether to ignore robots.txt rules. Not recommended.

isIgnoreRobotsTxt

public boolean isIgnoreRobotsTxt()

Whether to ignore robots.txt rules. Not recommended.

withIgnoreRobotsTxt

public ICrawler withIgnoreRobotsTxt(boolean ignoreRobotsTxt)

withIgnoreRobotsTxt

public ICrawler withIgnoreRobotsTxt(java.lang.Boolean ignoreRobotsTxt)

setEnableConvertProcessor

public void setEnableConvertProcessor(boolean enableConvertProcessor)

Whether to enable remoteconvert-based processor for links extracting in binary documents.

isEnableConvertProcessor

public boolean isEnableConvertProcessor()

Whether to enable remoteconvert-based processor for links extracting in binary documents.

withEnableConvertProcessor

public ICrawler withEnableConvertProcessor(boolean enableConvertProcessor)

withEnableConvertProcessor

public ICrawler withEnableConvertProcessor(java.lang.Boolean enableConvertProcessor)

setNearDuplicateDetector

public void setNearDuplicateDetector(boolean nearDuplicateDetector)

Whether to enable the near-duplicate content detector.

isNearDuplicateDetector

public boolean isNearDuplicateDetector()

Whether to enable the near-duplicate content detector.

withNearDuplicateDetector

public ICrawler withNearDuplicateDetector(boolean nearDuplicateDetector)

withNearDuplicateDetector

public ICrawler withNearDuplicateDetector(java.lang.Boolean nearDuplicateDetector)

setPatternsDetector

public void setPatternsDetector(boolean patternsDetector)

Whether to enable patterns detection in pages.

isPatternsDetector

public boolean isPatternsDetector()

Whether to enable patterns detection in pages.

withPatternsDetector

public ICrawler withPatternsDetector(boolean patternsDetector)

withPatternsDetector

public ICrawler withPatternsDetector(java.lang.Boolean patternsDetector)

setCrawlSitemaps

public void setCrawlSitemaps(boolean crawlSitemaps)

Whether to crawl sitemaps.

isCrawlSitemaps

public boolean isCrawlSitemaps()

Whether to crawl sitemaps.

withCrawlSitemaps

public ICrawler withCrawlSitemaps(boolean crawlSitemaps)

withCrawlSitemaps

public ICrawler withCrawlSitemaps(java.lang.Boolean crawlSitemaps)

setDisableConditionalGet

public void setDisableConditionalGet(boolean disableConditionalGet)

Whether to always fetch documents, even if the server tells it has not changed.

isDisableConditionalGet

public boolean isDisableConditionalGet()

Whether to always fetch documents, even if the server tells it has not changed.

withDisableConditionalGet

public ICrawler withDisableConditionalGet(boolean disableConditionalGet)

withDisableConditionalGet

public ICrawler withDisableConditionalGet(java.lang.Boolean disableConditionalGet)

setDefaultAccept

public void setDefaultAccept(boolean defaultAccept)

Whether to crawl a url by default when it matches no other accept rule.

isDefaultAccept

public boolean isDefaultAccept()

Whether to crawl a url by default when it matches no other accept rule.

withDefaultAccept

public ICrawler withDefaultAccept(boolean defaultAccept)

withDefaultAccept

public ICrawler withDefaultAccept(java.lang.Boolean defaultAccept)

setDefaultIndex

public void setDefaultIndex(boolean defaultIndex)

Whether to index by default when a url matches no index rule.

isDefaultIndex

public boolean isDefaultIndex()

Whether to index by default when a url matches no index rule.

withDefaultIndex

public ICrawler withDefaultIndex(boolean defaultIndex)

withDefaultIndex

public ICrawler withDefaultIndex(java.lang.Boolean defaultIndex)

setDefaultFollow

public void setDefaultFollow(boolean defaultFollow)

Whether to follow by default when a url matches no follow rule.

isDefaultFollow

public boolean isDefaultFollow()

Whether to follow by default when a url matches no follow rule.

withDefaultFollow

public ICrawler withDefaultFollow(boolean defaultFollow)

withDefaultFollow

public ICrawler withDefaultFollow(java.lang.Boolean defaultFollow)

setDefaultFollowRoots

public void setDefaultFollowRoots(boolean defaultFollowRoots)

Whether to automatically follow root urls

isDefaultFollowRoots

public boolean isDefaultFollowRoots()

Whether to automatically follow root urls

withDefaultFollowRoots

public ICrawler withDefaultFollowRoots(boolean defaultFollowRoots)

withDefaultFollowRoots

public ICrawler withDefaultFollowRoots(java.lang.Boolean defaultFollowRoots)

setEnableSimpleSiteCollapsing

public void setEnableSimpleSiteCollapsing(boolean enableSimpleSiteCollapsing)

Whether to generate a site ID suitable for document collapsing.

isEnableSimpleSiteCollapsing

public boolean isEnableSimpleSiteCollapsing()

Whether to generate a site ID suitable for document collapsing.

withEnableSimpleSiteCollapsing

public ICrawler withEnableSimpleSiteCollapsing(boolean enableSimpleSiteCollapsing)

withEnableSimpleSiteCollapsing

public ICrawler withEnableSimpleSiteCollapsing(java.lang.Boolean enableSimpleSiteCollapsing)

setSimpleSiteCollapsingDepth

public void setSimpleSiteCollapsingDepth(int simpleSiteCollapsingDepth)

How many path segments to use to generate the site collapsing ID.

getSimpleSiteCollapsingDepth

public int getSimpleSiteCollapsingDepth()

How many path segments to use to generate the site collapsing ID.

withSimpleSiteCollapsingDepth

public ICrawler withSimpleSiteCollapsingDepth(int simpleSiteCollapsingDepth)

withSimpleSiteCollapsingDepth

public ICrawler withSimpleSiteCollapsingDepth(java.lang.Integer simpleSiteCollapsingDepth)

setMimeTypesMode

public void setMimeTypesMode(java.lang.String mimeTypesMode)

Mime types white/black list

getMimeTypesMode

public java.lang.String getMimeTypesMode()

Mime types white/black list

withMimeTypesMode

public ICrawler withMimeTypesMode(java.lang.String mimeTypesMode)

getMimeTypes

public ICrawler.MimeTypes getMimeTypes()

setMimeTypes

public void setMimeTypes(ICrawler.MimeTypes __value)

withMimeTypes

public ICrawler withMimeTypes(StringConstantValue... __values)

withMimeTypes

public ICrawler withMimeTypes(java.util.Collection<StringConstantValue> __values)

withMimeTypes

public ICrawler withMimeTypes(ICrawler.MimeTypes __value)

getSessionIdBlacklist

public ICrawler.SessionIdBlacklist getSessionIdBlacklist()

setSessionIdBlacklist

public void setSessionIdBlacklist(ICrawler.SessionIdBlacklist __value)

withSessionIdBlacklist

public ICrawler withSessionIdBlacklist(StringConstantValue... __values)

withSessionIdBlacklist

public ICrawler withSessionIdBlacklist(java.util.Collection<StringConstantValue> __values)

withSessionIdBlacklist

public ICrawler withSessionIdBlacklist(ICrawler.SessionIdBlacklist __value)

setSmartRefresh

public void setSmartRefresh(boolean smartRefresh)

Whether to crawl a fraction of refreshed urls.

isSmartRefresh

public boolean isSmartRefresh()

Whether to crawl a fraction of refreshed urls.

withSmartRefresh

public ICrawler withSmartRefresh(boolean smartRefresh)

withSmartRefresh

public ICrawler withSmartRefresh(java.lang.Boolean smartRefresh)

setSmartRefreshMinAgeS

public void setSmartRefreshMinAgeS(int smartRefreshMinAgeS)

Age in seconds at which we may refresh old urls.

getSmartRefreshMinAgeS

public int getSmartRefreshMinAgeS()

Age in seconds at which we may refresh old urls.

withSmartRefreshMinAgeS

public ICrawler withSmartRefreshMinAgeS(int smartRefreshMinAgeS)

withSmartRefreshMinAgeS

public ICrawler withSmartRefreshMinAgeS(java.lang.Integer smartRefreshMinAgeS)

setSmartRefreshMaxAgeS

public void setSmartRefreshMaxAgeS(int smartRefreshMaxAgeS)

Age in seconds at which we force the refresh of old urls.

getSmartRefreshMaxAgeS

public int getSmartRefreshMaxAgeS()

Age in seconds at which we force the refresh of old urls.

withSmartRefreshMaxAgeS

public ICrawler withSmartRefreshMaxAgeS(int smartRefreshMaxAgeS)

withSmartRefreshMaxAgeS

public ICrawler withSmartRefreshMaxAgeS(java.lang.Integer smartRefreshMaxAgeS)

setArchiveDocuments

public void setArchiveDocuments(boolean archiveDocuments)

When enabled, deleted documents are not deleted, but kept with their deletion date.

isArchiveDocuments

public boolean isArchiveDocuments()

When enabled, deleted documents are not deleted, but kept with their deletion date.

withArchiveDocuments

public ICrawler withArchiveDocuments(boolean archiveDocuments)

withArchiveDocuments

public ICrawler withArchiveDocuments(java.lang.Boolean archiveDocuments)

getPushAPIFilter

public ICrawler.PushAPIFilter getPushAPIFilter()

setPushAPIFilter

public void setPushAPIFilter(ICrawler.PushAPIFilter __value)

withPushAPIFilter

public ICrawler withPushAPIFilter(KeyValue... __values)

withPushAPIFilter

public ICrawler withPushAPIFilter(java.util.Collection<KeyValue> __values)

withPushAPIFilter

public ICrawler withPushAPIFilter(ICrawler.PushAPIFilter __value)

setUrlTesterData

public void setUrlTesterData(UrlTesterData urlTesterData)

getUrlTesterData

public UrlTesterData getUrlTesterData()

withUrlTesterData

public ICrawler withUrlTesterData(UrlTesterData urlTesterData)

setEnableConsolidation

public void setEnableConsolidation(boolean enableConsolidation)

Define if we use a standard PAPI or a consolidation PAPI.

isEnableConsolidation

public boolean isEnableConsolidation()

Define if we use a standard PAPI or a consolidation PAPI.

withEnableConsolidation

public ICrawler withEnableConsolidation(boolean enableConsolidation)

withEnableConsolidation

public ICrawler withEnableConsolidation(java.lang.Boolean enableConsolidation)

clone

public abstract ICrawler clone()

Overrides:

clone in class java.lang.Object

makeCopy

public ICrawler makeCopy()

readFrom

public static ICrawler readFrom(java.io.InputStream is)
                         throws javax.xml.bind.JAXBException

Read this ICrawler from an XML fragment.

Throws:

javax.xml.bind.JAXBException

writeTo

public void writeTo(java.io.OutputStream os)
             throws javax.xml.bind.JAXBException,
                    java.io.IOException

Throws:

javax.xml.bind.JAXBException

java.io.IOException

fromString

public static ICrawler fromString(java.lang.String s)
                           throws javax.xml.bind.JAXBException,
                                  java.io.UnsupportedEncodingException

String representation of this ICrawler.

Throws:

javax.xml.bind.JAXBException

java.io.UnsupportedEncodingException

toString

public java.lang.String toString()

String representation of this ICrawler.

Overrides:

toString in class java.lang.Object

check

public void check(boolean deep,
                  java.lang.String errorContext)
           throws com.exalead.util.TypedException

Checks this ICrawler.

Specified by:

check in interface com.exalead.util.Checkable

Throws:

com.exalead.util.TypedException