com.exalead.mercury.mami.crawl.v21

Class Crawler

java.lang.Object

com.exalead.mercury.mami.crawl.v21.ICrawler

com.exalead.mercury.mami.crawl.v21.Crawler

All Implemented Interfaces:

com.exalead.util.Checkable, java.io.Serializable, java.lang.Cloneable

public class Crawler
extends ICrawler
implements com.exalead.util.Checkable, java.io.Serializable, Cloneable

A crawler configuration.
A crawler may contain a CrawlSchedulerConfig to overwrite the default fifo priorities.
A crawler may contain a CustomCrawlConfig to enable custom processors.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Crawler.Roots
static class	Crawler.Rootsets

Nested classes/interfaces inherited from class com.exalead.mercury.mami.crawl.v21.ICrawler

ICrawler.MimeTypes, ICrawler.PushAPIFilter, ICrawler.SessionIdBlacklist

Field Summary

Fields

Modifier and Type	Field and Description
protected Crawler.Roots	roots
protected Crawler.Rootsets	rootsets

Fields inherited from class com.exalead.mercury.mami.crawl.v21.ICrawler

aggressive, archiveDocuments, buildGroup, connectorServer, crawlerServer, crawlSchedulerConfig, crawlSitemaps, customCrawlConfig, dataModel, DEFAULT_AGGRESSIVE, DEFAULT_ARCHIVE_DOCUMENTS, DEFAULT_CRAWL_SITEMAPS, DEFAULT_DEFAULT_ACCEPT, DEFAULT_DEFAULT_FOLLOW, DEFAULT_DEFAULT_FOLLOW_ROOTS, DEFAULT_DEFAULT_INDEX, DEFAULT_DISABLE_CONDITIONAL_GET, DEFAULT_ENABLE_CONSOLIDATION, DEFAULT_ENABLE_CONVERT_PROCESSOR, DEFAULT_ENABLE_SIMPLE_SITE_COLLAPSING, DEFAULT_IGNORE_ROBOTS_TXT, DEFAULT_MIME_TYPES_MODE, DEFAULT_NEAR_DUPLICATE_DETECTOR, DEFAULT_NTHREADS, DEFAULT_PATTERNS_DETECTOR, DEFAULT_SIMPLE_SITE_COLLAPSING_DEPTH, DEFAULT_SMART_REFRESH, DEFAULT_SMART_REFRESH_MAX_AGE_S, DEFAULT_SMART_REFRESH_MIN_AGE_S, DEFAULT_STORE_TEXT_ONLY, DEFAULT_THROTTLE_TIME_M_S, defaultAccept, defaultFollow, defaultFollowRoots, defaultIndex, disableConditionalGet, documentsType, enableConsolidation, enableConvertProcessor, enableSimpleSiteCollapsing, fetcher, ignoreRobotsTxt, mimeTypes, mimeTypesMode, name, nearDuplicateDetector, nthreads, patternsDetector, PushAPIFilter, rules, sessionIdBlacklist, simpleSiteCollapsingDepth, smartRefresh, smartRefreshMaxAgeS, smartRefreshMinAgeS, storeTextOnly, throttleTimeMS, urlTesterData

Constructor Summary

Constructors

Constructor and Description
Crawler()
Crawler(Crawler o) Copy constructor

Method Summary

Modifier and Type	Method and Description
void	check(boolean deep, java.lang.String errorContext) Checks this Crawler.
Crawler	clone() Creates and returns a deep copy of this Crawler.
static Crawler	fromString(java.lang.String s) String representation of this Crawler.
Crawler.Roots	getRoots()
Crawler.Rootsets	getRootsets()
Crawler	makeCopy() Creates and returns a deep copy of this Crawler.
static Crawler	readFrom(java.io.InputStream is) Read this Crawler from an XML fragment.
void	setRoots(Crawler.Roots __value)
void	setRootsets(Crawler.Rootsets __value)
java.lang.String	toString() String representation of this Crawler.
Crawler	withAggressive(boolean aggressive)
Crawler	withAggressive(java.lang.Boolean aggressive)
Crawler	withArchiveDocuments(boolean archiveDocuments)
Crawler	withArchiveDocuments(java.lang.Boolean archiveDocuments)
Crawler	withBuildGroup(java.lang.String buildGroup)
Crawler	withConnectorServer(java.lang.String connectorServer)
Crawler	withCrawlerServer(java.lang.String crawlerServer)
Crawler	withCrawlSchedulerConfig(CrawlSchedulerConfig crawlSchedulerConfig)
Crawler	withCrawlSitemaps(boolean crawlSitemaps)
Crawler	withCrawlSitemaps(java.lang.Boolean crawlSitemaps)
Crawler	withCustomCrawlConfig(CustomCrawlConfig customCrawlConfig)
Crawler	withDataModel(java.lang.String dataModel)
Crawler	withDefaultAccept(boolean defaultAccept)
Crawler	withDefaultAccept(java.lang.Boolean defaultAccept)
Crawler	withDefaultFollow(boolean defaultFollow)
Crawler	withDefaultFollow(java.lang.Boolean defaultFollow)
Crawler	withDefaultFollowRoots(boolean defaultFollowRoots)
Crawler	withDefaultFollowRoots(java.lang.Boolean defaultFollowRoots)
Crawler	withDefaultIndex(boolean defaultIndex)
Crawler	withDefaultIndex(java.lang.Boolean defaultIndex)
Crawler	withDisableConditionalGet(boolean disableConditionalGet)
Crawler	withDisableConditionalGet(java.lang.Boolean disableConditionalGet)
Crawler	withDocumentsType(java.lang.String documentsType)
Crawler	withEnableConsolidation(boolean enableConsolidation)
Crawler	withEnableConsolidation(java.lang.Boolean enableConsolidation)
Crawler	withEnableConvertProcessor(boolean enableConvertProcessor)
Crawler	withEnableConvertProcessor(java.lang.Boolean enableConvertProcessor)
Crawler	withEnableSimpleSiteCollapsing(boolean enableSimpleSiteCollapsing)
Crawler	withEnableSimpleSiteCollapsing(java.lang.Boolean enableSimpleSiteCollapsing)
Crawler	withFetcher(java.lang.String fetcher)
Crawler	withIgnoreRobotsTxt(boolean ignoreRobotsTxt)
Crawler	withIgnoreRobotsTxt(java.lang.Boolean ignoreRobotsTxt)
Crawler	withMimeTypes(java.util.Collection<StringConstantValue> __values)
Crawler	withMimeTypes(StringConstantValue... __values)
Crawler	withMimeTypesMode(java.lang.String mimeTypesMode)
Crawler	withName(java.lang.String name)
Crawler	withNearDuplicateDetector(boolean nearDuplicateDetector)
Crawler	withNearDuplicateDetector(java.lang.Boolean nearDuplicateDetector)
Crawler	withNthreads(int nthreads)
Crawler	withNthreads(java.lang.Integer nthreads)
Crawler	withPatternsDetector(boolean patternsDetector)
Crawler	withPatternsDetector(java.lang.Boolean patternsDetector)
Crawler	withPushAPIFilter(java.util.Collection<KeyValue> __values)
Crawler	withPushAPIFilter(KeyValue... __values)
Crawler	withRoots(java.util.Collection<Root> __values)
Crawler	withRoots(Crawler.Roots __value)
Crawler	withRoots(Root... __values)
Crawler	withRootsets(java.util.Collection<RootSet> __values)
Crawler	withRootsets(Crawler.Rootsets __value)
Crawler	withRootsets(RootSet... __values)
Crawler	withRules(java.util.Collection<Rules> __values)
Crawler	withRules(Rules... __values)
Crawler	withSessionIdBlacklist(java.util.Collection<StringConstantValue> __values)
Crawler	withSessionIdBlacklist(StringConstantValue... __values) SessionId blacklist.
Crawler	withSimpleSiteCollapsingDepth(int simpleSiteCollapsingDepth)
Crawler	withSimpleSiteCollapsingDepth(java.lang.Integer simpleSiteCollapsingDepth)
Crawler	withSmartRefresh(boolean smartRefresh)
Crawler	withSmartRefresh(java.lang.Boolean smartRefresh)
Crawler	withSmartRefreshMaxAgeS(int smartRefreshMaxAgeS)
Crawler	withSmartRefreshMaxAgeS(java.lang.Integer smartRefreshMaxAgeS)
Crawler	withSmartRefreshMinAgeS(int smartRefreshMinAgeS)
Crawler	withSmartRefreshMinAgeS(java.lang.Integer smartRefreshMinAgeS)
Crawler	withStoreTextOnly(boolean storeTextOnly)
Crawler	withStoreTextOnly(java.lang.Boolean storeTextOnly)
Crawler	withThrottleTimeMS(int throttleTimeMS)
Crawler	withThrottleTimeMS(java.lang.Integer throttleTimeMS)
Crawler	withUrlTesterData(UrlTesterData urlTesterData)
void	writeTo(java.io.OutputStream os) Write this Crawler as an XML fragment

Methods inherited from class com.exalead.mercury.mami.crawl.v21.ICrawler

getBuildGroup, getConnectorServer, getCrawlerServer, getCrawlSchedulerConfig, getCustomCrawlConfig, getDataModel, getDocumentsType, getFetcher, getMimeTypes, getMimeTypesMode, getName, getNthreads, getPushAPIFilter, getRules, getSessionIdBlacklist, getSimpleSiteCollapsingDepth, getSmartRefreshMaxAgeS, getSmartRefreshMinAgeS, getThrottleTimeMS, getUrlTesterData, isAggressive, isArchiveDocuments, isCrawlSitemaps, isDefaultAccept, isDefaultFollow, isDefaultFollowRoots, isDefaultIndex, isDisableConditionalGet, isEnableConsolidation, isEnableConvertProcessor, isEnableSimpleSiteCollapsing, isIgnoreRobotsTxt, isNearDuplicateDetector, isPatternsDetector, isSmartRefresh, isStoreTextOnly, setAggressive, setArchiveDocuments, setBuildGroup, setConnectorServer, setCrawlerServer, setCrawlSchedulerConfig, setCrawlSitemaps, setCustomCrawlConfig, setDataModel, setDefaultAccept, setDefaultFollow, setDefaultFollowRoots, setDefaultIndex, setDisableConditionalGet, setDocumentsType, setEnableConsolidation, setEnableConvertProcessor, setEnableSimpleSiteCollapsing, setFetcher, setIgnoreRobotsTxt, setMimeTypes, setMimeTypesMode, setName, setNearDuplicateDetector, setNthreads, setPatternsDetector, setPushAPIFilter, setSessionIdBlacklist, setSimpleSiteCollapsingDepth, setSmartRefresh, setSmartRefreshMaxAgeS, setSmartRefreshMinAgeS, setStoreTextOnly, setThrottleTimeMS, setUrlTesterData, withMimeTypes, withPushAPIFilter, withSessionIdBlacklist

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

roots

protected Crawler.Roots roots

rootsets

protected Crawler.Rootsets rootsets

Constructor Detail

Crawler

public Crawler()

Crawler

public Crawler(Crawler o)

Copy constructor

Method Detail

withRules

public Crawler withRules(Rules... __values)

Overrides:

withRules in class ICrawler

withRules

public Crawler withRules(java.util.Collection<Rules> __values)

Overrides:

withRules in class ICrawler

withCrawlSchedulerConfig

public Crawler withCrawlSchedulerConfig(CrawlSchedulerConfig crawlSchedulerConfig)

Overrides:

withCrawlSchedulerConfig in class ICrawler

withCustomCrawlConfig

public Crawler withCustomCrawlConfig(CustomCrawlConfig customCrawlConfig)

Overrides:

withCustomCrawlConfig in class ICrawler

withName

public Crawler withName(java.lang.String name)

Overrides:

withName in class ICrawler

withDocumentsType

public Crawler withDocumentsType(java.lang.String documentsType)

Overrides:

withDocumentsType in class ICrawler

withFetcher

public Crawler withFetcher(java.lang.String fetcher)

Overrides:

withFetcher in class ICrawler

withCrawlerServer

public Crawler withCrawlerServer(java.lang.String crawlerServer)

Overrides:

withCrawlerServer in class ICrawler

withConnectorServer

public Crawler withConnectorServer(java.lang.String connectorServer)

Overrides:

withConnectorServer in class ICrawler

withBuildGroup

public Crawler withBuildGroup(java.lang.String buildGroup)

Overrides:

withBuildGroup in class ICrawler

withDataModel

public Crawler withDataModel(java.lang.String dataModel)

Overrides:

withDataModel in class ICrawler

withStoreTextOnly

public Crawler withStoreTextOnly(boolean storeTextOnly)

Overrides:

withStoreTextOnly in class ICrawler

withStoreTextOnly

public Crawler withStoreTextOnly(java.lang.Boolean storeTextOnly)

Overrides:

withStoreTextOnly in class ICrawler

withNthreads

public Crawler withNthreads(int nthreads)

Overrides:

withNthreads in class ICrawler

withNthreads

public Crawler withNthreads(java.lang.Integer nthreads)

Overrides:

withNthreads in class ICrawler

withAggressive

public Crawler withAggressive(boolean aggressive)

Overrides:

withAggressive in class ICrawler

withAggressive

public Crawler withAggressive(java.lang.Boolean aggressive)

Overrides:

withAggressive in class ICrawler

withThrottleTimeMS

public Crawler withThrottleTimeMS(int throttleTimeMS)

Overrides:

withThrottleTimeMS in class ICrawler

withThrottleTimeMS

public Crawler withThrottleTimeMS(java.lang.Integer throttleTimeMS)

Overrides:

withThrottleTimeMS in class ICrawler

withIgnoreRobotsTxt

public Crawler withIgnoreRobotsTxt(boolean ignoreRobotsTxt)

Overrides:

withIgnoreRobotsTxt in class ICrawler

withIgnoreRobotsTxt

public Crawler withIgnoreRobotsTxt(java.lang.Boolean ignoreRobotsTxt)

Overrides:

withIgnoreRobotsTxt in class ICrawler

withEnableConvertProcessor

public Crawler withEnableConvertProcessor(boolean enableConvertProcessor)

Overrides:

withEnableConvertProcessor in class ICrawler

withEnableConvertProcessor

public Crawler withEnableConvertProcessor(java.lang.Boolean enableConvertProcessor)

Overrides:

withEnableConvertProcessor in class ICrawler

withNearDuplicateDetector

public Crawler withNearDuplicateDetector(boolean nearDuplicateDetector)

Overrides:

withNearDuplicateDetector in class ICrawler

withNearDuplicateDetector

public Crawler withNearDuplicateDetector(java.lang.Boolean nearDuplicateDetector)

Overrides:

withNearDuplicateDetector in class ICrawler

withPatternsDetector

public Crawler withPatternsDetector(boolean patternsDetector)

Overrides:

withPatternsDetector in class ICrawler

withPatternsDetector

public Crawler withPatternsDetector(java.lang.Boolean patternsDetector)

Overrides:

withPatternsDetector in class ICrawler

withCrawlSitemaps

public Crawler withCrawlSitemaps(boolean crawlSitemaps)

Overrides:

withCrawlSitemaps in class ICrawler

withCrawlSitemaps

public Crawler withCrawlSitemaps(java.lang.Boolean crawlSitemaps)

Overrides:

withCrawlSitemaps in class ICrawler

withDisableConditionalGet

public Crawler withDisableConditionalGet(boolean disableConditionalGet)

Overrides:

withDisableConditionalGet in class ICrawler

withDisableConditionalGet

public Crawler withDisableConditionalGet(java.lang.Boolean disableConditionalGet)

Overrides:

withDisableConditionalGet in class ICrawler

withDefaultAccept

public Crawler withDefaultAccept(boolean defaultAccept)

Overrides:

withDefaultAccept in class ICrawler

withDefaultAccept

public Crawler withDefaultAccept(java.lang.Boolean defaultAccept)

Overrides:

withDefaultAccept in class ICrawler

withDefaultIndex

public Crawler withDefaultIndex(boolean defaultIndex)

Overrides:

withDefaultIndex in class ICrawler

withDefaultIndex

public Crawler withDefaultIndex(java.lang.Boolean defaultIndex)

Overrides:

withDefaultIndex in class ICrawler

withDefaultFollow

public Crawler withDefaultFollow(boolean defaultFollow)

Overrides:

withDefaultFollow in class ICrawler

withDefaultFollow

public Crawler withDefaultFollow(java.lang.Boolean defaultFollow)

Overrides:

withDefaultFollow in class ICrawler

withDefaultFollowRoots

public Crawler withDefaultFollowRoots(boolean defaultFollowRoots)

Overrides:

withDefaultFollowRoots in class ICrawler

withDefaultFollowRoots

public Crawler withDefaultFollowRoots(java.lang.Boolean defaultFollowRoots)

Overrides:

withDefaultFollowRoots in class ICrawler

withEnableSimpleSiteCollapsing

public Crawler withEnableSimpleSiteCollapsing(boolean enableSimpleSiteCollapsing)

Overrides:

withEnableSimpleSiteCollapsing in class ICrawler

withEnableSimpleSiteCollapsing

public Crawler withEnableSimpleSiteCollapsing(java.lang.Boolean enableSimpleSiteCollapsing)

Overrides:

withEnableSimpleSiteCollapsing in class ICrawler

withSimpleSiteCollapsingDepth

public Crawler withSimpleSiteCollapsingDepth(int simpleSiteCollapsingDepth)

Overrides:

withSimpleSiteCollapsingDepth in class ICrawler

withSimpleSiteCollapsingDepth

public Crawler withSimpleSiteCollapsingDepth(java.lang.Integer simpleSiteCollapsingDepth)

Overrides:

withSimpleSiteCollapsingDepth in class ICrawler

withMimeTypesMode

public Crawler withMimeTypesMode(java.lang.String mimeTypesMode)

Overrides:

withMimeTypesMode in class ICrawler

withMimeTypes

public Crawler withMimeTypes(StringConstantValue... __values)

Overrides:

withMimeTypes in class ICrawler

withMimeTypes

public Crawler withMimeTypes(java.util.Collection<StringConstantValue> __values)

Overrides:

withMimeTypes in class ICrawler

withSessionIdBlacklist

public Crawler withSessionIdBlacklist(StringConstantValue... __values)

SessionId blacklist. These parameters are removed from URLs with a path or query part containing them.

Overrides:

withSessionIdBlacklist in class ICrawler

withSessionIdBlacklist

public Crawler withSessionIdBlacklist(java.util.Collection<StringConstantValue> __values)

Overrides:

withSessionIdBlacklist in class ICrawler

withSmartRefresh

public Crawler withSmartRefresh(boolean smartRefresh)

Overrides:

withSmartRefresh in class ICrawler

withSmartRefresh

public Crawler withSmartRefresh(java.lang.Boolean smartRefresh)

Overrides:

withSmartRefresh in class ICrawler

withSmartRefreshMinAgeS

public Crawler withSmartRefreshMinAgeS(int smartRefreshMinAgeS)

Overrides:

withSmartRefreshMinAgeS in class ICrawler

withSmartRefreshMinAgeS

public Crawler withSmartRefreshMinAgeS(java.lang.Integer smartRefreshMinAgeS)

Overrides:

withSmartRefreshMinAgeS in class ICrawler

withSmartRefreshMaxAgeS

public Crawler withSmartRefreshMaxAgeS(int smartRefreshMaxAgeS)

Overrides:

withSmartRefreshMaxAgeS in class ICrawler

withSmartRefreshMaxAgeS

public Crawler withSmartRefreshMaxAgeS(java.lang.Integer smartRefreshMaxAgeS)

Overrides:

withSmartRefreshMaxAgeS in class ICrawler

withArchiveDocuments

public Crawler withArchiveDocuments(boolean archiveDocuments)

Overrides:

withArchiveDocuments in class ICrawler

withArchiveDocuments

public Crawler withArchiveDocuments(java.lang.Boolean archiveDocuments)

Overrides:

withArchiveDocuments in class ICrawler

withPushAPIFilter

public Crawler withPushAPIFilter(KeyValue... __values)

Overrides:

withPushAPIFilter in class ICrawler

withPushAPIFilter

public Crawler withPushAPIFilter(java.util.Collection<KeyValue> __values)

Overrides:

withPushAPIFilter in class ICrawler

withUrlTesterData

public Crawler withUrlTesterData(UrlTesterData urlTesterData)

Overrides:

withUrlTesterData in class ICrawler

withEnableConsolidation

public Crawler withEnableConsolidation(boolean enableConsolidation)

Overrides:

withEnableConsolidation in class ICrawler

withEnableConsolidation

public Crawler withEnableConsolidation(java.lang.Boolean enableConsolidation)

Overrides:

withEnableConsolidation in class ICrawler

getRoots

public Crawler.Roots getRoots()

setRoots

public void setRoots(Crawler.Roots __value)

withRoots

public Crawler withRoots(Root... __values)

withRoots

public Crawler withRoots(java.util.Collection<Root> __values)

withRoots

public Crawler withRoots(Crawler.Roots __value)

getRootsets

public Crawler.Rootsets getRootsets()

setRootsets

public void setRootsets(Crawler.Rootsets __value)

withRootsets

public Crawler withRootsets(RootSet... __values)

withRootsets

public Crawler withRootsets(java.util.Collection<RootSet> __values)

withRootsets

public Crawler withRootsets(Crawler.Rootsets __value)

clone

public Crawler clone()

Creates and returns a deep copy of this Crawler.

Specified by:

clone in class ICrawler

makeCopy

public Crawler makeCopy()

Creates and returns a deep copy of this Crawler.

Overrides:

makeCopy in class ICrawler

readFrom

public static Crawler readFrom(java.io.InputStream is)
                        throws javax.xml.bind.JAXBException

Read this Crawler from an XML fragment.

Throws:

javax.xml.bind.JAXBException

writeTo

public void writeTo(java.io.OutputStream os)
             throws javax.xml.bind.JAXBException,
                    java.io.IOException

Write this Crawler as an XML fragment

Overrides:

writeTo in class ICrawler

Throws:

javax.xml.bind.JAXBException

java.io.IOException

fromString

public static Crawler fromString(java.lang.String s)
                          throws javax.xml.bind.JAXBException,
                                 java.io.UnsupportedEncodingException

String representation of this Crawler.

Throws:

javax.xml.bind.JAXBException

java.io.UnsupportedEncodingException

toString

public java.lang.String toString()

String representation of this Crawler.

Overrides:

toString in class ICrawler

check

public void check(boolean deep,
                  java.lang.String errorContext)
           throws com.exalead.util.TypedException

Checks this Crawler.

Specified by:

check in interface com.exalead.util.Checkable

Overrides:

check in class ICrawler

Throws:

com.exalead.util.TypedException