com.exalead.mercury.mami.crawl.v20

Class Crawler

java.lang.Object

com.exalead.mercury.mami.crawl.v20.ICrawler

com.exalead.mercury.mami.crawl.v20.Crawler

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable

public class Crawler
extends ICrawler
implements java.io.Serializable, java.lang.Cloneable

A crawler configuration. A crawler may contain a CrawlSchedulerConfig to overwrite the default fifo priorities. A crawler may contain a CustomCrawlConfig to enable custom processors.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Crawler.Roots Java class for anonymous complex type.
static class	Crawler.Rootsets Java class for anonymous complex type.

Nested classes/interfaces inherited from class com.exalead.mercury.mami.crawl.v20.ICrawler

ICrawler.MimeTypes, ICrawler.SessionIdBlacklist

Field Summary

Fields

Modifier and Type	Field and Description
protected Crawler.Roots	roots
protected Crawler.Rootsets	rootsets

Fields inherited from class com.exalead.mercury.mami.crawl.v20.ICrawler

aggressive, buildGroup, crawlerServer, crawlSchedulerConfig, crawlSitemaps, customCrawlConfig, defaultAccept, defaultFollow, defaultFollowRoots, defaultIndex, disableConditionalGet, enableConvertProcessor, enableSimpleSiteCollapsing, fetcher, ignoreRobotsTxt, indexRedirectSources, mimeTypes, mimeTypesMode, name, nearDuplicateDetector, nthreads, patternsDetector, rules, sessionIdBlacklist, simpleSiteCollapsingDepth, smartRefresh, smartRefreshMaxAgeS, smartRefreshMinAgeS, storeTextOnly, throttleTimeMS

Constructor Summary

Constructors

Constructor and Description
Crawler() Creates a new Crawler instance.
Crawler(Crawler o) Creates a new Crawler instance by deeply copying a given Crawler instance.

Method Summary

Modifier and Type	Method and Description
Crawler	clone() Creates and returns a deep copy of this object.
Crawler.Roots	getRoots() A list of root urls to start the crawl from.
Crawler.Rootsets	getRootsets() A list of files to load urls/sites from.
void	setRoots(Crawler.Roots value) Sets the value of the roots property.
void	setRootsets(Crawler.Rootsets value) Sets the value of the rootsets property.
Crawler	withAggressive(java.lang.Boolean value)
Crawler	withBuildGroup(java.lang.String value)
Crawler	withCrawlerServer(java.lang.String value)
Crawler	withCrawlSchedulerConfig(CrawlSchedulerConfig value)
Crawler	withCrawlSitemaps(java.lang.Boolean value)
Crawler	withCustomCrawlConfig(CustomCrawlConfig value)
Crawler	withDefaultAccept(java.lang.Boolean value)
Crawler	withDefaultFollow(java.lang.Boolean value)
Crawler	withDefaultFollowRoots(java.lang.Boolean value)
Crawler	withDefaultIndex(java.lang.Boolean value)
Crawler	withDisableConditionalGet(java.lang.Boolean value)
Crawler	withEnableConvertProcessor(java.lang.Boolean value)
Crawler	withEnableSimpleSiteCollapsing(java.lang.Boolean value)
Crawler	withFetcher(java.lang.String value)
Crawler	withIgnoreRobotsTxt(java.lang.Boolean value)
Crawler	withIndexRedirectSources(java.lang.Boolean value)
Crawler	withMimeTypes(ICrawler.MimeTypes value)
Crawler	withMimeTypesMode(java.lang.String value)
Crawler	withName(java.lang.String value)
Crawler	withNearDuplicateDetector(java.lang.Boolean value)
Crawler	withNthreads(java.lang.Integer value)
Crawler	withPatternsDetector(java.lang.Boolean value)
Crawler	withRoots(Crawler.Roots value)
Crawler	withRootsets(Crawler.Rootsets value)
Crawler	withRules(java.util.Collection<Rules> values)
Crawler	withRules(Rules... values)
Crawler	withSessionIdBlacklist(ICrawler.SessionIdBlacklist value)
Crawler	withSimpleSiteCollapsingDepth(java.lang.Long value)
Crawler	withSmartRefresh(java.lang.Boolean value)
Crawler	withSmartRefreshMaxAgeS(java.lang.Integer value)
Crawler	withSmartRefreshMinAgeS(java.lang.Integer value)
Crawler	withStoreTextOnly(java.lang.Boolean value)
Crawler	withThrottleTimeMS(java.lang.Integer value)

Methods inherited from class com.exalead.mercury.mami.crawl.v20.ICrawler

getBuildGroup, getCrawlerServer, getCrawlSchedulerConfig, getCustomCrawlConfig, getFetcher, getMimeTypes, getMimeTypesMode, getName, getNthreads, getRules, getSessionIdBlacklist, getSimpleSiteCollapsingDepth, getSmartRefreshMaxAgeS, getSmartRefreshMinAgeS, getThrottleTimeMS, isAggressive, isCrawlSitemaps, isDefaultAccept, isDefaultFollow, isDefaultFollowRoots, isDefaultIndex, isDisableConditionalGet, isEnableConvertProcessor, isEnableSimpleSiteCollapsing, isIgnoreRobotsTxt, isIndexRedirectSources, isNearDuplicateDetector, isPatternsDetector, isSmartRefresh, isStoreTextOnly, setAggressive, setBuildGroup, setCrawlerServer, setCrawlSchedulerConfig, setCrawlSitemaps, setCustomCrawlConfig, setDefaultAccept, setDefaultFollow, setDefaultFollowRoots, setDefaultIndex, setDisableConditionalGet, setEnableConvertProcessor, setEnableSimpleSiteCollapsing, setFetcher, setIgnoreRobotsTxt, setIndexRedirectSources, setMimeTypes, setMimeTypesMode, setName, setNearDuplicateDetector, setNthreads, setPatternsDetector, setSessionIdBlacklist, setSimpleSiteCollapsingDepth, setSmartRefresh, setSmartRefreshMaxAgeS, setSmartRefreshMinAgeS, setStoreTextOnly, setThrottleTimeMS

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

roots

protected Crawler.Roots roots

rootsets

protected Crawler.Rootsets rootsets

Constructor Detail

Crawler

public Crawler()

Creates a new Crawler instance.

Crawler

public Crawler(Crawler o)

Creates a new Crawler instance by deeply copying a given Crawler instance.

Parameters:

o - The instance to copy.

Throws:

java.lang.NullPointerException - if o is null.

Method Detail

getRoots

public Crawler.Roots getRoots()

A list of root urls to start the crawl from.

Returns:

possible object is Crawler.Roots

setRoots

public void setRoots(Crawler.Roots value)

Sets the value of the roots property.

Parameters:

value - allowed object is Crawler.Roots

getRootsets

public Crawler.Rootsets getRootsets()

A list of files to load urls/sites from.

Returns:

possible object is Crawler.Rootsets

setRootsets

public void setRootsets(Crawler.Rootsets value)

Sets the value of the rootsets property.

Parameters:

value - allowed object is Crawler.Rootsets

withRoots

public Crawler withRoots(Crawler.Roots value)

withRootsets

public Crawler withRootsets(Crawler.Rootsets value)

withRules

public Crawler withRules(Rules... values)

Overrides:

withRules in class ICrawler

withRules

public Crawler withRules(java.util.Collection<Rules> values)

Overrides:

withRules in class ICrawler

withCrawlSchedulerConfig

public Crawler withCrawlSchedulerConfig(CrawlSchedulerConfig value)

Overrides:

withCrawlSchedulerConfig in class ICrawler

withCustomCrawlConfig

public Crawler withCustomCrawlConfig(CustomCrawlConfig value)

Overrides:

withCustomCrawlConfig in class ICrawler

withMimeTypes

public Crawler withMimeTypes(ICrawler.MimeTypes value)

Overrides:

withMimeTypes in class ICrawler

withSessionIdBlacklist

public Crawler withSessionIdBlacklist(ICrawler.SessionIdBlacklist value)

Overrides:

withSessionIdBlacklist in class ICrawler

withName

public Crawler withName(java.lang.String value)

Overrides:

withName in class ICrawler

withFetcher

public Crawler withFetcher(java.lang.String value)

Overrides:

withFetcher in class ICrawler

withCrawlerServer

public Crawler withCrawlerServer(java.lang.String value)

Overrides:

withCrawlerServer in class ICrawler

withBuildGroup

public Crawler withBuildGroup(java.lang.String value)

Overrides:

withBuildGroup in class ICrawler

withStoreTextOnly

public Crawler withStoreTextOnly(java.lang.Boolean value)

Overrides:

withStoreTextOnly in class ICrawler

withNthreads

public Crawler withNthreads(java.lang.Integer value)

Overrides:

withNthreads in class ICrawler

withAggressive

public Crawler withAggressive(java.lang.Boolean value)

Overrides:

withAggressive in class ICrawler

withThrottleTimeMS

public Crawler withThrottleTimeMS(java.lang.Integer value)

Overrides:

withThrottleTimeMS in class ICrawler

withIgnoreRobotsTxt

public Crawler withIgnoreRobotsTxt(java.lang.Boolean value)

Overrides:

withIgnoreRobotsTxt in class ICrawler

withEnableConvertProcessor

public Crawler withEnableConvertProcessor(java.lang.Boolean value)

Overrides:

withEnableConvertProcessor in class ICrawler

withNearDuplicateDetector

public Crawler withNearDuplicateDetector(java.lang.Boolean value)

Overrides:

withNearDuplicateDetector in class ICrawler

withPatternsDetector

public Crawler withPatternsDetector(java.lang.Boolean value)

Overrides:

withPatternsDetector in class ICrawler

withCrawlSitemaps

public Crawler withCrawlSitemaps(java.lang.Boolean value)

Overrides:

withCrawlSitemaps in class ICrawler

withDisableConditionalGet

public Crawler withDisableConditionalGet(java.lang.Boolean value)

Overrides:

withDisableConditionalGet in class ICrawler

withDefaultAccept

public Crawler withDefaultAccept(java.lang.Boolean value)

Overrides:

withDefaultAccept in class ICrawler

withDefaultIndex

public Crawler withDefaultIndex(java.lang.Boolean value)

Overrides:

withDefaultIndex in class ICrawler

withDefaultFollow

public Crawler withDefaultFollow(java.lang.Boolean value)

Overrides:

withDefaultFollow in class ICrawler

withDefaultFollowRoots

public Crawler withDefaultFollowRoots(java.lang.Boolean value)

Overrides:

withDefaultFollowRoots in class ICrawler

withEnableSimpleSiteCollapsing

public Crawler withEnableSimpleSiteCollapsing(java.lang.Boolean value)

Overrides:

withEnableSimpleSiteCollapsing in class ICrawler

withSimpleSiteCollapsingDepth

public Crawler withSimpleSiteCollapsingDepth(java.lang.Long value)

Overrides:

withSimpleSiteCollapsingDepth in class ICrawler

withMimeTypesMode

public Crawler withMimeTypesMode(java.lang.String value)

Overrides:

withMimeTypesMode in class ICrawler

withIndexRedirectSources

public Crawler withIndexRedirectSources(java.lang.Boolean value)

Overrides:

withIndexRedirectSources in class ICrawler

withSmartRefresh

public Crawler withSmartRefresh(java.lang.Boolean value)

Overrides:

withSmartRefresh in class ICrawler

withSmartRefreshMinAgeS

public Crawler withSmartRefreshMinAgeS(java.lang.Integer value)

Overrides:

withSmartRefreshMinAgeS in class ICrawler

withSmartRefreshMaxAgeS

public Crawler withSmartRefreshMaxAgeS(java.lang.Integer value)

Overrides:

withSmartRefreshMaxAgeS in class ICrawler

clone

public Crawler clone()

Creates and returns a deep copy of this object.

Overrides:

clone in class ICrawler

Returns:

A deep copy of this object.