com.exalead.mercury.mami.crawl.v10

Class Crawler

java.lang.Object

com.exalead.mercury.mami.crawl.v10.ICrawler

com.exalead.mercury.mami.crawl.v10.Crawler

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable

public class Crawler
extends ICrawler
implements java.io.Serializable, java.lang.Cloneable

A crawler configuration. A crawler can and should contain sources. A crawler may contain a CrawlSchedulerConfig to overwrite the default fifo priorities. A crawler may contain a CustomCrawlConfig to enable custom processors. A crawler may contain KeyValues that are pushed as metas with every crawled document.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected CrawlSchedulerConfig	crawlSchedulerConfig
protected CustomCrawlConfig	customCrawlConfig
protected java.util.List<KeyValue>	keyValue
protected java.util.List<Source>	source

Fields inherited from class com.exalead.mercury.mami.crawl.v10.ICrawler

aggressive, buildGroup, crawlerServer, crawlSitemaps, defaultFetcher, enableConvertProcessor, ignoreRobotsTxt, name, nearDuplicateDetector, nthreads, patternsDetector, throttleTimeMS

Constructor Summary

Constructors

Constructor and Description
Crawler() Creates a new Crawler instance.
Crawler(Crawler o) Creates a new Crawler instance by deeply copying a given Crawler instance.

Method Summary

Modifier and Type	Method and Description
Crawler	clone() Creates and returns a deep copy of this object.
CrawlSchedulerConfig	getCrawlSchedulerConfig() Gets the value of the crawlSchedulerConfig property.
CustomCrawlConfig	getCustomCrawlConfig() Gets the value of the customCrawlConfig property.
java.util.List<KeyValue>	getKeyValue() Gets the value of the keyValue property.
java.util.List<Source>	getSource() Gets the value of the source property.
void	setCrawlSchedulerConfig(CrawlSchedulerConfig value) Sets the value of the crawlSchedulerConfig property.
void	setCustomCrawlConfig(CustomCrawlConfig value) Sets the value of the customCrawlConfig property.
Crawler	withAggressive(java.lang.Boolean value)
Crawler	withBuildGroup(java.lang.String value)
Crawler	withCrawlerServer(java.lang.String value)
Crawler	withCrawlSchedulerConfig(CrawlSchedulerConfig value)
Crawler	withCrawlSitemaps(java.lang.Boolean value)
Crawler	withCustomCrawlConfig(CustomCrawlConfig value)
Crawler	withDefaultFetcher(java.lang.String value)
Crawler	withEnableConvertProcessor(java.lang.Boolean value)
Crawler	withIgnoreRobotsTxt(java.lang.Boolean value)
Crawler	withKeyValue(java.util.Collection<KeyValue> values)
Crawler	withKeyValue(KeyValue... values)
Crawler	withName(java.lang.String value)
Crawler	withNearDuplicateDetector(java.lang.Boolean value)
Crawler	withNthreads(java.lang.Integer value)
Crawler	withPatternsDetector(java.lang.Boolean value)
Crawler	withSource(java.util.Collection<Source> values)
Crawler	withSource(Source... values)
Crawler	withThrottleTimeMS(java.lang.Integer value)

Methods inherited from class com.exalead.mercury.mami.crawl.v10.ICrawler

getBuildGroup, getCrawlerServer, getDefaultFetcher, getName, getNthreads, getThrottleTimeMS, isAggressive, isCrawlSitemaps, isEnableConvertProcessor, isIgnoreRobotsTxt, isNearDuplicateDetector, isPatternsDetector, setAggressive, setBuildGroup, setCrawlerServer, setCrawlSitemaps, setDefaultFetcher, setEnableConvertProcessor, setIgnoreRobotsTxt, setName, setNearDuplicateDetector, setNthreads, setPatternsDetector, setThrottleTimeMS

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

crawlSchedulerConfig

protected CrawlSchedulerConfig crawlSchedulerConfig

customCrawlConfig

protected CustomCrawlConfig customCrawlConfig

keyValue

protected java.util.List<KeyValue> keyValue

source

protected java.util.List<Source> source

Constructor Detail

Crawler

public Crawler()

Creates a new Crawler instance.

Crawler

public Crawler(Crawler o)

Creates a new Crawler instance by deeply copying a given Crawler instance.

Parameters:

o - The instance to copy.

Throws:

java.lang.NullPointerException - if o is null.

Method Detail

getCrawlSchedulerConfig

public CrawlSchedulerConfig getCrawlSchedulerConfig()

Gets the value of the crawlSchedulerConfig property.

Returns:

possible object is CrawlSchedulerConfig

setCrawlSchedulerConfig

public void setCrawlSchedulerConfig(CrawlSchedulerConfig value)

Sets the value of the crawlSchedulerConfig property.

Parameters:

value - allowed object is CrawlSchedulerConfig

getCustomCrawlConfig

public CustomCrawlConfig getCustomCrawlConfig()

Gets the value of the customCrawlConfig property.

Returns:

possible object is CustomCrawlConfig

setCustomCrawlConfig

public void setCustomCrawlConfig(CustomCrawlConfig value)

Sets the value of the customCrawlConfig property.

Parameters:

value - allowed object is CustomCrawlConfig

getKeyValue

public java.util.List<KeyValue> getKeyValue()

Gets the value of the keyValue property.

This accessor method returns a reference to the live list, not a snapshot. Therefore any modification you make to the returned list will be present inside the JAXB object. This is why there is not a set method for the keyValue property.

For example, to add a new item, do as follows:

    getKeyValue().add(newItem);
 

Objects of the following type(s) are allowed in the list KeyValue

getSource

public java.util.List<Source> getSource()

Gets the value of the source property.

This accessor method returns a reference to the live list, not a snapshot. Therefore any modification you make to the returned list will be present inside the JAXB object. This is why there is not a set method for the source property.

For example, to add a new item, do as follows:

    getSource().add(newItem);
 

Objects of the following type(s) are allowed in the list Source

withCrawlSchedulerConfig

public Crawler withCrawlSchedulerConfig(CrawlSchedulerConfig value)

withCustomCrawlConfig

public Crawler withCustomCrawlConfig(CustomCrawlConfig value)

withKeyValue

public Crawler withKeyValue(KeyValue... values)

withKeyValue

public Crawler withKeyValue(java.util.Collection<KeyValue> values)

withSource

public Crawler withSource(Source... values)

withSource

public Crawler withSource(java.util.Collection<Source> values)

withName

public Crawler withName(java.lang.String value)

Overrides:

withName in class ICrawler

withCrawlerServer

public Crawler withCrawlerServer(java.lang.String value)

Overrides:

withCrawlerServer in class ICrawler

withBuildGroup

public Crawler withBuildGroup(java.lang.String value)

Overrides:

withBuildGroup in class ICrawler

withNthreads

public Crawler withNthreads(java.lang.Integer value)

Overrides:

withNthreads in class ICrawler

withAggressive

public Crawler withAggressive(java.lang.Boolean value)

Overrides:

withAggressive in class ICrawler

withThrottleTimeMS

public Crawler withThrottleTimeMS(java.lang.Integer value)

Overrides:

withThrottleTimeMS in class ICrawler

withIgnoreRobotsTxt

public Crawler withIgnoreRobotsTxt(java.lang.Boolean value)

Overrides:

withIgnoreRobotsTxt in class ICrawler

withEnableConvertProcessor

public Crawler withEnableConvertProcessor(java.lang.Boolean value)

Overrides:

withEnableConvertProcessor in class ICrawler

withNearDuplicateDetector

public Crawler withNearDuplicateDetector(java.lang.Boolean value)

Overrides:

withNearDuplicateDetector in class ICrawler

withPatternsDetector

public Crawler withPatternsDetector(java.lang.Boolean value)

Overrides:

withPatternsDetector in class ICrawler

withDefaultFetcher

public Crawler withDefaultFetcher(java.lang.String value)

Overrides:

withDefaultFetcher in class ICrawler

withCrawlSitemaps

public Crawler withCrawlSitemaps(java.lang.Boolean value)

Overrides:

withCrawlSitemaps in class ICrawler

clone

public Crawler clone()

Creates and returns a deep copy of this object.

Overrides:

clone in class ICrawler

Returns:

A deep copy of this object.