com.exalead.mercury.mami.crawl.v20

Class FeedFetcher

java.lang.Object

com.exalead.mercury.mami.crawl.v20.ICrawler

com.exalead.mercury.mami.crawl.v20.FeedFetcher

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable

public class FeedFetcher
extends ICrawler
implements java.io.Serializable, java.lang.Cloneable

A feedfetcher configuration.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FeedFetcher.Feeds Java class for anonymous complex type.
static class	FeedFetcher.Rootsets Java class for anonymous complex type.

Nested classes/interfaces inherited from class com.exalead.mercury.mami.crawl.v20.ICrawler

ICrawler.MimeTypes, ICrawler.SessionIdBlacklist

Field Summary

Fields

Modifier and Type	Field and Description
protected FeedFetcher.Feeds	feeds
protected java.lang.Integer	refreshDelayS
protected FeedFetcher.Rootsets	rootsets

Fields inherited from class com.exalead.mercury.mami.crawl.v20.ICrawler

aggressive, buildGroup, crawlerServer, crawlSchedulerConfig, crawlSitemaps, customCrawlConfig, defaultAccept, defaultFollow, defaultFollowRoots, defaultIndex, disableConditionalGet, enableConvertProcessor, enableSimpleSiteCollapsing, fetcher, ignoreRobotsTxt, indexRedirectSources, mimeTypes, mimeTypesMode, name, nearDuplicateDetector, nthreads, patternsDetector, rules, sessionIdBlacklist, simpleSiteCollapsingDepth, smartRefresh, smartRefreshMaxAgeS, smartRefreshMinAgeS, storeTextOnly, throttleTimeMS

Constructor Summary

Constructors

Constructor and Description
FeedFetcher() Creates a new FeedFetcher instance.
FeedFetcher(FeedFetcher o) Creates a new FeedFetcher instance by deeply copying a given FeedFetcher instance.

Method Summary

Modifier and Type	Method and Description
FeedFetcher	clone() Creates and returns a deep copy of this object.
FeedFetcher.Feeds	getFeeds() a list of feeds
int	getRefreshDelayS() minimum delay before refreshing any url, default is 1 minute (default: 60)
FeedFetcher.Rootsets	getRootsets() A list of files to load urls/sites from.
void	setFeeds(FeedFetcher.Feeds value) Sets the value of the feeds property.
void	setRefreshDelayS(java.lang.Integer value) Sets the value of the refreshDelayS property.
void	setRootsets(FeedFetcher.Rootsets value) Sets the value of the rootsets property.
FeedFetcher	withAggressive(java.lang.Boolean value)
FeedFetcher	withBuildGroup(java.lang.String value)
FeedFetcher	withCrawlerServer(java.lang.String value)
FeedFetcher	withCrawlSchedulerConfig(CrawlSchedulerConfig value)
FeedFetcher	withCrawlSitemaps(java.lang.Boolean value)
FeedFetcher	withCustomCrawlConfig(CustomCrawlConfig value)
FeedFetcher	withDefaultAccept(java.lang.Boolean value)
FeedFetcher	withDefaultFollow(java.lang.Boolean value)
FeedFetcher	withDefaultFollowRoots(java.lang.Boolean value)
FeedFetcher	withDefaultIndex(java.lang.Boolean value)
FeedFetcher	withDisableConditionalGet(java.lang.Boolean value)
FeedFetcher	withEnableConvertProcessor(java.lang.Boolean value)
FeedFetcher	withEnableSimpleSiteCollapsing(java.lang.Boolean value)
FeedFetcher	withFeeds(FeedFetcher.Feeds value)
FeedFetcher	withFetcher(java.lang.String value)
FeedFetcher	withIgnoreRobotsTxt(java.lang.Boolean value)
FeedFetcher	withIndexRedirectSources(java.lang.Boolean value)
FeedFetcher	withMimeTypes(ICrawler.MimeTypes value)
FeedFetcher	withMimeTypesMode(java.lang.String value)
FeedFetcher	withName(java.lang.String value)
FeedFetcher	withNearDuplicateDetector(java.lang.Boolean value)
FeedFetcher	withNthreads(java.lang.Integer value)
FeedFetcher	withPatternsDetector(java.lang.Boolean value)
FeedFetcher	withRefreshDelayS(java.lang.Integer value)
FeedFetcher	withRootsets(FeedFetcher.Rootsets value)
FeedFetcher	withRules(java.util.Collection<Rules> values)
FeedFetcher	withRules(Rules... values)
FeedFetcher	withSessionIdBlacklist(ICrawler.SessionIdBlacklist value)
FeedFetcher	withSimpleSiteCollapsingDepth(java.lang.Long value)
FeedFetcher	withSmartRefresh(java.lang.Boolean value)
FeedFetcher	withSmartRefreshMaxAgeS(java.lang.Integer value)
FeedFetcher	withSmartRefreshMinAgeS(java.lang.Integer value)
FeedFetcher	withStoreTextOnly(java.lang.Boolean value)
FeedFetcher	withThrottleTimeMS(java.lang.Integer value)

Methods inherited from class com.exalead.mercury.mami.crawl.v20.ICrawler

getBuildGroup, getCrawlerServer, getCrawlSchedulerConfig, getCustomCrawlConfig, getFetcher, getMimeTypes, getMimeTypesMode, getName, getNthreads, getRules, getSessionIdBlacklist, getSimpleSiteCollapsingDepth, getSmartRefreshMaxAgeS, getSmartRefreshMinAgeS, getThrottleTimeMS, isAggressive, isCrawlSitemaps, isDefaultAccept, isDefaultFollow, isDefaultFollowRoots, isDefaultIndex, isDisableConditionalGet, isEnableConvertProcessor, isEnableSimpleSiteCollapsing, isIgnoreRobotsTxt, isIndexRedirectSources, isNearDuplicateDetector, isPatternsDetector, isSmartRefresh, isStoreTextOnly, setAggressive, setBuildGroup, setCrawlerServer, setCrawlSchedulerConfig, setCrawlSitemaps, setCustomCrawlConfig, setDefaultAccept, setDefaultFollow, setDefaultFollowRoots, setDefaultIndex, setDisableConditionalGet, setEnableConvertProcessor, setEnableSimpleSiteCollapsing, setFetcher, setIgnoreRobotsTxt, setIndexRedirectSources, setMimeTypes, setMimeTypesMode, setName, setNearDuplicateDetector, setNthreads, setPatternsDetector, setSessionIdBlacklist, setSimpleSiteCollapsingDepth, setSmartRefresh, setSmartRefreshMaxAgeS, setSmartRefreshMinAgeS, setStoreTextOnly, setThrottleTimeMS

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

feeds

protected FeedFetcher.Feeds feeds

rootsets

protected FeedFetcher.Rootsets rootsets

refreshDelayS

protected java.lang.Integer refreshDelayS

Constructor Detail

FeedFetcher

public FeedFetcher()

Creates a new FeedFetcher instance.

FeedFetcher

public FeedFetcher(FeedFetcher o)

Creates a new FeedFetcher instance by deeply copying a given FeedFetcher instance.

Parameters:

o - The instance to copy.

Throws:

java.lang.NullPointerException - if o is null.

Method Detail

getFeeds

public FeedFetcher.Feeds getFeeds()

a list of feeds

Returns:

possible object is FeedFetcher.Feeds

setFeeds

public void setFeeds(FeedFetcher.Feeds value)

Sets the value of the feeds property.

Parameters:

value - allowed object is FeedFetcher.Feeds

getRootsets

public FeedFetcher.Rootsets getRootsets()

A list of files to load urls/sites from.

Returns:

possible object is FeedFetcher.Rootsets

setRootsets

public void setRootsets(FeedFetcher.Rootsets value)

Sets the value of the rootsets property.

Parameters:

value - allowed object is FeedFetcher.Rootsets

getRefreshDelayS

public int getRefreshDelayS()

minimum delay before refreshing any url, default is 1 minute (default: 60)

Returns:

possible object is Integer

setRefreshDelayS

public void setRefreshDelayS(java.lang.Integer value)

Sets the value of the refreshDelayS property.

Parameters:

value - allowed object is Integer

withFeeds

public FeedFetcher withFeeds(FeedFetcher.Feeds value)

withRootsets

public FeedFetcher withRootsets(FeedFetcher.Rootsets value)

withRefreshDelayS

public FeedFetcher withRefreshDelayS(java.lang.Integer value)

withRules

public FeedFetcher withRules(Rules... values)

Overrides:

withRules in class ICrawler

withRules

public FeedFetcher withRules(java.util.Collection<Rules> values)

Overrides:

withRules in class ICrawler

withCrawlSchedulerConfig

public FeedFetcher withCrawlSchedulerConfig(CrawlSchedulerConfig value)

Overrides:

withCrawlSchedulerConfig in class ICrawler

withCustomCrawlConfig

public FeedFetcher withCustomCrawlConfig(CustomCrawlConfig value)

Overrides:

withCustomCrawlConfig in class ICrawler

withMimeTypes

public FeedFetcher withMimeTypes(ICrawler.MimeTypes value)

Overrides:

withMimeTypes in class ICrawler

withSessionIdBlacklist

public FeedFetcher withSessionIdBlacklist(ICrawler.SessionIdBlacklist value)

Overrides:

withSessionIdBlacklist in class ICrawler

withName

public FeedFetcher withName(java.lang.String value)

Overrides:

withName in class ICrawler

withFetcher

public FeedFetcher withFetcher(java.lang.String value)

Overrides:

withFetcher in class ICrawler

withCrawlerServer

public FeedFetcher withCrawlerServer(java.lang.String value)

Overrides:

withCrawlerServer in class ICrawler

withBuildGroup

public FeedFetcher withBuildGroup(java.lang.String value)

Overrides:

withBuildGroup in class ICrawler

withStoreTextOnly

public FeedFetcher withStoreTextOnly(java.lang.Boolean value)

Overrides:

withStoreTextOnly in class ICrawler

withNthreads

public FeedFetcher withNthreads(java.lang.Integer value)

Overrides:

withNthreads in class ICrawler

withAggressive

public FeedFetcher withAggressive(java.lang.Boolean value)

Overrides:

withAggressive in class ICrawler

withThrottleTimeMS

public FeedFetcher withThrottleTimeMS(java.lang.Integer value)

Overrides:

withThrottleTimeMS in class ICrawler

withIgnoreRobotsTxt

public FeedFetcher withIgnoreRobotsTxt(java.lang.Boolean value)

Overrides:

withIgnoreRobotsTxt in class ICrawler

withEnableConvertProcessor

public FeedFetcher withEnableConvertProcessor(java.lang.Boolean value)

Overrides:

withEnableConvertProcessor in class ICrawler

withNearDuplicateDetector

public FeedFetcher withNearDuplicateDetector(java.lang.Boolean value)

Overrides:

withNearDuplicateDetector in class ICrawler

withPatternsDetector

public FeedFetcher withPatternsDetector(java.lang.Boolean value)

Overrides:

withPatternsDetector in class ICrawler

withCrawlSitemaps

public FeedFetcher withCrawlSitemaps(java.lang.Boolean value)

Overrides:

withCrawlSitemaps in class ICrawler

withDisableConditionalGet

public FeedFetcher withDisableConditionalGet(java.lang.Boolean value)

Overrides:

withDisableConditionalGet in class ICrawler

withDefaultAccept

public FeedFetcher withDefaultAccept(java.lang.Boolean value)

Overrides:

withDefaultAccept in class ICrawler

withDefaultIndex

public FeedFetcher withDefaultIndex(java.lang.Boolean value)

Overrides:

withDefaultIndex in class ICrawler

withDefaultFollow

public FeedFetcher withDefaultFollow(java.lang.Boolean value)

Overrides:

withDefaultFollow in class ICrawler

withDefaultFollowRoots

public FeedFetcher withDefaultFollowRoots(java.lang.Boolean value)

Overrides:

withDefaultFollowRoots in class ICrawler

withEnableSimpleSiteCollapsing

public FeedFetcher withEnableSimpleSiteCollapsing(java.lang.Boolean value)

Overrides:

withEnableSimpleSiteCollapsing in class ICrawler

withSimpleSiteCollapsingDepth

public FeedFetcher withSimpleSiteCollapsingDepth(java.lang.Long value)

Overrides:

withSimpleSiteCollapsingDepth in class ICrawler

withMimeTypesMode

public FeedFetcher withMimeTypesMode(java.lang.String value)

Overrides:

withMimeTypesMode in class ICrawler

withIndexRedirectSources

public FeedFetcher withIndexRedirectSources(java.lang.Boolean value)

Overrides:

withIndexRedirectSources in class ICrawler

withSmartRefresh

public FeedFetcher withSmartRefresh(java.lang.Boolean value)

Overrides:

withSmartRefresh in class ICrawler

withSmartRefreshMinAgeS

public FeedFetcher withSmartRefreshMinAgeS(java.lang.Integer value)

Overrides:

withSmartRefreshMinAgeS in class ICrawler

withSmartRefreshMaxAgeS

public FeedFetcher withSmartRefreshMaxAgeS(java.lang.Integer value)

Overrides:

withSmartRefreshMaxAgeS in class ICrawler

clone

public FeedFetcher clone()

Creates and returns a deep copy of this object.

Overrides:

clone in class ICrawler

Returns:

A deep copy of this object.