ICrawler (NETVIBES Exalead CloudView Clients SDK)

java.lang.Object
- com.exalead.mercury.mami.crawl.v20.ICrawler

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable

Direct Known Subclasses:

Crawler, FeedFetcher
```
public abstract class ICrawler
extends java.lang.Object
implements java.io.Serializable, java.lang.Cloneable
```
See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`ICrawler.MimeTypes` Java class for anonymous complex type.
`static class`	`ICrawler.SessionIdBlacklist` Java class for anonymous complex type.

Field Summary

Fields
Modifier and Type	Field and Description
`protected java.lang.Boolean`	`aggressive`
`protected java.lang.String`	`buildGroup`
`protected java.lang.String`	`crawlerServer`
`protected CrawlSchedulerConfig`	`crawlSchedulerConfig`
`protected java.lang.Boolean`	`crawlSitemaps`
`protected CustomCrawlConfig`	`customCrawlConfig`
`protected java.lang.Boolean`	`defaultAccept`
`protected java.lang.Boolean`	`defaultFollow`
`protected java.lang.Boolean`	`defaultFollowRoots`
`protected java.lang.Boolean`	`defaultIndex`
`protected java.lang.Boolean`	`disableConditionalGet`
`protected java.lang.Boolean`	`enableConvertProcessor`
`protected java.lang.Boolean`	`enableSimpleSiteCollapsing`
`protected java.lang.String`	`fetcher`
`protected java.lang.Boolean`	`ignoreRobotsTxt`
`protected java.lang.Boolean`	`indexRedirectSources`
`protected ICrawler.MimeTypes`	`mimeTypes`
`protected java.lang.String`	`mimeTypesMode`
`protected java.lang.String`	`name`
`protected java.lang.Boolean`	`nearDuplicateDetector`
`protected java.lang.Integer`	`nthreads`
`protected java.lang.Boolean`	`patternsDetector`
`protected java.util.List<Rules>`	`rules`
`protected ICrawler.SessionIdBlacklist`	`sessionIdBlacklist`
`protected java.lang.Long`	`simpleSiteCollapsingDepth`
`protected java.lang.Boolean`	`smartRefresh`
`protected java.lang.Integer`	`smartRefreshMaxAgeS`
`protected java.lang.Integer`	`smartRefreshMinAgeS`
`protected java.lang.Boolean`	`storeTextOnly`
`protected java.lang.Integer`	`throttleTimeMS`

Constructor Summary

Constructors
Constructor and Description

ICrawler()
Creates a new ICrawler instance.

ICrawler(ICrawler o)
Creates a new ICrawler instance by deeply copying a given ICrawler instance.

Constructors
Constructor and Description
`ICrawler()` Creates a new `ICrawler` instance.
`ICrawler(ICrawler o)` Creates a new `ICrawler` instance by deeply copying a given `ICrawler` instance.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`ICrawler`	`clone()` Creates and returns a deep copy of this object.
`java.lang.String`	`getBuildGroup()` Target build group.
`java.lang.String`	`getCrawlerServer()` Crawler server hosting this crawler.
`CrawlSchedulerConfig`	`getCrawlSchedulerConfig()` Gets the value of the crawlSchedulerConfig property.
`CustomCrawlConfig`	`getCustomCrawlConfig()` Gets the value of the customCrawlConfig property.
`java.lang.String`	`getFetcher()` Which fetcher to use.
`ICrawler.MimeTypes`	`getMimeTypes()` Gets the value of the mimeTypes property.
`java.lang.String`	`getMimeTypesMode()` Mime types white/black list (default: exclude)
`java.lang.String`	`getName()` The crawler name.
`int`	`getNthreads()` The number of crawl threads which must be strictly positive.
`java.util.List<Rules>`	`getRules()` Gets the value of the rules property.
`ICrawler.SessionIdBlacklist`	`getSessionIdBlacklist()` SessionId blacklist.
`long`	`getSimpleSiteCollapsingDepth()` How many path segments to use to generate the site collapsing ID.
`int`	`getSmartRefreshMaxAgeS()` Age in seconds at which we force the refresh of old urls.
`int`	`getSmartRefreshMinAgeS()` Age in seconds at which we may refresh old urls.
`int`	`getThrottleTimeMS()` In the case of non-aggressive crawl, this defines the sleep interval between requests to the same host.
`boolean`	`isAggressive()` Whether to enable aggressive crawl, that never sleeps between two requests to the same host.
`boolean`	`isCrawlSitemaps()` Whether to crawl sitemaps.
`boolean`	`isDefaultAccept()` Whether to crawl a url by default when it matches no other accept rule.
`boolean`	`isDefaultFollow()` Whether to follow by default when a url matches no follow rule.
`boolean`	`isDefaultFollowRoots()` Whether to automatically follow root urls (default: true)
`boolean`	`isDefaultIndex()` Whether to index by default when a url matches no index rule.
`boolean`	`isDisableConditionalGet()` Whether to always fetch documents, even if the server tells it has not changed.
`boolean`	`isEnableConvertProcessor()` Whether to enable remoteconvert-based processor for links extracting in binary documents.
`boolean`	`isEnableSimpleSiteCollapsing()` Whether to generate a site ID suitable for document collapsing.
`boolean`	`isIgnoreRobotsTxt()` Whether to ignore robots.txt rules.
`boolean`	`isIndexRedirectSources()` Whether to index redirections with target data.
`boolean`	`isNearDuplicateDetector()` Whether to enable the near-duplicate content detector.
`boolean`	`isPatternsDetector()` Whether to enable patterns detection in pages.
`boolean`	`isSmartRefresh()` Whether to crawl a fraction of refreshed urls.
`boolean`	`isStoreTextOnly()` Whether to store original binary documents, or only converted text.
`void`	`setAggressive(java.lang.Boolean value)` Sets the value of the aggressive property.
`void`	`setBuildGroup(java.lang.String value)` Sets the value of the buildGroup property.
`void`	`setCrawlerServer(java.lang.String value)` Sets the value of the crawlerServer property.
`void`	`setCrawlSchedulerConfig(CrawlSchedulerConfig value)` Sets the value of the crawlSchedulerConfig property.
`void`	`setCrawlSitemaps(java.lang.Boolean value)` Sets the value of the crawlSitemaps property.
`void`	`setCustomCrawlConfig(CustomCrawlConfig value)` Sets the value of the customCrawlConfig property.
`void`	`setDefaultAccept(java.lang.Boolean value)` Sets the value of the defaultAccept property.
`void`	`setDefaultFollow(java.lang.Boolean value)` Sets the value of the defaultFollow property.
`void`	`setDefaultFollowRoots(java.lang.Boolean value)` Sets the value of the defaultFollowRoots property.
`void`	`setDefaultIndex(java.lang.Boolean value)` Sets the value of the defaultIndex property.
`void`	`setDisableConditionalGet(java.lang.Boolean value)` Sets the value of the disableConditionalGet property.
`void`	`setEnableConvertProcessor(java.lang.Boolean value)` Sets the value of the enableConvertProcessor property.
`void`	`setEnableSimpleSiteCollapsing(java.lang.Boolean value)` Sets the value of the enableSimpleSiteCollapsing property.
`void`	`setFetcher(java.lang.String value)` Sets the value of the fetcher property.
`void`	`setIgnoreRobotsTxt(java.lang.Boolean value)` Sets the value of the ignoreRobotsTxt property.
`void`	`setIndexRedirectSources(java.lang.Boolean value)` Sets the value of the indexRedirectSources property.
`void`	`setMimeTypes(ICrawler.MimeTypes value)` Sets the value of the mimeTypes property.
`void`	`setMimeTypesMode(java.lang.String value)` Sets the value of the mimeTypesMode property.
`void`	`setName(java.lang.String value)` Sets the value of the name property.
`void`	`setNearDuplicateDetector(java.lang.Boolean value)` Sets the value of the nearDuplicateDetector property.
`void`	`setNthreads(java.lang.Integer value)` Sets the value of the nthreads property.
`void`	`setPatternsDetector(java.lang.Boolean value)` Sets the value of the patternsDetector property.
`void`	`setSessionIdBlacklist(ICrawler.SessionIdBlacklist value)` Sets the value of the sessionIdBlacklist property.
`void`	`setSimpleSiteCollapsingDepth(java.lang.Long value)` Sets the value of the simpleSiteCollapsingDepth property.
`void`	`setSmartRefresh(java.lang.Boolean value)` Sets the value of the smartRefresh property.
`void`	`setSmartRefreshMaxAgeS(java.lang.Integer value)` Sets the value of the smartRefreshMaxAgeS property.
`void`	`setSmartRefreshMinAgeS(java.lang.Integer value)` Sets the value of the smartRefreshMinAgeS property.
`void`	`setStoreTextOnly(java.lang.Boolean value)` Sets the value of the storeTextOnly property.
`void`	`setThrottleTimeMS(java.lang.Integer value)` Sets the value of the throttleTimeMS property.
`ICrawler`	`withAggressive(java.lang.Boolean value)`
`ICrawler`	`withBuildGroup(java.lang.String value)`
`ICrawler`	`withCrawlerServer(java.lang.String value)`
`ICrawler`	`withCrawlSchedulerConfig(CrawlSchedulerConfig value)`
`ICrawler`	`withCrawlSitemaps(java.lang.Boolean value)`
`ICrawler`	`withCustomCrawlConfig(CustomCrawlConfig value)`
`ICrawler`	`withDefaultAccept(java.lang.Boolean value)`
`ICrawler`	`withDefaultFollow(java.lang.Boolean value)`
`ICrawler`	`withDefaultFollowRoots(java.lang.Boolean value)`
`ICrawler`	`withDefaultIndex(java.lang.Boolean value)`
`ICrawler`	`withDisableConditionalGet(java.lang.Boolean value)`
`ICrawler`	`withEnableConvertProcessor(java.lang.Boolean value)`
`ICrawler`	`withEnableSimpleSiteCollapsing(java.lang.Boolean value)`
`ICrawler`	`withFetcher(java.lang.String value)`
`ICrawler`	`withIgnoreRobotsTxt(java.lang.Boolean value)`
`ICrawler`	`withIndexRedirectSources(java.lang.Boolean value)`
`ICrawler`	`withMimeTypes(ICrawler.MimeTypes value)`
`ICrawler`	`withMimeTypesMode(java.lang.String value)`
`ICrawler`	`withName(java.lang.String value)`
`ICrawler`	`withNearDuplicateDetector(java.lang.Boolean value)`
`ICrawler`	`withNthreads(java.lang.Integer value)`
`ICrawler`	`withPatternsDetector(java.lang.Boolean value)`
`ICrawler`	`withRules(java.util.Collection<Rules> values)`
`ICrawler`	`withRules(Rules... values)`
`ICrawler`	`withSessionIdBlacklist(ICrawler.SessionIdBlacklist value)`
`ICrawler`	`withSimpleSiteCollapsingDepth(java.lang.Long value)`
`ICrawler`	`withSmartRefresh(java.lang.Boolean value)`
`ICrawler`	`withSmartRefreshMaxAgeS(java.lang.Integer value)`
`ICrawler`	`withSmartRefreshMinAgeS(java.lang.Integer value)`
`ICrawler`	`withStoreTextOnly(java.lang.Boolean value)`
`ICrawler`	`withThrottleTimeMS(java.lang.Integer value)`

Methods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - rules
```
protected java.util.List<Rules> rules
```
  - crawlSchedulerConfig
```
protected CrawlSchedulerConfig crawlSchedulerConfig
```
  - customCrawlConfig
```
protected CustomCrawlConfig customCrawlConfig
```
  - mimeTypes
```
protected ICrawler.MimeTypes mimeTypes
```
  - sessionIdBlacklist
```
protected ICrawler.SessionIdBlacklist sessionIdBlacklist
```
  - name
```
protected java.lang.String name
```
  - fetcher
```
protected java.lang.String fetcher
```
  - crawlerServer
```
protected java.lang.String crawlerServer
```
  - buildGroup
```
protected java.lang.String buildGroup
```
  - storeTextOnly
```
protected java.lang.Boolean storeTextOnly
```
  - nthreads
```
protected java.lang.Integer nthreads
```
  - aggressive
```
protected java.lang.Boolean aggressive
```
  - throttleTimeMS
```
protected java.lang.Integer throttleTimeMS
```
  - ignoreRobotsTxt
```
protected java.lang.Boolean ignoreRobotsTxt
```
  - enableConvertProcessor
```
protected java.lang.Boolean enableConvertProcessor
```
  - nearDuplicateDetector
```
protected java.lang.Boolean nearDuplicateDetector
```
  - patternsDetector
```
protected java.lang.Boolean patternsDetector
```
  - crawlSitemaps
```
protected java.lang.Boolean crawlSitemaps
```
  - disableConditionalGet
```
protected java.lang.Boolean disableConditionalGet
```
  - defaultAccept
```
protected java.lang.Boolean defaultAccept
```
  - defaultIndex
```
protected java.lang.Boolean defaultIndex
```
  - defaultFollow
```
protected java.lang.Boolean defaultFollow
```
  - defaultFollowRoots
```
protected java.lang.Boolean defaultFollowRoots
```
  - enableSimpleSiteCollapsing
```
protected java.lang.Boolean enableSimpleSiteCollapsing
```
  - simpleSiteCollapsingDepth
```
protected java.lang.Long simpleSiteCollapsingDepth
```
  - mimeTypesMode
```
protected java.lang.String mimeTypesMode
```
  - indexRedirectSources
```
protected java.lang.Boolean indexRedirectSources
```
  - smartRefresh
```
protected java.lang.Boolean smartRefresh
```
  - smartRefreshMinAgeS
```
protected java.lang.Integer smartRefreshMinAgeS
```
  - smartRefreshMaxAgeS
```
protected java.lang.Integer smartRefreshMaxAgeS
```
- Constructor Detail
  - ICrawler
```
public ICrawler()
```
    Creates a new ICrawler instance.
  - ICrawler
```
public ICrawler(ICrawler o)
```
    Creates a new ICrawler instance by deeply copying a given ICrawler instance.
    
    Parameters:
    
    o - The instance to copy.
    
    Throws:
    
    java.lang.NullPointerException - if o is null.
- Method Detail
  - getRules
```
public java.util.List<Rules> getRules()
```
    Gets the value of the rules property.
    This accessor method returns a reference to the live list, not a snapshot. Therefore any modification you make to the returned list will be present inside the JAXB object. This is why there is not a set method for the rules property.
    For example, to add a new item, do as follows:
```
    getRules().add(newItem);
 
```
    Objects of the following type(s) are allowed in the list Rules
  - getCrawlSchedulerConfig
```
public CrawlSchedulerConfig getCrawlSchedulerConfig()
```
    Gets the value of the crawlSchedulerConfig property.
    
    Returns:
    
    possible object is CrawlSchedulerConfig
  - setCrawlSchedulerConfig
```
public void setCrawlSchedulerConfig(CrawlSchedulerConfig value)
```
    Sets the value of the crawlSchedulerConfig property.
    
    Parameters:
    
    value - allowed object is CrawlSchedulerConfig
  - getCustomCrawlConfig
```
public CustomCrawlConfig getCustomCrawlConfig()
```
    Gets the value of the customCrawlConfig property.
    
    Returns:
    
    possible object is CustomCrawlConfig
  - setCustomCrawlConfig
```
public void setCustomCrawlConfig(CustomCrawlConfig value)
```
    Sets the value of the customCrawlConfig property.
    
    Parameters:
    
    value - allowed object is CustomCrawlConfig
  - getMimeTypes
```
public ICrawler.MimeTypes getMimeTypes()
```
    Gets the value of the mimeTypes property.
    
    Returns:
    
    possible object is ICrawler.MimeTypes
  - setMimeTypes
```
public void setMimeTypes(ICrawler.MimeTypes value)
```
    Sets the value of the mimeTypes property.
    
    Parameters:
    
    value - allowed object is ICrawler.MimeTypes
  - getSessionIdBlacklist
```
public ICrawler.SessionIdBlacklist getSessionIdBlacklist()
```
    SessionId blacklist. These parameters are removed from URLs with a path or query part containing them.
    
    Returns:
    
    possible object is ICrawler.SessionIdBlacklist
  - setSessionIdBlacklist
```
public void setSessionIdBlacklist(ICrawler.SessionIdBlacklist value)
```
    Sets the value of the sessionIdBlacklist property.
    
    Parameters:
    
    value - allowed object is ICrawler.SessionIdBlacklist
  - getName
```
public java.lang.String getName()
```
    The crawler name. It must be unique across all crawlers. (default: null)
    
    Returns:
    
    possible object is String
  - setName
```
public void setName(java.lang.String value)
```
    Sets the value of the name property.
    
    Parameters:
    
    value - allowed object is String
  - getFetcher
```
public java.lang.String getFetcher()
```
    Which fetcher to use. (default: null)
    
    Returns:
    
    possible object is String
  - setFetcher
```
public void setFetcher(java.lang.String value)
```
    Sets the value of the fetcher property.
    
    Parameters:
    
    value - allowed object is String
  - getCrawlerServer
```
public java.lang.String getCrawlerServer()
```
    Crawler server hosting this crawler. See Deployment configuration. (default: null)
    
    Returns:
    
    possible object is String
  - setCrawlerServer
```
public void setCrawlerServer(java.lang.String value)
```
    Sets the value of the crawlerServer property.
    
    Parameters:
    
    value - allowed object is String
  - getBuildGroup
```
public java.lang.String getBuildGroup()
```
    Target build group. (default: null)
    
    Returns:
    
    possible object is String
  - setBuildGroup
```
public void setBuildGroup(java.lang.String value)
```
    Sets the value of the buildGroup property.
    
    Parameters:
    
    value - allowed object is String
  - isStoreTextOnly
```
public boolean isStoreTextOnly()
```
    Whether to store original binary documents, or only converted text. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setStoreTextOnly
```
public void setStoreTextOnly(java.lang.Boolean value)
```
    Sets the value of the storeTextOnly property.
    
    Parameters:
    
    value - allowed object is Boolean
  - getNthreads
```
public int getNthreads()
```
    The number of crawl threads which must be strictly positive. (default: 1)
    
    Returns:
    
    possible object is Integer
  - setNthreads
```
public void setNthreads(java.lang.Integer value)
```
    Sets the value of the nthreads property.
    
    Parameters:
    
    value - allowed object is Integer
  - isAggressive
```
public boolean isAggressive()
```
    Whether to enable aggressive crawl, that never sleeps between two requests to the same host. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setAggressive
```
public void setAggressive(java.lang.Boolean value)
```
    Sets the value of the aggressive property.
    
    Parameters:
    
    value - allowed object is Boolean
  - getThrottleTimeMS
```
public int getThrottleTimeMS()
```
    In the case of non-aggressive crawl, this defines the sleep interval between requests to the same host. (default: 2500)
    
    Returns:
    
    possible object is Integer
  - setThrottleTimeMS
```
public void setThrottleTimeMS(java.lang.Integer value)
```
    Sets the value of the throttleTimeMS property.
    
    Parameters:
    
    value - allowed object is Integer
  - isIgnoreRobotsTxt
```
public boolean isIgnoreRobotsTxt()
```
    Whether to ignore robots.txt rules. Not recommended. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setIgnoreRobotsTxt
```
public void setIgnoreRobotsTxt(java.lang.Boolean value)
```
    Sets the value of the ignoreRobotsTxt property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isEnableConvertProcessor
```
public boolean isEnableConvertProcessor()
```
    Whether to enable remoteconvert-based processor for links extracting in binary documents. (default: true)
    
    Returns:
    
    possible object is Boolean
  - setEnableConvertProcessor
```
public void setEnableConvertProcessor(java.lang.Boolean value)
```
    Sets the value of the enableConvertProcessor property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isNearDuplicateDetector
```
public boolean isNearDuplicateDetector()
```
    Whether to enable the near-duplicate content detector. (default: true)
    
    Returns:
    
    possible object is Boolean
  - setNearDuplicateDetector
```
public void setNearDuplicateDetector(java.lang.Boolean value)
```
    Sets the value of the nearDuplicateDetector property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isPatternsDetector
```
public boolean isPatternsDetector()
```
    Whether to enable patterns detection in pages. (default: true)
    
    Returns:
    
    possible object is Boolean
  - setPatternsDetector
```
public void setPatternsDetector(java.lang.Boolean value)
```
    Sets the value of the patternsDetector property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isCrawlSitemaps
```
public boolean isCrawlSitemaps()
```
    Whether to crawl sitemaps. (default: true)
    
    Returns:
    
    possible object is Boolean
  - setCrawlSitemaps
```
public void setCrawlSitemaps(java.lang.Boolean value)
```
    Sets the value of the crawlSitemaps property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isDisableConditionalGet
```
public boolean isDisableConditionalGet()
```
    Whether to always fetch documents, even if the server tells it has not changed. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setDisableConditionalGet
```
public void setDisableConditionalGet(java.lang.Boolean value)
```
    Sets the value of the disableConditionalGet property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isDefaultAccept
```
public boolean isDefaultAccept()
```
    Whether to crawl a url by default when it matches no other accept rule. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setDefaultAccept
```
public void setDefaultAccept(java.lang.Boolean value)
```
    Sets the value of the defaultAccept property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isDefaultIndex
```
public boolean isDefaultIndex()
```
    Whether to index by default when a url matches no index rule. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setDefaultIndex
```
public void setDefaultIndex(java.lang.Boolean value)
```
    Sets the value of the defaultIndex property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isDefaultFollow
```
public boolean isDefaultFollow()
```
    Whether to follow by default when a url matches no follow rule. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setDefaultFollow
```
public void setDefaultFollow(java.lang.Boolean value)
```
    Sets the value of the defaultFollow property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isDefaultFollowRoots
```
public boolean isDefaultFollowRoots()
```
    Whether to automatically follow root urls (default: true)
    
    Returns:
    
    possible object is Boolean
  - setDefaultFollowRoots
```
public void setDefaultFollowRoots(java.lang.Boolean value)
```
    Sets the value of the defaultFollowRoots property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isEnableSimpleSiteCollapsing
```
public boolean isEnableSimpleSiteCollapsing()
```
    Whether to generate a site ID suitable for document collapsing. (default: true)
    
    Returns:
    
    possible object is Boolean
  - setEnableSimpleSiteCollapsing
```
public void setEnableSimpleSiteCollapsing(java.lang.Boolean value)
```
    Sets the value of the enableSimpleSiteCollapsing property.
    
    Parameters:
    
    value - allowed object is Boolean
  - getSimpleSiteCollapsingDepth
```
public long getSimpleSiteCollapsingDepth()
```
    How many path segments to use to generate the site collapsing ID. (default: 0)
    
    Returns:
    
    possible object is Long
  - setSimpleSiteCollapsingDepth
```
public void setSimpleSiteCollapsingDepth(java.lang.Long value)
```
    Sets the value of the simpleSiteCollapsingDepth property.
    
    Parameters:
    
    value - allowed object is Long
  - getMimeTypesMode
```
public java.lang.String getMimeTypesMode()
```
    Mime types white/black list (default: exclude)
    
    Returns:
    
    possible object is String
  - setMimeTypesMode
```
public void setMimeTypesMode(java.lang.String value)
```
    Sets the value of the mimeTypesMode property.
    
    Parameters:
    
    value - allowed object is String
  - isIndexRedirectSources
```
public boolean isIndexRedirectSources()
```
    Whether to index redirections with target data. (default: true)
    
    Returns:
    
    possible object is Boolean
  - setIndexRedirectSources
```
public void setIndexRedirectSources(java.lang.Boolean value)
```
    Sets the value of the indexRedirectSources property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isSmartRefresh
```
public boolean isSmartRefresh()
```
    Whether to crawl a fraction of refreshed urls. (default: true)
    
    Returns:
    
    possible object is Boolean
  - setSmartRefresh
```
public void setSmartRefresh(java.lang.Boolean value)
```
    Sets the value of the smartRefresh property.
    
    Parameters:
    
    value - allowed object is Boolean
  - getSmartRefreshMinAgeS
```
public int getSmartRefreshMinAgeS()
```
    Age in seconds at which we may refresh old urls. (default: 3600)
    
    Returns:
    
    possible object is Integer
  - setSmartRefreshMinAgeS
```
public void setSmartRefreshMinAgeS(java.lang.Integer value)
```
    Sets the value of the smartRefreshMinAgeS property.
    
    Parameters:
    
    value - allowed object is Integer
  - getSmartRefreshMaxAgeS
```
public int getSmartRefreshMaxAgeS()
```
    Age in seconds at which we force the refresh of old urls. (default: 604800)
    
    Returns:
    
    possible object is Integer
  - setSmartRefreshMaxAgeS
```
public void setSmartRefreshMaxAgeS(java.lang.Integer value)
```
    Sets the value of the smartRefreshMaxAgeS property.
    
    Parameters:
    
    value - allowed object is Integer
  - withRules
```
public ICrawler withRules(Rules... values)
```
  - withRules
```
public ICrawler withRules(java.util.Collection<Rules> values)
```
  - withCrawlSchedulerConfig
```
public ICrawler withCrawlSchedulerConfig(CrawlSchedulerConfig value)
```
  - withCustomCrawlConfig
```
public ICrawler withCustomCrawlConfig(CustomCrawlConfig value)
```
  - withMimeTypes
```
public ICrawler withMimeTypes(ICrawler.MimeTypes value)
```
  - withSessionIdBlacklist
```
public ICrawler withSessionIdBlacklist(ICrawler.SessionIdBlacklist value)
```
  - withName
```
public ICrawler withName(java.lang.String value)
```
  - withFetcher
```
public ICrawler withFetcher(java.lang.String value)
```
  - withCrawlerServer
```
public ICrawler withCrawlerServer(java.lang.String value)
```
  - withBuildGroup
```
public ICrawler withBuildGroup(java.lang.String value)
```
  - withStoreTextOnly
```
public ICrawler withStoreTextOnly(java.lang.Boolean value)
```
  - withNthreads
```
public ICrawler withNthreads(java.lang.Integer value)
```
  - withAggressive
```
public ICrawler withAggressive(java.lang.Boolean value)
```
  - withThrottleTimeMS
```
public ICrawler withThrottleTimeMS(java.lang.Integer value)
```
  - withIgnoreRobotsTxt
```
public ICrawler withIgnoreRobotsTxt(java.lang.Boolean value)
```
  - withEnableConvertProcessor
```
public ICrawler withEnableConvertProcessor(java.lang.Boolean value)
```
  - withNearDuplicateDetector
```
public ICrawler withNearDuplicateDetector(java.lang.Boolean value)
```
  - withPatternsDetector
```
public ICrawler withPatternsDetector(java.lang.Boolean value)
```
  - withCrawlSitemaps
```
public ICrawler withCrawlSitemaps(java.lang.Boolean value)
```
  - withDisableConditionalGet
```
public ICrawler withDisableConditionalGet(java.lang.Boolean value)
```
  - withDefaultAccept
```
public ICrawler withDefaultAccept(java.lang.Boolean value)
```
  - withDefaultIndex
```
public ICrawler withDefaultIndex(java.lang.Boolean value)
```
  - withDefaultFollow
```
public ICrawler withDefaultFollow(java.lang.Boolean value)
```
  - withDefaultFollowRoots
```
public ICrawler withDefaultFollowRoots(java.lang.Boolean value)
```
  - withEnableSimpleSiteCollapsing
```
public ICrawler withEnableSimpleSiteCollapsing(java.lang.Boolean value)
```
  - withSimpleSiteCollapsingDepth
```
public ICrawler withSimpleSiteCollapsingDepth(java.lang.Long value)
```
  - withMimeTypesMode
```
public ICrawler withMimeTypesMode(java.lang.String value)
```
  - withIndexRedirectSources
```
public ICrawler withIndexRedirectSources(java.lang.Boolean value)
```
  - withSmartRefresh
```
public ICrawler withSmartRefresh(java.lang.Boolean value)
```
  - withSmartRefreshMinAgeS
```
public ICrawler withSmartRefreshMinAgeS(java.lang.Integer value)
```
  - withSmartRefreshMaxAgeS
```
public ICrawler withSmartRefreshMaxAgeS(java.lang.Integer value)
```
  - clone
```
public ICrawler clone()
```
    Creates and returns a deep copy of this object.
    
    Overrides:
    
    clone in class java.lang.Object
    
    Returns:
    
    A deep copy of this object.

Class ICrawler

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

rules

crawlSchedulerConfig

customCrawlConfig

mimeTypes

sessionIdBlacklist

name

fetcher

crawlerServer

buildGroup

storeTextOnly

nthreads

aggressive

throttleTimeMS

ignoreRobotsTxt

enableConvertProcessor

nearDuplicateDetector

patternsDetector

crawlSitemaps

disableConditionalGet

defaultAccept

defaultIndex

defaultFollow

defaultFollowRoots

enableSimpleSiteCollapsing

simpleSiteCollapsingDepth

mimeTypesMode

indexRedirectSources

smartRefresh

smartRefreshMinAgeS

smartRefreshMaxAgeS

Constructor Detail

ICrawler

ICrawler

Method Detail

getRules

getCrawlSchedulerConfig

setCrawlSchedulerConfig

getCustomCrawlConfig

setCustomCrawlConfig

getMimeTypes

setMimeTypes

getSessionIdBlacklist

setSessionIdBlacklist

getName

setName

getFetcher

setFetcher

getCrawlerServer

setCrawlerServer

getBuildGroup

setBuildGroup

isStoreTextOnly

setStoreTextOnly

getNthreads

setNthreads

isAggressive

setAggressive

getThrottleTimeMS

setThrottleTimeMS

isIgnoreRobotsTxt

setIgnoreRobotsTxt

isEnableConvertProcessor

setEnableConvertProcessor

isNearDuplicateDetector

setNearDuplicateDetector

isPatternsDetector

setPatternsDetector

isCrawlSitemaps

setCrawlSitemaps

isDisableConditionalGet

setDisableConditionalGet

isDefaultAccept

setDefaultAccept