ICrawler (NETVIBES Exalead CloudView Clients SDK)

java.lang.Object
- com.exalead.mercury.mami.crawl.v10.ICrawler

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable

Direct Known Subclasses:

Crawler, FeedFetcher
```
public abstract class ICrawler
extends java.lang.Object
implements java.io.Serializable, java.lang.Cloneable
```
See Also:

Serialized Form

Field Summary

Fields
Modifier and Type	Field and Description
`protected java.lang.Boolean`	`aggressive`
`protected java.lang.String`	`buildGroup`
`protected java.lang.String`	`crawlerServer`
`protected java.lang.Boolean`	`crawlSitemaps`
`protected java.lang.String`	`defaultFetcher`
`protected java.lang.Boolean`	`enableConvertProcessor`
`protected java.lang.Boolean`	`ignoreRobotsTxt`
`protected java.lang.String`	`name`
`protected java.lang.Boolean`	`nearDuplicateDetector`
`protected java.lang.Integer`	`nthreads`
`protected java.lang.Boolean`	`patternsDetector`
`protected java.lang.Integer`	`throttleTimeMS`

Constructor Summary

Constructors
Constructor and Description

ICrawler()
Creates a new ICrawler instance.

ICrawler(ICrawler o)
Creates a new ICrawler instance by deeply copying a given ICrawler instance.

Constructors
Constructor and Description
`ICrawler()` Creates a new `ICrawler` instance.
`ICrawler(ICrawler o)` Creates a new `ICrawler` instance by deeply copying a given `ICrawler` instance.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`ICrawler`	`clone()` Creates and returns a deep copy of this object.
`java.lang.String`	`getBuildGroup()` Target build group.
`java.lang.String`	`getCrawlerServer()` Crawler server hosting this crawler.
`java.lang.String`	`getDefaultFetcher()` Which fetcher to use with urls that belong to no source.
`java.lang.String`	`getName()` The crawler name.
`int`	`getNthreads()` The number of crawl threads which must be strictly positive.
`int`	`getThrottleTimeMS()` In the case of non-aggressive crawl, this defines the sleep interval between requests to the same host.
`boolean`	`isAggressive()` Whether to activate aggressive crawl, that never sleeps between two requests to the same host.
`boolean`	`isCrawlSitemaps()` Whether to crawl sitemaps.
`boolean`	`isEnableConvertProcessor()` Whether to enable remoteconvert-based processor for links extracting in binary documents.
`boolean`	`isIgnoreRobotsTxt()` Whether to ignore robots.txt rules.
`boolean`	`isNearDuplicateDetector()` Whether to enable near-duplicate content detector.
`boolean`	`isPatternsDetector()` Whether to enable patterns detection in pages.
`void`	`setAggressive(java.lang.Boolean value)` Sets the value of the aggressive property.
`void`	`setBuildGroup(java.lang.String value)` Sets the value of the buildGroup property.
`void`	`setCrawlerServer(java.lang.String value)` Sets the value of the crawlerServer property.
`void`	`setCrawlSitemaps(java.lang.Boolean value)` Sets the value of the crawlSitemaps property.
`void`	`setDefaultFetcher(java.lang.String value)` Sets the value of the defaultFetcher property.
`void`	`setEnableConvertProcessor(java.lang.Boolean value)` Sets the value of the enableConvertProcessor property.
`void`	`setIgnoreRobotsTxt(java.lang.Boolean value)` Sets the value of the ignoreRobotsTxt property.
`void`	`setName(java.lang.String value)` Sets the value of the name property.
`void`	`setNearDuplicateDetector(java.lang.Boolean value)` Sets the value of the nearDuplicateDetector property.
`void`	`setNthreads(java.lang.Integer value)` Sets the value of the nthreads property.
`void`	`setPatternsDetector(java.lang.Boolean value)` Sets the value of the patternsDetector property.
`void`	`setThrottleTimeMS(java.lang.Integer value)` Sets the value of the throttleTimeMS property.
`ICrawler`	`withAggressive(java.lang.Boolean value)`
`ICrawler`	`withBuildGroup(java.lang.String value)`
`ICrawler`	`withCrawlerServer(java.lang.String value)`
`ICrawler`	`withCrawlSitemaps(java.lang.Boolean value)`
`ICrawler`	`withDefaultFetcher(java.lang.String value)`
`ICrawler`	`withEnableConvertProcessor(java.lang.Boolean value)`
`ICrawler`	`withIgnoreRobotsTxt(java.lang.Boolean value)`
`ICrawler`	`withName(java.lang.String value)`
`ICrawler`	`withNearDuplicateDetector(java.lang.Boolean value)`
`ICrawler`	`withNthreads(java.lang.Integer value)`
`ICrawler`	`withPatternsDetector(java.lang.Boolean value)`
`ICrawler`	`withThrottleTimeMS(java.lang.Integer value)`

Methods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - name
```
protected java.lang.String name
```
  - crawlerServer
```
protected java.lang.String crawlerServer
```
  - buildGroup
```
protected java.lang.String buildGroup
```
  - nthreads
```
protected java.lang.Integer nthreads
```
  - aggressive
```
protected java.lang.Boolean aggressive
```
  - throttleTimeMS
```
protected java.lang.Integer throttleTimeMS
```
  - ignoreRobotsTxt
```
protected java.lang.Boolean ignoreRobotsTxt
```
  - enableConvertProcessor
```
protected java.lang.Boolean enableConvertProcessor
```
  - nearDuplicateDetector
```
protected java.lang.Boolean nearDuplicateDetector
```
  - patternsDetector
```
protected java.lang.Boolean patternsDetector
```
  - defaultFetcher
```
protected java.lang.String defaultFetcher
```
  - crawlSitemaps
```
protected java.lang.Boolean crawlSitemaps
```
- Constructor Detail
  - ICrawler
```
public ICrawler()
```
    Creates a new ICrawler instance.
  - ICrawler
```
public ICrawler(ICrawler o)
```
    Creates a new ICrawler instance by deeply copying a given ICrawler instance.
    
    Parameters:
    
    o - The instance to copy.
    
    Throws:
    
    java.lang.NullPointerException - if o is null.
- Method Detail
  - getName
```
public java.lang.String getName()
```
    The crawler name. It must be unique across all crawlers. (default: null)
    
    Returns:
    
    possible object is String
  - setName
```
public void setName(java.lang.String value)
```
    Sets the value of the name property.
    
    Parameters:
    
    value - allowed object is String
  - getCrawlerServer
```
public java.lang.String getCrawlerServer()
```
    Crawler server hosting this crawler. See Deployment configuration. (default: null)
    
    Returns:
    
    possible object is String
  - setCrawlerServer
```
public void setCrawlerServer(java.lang.String value)
```
    Sets the value of the crawlerServer property.
    
    Parameters:
    
    value - allowed object is String
  - getBuildGroup
```
public java.lang.String getBuildGroup()
```
    Target build group. (default: null)
    
    Returns:
    
    possible object is String
  - setBuildGroup
```
public void setBuildGroup(java.lang.String value)
```
    Sets the value of the buildGroup property.
    
    Parameters:
    
    value - allowed object is String
  - getNthreads
```
public int getNthreads()
```
    The number of crawl threads which must be strictly positive. (default: 1)
    
    Returns:
    
    possible object is Integer
  - setNthreads
```
public void setNthreads(java.lang.Integer value)
```
    Sets the value of the nthreads property.
    
    Parameters:
    
    value - allowed object is Integer
  - isAggressive
```
public boolean isAggressive()
```
    Whether to activate aggressive crawl, that never sleeps between two requests to the same host. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setAggressive
```
public void setAggressive(java.lang.Boolean value)
```
    Sets the value of the aggressive property.
    
    Parameters:
    
    value - allowed object is Boolean
  - getThrottleTimeMS
```
public int getThrottleTimeMS()
```
    In the case of non-aggressive crawl, this defines the sleep interval between requests to the same host. (default: 2500)
    
    Returns:
    
    possible object is Integer
  - setThrottleTimeMS
```
public void setThrottleTimeMS(java.lang.Integer value)
```
    Sets the value of the throttleTimeMS property.
    
    Parameters:
    
    value - allowed object is Integer
  - isIgnoreRobotsTxt
```
public boolean isIgnoreRobotsTxt()
```
    Whether to ignore robots.txt rules. Not recommended. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setIgnoreRobotsTxt
```
public void setIgnoreRobotsTxt(java.lang.Boolean value)
```
    Sets the value of the ignoreRobotsTxt property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isEnableConvertProcessor
```
public boolean isEnableConvertProcessor()
```
    Whether to enable remoteconvert-based processor for links extracting in binary documents. (default: true)
    
    Returns:
    
    possible object is Boolean
  - setEnableConvertProcessor
```
public void setEnableConvertProcessor(java.lang.Boolean value)
```
    Sets the value of the enableConvertProcessor property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isNearDuplicateDetector
```
public boolean isNearDuplicateDetector()
```
    Whether to enable near-duplicate content detector. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setNearDuplicateDetector
```
public void setNearDuplicateDetector(java.lang.Boolean value)
```
    Sets the value of the nearDuplicateDetector property.
    
    Parameters:
    
    value - allowed object is Boolean
  - isPatternsDetector
```
public boolean isPatternsDetector()
```
    Whether to enable patterns detection in pages. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setPatternsDetector
```
public void setPatternsDetector(java.lang.Boolean value)
```
    Sets the value of the patternsDetector property.
    
    Parameters:
    
    value - allowed object is Boolean
  - getDefaultFetcher
```
public java.lang.String getDefaultFetcher()
```
    Which fetcher to use with urls that belong to no source. (default: null)
    
    Returns:
    
    possible object is String
  - setDefaultFetcher
```
public void setDefaultFetcher(java.lang.String value)
```
    Sets the value of the defaultFetcher property.
    
    Parameters:
    
    value - allowed object is String
  - isCrawlSitemaps
```
public boolean isCrawlSitemaps()
```
    Whether to crawl sitemaps. Warning: urls crawled through sitemaps lose their forced metas. (default: false)
    
    Returns:
    
    possible object is Boolean
  - setCrawlSitemaps
```
public void setCrawlSitemaps(java.lang.Boolean value)
```
    Sets the value of the crawlSitemaps property.
    
    Parameters:
    
    value - allowed object is Boolean
  - withName
```
public ICrawler withName(java.lang.String value)
```
  - withCrawlerServer
```
public ICrawler withCrawlerServer(java.lang.String value)
```
  - withBuildGroup
```
public ICrawler withBuildGroup(java.lang.String value)
```
  - withNthreads
```
public ICrawler withNthreads(java.lang.Integer value)
```
  - withAggressive
```
public ICrawler withAggressive(java.lang.Boolean value)
```
  - withThrottleTimeMS
```
public ICrawler withThrottleTimeMS(java.lang.Integer value)
```
  - withIgnoreRobotsTxt
```
public ICrawler withIgnoreRobotsTxt(java.lang.Boolean value)
```
  - withEnableConvertProcessor
```
public ICrawler withEnableConvertProcessor(java.lang.Boolean value)
```
  - withNearDuplicateDetector
```
public ICrawler withNearDuplicateDetector(java.lang.Boolean value)
```
  - withPatternsDetector
```
public ICrawler withPatternsDetector(java.lang.Boolean value)
```
  - withDefaultFetcher
```
public ICrawler withDefaultFetcher(java.lang.String value)
```
  - withCrawlSitemaps
```
public ICrawler withCrawlSitemaps(java.lang.Boolean value)
```
  - clone
```
public ICrawler clone()
```
    Creates and returns a deep copy of this object.
    
    Overrides:
    
    clone in class java.lang.Object
    
    Returns:
    
    A deep copy of this object.

Class ICrawler

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

name

crawlerServer

buildGroup

nthreads

aggressive

throttleTimeMS

ignoreRobotsTxt

enableConvertProcessor

nearDuplicateDetector

patternsDetector

defaultFetcher

crawlSitemaps

Constructor Detail

ICrawler

ICrawler

Method Detail

getName

setName

getCrawlerServer

setCrawlerServer

getBuildGroup

setBuildGroup

getNthreads

setNthreads

isAggressive

setAggressive

getThrottleTimeMS

setThrottleTimeMS

isIgnoreRobotsTxt

setIgnoreRobotsTxt

isEnableConvertProcessor

setEnableConvertProcessor

isNearDuplicateDetector

setNearDuplicateDetector

isPatternsDetector

setPatternsDetector

getDefaultFetcher

setDefaultFetcher

isCrawlSitemaps

setCrawlSitemaps

withName

withCrawlerServer

withBuildGroup

withNthreads

withAggressive

withThrottleTimeMS

withIgnoreRobotsTxt

withEnableConvertProcessor

withNearDuplicateDetector

withPatternsDetector

withDefaultFetcher

withCrawlSitemaps

clone