Class: RequestItem

Defined in:

src/kermit/RequestItem.coffee

Overview

The RequestItem is the central object in the processing of a single URL. The Crawler will funnel each item through the different ProcessingPhases - applying all Extensions registered for the particular phase.

During its lifecycle the item is enriched with listeners and properties by the Extensions that take care of its processing.

(Meta-)Information necessary for its processing is usually attached to the item in order to centralize state.

Instance Method Summary

# (RequestItem) onChange(property, listener)
Register a listener {Function} to be invoked whenever the specified property value is changed changes.
# (String) url(url) Get the string representation of the uri
# (String) id()
# (void) useSSL() Check whether https should be used to fetch this item
# (String) phase(phase) Private Change the phase and notify subscribed listeners or retrieve the current phase value
# (void) stamp(tag) Add a new timestamp to the collection of timestamps for the given tag.
# (void) stamps(tag) Get all timestamps stored for the given tag
# (Number) durationOf(phase) Compute the duration of a phase
# (void) timeToComplete() Calculate processing time from INITIAL to COMPLETE
# (RequestItem) onPhase(phase, listener) Register a change listener for a specific value of the phase property
# (RequestItem) spool() Change the items phase to SPOOLED
# (RequestItem) ready() Change the items phase to READY
# (RequestItem) fetching() Change the items phase to FETCHING
# (RequestItem) fetched() Bound Change the items phase to FETCHED
# (RequestItem) complete() Change the items phase to COMPLETE
# (RequestItem) error(error) Change the items phase to ERROR
# (RequestItem) cancel() Change the items phase to CANCELED
# (Boolean) isInitial() Check whether this item has phase INITIAL
# (Boolean) isSpooled() Check whether this item has phase SPOOLED
# (Boolean) isReady() Check whether this item has phase READY
# (Boolean) isFetching() Check whether this item has phase FETCHING
# (Boolean) isFetched() Check whether this item has phase FETCHED
# (Boolean) isComplete() Check whether this item has phase COMPLETE
# (Boolean) isCanceled() Check whether this item has phase CANCELED
# (Boolean) isError() Check whether this item has phase ERROR
# (void) cleanup() Clean all item data that potentially occupies much memory
# (void) pipeline() Access the Pipeline of this item
# (Number) parents() A item might have been created by another item (its parent).
# (void) toString() Generate a human readable representation of this item

Constructor Details

# (void) constructor(url, meta, log)

Create a new item for the given url and with the given metadata attached

Instance Method Details

# (RequestItem) onChange(property, listener)

Register a listener {Function} to be invoked whenever the specified property value is changed changes. The post-change state of the item will be passed to the handler

Parameters:

property ( String ) — The name of the property to watch
listener ( Function ) — The handler to be invoked whenever the property

Returns:

( RequestItem ) — This item

# (String) url(url)

Get the string representation of the uri

Returns:

( String ) — The URI as string

# (String) id()

Returns:

( String ) — The synthetic id of this item

# (void) useSSL()

Check whether https should be used to fetch this item

# (String) phase(phase) Private

Change the phase and notify subscribed listeners or retrieve the current phase value

Parameters:

phase ( String ) — The phase value to set

Returns:

( String ) — The current value of phase

# (void) stamp(tag)

Add a new timestamp to the collection of timestamps for the given tag. Timestamps are useful to keep track of processing durations.

# (void) stamps(tag)

Get all timestamps stored for the given tag

# (Number) durationOf(phase)

Compute the duration of a phase

Returns:

( Number ) — The duration of the respective phase in ms or -1 if phase not completed

# (void) timeToComplete()

Calculate processing time from INITIAL to COMPLETE

# (RequestItem) onPhase(phase, listener)

Parameters:

phase ( String ) — The phase value that will trigger invocation of the listener
listener ( Function ) — The listener to be invoked if phase changes

Returns:

( RequestItem ) — This item

# (RequestItem) spool()

Change the items phase to SPOOLED

Throws:

( ? ) — Error if item does have other phase than INITIAL

Returns:

( RequestItem ) — This item

# (RequestItem) ready()

Change the items phase to READY

Throws:

( ? ) — Error if item does have other phase than SPOOLED

Returns:

( RequestItem ) — This item

# (RequestItem) fetching()

Change the items phase to FETCHING

Throws:

( ? ) — Error if item does have other phase than READY

Returns:

( RequestItem ) — This item

# (RequestItem) fetched() Bound

Change the items phase to FETCHED

Throws:

( ? ) — Error if item item does have other phase than FETCHING

Returns:

( RequestItem ) — This item

# (RequestItem) complete()

Change the items phase to COMPLETE

Throws:

( ? ) — Error if item does have other phase than FETCHED

Returns:

( RequestItem ) — This item

# (RequestItem) error(error)

Change the items phase to ERROR

Returns:

( RequestItem ) — This item

# (RequestItem) cancel()

Change the items phase to CANCELED

Returns:

( RequestItem ) — This item

# (Boolean) isInitial()

Check whether this item has phase INITIAL

Returns:

( Boolean ) — True if phase is INITIAL, false otherwise

# (Boolean) isSpooled()

Check whether this item has phase SPOOLED

Returns:

( Boolean ) — True if phase is SPOOLED, false otherwise

# (Boolean) isReady()

Check whether this item has phase READY

Returns:

( Boolean ) — True if phase is READY, false otherwise

# (Boolean) isFetching()

Check whether this item has phase FETCHING

Returns:

( Boolean ) — True if phase is FETCHING, false otherwise

# (Boolean) isFetched()

Check whether this item has phase FETCHED

Returns:

( Boolean ) — True if phase is FETCHED, false otherwise

# (Boolean) isComplete()

Check whether this item has phase COMPLETE

Returns:

( Boolean ) — True if phase is COMPLETE, false otherwise

# (Boolean) isCanceled()

Check whether this item has phase CANCELED

Returns:

( Boolean ) — True if phase is CANCELED, false otherwise

# (Boolean) isError()

Check whether this item has phase ERROR

Returns:

( Boolean ) — True if phase is ERROR, false otherwise

# (void) cleanup()

Clean all item data that potentially occupies much memory

# (void) pipeline()

Access the Pipeline of this item

# (Number) parents()

A item might have been created by another item (its parent). That parent might in turn have been created by another item and so on.

Returns:

( Number ) — The number of parents of this item

# (void) toString()

Generate a human readable representation of this item