Class: QueueManager
| Defined in: | src/kermit/QueueManager.coffee |
Overview
Provides access to a queue like system that allows to access RequestItems using lokijs query interface. Queues are emulated with dynamic views on a single item collection.
Instance Method Summary
- # (void) initialize() Initialize this queue manager
- # (Object) itemCountByPhase(phases = Phase.ALL, result = {}) Count the number of items per phase number of items in that phase
- # (void) insert(item) Insert a item into the queue
- # (void) updateUrl(url, phase, meta) Private Update the url collection such that it reflects the current status of the given URL
- # (void) scheduleUrl(url, meta) Add a url as scheduled
- # (void) nextUrlBatch(size = 100) Retrieve the next batch of scheduled URLs (FIFO ordered)
- # (void) update(item) Update a known item
- # (Boolean) hasUrl(url, phase) Check whether the given url has already been processed or is on its way to being processed
- # (void) isVisited(url) Whether the given url has already been visited
- # (void) isScheduled(url) Whether the given url is already scheduled
- # (void) isProcessing(url) Whether the given url is currently processing
- # (void) isKnown(url) Whether the given url is known, i.e.
- # (void) completed(item) Handle a item that successfully completed processing (run cleanup and remember the url as successfully processed).
- # (void) itemsWaiting() Retrieve a set of all items with phases defined as "WAITING"
- # (void) hasItemsWaiting() Determines whether there are items left for Spooling
- # (void) itemsProcessing(pattern) Retrieve
- # (void) initial() Get all RequestItems with phase INITIAL
- # (Array<RequestItem.state>) spooled(batchSize = 20) Retrieve the next batch of SPOOLED items
Constructor Details
#
(void)
constructor(file)
Construct a new QueueManager with its own data file
Instance Method Details
#
(void)
initialize()
Initialize this queue manager
#
(Object)
itemCountByPhase(phases = Phase.ALL, result = {})
Count the number of items per phase number of items in that phase
#
(void)
insert(item)
Insert a item into the queue
#
(void)
updateUrl(url, phase, meta)
Private
Update the url collection such that it reflects the current status of the given URL
#
(void)
scheduleUrl(url, meta)
Add a url as scheduled
#
(void)
nextUrlBatch(size = 100)
Retrieve the next batch of scheduled URLs (FIFO ordered)
#
(void)
update(item)
Update a known item
#
(Boolean)
hasUrl(url, phase)
Check whether the given url has already been processed or is on its way to being processed
#
(void)
isVisited(url)
Whether the given url has already been visited
#
(void)
isScheduled(url)
Whether the given url is already scheduled
#
(void)
isProcessing(url)
Whether the given url is currently processing
#
(void)
isKnown(url)
Whether the given url is known, i.e. one of [scheduled|processing|visited]
#
(void)
completed(item)
Handle a item that successfully completed processing (run cleanup and remember the url as successfully processed).
#
(void)
itemsWaiting()
Retrieve a set of all items with phases defined as "WAITING"
#
(void)
hasItemsWaiting()
Determines whether there are items left for Spooling
#
(void)
itemsProcessing(pattern)
Retrieve
#
(void)
initial()
Get all RequestItems with phase INITIAL
#
(Array<RequestItem.state>)
spooled(batchSize = 20)
Retrieve the next batch of SPOOLED items