Class: UrlStore
| Defined in: | src/kermit/QueueSystem.coffee |
| Inherits: | Mixin |
Overview
Manage the collection of URLs that are scheduled => to be processed in the future processing => being processed in form of a RequestItem visited => completed processing (RequestItem reached phase COMPLETE)
Instance Method Summary
- # (void) processing(url, phase, meta) Private Transition the given URL from 'scheduled' to 'processing'.
- # (void) count(phase) Returns the number of URLs in given phase
- # (void) reschedule(url)
- # (void) schedule(url, meta) Add the given URL to the collection of scheduled URLs
- # (void) visited(url) Mark a known URL as visited (silently ignores cases of unknown URLs)
- # (void) scheduled(size = 100) Retrieve the next batch of scheduled URLs
- # (void) save()
Constructor Details
#
(void)
constructor(options)
Create a new URL manager
Instance Method Details
#
(void)
processing(url, phase, meta)
Private
Transition the given URL from 'scheduled' to 'processing'. Inserts a new entry if no scheduled URL has been found.
#
(void)
count(phase)
Returns the number of URLs in given phase
#
(void)
reschedule(url)
#
(void)
schedule(url, meta)
Add the given URL to the collection of scheduled URLs
#
(void)
visited(url)
Mark a known URL as visited (silently ignores cases of unknown URLs)
#
(void)
scheduled(size = 100)
Retrieve the next batch of scheduled URLs
#
(void)
save()