Live remote crawling

Jump to bottom

Benjamin Ooghe-Tabanou edited this page Dec 21, 2012 · 1 revision

live crawling
the URLs candidates found in the outbound links to be submited to researchers for review are asynchronously sent back to UI while crawling.
The researchers work on review and qualification while the crawl is still on-going without having to wait for the end of the iteration
remote crawling
UI and crawlers modules can be hosted on distant machines on a network.