A NodeJS crawler library to quick and easy build versatile crawlers. Just to make working with request and cheerio a little easier and to not have to write all the standard stuff over and over again.

Functions

• Play nice with servers: Wait between each request.
• Get ´next´ and ´last´ URL for pagination scenario.
• Write list syncronusly to file at the end
• Serving header info

Examples

• List crawling: Crawl paginated lists for URLs

Functionality to be

• [ ] Item crawling
• [ ] Pagination iteration, second version
• [ ] Define which domain(s) to crawl
• [ ] Site-crawl - Add found URLs to crawl queue
• [ ] Write content asyncronusly (add to file) throughout crawling.
• [ ] Follow robots.txt
• [ ] Check if new content
• [ ] Check if updated content
• [ ] Overwrite crawler header and set ´from´-field.
• [ ] Crawl with headless browser.