fetch-site
v1.8.2
Published
Fetch website with all the resources/responses/requests to local files, using puppeteer.
Downloads
36
Maintainers
Readme
fetch-site
Fetch website with all the resources/responses/requests to local files, using puppeteer.
Install
CLI
npm install -g fetch-site
Module
npm install --save fetch-site
Usage
CLI Usage
$ fetch-site url [options]
Options
--version Show version info
--help Show help info
--no-headless, -h Set {headless: false} for 'launch-option'
--dir, -d Dir to save result to
--shot, -s Filename to save a screenshot after page open
--user-agent, -u Set userAgent, string
--viewport, -v Set viewport, e.g. `'{width:1024, height: 768}'`
--timeout, -t Set maximum navigation time in milliseconds
--cookies, -c Set cookies
--wait-for, -w Wait for milliseconds/selector/function/closed(true)
--index-file Default name of index file, like index.html
--on-response onResponse event, function(response) as string
--launch-option, -l Launch option passed into puppeteer, object as string
--open-option, -o Open option to passed into page, object as string
--on-before-open Before open page event, function(page) as string
--on-after-open After open page event, function(page) as string
--on-finish Finish fetch event, function(page) as string
Examples
$ fetch-site http://baidu.com -h -w -o '{waitUntil:"networkidle0"}' -u 'My-UA-String' -t 0
Above example will open the url with {headless:false}
, wait until networkidle0
, and wait for page close to exit, using UAString "My-UA-String"
.
Module Usage
const fetchSite = require('fetch-site')
fetchSite({
url: 'http://www.baidu.com',
// whether to save a screenshot
shot: 'shot.png',
dir: 'baidu.com',
launchOption:{
headless: false
},
openOption:{
timeout: 100*1e3,
waitUntil: 'networkidle0'
},
onResponse: async response=>{
if(/his/.test(response.url)) return false
// return false will skip the resource
},
onBeforeOpen: async page=>{
page.setViewport({width: 1440, height: 1000})
},
onAfterOpen: async page=>{
await new Promise(r=>setTimeout(r, 5000))
},
onFinish: async page=>console.log('ok')
})
Then check the folder baidu.com
for results.