Web fetch

Asset Crawler for common Web pages

Usage

As a Node.js package

import { savePage } from 'web-fetch';

savePage({
    source: 'http://URL.to/one/of/your/old/posts/'
});

Docker example

https://github.com/kaiyuanshe/KYS-service

As a Command Line tool

web-fetch http://URL.to/one/of/your/old/posts/

In GitHub actions

name: Fetch Web pages
on:
    issues:
        types:
            - opened
jobs:
    Fetch-and-Save:
        runs-on: ubuntu-latest
        permissions:
            contents: write
            issues: write
        steps:
            - uses: actions/checkout@v4

            - uses: pnpm/action-setup@v4
              with:
                  version: 9
            - uses: actions/setup-node@v4
              with:
                  node-version: 20
                  cache: pnpm
            - uses: browser-actions/setup-chrome@v1

            - name: Setup Web-fetch
              run: pnpm i web-fetch -g

            - name: Fetch first URL in Issue Body
              run: web-fetch $(echo "${{ github.event.issue.Body }}" | grep -Eo "https?://\S+")

            - uses: stefanzweifel/git-auto-commit-action@v5
              with:
                  commit_message: '${{ github.event.issue.title }}'

Supported Structure

https://github.com/TechQuery/Web-fetch/blob/d4c574adb6ed4b3030995f4ea26911e88d00c1fc/source/config.ts#L10-L29

Renderer

1. Puppeteer (default)

2. JSDOM

Wrapper

1. Hexo plugin