Web Metadata Scraper

A TypeScript-based tool for scraping and analyzing metadata from web pages. It can extract various types of metadata including basic meta tags, Open Graph tags, Twitter Card data, and structured data.

Features

• Extract comprehensive metadata from web pages
• Support for custom HTTP headers
• HTML report generation
• Website comparison functionality
• Configurable through command line or config files

Installation

npm install web-metadata-scraper

Usage

Basic Usage

1. Scrape metadata and output to console:

npm start -- https://example.com

2. Use custom headers:

npm start -- https://example.com '{"Authorization": "Bearer token"}'

3. Generate HTML report:

npm start -- https://example.com --html ./reports/report.html --open

Programmatic Usage

import { scrapeMetadata } from "web-metadata-scraper";

// Basic usage
const metadata = await scrapeMetadata("https://example.com");

// With custom headers
const metadata = await scrapeMetadata("https://example.com", {
  Authorization: "Bearer token",
  "User-Agent": "Custom Bot 1.0",
});

// With custom options
const metadata = await scrapeMetadata(
  "https://example.com",
  {},
  {
    timeout: 5000,
    followRedirect: true,
  },
);

// Access specific metadata
console.log(metadata.basic.title); // Page title
console.log(metadata.opengraph.image); // OG image URL
console.log(metadata.scripts.gtm.ids); // Google Tag Manager IDs
console.log(metadata.structured); // JSON-LD data

Comparison Mode

Compare metadata between two websites:

1. Using command line:

npm start -- --compare https://site1.com https://site2.com --html ./reports/comparison.html --open

2. Using config file (config.json):

{
  "urls": ["https://site1.com", "https://site2.com"],
  "headers": {
    "Authorization": "Bearer token"
  },
  "output": {
    "html": "./reports/comparison.html",
    "openHtml": true
  }
}

Then run:

npm start -- --config config.json

API Reference

Main Functions

scrapeMetadata(url, headers?, options?)

Scrapes metadata from a given URL.

• url: string - The URL to scrape
• headers: object (optional) - Custom HTTP headers
• options: object (optional) - Request options
• Returns: Promise

interface Metadata {
  basic: {
    title: string;
    description: string;
    keywords: string[];
    // ...
  };
  opengraph: {
    title: string;
    description: string;
    image: string;
    // ...
  };
  twitter: {
    card: string;
    title: string;
    description: string;
    // ...
  };
  scripts: {
    gtm: {
      ids: string[];
      implementation: string;
    };
    analytics: {
      ga: string[];
      gtag: string[];
      ua: string[];
      ga4: string[];
    };
    other: string[];
  };
  structured: any[];
}

Advanced Examples

1. Extract only Google Analytics information:

const { scripts } = await scrapeMetadata("https://example.com");
const { ga4, ua } = scripts.analytics;
console.log("GA4 IDs:", ga4);
console.log("Universal Analytics IDs:", ua);

2. Extract all JSON-LD data:

const { structured } = await scrapeMetadata("https://example.com");
const jsonLdData = structured.filter((item) =>
  item["@context"]?.includes("schema.org"),
);

3. Custom error handling:

try {
  const metadata = await scrapeMetadata(
    "https://example.com",
    {},
    {
      timeout: 3000,
      followRedirect: false,
    },
  );
} catch (error) {
  if (error.message.includes("timeout")) {
    console.error("Request timed out");
  } else if (error.message.includes("Invalid URL")) {
    console.error("Invalid URL provided");
  } else {
    console.error("Failed to fetch metadata:", error);
  }
}

Configuration

You can provide configuration either through command line arguments or a config file.

Command Line Arguments

• <url> - URL to scrape
• --compare <url1> <url2> - Compare two URLs
• --html <path> - Generate HTML report
• --open - Open HTML report after generation
• --config <path> - Use config file

Config File Format

{
  // Single URL mode
  "url": "https://example.com",

  // OR Comparison mode
  "urls": ["https://site1.com", "https://site2.com"],

  // Optional fields
  "headers": {
    "Authorization": "Bearer token"
  },
  "options": {
    "timeout": 5000,
    "followRedirect": true
  },
  "output": {
    "html": "./reports/report.html",
    "openHtml": true
  }
}

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.