npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

sitemapper

v3.2.20

Published

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

Downloads

130,934

Readme

Sitemap-parser

Code Scanning NPM Publish Version Bump Test Codecov CodeFactor GitHub license GitHub release date Inline docs Libraries.io dependency status for latest release license Monthly Downloads npm version release scrutinizer

Parse through a sitemaps xml to get all the urls for your crawler.

Version 2

Installation

npm install sitemapper --save

Simple Example

const Sitemapper = require('sitemapper');

const sitemap = new Sitemapper();

sitemap.fetch('https://wp.seantburke.com/sitemap.xml').then(function (sites) {
  console.log(sites);
});

Examples

import Sitemapper from 'sitemapper';

(async () => {
  const Google = new Sitemapper({
    url: 'https://www.google.com/work/sitemap.xml',
    timeout: 15000, // 15 seconds
  });

  try {
    const { sites } = await Google.fetch();
    console.log(sites);
  } catch (error) {
    console.log(error);
  }
})();

// or

const sitemapper = new Sitemapper();
sitemapper.timeout = 5000;

sitemapper
  .fetch('https://wp.seantburke.com/sitemap.xml')
  .then(({ url, sites }) => console.log(`url:${url}`, 'sites:', sites))
  .catch((error) => console.log(error));

Options

You can add options on the initial Sitemapper object when instantiating it.

  • requestHeaders: (Object) - Additional Request Headers (e.g. User-Agent)
  • timeout: (Number) - Maximum timeout in ms for a single URL. Default: 15000 (15 seconds)
  • url: (String) - Sitemap URL to crawl
  • debug: (Boolean) - Enables/Disables debug console logging. Default: False
  • concurrency: (Number) - Sets the maximum number of concurrent sitemap crawling threads. Default: 10
  • retries: (Number) - Sets the maximum number of retries to attempt in case of an error response (e.g. 404 or Timeout). Default: 0
  • rejectUnauthorized: (Boolean) - If true, it will throw on invalid certificates, such as expired or self-signed ones. Default: True
  • lastmod: (Number) - Timestamp of the minimum lastmod value allowed for returned urls
  • proxyAgent: (HttpProxyAgent|HttpsProxyAgent) - instance of npm "hpagent" HttpProxyAgent or HttpsProxyAgent to be passed to npm "got"
  • exclusions: (Array) - Array of regex patterns to exclude URLs from being processed
  • field: (Object) - An object of fields to be returned from the sitemap. Leaving a field out has the same effect as <field>: false. If not specified sitemapper defaults to returning the 'classic' array of urls. Available fields:
    • loc: (Boolean) - The URL location of the page
    • lastmod: (Boolean) - The date of last modification of the page
    • changefreq: (Boolean) - How frequently the page is likely to change
    • priority: (Boolean) - The priority of this URL relative to other URLs on your site
    • image:loc: (Boolean) - The URL location of the image (for image sitemaps)
    • image:title: (Boolean) - The title of the image (for image sitemaps)
    • image:caption: (Boolean) - The caption of the image (for image sitemaps)
    • video:title: (Boolean) - The title of the video (for video sitemaps)
    • video:description: (Boolean) - The description of the video (for video sitemaps)
    • video:thumbnail_loc: (Boolean) - The thumbnail URL of the video (for video sitemaps)

For Example:

field: {
  loc: true,
  lastmod: true,
  changefreq: true,
  priority: true,
}

Leaving a field out has the same effect as <field>: false. If not specified sitemapper defaults to returning the 'classic' array of urls.

An example using all available options:

const sitemapper = new Sitemapper({
  url: 'https://art-works.community/sitemap.xml',
  timeout: 15000,
  requestHeaders: {
    'User-Agent':
      'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0',
  },
  debug: true,
  concurrency: 2,
  retries: 1,
  rejectUnauthorized: false,
  field: {
    loc: true,
    lastmod: true,
    changefreq: true,
    priority: true,
  },
  proxyAgent: new HttpProxyAgent('http://localhost:8080'),
});