Markify

Typescript library to seamlessly convert web pages and HTML snippets into well-structured Markdown content.

Features

• Convert web pages to Markdown from a URL
• Convert raw HTML strings to Markdown
• Strip navigation, headers, footers and other boilerplate, keeping just the main content
• Extract page metadata (title, description, canonical URL, OpenGraph, JSON-LD) as YAML frontmatter
• GitHub-flavoured tables, nested lists, task lists, and fenced code blocks
• Ignore specific elements by providing a list of HTML selectors
• Command line interface

Installation

npm install markify-ts

Usage

Convert a webpage to Markdown

const result = await markify({
  url: 'https://example.com'
});

console.log(result.markdown);

Convert a raw HTML string to Markdown

const result = await markify({
  htmlContent: '<h1>Hello World</h1>'
})

console.log(result.markdown)

Ignore specific elements

const result = await markify({
  url: 'https://example.com',
  ignoreSelectors: ['p', '.container']
});

• Ignore all spans

['span']

• Ignore all elements with the class container

['.container']

Ignore all elements with a data-testid attribute equal to featured-item

['[data-testid="featured-item"]']

Ignore all elements with id equals to 'banner'

['#banner']

Extract only the main content

Pages are mostly navigation, cookie banners and footers. extractContent keeps just the article body. It defaults to true when you pass a url, and false for an htmlContent snippet (where you have already chosen what to convert).

const result = await markify({
  url: 'https://example.com/blog/post',
  extractContent: true
});

Include page metadata as frontmatter

const result = await markify({
  url: 'https://example.com/blog/post',
  includeFrontmatter: true
});

---
title: "Post title"
description: "A short description."
canonical: "https://example.com/blog/post"
lang: "en"
---

# Post title

List of available parameters

• url: URL to be converted to Markdown
• htmlContent: HTML string to be converted to Markdown
• ignoreSelectors: Array of selectors to ignore
• ignoreHiddenElements: Boolean to ignore hidden elements, like those with display: none (default: false)
• extractContent: Strip boilerplate and keep only the main content (default: true for url, false for htmlContent)
• includeFrontmatter: Prepend YAML frontmatter built from the page's metadata (default: false)
• fetchOptions: Fetch options to be used when fetching the URL (default: {})
• timeout: Milliseconds to wait for the URL before aborting (default: 30000)
• maxBytes: Maximum response size in bytes (default: 10485760, i.e. 10MB)
• retries: Number of times to retry a failed fetch (default: 2)
• fetchImpl: Replacement for the global fetch, so a server can apply its own network policy (for example, rejecting private addresses to prevent SSRF)

Fetching untrusted URLs

If you accept a URL from your users and convert it server-side, that request is made from your network. Guard it with fetchImpl so a URL pointing at localhost or a cloud metadata endpoint cannot be used to reach your internal services:

const result = await markify({ url: userSuppliedUrl, fetchImpl: mySafeFetch });

Run from the command line

# Clone the repository
git clone "https://github.com/thiagobarbosa/markify-ts.git"
cd markify-ts/markify

# Install dependencies
npm install

# Run with a specific URL
npm run markify -- --url "https://example.com"
# an output file will be created under "outputs/markdown.md"

# Run with a HTML string
npm run markify -- --html "<h1>Hello World</h1>"

# Run with a specific URL and ignore specific elements
npm run markify -- --url "https://example.com" --ignore-selectors ".sidebar,#promo"

# Run with ignore hidden elements
npm run markify -- --url "https://example.com" --ignore-hidden

# Convert the whole page instead of just the main content
npm run markify -- --url "https://example.com" --no-extract-content

# Prepend YAML frontmatter built from the page metadata
npm run markify -- --url "https://example.com" --frontmatter

# Define the output directory/file
npm run markify -- --url "https://example.com" --output "files/output.md"

# All options
npm run markify -- --help

Development

npm test        # run the test suite
npm run lint    # lint
npm run build   # build to dist/

License

MIT license