451 Tools

Censorship resilient and distributed publishing. This repository contains a collection of scripts and tools to help you run your own censorship resilient web services. We mostly use Service Workers to achieve this.

451 Tools started as a censorship circumvention technology that we were developing for RadioZamaneh.com which is permanently blocked in Iran. We just wanted to make our own content accessible even during shutdowns. However, we realised that it had potential for others as well, so we eventually turned it into a NPM package. So it really has a bottom up origin story.

Installation

npm install 451-tools

Table of contents

• Getting started
• Bookmarking
• Fallback pages
• Offline assets
• Content bundles
• Fallback image
• Caching strategies
• Mirroring
• UI components

Getting started

To ensure smooth integration between our modules it is necessary to use the registerServiceWorkerController function. Ensure serviceWorkerController is included within the options parameter when calling the module, as shown in the example below.

Inside your service worker:

import { registerBookmarkApi, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerBookmarkApi({ serviceWorkerController });

Individual modules can be configured from a configuration file. You need to host that file on your server and pass the path of the file as a url parameter to your service worker registration path, as illustrated below.

From the client:

if ('serviceWorker' in navigator) {
  window.addEventListener('load', () => {
    const configurationVersion = 1;
    const serviceWorkerUrl = `/service-worker.js?configuration=${encodeURIComponent(`/static/service-worker-configuration.json?v=${configurationVersion}`)}`;
    navigator.serviceWorker
      .register(serviceWorkerUrl)
      .then((registration) => registration.update())
  });
}

Make sure to revision hash or version the configuration file, so that you can update it without having to update your service worker.

The configuration file should look like this, all modules are optional.

{
  "fallbackPages": {},
  "offlineAssets": {},
  "contentBundles": {},
  "mirroring": {}
}

For additional logging, you can enable debug mode by setting the debugMode flag to true in your configuration.

{
  ...
  "debugMode": true,
}

For more information about the configuration options, see the documentation of the individual modules.

API endpoint: GET /451-tools/configuration/update/

Signals the service worker to update the configuration. It will try to refetch and update the configuration.

Example

fetch('/451-tools/configuration/update/', { method: 'GET' });

API endpoint: PUT /451-tools/configuration/

Overwrites the current configuration with a new one. ⚠️ Warning: This is an advanced feature. Ensure you understand the implications of changing the configuration before using this endpoint.

Example

fetch('/451-tools/configuration/', {
  method: 'PUT',
  headers: {
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    "fallbackPages": {
      "customProperties": {
        "primary-color": "darkblue"
      }
    },
  }),
});

Implementing a Kill Switch

If any issues arise with the service worker, whether due to implementation or unforeseen circumstances, you can implement a "kill switch" as a temporary measure to quickly gain time while you work on diagnosing and fixing the issue.

Unregistering the Service Worker

To immediately stop the service worker from running and unregister it from all clients, use the following code snippet. This should be deployed in place of your usual service worker registration script to prevent a new service worker from being installed after unregistration:

if ('serviceWorker' in navigator) {
  navigator.serviceWorker.getRegistrations().then(registrations => {
    registrations.forEach(registration => {
      registration.unregister();
    });

    // Optional: Clear all caches to ensure no old cached resources are causing issues
    caches.keys().then(cacheNames => {
      cacheNames.forEach(cacheName => caches.delete(cacheName));
    });
  });
}

This approach ensures that the problematic service worker is completely unregistered and won't affect users who load the page afterward. Additionally, the optional cache-clearing step ensures that any cached data that might be contributing to the issue is removed.

Make sure to deploy this kill switch script instead of the service worker registration script. Otherwise, there is a risk of unregistering all service workers and immediately installing a new one.

Bookmarking

Your readers can bookmark content for offline reading.

We provide some API endpoints defined in the service worker to save pages for offline availability.

Inside your service worker:

import { registerBookmarkApi, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerBookmarkApi({ serviceWorkerController });

When your service worker is installed, you can now call the bookmark API, which is available at /451-tools/bookmark/.

GET /451-tools/bookmark/

Get all pages from the bookmark cache.

Response

An array of bookmarked pages.

| Property | Type | Description | Example | | -------- | -------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | url | string | The url of the bookmarked page | htts://domain.org/path/to/the/page/ | | path | string | The path of the bookmarked page | /path/to/the/page/ | | html | string | The html of the bookmarked page | <html><head><title>Page title</title></head><body>Page content</body></html> | | metadata | object | Optional metadata of the bookmarked page.Passed when a page was bookmarked | { title: 'Page title' } |

Example

fetch('/451-tools/bookmark/', { method: 'GET' })
  .then(response => response.json())
  .then(pages => {
    console.log(pages);
  });

GET /451-tools/bookmark/{path}

Get a single page from the bookmark cache. Where the path is a url encoded path.

Response

200: A single bookmarked page.

| Property | Type | Description | Example | | -------- | -------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | url | string | The url of the bookmarked page | htts://domain.org/path/to/the/page | | path | string | The path of the bookmarked page | /path/to/the/page | | html | string | The html of the bookmarked page | <html><head><title>Page title</title></head><body>Page content</body></html> | | metadata | object | Optional metadata of the bookmarked page.Passed when a page was bookmarked | { title: 'Page title' } |

404: The page was not found in the bookmark cache.

400: The path is not a valid url encoded path.

Example

fetch('/451-tools/bookmark/%2Fpath%2Fto%2Fthe%2Fpage%2F', { method: 'GET' })
  .then(response => response.json())
  .then(page => {
    console.log(page);
  });

POST /451-tools/bookmark/{path}

Add a page to the bookmark cache. Where the path is a url encoded path.

Request body

| Property | Type | Description | Example | | -------- | -------- | ---------------------------------------------------------------------------------------------------------- | ------------------------- | | metadata | object | Optional metadata for the bookmarked page.It can be anything you want as long as it is a valid object. | { title: 'Page title' } |

Example

fetch('/451-tools/bookmark/%2Fpath%2Fto%2Fthe%2Fpage%2F', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({ metadata: { title: 'Page title' } }),
});

DELETE /451-tools/bookmark/{path}

Delete a page from the bookmark cache. Where the path is a url encoded path.

Example

fetch('/451-tools/bookmark/%2Fpath%2Fto%2Fthe%2Fpage%2F', { method: 'DELETE' });

Fallback pages

Instead of seeing a 404 or a No internet page, your users will see a custom page with information that you want to communicate to them (great during shutdowns, if censored, and when they are offline). When visitors find you through a search engine, on social media, or other places, you can have a different, custom fallback page for each of these referrers.

Inside your service worker:

import { registerFallbackPages, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerFallbackPages({ serviceWorkerController });

Calling registerFallbackPages is enough to get you started. It will render a default offline page whenever the network is not available. By default, it looks like this:

This page is somewhat customizable, it has support for basic theming and the default UI texts can be overwritten. However, in case this is not enough, we also support custom offline pages.

import { registerFallbackPages, registerFallbackHandlersPlugin } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerFallbackPages({ serviceWorkerController });

And the configuration for the custom offline page:

{
  "fallbackPages": {
    "pages": [
      {
        "url": "/my-offline-page/",
        "revision": "1"
      }
    ],
    "excludeByPath": [
      "/wp-admin",     // Matches any path starting with "/wp-admin".
      "/wp-login.php$" // Matches exactly "/wp-login.php".
    ]
  }
}

revision is not required, but it is highly recommended. Changing this value will cause a cache invalidation, which means we will re-cache the latest version.

⚠️ Please note that this module takes over the /451-tools/dashboard/ route regardless of the user's online status. The route is used to make the default offline page available for online users.

See below the possible options for the configuring the fallback pages:

| Option | Description | Required | Type | Default | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------- | ------- | | pages | A single or a list of offline fallback pages. | true | String \| Object \| Array | - | | excludeByPath | A list of paths to exclude from fallback pages, supports strings containing regular expressions. | false | String[] | [] | | language | The language for the default offline page. The value will be used as a lang attribute on the HTML tag. See MDN for possible values. | false | String | en | | textDirection | The text direction for the default offline page. The value will be used as a dir attribute on the HTML tag. Can be either ltr or rtl. | false | String | ltr | | customProperties | Will be used to overwrite the default styles of the default offline page. The full list of over writable values can be found below. | false | Object | {} | | templateStrings | Will be used to overwrite the default UI text strings of the default offline page. The full list of over writable values can be found below. | false | Object | {} | | showHiddenTabs | Toggle the display of tabs for unused features on the default offline page. | false | Boolean | false |

customProperties

A list of all the over writable custom properties and their default values.

| Key | Description | Default | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------- | | background-color | Color used as the page's background. | #f9fbfc | | border-color | Color used for stylistic borders and separators. | #e6e8ec | | card-background-color | Color used as the card's background. | #ffffff | | card-box-shadow | Box shadow used by the cards, it can be removed by passing it a value of none. | 0px 8px 24px rgba(149, 157, 165, 0.2) | | container-width | The main container's max width. | 1200px | | icon-color | Color used for the header's offline icon. | #e6e8ec | | primary-color | Color used for links, highlights, active & focus states, etc. Essentially, whenever we need to pull a user's attention to something. | #1993f6 | | tab-text-color | Color used as the tab's primary text color (non-active state). | #7a7d95 | | text-color | Color used as the primary text color. | #2d2d3b |

templateStrings

A list of all the over writable UI text strings and their default values.

| Key | Description | Default | |---------------------|--------------------------------------------------|-----------------------------------------------------------------| | pageTitle | Page title | Woops, No Internet Connection | | pageDescription | Page description | Please check your internet connection and try again. | | homeLinkText | The link text for the home link in the header | Home | | bookmarksTabTitle | The title for the bookmarks tab | Bookmarks | | editorialTabTitle | The title for the editorial(content bundles) tab | Editorial | | aboutTabTitle | The title for the about tab | About | | copyright | Copyright | Copyright © ${new Date().getFullYear()}. All rights reserved. | | aboutInnerHTML | Inner HTML used to render the about tab | - |

Default offline page with customization

import { registerFallbackPages, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerFallbackPages({ serviceWorkerController });

And the configuration for the default offline page with customization:

{
  "fallbackPages": {
    "customProperties": {
      "primary-color": "blue"
    },
    "templateStrings": {
      "pageTitle": "My Offline Page"
    }
  }
}

Custom offline page

import { registerFallbackPages, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerFallbackPages({ serviceWorkerController });

And the configuration for the custom offline page:

{
  "fallbackPages": {
    "pages": [
      {
        "url": "/my-offline-page/",
        "revision": "1"
      }
    ]
  }
}

Multiple offline pages

For most cases, a single fallback is enough. However, we support an optional refererRegex property. This can, for example, be used to render a different fallback page for users who came from a search engine:

import { registerFallbackPages, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerFallbackPages({ serviceWorkerController });

And the configuration for multiple offline pages:

{
  "fallbackPages": {
    "pages": [
      {
        "url": "/my-offline-page/",
        "revision": "1"
      },
      {
        "url": "/google-search-offline-page/",
        "revision": "1",
        "refererRegex": "https://(www.)?(google).com/"
      }
    ]
  }
}

Offline assets

We provide a offline strategy for assets to use when the network is not available.

Inside your service worker:

import { registerOfflineAssets, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerOfflineAssets({ serviceWorkerController });

And the configuration for the offline assets:

{
  "offlineAssets": {
    "assets": [
      {
        "url": "/main.css",
        "revision": "1"
      },
      {
        "url": "/main.js",
        "revision": "1"
      }
    ]
  }
}

revision is not required, but it is highly recommended. Changing this value will cause a cache invalidation, which means we will re-cache the latest version.

See below the possible options for the configuring the offline assets:

| Option | Description | Required | Type | |---------------------------|------------------------------------------------------|----------|----------------------------| | assets | A list of offline assets. | true | String \| Object \| Array |

Content bundles

You can mark articles for precaching so your readers always have access to them (great for evergreen articles or ones with particularly important information in case of a shutdown).

You can define a list of pages as content bundles to be cached by the service worker.

Inside your service worker:

import { registerContentBundles, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerContentBundles({ serviceWorkerController });

And the configuration for the content bundles:

{
  "contentBundles": {
    "pages": [
      {
        "url": "/content-bundle-1/",
        "revision": "1"
      },
      {
        "url": "/content-bundle-2/",
        "revision": "1"
      }
    ]
  }
}

revision is not required, but it is highly recommended. Changing this value will cause a cache invalidation, which means we will re-cache the latest version.

| Option | Description | Required | Type | | ------------------------- |---------------------------------------------------------------|---------|-----------------------------| | pages | A list of pages. | true | String \| Object \| Array |

A page object can have the following properties:

| Property | Description | Required | Type | Default | |------------|--------------------------------------------------------------------------------------------------------------------| -------- |----------|---------| | url | The url of the page. | true | String | - | | revision | The revision of the page. | false | String | - | | metadata | Optional metadata for the content bundle.Passed back when retrieving content bundles through our API endpoint. | false | Object | {} |

GET /451-tools/content-bundles/

Get all content from the content bundles cache.

Response

An array of content.

| Property | Type | Description | Example | | ----------- | -------- | --------------------------------- | ------------------------------------------------------------------------------ | | url | string | The url of the content. | htts://domain.org/path/to/the/page/ | | path | string | The path of the content. | /path/to/the/page/ | | content | string | The text string of the content. | <html><head><title>Page title</title></head><body>Page content</body></html> | | contentType | string | The type of the content. | text/html; charset=utf-8 | | metadata | object | Optional metadata of the content. | { title: 'Page title' } |

Example

fetch('/451-tools/content-bundles/', { method: 'GET' })
  .then(response => response.json())
  .then(content => {
    console.log(content);
  });

Fallback image

We provide a fallback strategy for images, it returns a fallback SVG for uncached images, when the network is not available.

Inside your service worker:

import { registerFallbackImage, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerFallbackImage({ serviceWorkerController });

Mirroring

In order to keep a website more available you can configure alternative mirrors where the website is hosted.

You can feed the URLs for your mirrors as configuration, and it will fetch content for your readers from the mirror if the primary website is unavailable. This way your readers do not have to go looking for your mirrors, and you do not have to separately communicate them to your readers.

The service worker will always try to get resources from the primary domain. But once the primary domain fails and mirrors are configured content will be fetched from one of the configured mirrors. By default, we will time out and cancel a request after 3 seconds.

We mark a mirror and its status, which means that for subsequent requests a mirror that was marked as up will be used for the next 10 minutes. We feel that this leads to a faster user experience because we will not encounter the timeout for every request that is done. When all mirrors are marked as down we will reset the status of the mirrors and try the primary domain again on the next document request.

If all mirrors fail the offline fallback pages strategy as defined in fallback pages will be used. If the offline fallback page is not configured the request will simply fail.

In order to further speed up the user experience and/or still be able to serve as much content even when all mirrors are down, you can configure a caching strategy for the requested documents. That strategy is used on top of mirroring failing requests for fresh content. The available strategies are: network-only(default), network-first and stale-while-revalidate.

Inside your service worker:

import { registerMirroring, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerMirroring({ serviceWorkerController });

⚠️ Please note that the order of registering modules is important. Especially with mirroring. Since it will match all document requests and same origin requests it is important to register it last.

Once the service worker is installed, you can call the mirror status API endpoint, which is available at /451-tools/mirror-status/. It will return the status of where the content is coming from. The response looks like this:

{
  "status": "up" // or "unknown", "down", "low" 
}

| Status | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | unknown | The status of the primary domain is not known (yet). | | up | Content is coming from the primary domain. | | down | It is not possible to get up to date resources from either the primary domain or any (all) of the defined mirrors. | | low | It is not possible to get resources from the primary domain, but it is still possible to get up to date resources from one of the mirrors. |

Add an object with the following properties to your configuration file:

| Option | Description | Required | Type | Default | |-----------------|---------------------------------------------------------------------------------------------| -------- |---------------------------------------------------------------|----------------| | urls | A list of alternative URLs where the website is available. | true | String[] | - | | excludeByPath | A list of paths to exclude from mirroring, supports strings containing regular expressions. | false | String[] | [] | | timeout | Maximum time in milliseconds to wait for a response before cancelling the request. | false | Number \| String | 3000 | | strategy | A pattern that determines how a response is generated after receiving a fetch event | false | network-only | network-first | stale-while-revalidate | network-only |

{
  "mirroring": {
    "urls": [
      "https://mirror1.com/",
      "https://mirror2.com/"
    ],
    "excludeByPath": [
      "/wp-admin",     // Matches any path starting with "/wp-admin".
      "/wp-login.php$" // Matches exactly "/wp-login.php".
    ]
  }
}

Cross-Origin Resource Sharing (CORS)

To ensure that your website or mirrors work seamlessly, your server must be configured to include the appropriate CORS response headers.

Mandatory Headers

To enable CORS for the mirroring functionality, your server must set the following headers in response to CORS preflight (OPTIONS) requests:

Access-Control-Allow-Origin:    <https://your-domain.com> | *

• Access-Control-Allow-Origin: This header specifies which origins are permitted to access the resources. You can allow all origins (*) or specify a particular origin, such as <https://your-domain.com>.

These headers are crucial for ensuring that browsers permit the necessary cross-origin requests.

Optional and Additional Headers

In addition to the mandatory headers, you may need to set additional headers under certain conditions to fine-tune your CORS policy:

• Vary: Origin: If the Access-Control-Allow-Origin header specifies a particular origin (e.g., <https://your-domain.com>), it is recommended to include the Vary: Origin header. This ensures that caches correctly handle responses that vary based on the Origin request header.

  Access-Control-Allow-Origin:    <https://your-domain.com>
  Vary:                           Origin

These additional headers help ensure that your site or mirror behaves consistently across different client scenarios and caching configurations.

Caching strategies

We provide a mechanism fo registering different caching strategies for different requests. Requests can be matched by using a regular expression on the request URL. If the request is matched the configured strategy is used when requesting/serving the request URL. The available strategies are:

network-only(default), network-first and stale-while-revalidate.

⚠️ Please note that when the mirroring module is also used, the network part of any strategy will use the mirroring module to fetch the content.

Inside your service worker:

import { registerCachingStrategies, registerServiceWorkerController } from '451-tools';

const serviceWorkerController = registerServiceWorkerController();
registerCachingStrategies({ serviceWorkerController });

Add an array of objects with the following properties to your configuration file:

| Option | Description | Required | Type | Default | |-----------------|--------------------------------------------------------------------------------------| -------- |---------------------------------------------------------------|-----------------| | urlPattern | A regular expression to match the request URL. | true | String | - | | strategy | A pattern that determines how a response is generated after receiving a fetch event. | true | network-only | network-first | stale-while-revalidate | - |

{
  "cachingStrategies": [
    {
      "urlPattern": "/api/.+$",
      "strategy": "network-only"
    }
  ]
}

UI components

In addition to our modules, we offer a set of UI components that come in the form of web components. While these components may offer added functionality, their primary function is to provide an alternative approach to implementing our modules without requiring any JavaScript coding.

Bookmark button

The button displays the bookmark status of the related page - whether it is currently bookmarked or not. In addition, it enables users to bookmark or unbookmark the page with just a click. Please note that this button requires the bookmarking module to function properly.

Usage

Inside your HTML:

<bookmark-button
  path="/my-page"
  bookmark-text="Bookmark"
  remove-bookmark-text="Remove Bookmark"
>
</bookmark-button>

Attributes

Pass the following attributes to the bookmark-button component:

| Attribute | Description | Required | Default | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------- | | path | The page's path. | true | - | | bookmark-text | The label displayed when the page is not bookmarked. | false | Bookmark | | remove-bookmark-text | The label displayed when the page is bookmarked. | false | Remove Bookmark | | metadata-title | Optional page metadata can be added to provide additional information about the page. This is also used by the default offline page to render the cards. | false | - | | metadata-description | Optional page metadata can be added to provide additional information about the page. This is also used by the default offline page to render the cards. | false | - | | metadata-image-src | Optional page metadata can be added to provide additional information about the page. This is also used by the default offline page to render the cards. | false | - | | metadata-image-height | Optional page metadata can be added to provide additional information about the page. This is also used by the default offline page to render the cards. | false | - | | metadata-image-width | Optional page metadata can be added to provide additional information about the page. This is also used by the default offline page to render the cards. | false | - |

Styling

Use the following selector:

bookmark-button::part(button) {
  background-color: rebeccapurple;
  color: white;
}

Traffic light

A traffic light component that indicates to the user whether they are accessing the primary website (green), a mirror (yellow) or cached content (red).

Usage

Inside your HTML:

<traffic-light
  href="/offline/"
  unknown-text="The status of the main site is currently unknown."
  up-text="The main site is online and accessible."
  low-text="The main site is unavailable; a mirror site is being used."
  down-text="The main site is currently inaccessible."
>
</traffic-light>

Attributes

Pass the following attributes to the traffic-light component:

| Attribute | Description | Required | Default | | -------------- | ------------------------------------------------------------------------------------------------------------- | -------- | --------- | | unknown-text | The label read out loud by screen readers when the status of the site is unknown. | false | Unknown | | up-text | The label read out loud by screen readers when the main site is online and accessible. | false | Online | | low-text | The label read out loud by screen readers when the main site is unavailable, and a mirror site is being used. | false | Low | | down-text | The label read out loud by screen readers when the main site is inaccessible. | false | Offline | | href | The URL that the hyperlink points to. | false | - | | target | Where to display the linked URL. | false | - | | rel | The relationship of the linked URL as space-separated link types. | false | - |

Styling

Use the following selectors:

traffic-light::part(button) {
  /* General button selector. */
}

traffic-light::part(online) {
  /* Online button selector. */
}

traffic-light::part(offline) {
  /* Offline button selector. */
}