ES Module Shims

86% of users are now running browsers with baseline support for ES modules.

But a lot of the useful features of modules come from new specifications which either aren't implemented yet, or are only available in some browsers.

It turns out that we can actually polyfill most of the newer modules specifications on top of these baseline implementations in a performant 7KB shim.

This includes support for:

• Dynamic import() shimming when necessary in eg older Firefox versions.
• import.meta and import.meta.url.
• Import Maps support.
• Importing JSON
• Importing Web Assembly (note, there is an open issue on how to handle the 4KB imposed limit)

In addition a custom fetch hook can be implemented allowing for streaming in-browser transform workflows.

Because we are still using the native module loader the edge cases work out comprehensively, including:

• Live bindings in ES modules
• Dynamic import expressions (import('src/' + varname'))
• Circular references, with the execption that live bindings are disabled for the first unexecuted circular parent.

Due to the use of a tiny Web Assembly JS tokenizer for ES module syntax only, with very simple rewriting rules, transformation is instant.

TS Module Shim

Extends the 'es-module-shim' module to support TypeScript files

This should really only be used for development purposes.

TypeScript compiler is large (8 Mb) and performance expensive.

How to Use

<!-- Include typescript and this module -->
<script src="//unpkg.com/typescript/lib/typescript.js"></script>
<script type="module" src="//unpkg.com/@mothepro/es-module-shims/dist/esm/index.js"></script>

<!-- Then include your typescript file as a shim -->
<script type="module-shim" src="index.ts"></script>

TODO

• Properly handle typescript file extensions
  • May need to use the transpileOptions field
• Conditionally load Typescript
  • Import Typescript compiler within this module.
  • Require only one module to be leaded in the HTML
• Better source map suppoert
• Support custom tsconfig files
  • Support the extends field used in some TSConfigs
• Add tests

Browser Support

Works in all browsers with baseline ES module support.

Current browser compatibility of modules features without ES module shims:

| ES Modules Features | Chrome (61+) | Firefox (60+) | Safari (10.1+) | Edge (16+) | | ---------------------------------- | ------------------------------------ | ------------------------------------ | ------------------------------------ | ------------------------------------ | | Executes Modules in Correct Order | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :x:1 | | Dynamic Import | :heavy_check_mark: 63+ | :heavy_check_mark: 67+ | :heavy_check_mark: 11.1+ | :x: | | import.meta.url | :heavy_check_mark: ~76+ | :heavy_check_mark: ~67+ | :heavy_check_mark: ~12+ | :x: | | Module Workers | :heavy_check_mark: ~68+ | :x: | :x: | :x: | | Import Maps | :x:2 | :x: | :x: | :x: | | JSON Modules | :x: | :x: | :x: | :x: | | CSS Modules | :x: | :x: | :x: | :x: | | Wasm Modules | :x: | :x: | :x: | :x: |

• 1: Edge executes parallel dependencies in non-deterministic order. (ChakraCore bug).
• 2: Enabled under the Experimental Web Platform Features flag in Chrome 76.
• ~: Indicates the exact first version support has not yet been determined (PR's welcome!).

Browser Compatibility with ES Module Shims:

| ES Modules Features | Chrome (61+) | Firefox (60+) | Safari (10.1+) | Edge (16+) | | ---------------------------------- | ------------------------------------ | ------------------------------------ | ------------------------------------ | ------------------------------------ | | Executes Modules in Correct Order | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:1 | | Dynamic Import | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | import.meta.url | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Module Workers | :heavy_check_mark: 63+ | :x:2 | :x:2 | :x:2 | | Import Maps | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | JSON Modules | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | CSS Modules | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Wasm Modules | :heavy_multiplication_x:3 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |

• 1: The Edge parallel execution ordering bug is corrected by ES Module Shims with an execution chain inlining approach.
• 2: Module worker support cannot be implemented without dynamic import support in web workers.
• 3: Chrome limits Web Assembly to 4KiB synchronous instantiations. Fix tracking in #1.

Import Maps

The goal is for this project to eventually become a true polyfill for import maps in older browsers, but this will only happen once the spec is implemented in more than one browser and demonstrated to be stable.

In order to import bare package specifiers like import "lodash" we need import maps, which are still an experimental specification.

Using this polyfill we can write:

<!doctype html>
<!-- either user "defer" or load this polyfill after the scripts below-->
<script defer src="es-module-shims.js"></script>
<script type="importmap-shim">
{
  "imports": {
    "test": "/test.js"
  },
  "scopes": {
    "/": {
      "test-dep": "/test-dep.js"
    }
  }
}
</script>
<script type="module-shim">
  import test from "test";
  console.log(test);
</script>

All modules are still loaded with the native browser module loader, just as Blob URLs, meaning there is minimal overhead to using a polyfill approach like this.

Dynamic Import

Dynamic import(...) within any modules loaded will be rewritten as importShim(...) automatically providing full support for all es-module-shims features through dynamic import.

To load code dynamically (say from the browser console), importShim can be called similarly:

importShim('/path/to/module.js').then(x => console.log(x));

import.meta.url

import.meta.url provides the full URL of the current module within the context of the module execution.

JSON Modules

To load JSON Modules, import any JSON file served with application/json:

import json from './test.json';

CSS Modules

To load CSS Modules, import any file with served with text/css:

import css from './style.css';
document.adoptedStyleSheets = [...document.adoptedStyleSheets, css];

Support relies on the new CSSStyleSheet constructor, which is currently only available in Chromium.

For other browsers a polyfill can be used.

Web Assembly

To load Web Assembly Modules, import a module served with application/wasm:

import { fn } from './test.wasm';

Web Assembly imports are in turn supported.

Import map support is provided both for mapping into Web Assembly URLs, as well as mapping import specifiers to JS or WebAssembly from within WASM.

Note some servers don't yet support serving .wasm files as application/wasm by default, in which case an invalid content type error will be thrown. The application/wasm MIME type is necessary for loading Web Assembly in line with the specification for browser modules.

Module Workers

To load workers with full import shims support, the WorkerShim constructor can be used:

const worker = new WorkerShim('./module.js', {
  type: 'module',
  // optional import map for worker:
  importMap: {...}
});

This matches the specification for ES module workers, supporting all features of import shims within the workers.

Module workers are only supported in browsers that provide dynamic import in worker environments, which is only Chrome currently.

Fetch Hook

Note: This hook is non spec-compliant, but is provided as a convenience feature since the pipeline handles the same data URL rewriting and circular handling of the module graph that applies when trying to implement any module transform system.

The ES Module Shims fetch hook can be used to implement transform plugins.

For example:

importShim.fetch = async function (url) {
  const response = await fetch(url);
  if (response.url.endsWith('.ts')) {
    const source = await response.body();
    const transformed = tsCompile(source);
    return new Response(new Blob([transformed], { type: 'application/javascript' }));
  }
  return response;
};

Because the dependency analysis applies by ES Module Shims takes care of ensuring all dependencies run through the same fetch hook, the above is all that is needed to implement custom plugins.

Streaming support can be handled through the above as well, although most compilers likely want synchronous sources as in the above.

Plugins

Since the Fetch Hook is very new, there are no plugin examples of it yet, but it should be easy to support various workflows such as TypeScript and new JS features this way.

If you work on something here please do share to link to from here.

Implementation Details

Import Rewriting

• Sources are fetched, import specifiers are rewritten to reference exact URLs, and then executed as BlobURLs through the whole module graph.
• CSP is not supported as we're using fetch and modular evaluation.
• The tokenizer handles the full language grammar including nested template strings, comments, regexes and division operator ambiguity based on backtracking.
• When executing a circular reference A -> B -> A, a shell module technique is used to "shim" the circular reference into an acyclic graph. As a result, live bindings for the circular parent A are not supported, and instead the bindings are captured immediately after the execution of A.

Import Maps

• The import maps specification is under active development and will change, all of the current specification features are implemented, but the edge cases are not currently fully handled. These will be refined as the specification and reference implementation continue to develop.

Web Assembly

• In order for Web Assembly to execute in the module graph as a blob: URL we need to use new WebAssembly.Instance for synchronous execution, but this has a 4KB size limit in Chrome and Firefox which will throw for larger binaries. There is no known workaround currently. Tracking in https://github.com/guybedford/es-module-shims/issues/1.
• Exports are snapshotted on execution. Unexecuted circular dependencies will be snapshotted as empty imports. This matches the current integration plans for Web Assembly.

Inspiration

Huge thanks to Rich Harris for inspiring this approach with Shimport.

License

MIT