qoopido.demand

v8.0.2

Published

2 years ago

Promise like module loader using XHR requests and localStorage caching to dynamically load JavaScript and CSS + dynamic dependency resolution + support for custom handlers

Downloads

305

0High
0Medium
0Low

qoopido

module modular script loader javascript css promise xhr ajax localStorage cache caching asset storage require basket

Qoopido.demand

Qoopido.demand is a modular, flexible and 100% async JavaScript module loader with a promise like interface that utilizes localStorage as a caching layer. It comes in a rather tiny package of ~7kB minified and gzipped.

Qoopido.demand originated from my daily use of require.js for the initial development of my Qoopido.nucleus library which is strictly atomic by nature, unbundled.

Key features in detail

promise like interface (no native promise support required)
localStorage caching for blazingly fast performance
dependency resolution for modules
automatic cache invalidation by version or lifetime
per module/path/subpath setting of cache parameters
relative and absolute module path resolution
fallback URLs for any dependency
support for handling modules, legacy scripts, bundles (concatenated scripts like from jsdelivr), text, CSS and JSON included
plugins for cookie support, lzstring compression and SRI included
support for custom handlers & plugins built in
optional support for auto-bundles via genie included

Compatibility

Qoopido.demand is developed for Chrome, Firefox, Safari, Edge, Opera and IE11+.

Active Support for IE9 and IE10 has been removed in Qoopido.demand 6.0.0 due to upcoming refactorings/cleanups and the lack of market share of these browsers today.

Support for IE8 has been actively removed in Qoopido.demand 4.0.0 due to the lack of justifiable polyfills for parts of some underlying pattern.

I do test on MacOS Sierra where Qoopido.demand is fully working on Chrome, Firefox, Safari and Opera. IE9, 10, 11 as well as Edge are testet on the official Microsoft VMs via VirtualBox.

Requirements

Due to modules getting loaded via XHR a remote server has to have CORS enabled. Be assured that most of the usual CDNs have CORS enabled by default.

External dependencies

None!

Availability

Qoopido.demand is available on GitHub as well as jsdelivr, npm and bower at the moment.

Loading demand

Use the following minified code snippet in a standalone script tag before the closing body tag to include demand:

(function(url, main, settings) {
	!function(e,n,t,r,s){r=n.getElementsByTagName(t)[0],s=n.createElement(t),e.demand={url:url,main:main,settings:settings},s.async=1,s.src=url,r.parentNode.insertBefore(s,r)}(window,document,"script");
}('../dist/demand.js', './app/main', { base: './', version: '1.0.0', cache: true }));

The snippet is very similar to Google Analytics. The outer function allows you to specify an URL from which to load demand itself as well as a path to the main module and configuration object for demand. The script tag that actually loads Qoopido.demand will be injected with its async attribute set.

As an alternative to the above snippet Qoopido.demand can now also be loaded with an alternative snippet that uses an iframe. The async method above, while not blocking rendering, seems to delay the onload event at least on some browsers. The iframe method solves this minor annoyance.

(function(url, main, settings) {
	!function(e,t,n,o,i,d,a){e.demand={url:url,main:main,settings:settings},o=t.getElementsByTagName(n)[0],i=t.createElement("iframe"),i.src="javascript:void(0)",i.name="demand-loader",i.title="",i.role="presentation",(i.frameElement||i).style.cssText="display:none;width:0;height:0;border:0;",o.parentNode.insertBefore(i,o);try{i=i.contentWindow.document}catch(e){d=t.domain,i.src='javascript:var d=document.open();d.domain="'+d+'";void(0);',i=i.contentWindow.document}i.open()._=function(){d&&(this.domain=d),a=this.createElement(n),a.src=url,this.body.appendChild(a)},i.write('<body onload="document._();" />'),i.close()}(this,document,"script");
}('../dist/demand.js', './app/main', { base: './', version: '1.0.0', cache: true }));

Remember to adjust parameters according to the async method.

Configuration

The last parameter of the above code snippet is a configuration object. It just contains base and version as these are the properties you will most likely set. There are some more, less frequently used, options that can be either specified here or as part of a demand.configure call in your main module (being described in the next section):

{
	// enables or disables caching in general (when true/false)
	// optional, defaults to "true"
	cache: true,

	// fine grained cache control (when object)
	// any path or part of a path can be set to true to
	// activate caching or false to disable it.
	// The longest matching path wins over others.
	cache: {
		'/demand/':     true,
		'/app/':        true,
		'/app/nocache': false
	},

	// cache will be validated against version
	// optional, defaults to "undefined"
	version: '1.0.0',

	// cache will be validated against lifetime, if > 0
	// optional, defaults to "0"
	// unit: seconds
	lifetime: 60,

	// sets the timeout for XHR requests
	// optional, defaults to "8" (limited to "2" up to "20")
	// unit: seconds
	timeout: 8,

	// base path from where your relative
	// dependencies get loaded
	// optional, defaults to "/"
	base: '[path/url to your scripts]',

	// optional
	pattern: {
		'/nucleus':   ['[path/url to Qoopido.nucleus]', '[fallback path/url to Qoopido.nucleus]'],
		'/app':       '[path/url to your modules]',
		// just an example, loading jQuery + bundle
		// will not work due to the nature of jQuery
		'/jquery':    '//cdn.jsdelivr.net/jquery/2.1.4/jquery.min',
		'/jquery+ui': '//cdn.jsdelivr.net/g/[email protected],[email protected]'
	},

	// per module configuration (if applicable)
	modules: {
		// configure the legacy handler
		'/demand/handler/legacy': {
			'/jquery': {
				probe: function() { return global.jQuery; }
			},
			'/jquery/ui': {
				probe:        function() { return global.jQuery.ui; },
				dependencies: [ 'legacy!/jquery' ]
			},
			'/velocity': {
				probe: function() { return global.Velocity || (global.jQuery && global.jQuery.fn.velocity); }
			},
			'/leaflet': {
				probe: function() { return global.L; }
			}
		}
		// configure the bundle handler
		'/demand/handler/bundle': {
			// declare which modules are included in the bundle
			// order is important
			'/jquery+ui': [ '/jquery', '/jquery/ui' ]
		},
		// configure genie plugin
		'/demand/plugin/genie': {
			// handle creation of auto-bundle URL for Qoopido.nucleus from jsdelivr
			'/nucleus/': function(dependencies) {
				var fragments = [],
            		i = 0, dependency;

            	for(; (dependency = dependencies[i]); i++) {
            		fragments.push(dependency.id.replace(/^\/nucleus\//, '') + '.js');
            	}

            	return '//cdn.jsdelivr.net/g/[email protected](' + fragments.join('+') + ')';
            },
            // handle creation of auto-bundle URL for your modules from your server
            '/app/': function(dependencies) {
            	var fragments = [],
            		i = 0, dependency;

            	for(; (dependency = dependencies[i]); i++) {
            		fragments.push(dependency.id.replace(/^\/js\//, '') + '.js');
            	}

            	return '/genie/?module[]=' + fragments.join('&module[]=');
            }
		}
	}

Usage

The demanded main module from the above script might look like the following example:

(function(global) {
	'use strict';

	function definition(demand, provide) {
		demand
			.configure({
				// any option from the previous section
				// most likely something like:
				pattern: {
				},
				modules: {
				}
			});

		return true; // just return true if there really is nothing to return
	}

	provide([ 'demand', 'provide' ], definition);
}(this));

Qoopido.demand consists of two components demand and provide just like require.js require and define.

Once demand is loaded anything that is either explicitly requested via demand or as a dependency of a provide call will be loaded via XHR as well as modified and injected into the DOM with the help of a handler. The result will be cached in localStorage (if caching is enabled and localStorage is available) that will get validated against an optional version and lifetime set via demand.configure or the modules path declaration (more on that later).

As main itself is also loaded as a module it will get cached in localStorage as well.

Controlling the cache

If caching is enabled, localStorage available and its quota not exceeded chances are good you will never have to manually deal with the cache.

By default demand will invalidate a modules cache under the following conditions:

module's version changed
module's lifetime is exceeded

Demand will, in addition, do its best to keep leftover garbage to a minimum. It does so by starting an automatic garbage collection for expired caches on load. In addition it will also clear a specific cache if it gets requested and is found to be invalid for any reason.

When localStorage quota is exceeded while trying to cache yet another module Qoopido.demand will load a special module /demand/cache/dispose and will try to free the required space by clearing existing caches in order of last access time, from oldest to newest.

Beside the automatic cache invalidation demand still offers manual control by registering a demand.cache object to the global demand function. This object offers the following methods to control the cache:

// only clear a single module's cache
demand.cache.clear('[module path]');

// clear all expired caches
demand.cache.clear.expired();

// completely clear the cache
demand.cache.clear.all();

Sidenote

Demand does use a prefix for its localStorage keys to prevent conflicts with other scripts. Each cache will consist of two keys, one to store the state information and one for the actual value (source) of the module. By separating the two only a very small string will have to get parsed to retrieve the state.

Demanding modules

After your project is set up accordingly you can load further modules like in the following example:

demand('./app/test', '/nucleus/component/iterator')
	.then(
		function(appTest, nucleusComponentIterator) {
			console.log('=> success', appTest, nucleusComponentIterator);

			new nucleusComponentIterator();
		},
		function() {
			console.log('=> error', arguments);
		}
	);

Relative module paths will be resolved relative to the base path or the path of an eventual parent module (see section Path resolution). The resulting path will afterwards get matched against patterns defined via demand.configure which will finally lead to an absolute URL to fetch the module from.

Sidenote

The rejection function will be passed all rejected dependencies as arguments, not only the first one rejected.

If no handler is specified it will default to the module handler. If you would like to load e.g. CSS simply prefix your path with css!.

Beside its global configuration Qoopido.demand also allows per module or subpath configuration for general cacheability, versioning and lifetime. All per module settings are optional parts of its path declaration.

You already learnt that a prefix of css! tells Qoopido.demand to use the CSS handler for the module. All other possible options are also part of the path declaration, for example

demand('[email protected]#2000!AnyCssModule').then(
	function() {}
);

will tell Qoopido.demand to load your AnyCssModule via the CSS handler and cache it at version 2.0.4 for 2000 seconds if cache is enabled either globally or for this specific module via demand.configure. You may, in rare cases, want to force a module to either be cached or not overriding any global configuration which can be done by:

demand('-css!AnyCssModule').then(
	function() {}
);

Prefixing the module's path with a - will completely disable any caching for this specific demand call whereas prefixing it with a + will force it to be cached disregarding any global cache settings.

As any parameter that is part of the path declaration is optional you gain total control over when and how Qoopido.demand caches your modules!

Auto-bundling with genie

Qoopido.demand's original idea was (and still is) to not need a server-side built-process to pre-compile static bundles but to directly load any module required on demand. This decision really embraces new technologies like HTTP/2 that do not establish a new connection for each single request but instead are able to handle all requests with a single connection.

While this is absolutely great HTTP/2 is not 100% supported by servers and clients yet and even if it is, requesting many assets may still slow down your perceived loading experience.

To handle this Qoopido.demand has a built-in plugin called genie which can be configured to create auto-bundle requests for all direct dependencies of a module. To give you a more detailed example think about a module depending on /nucleus/dom/element, /nucleus/dom/collection and /nucleus/component/sense.

If genie is enabled for paths prefixed with/nucleus/ it will determine if any of the dependencies are already loaded and if there are at least two left for any auto-bundle configured they will get loaded via a single request.

So if none of the dependencies of the aforementioned example are yet loaded all three will be loaded by a single request.

Sidenote

CDNs like jsdelivr allow to request bundles already. A very simple PHP script is part of this repository and can be found under /genie/index.php (adjust the BASE path accordingly). To be able to adopt genie for any kind of bundle URL it uses a callback function which is explained in the section Configuration.

Providing inline modules

Beside demanding other modules you can as well provide your own, just like in the following example:

function definition(appTest, qoopidoBase) {
	return function appMain() {

	}
}

provide('/app/main', [ './test', '/qoopido/base' ], definition);

This is an example for an inline module. The provide call, in this case, consists of three arguments:

path of the module
dependencies of the module
definition of the module

When dynamically loading modules path will have to be omitted and gets internally resolved via loading queue handling instead.

Developing loadable modules

Providing inline modules is only slightly different from building an external, loadable module.

In addition to inline modules you just need some minimal boilerplate code and an anynymous provide call without the path argument like in the following example:

(function() {
	'use strict';

	function definition(nucleusBase) {
		return function appTest() {

		}
	}

	provide([ '/nucleus/base' ], definition);
}());

This example illustrates a module named /app/test which we already know as the first dependency of the prior example. As with the inline module the definition function will receive all dependencies as arguments passed so they are in scope of the actual module.

See these gists to find some skeletons for frequently used types of modules ;)

Sidenote

Starting with Qoopido.demand 7.1.0 a module definition may also return a thenable and if it does, demand will resolve/reject the module accordingly.

Path resolution

Path definitions in demand are totally flexible. Relative paths as well as absolute paths starting with a single / will, by default, be resolved against the base configuration parameter and might get altered afterwards when matching a certain pattern configured.

There is only one exception to this rule: when providing a module with dependencies these dependencies will always get resolved against the modules own path, if the dependencies path is relative.

Absolute URLs starting either with a protocol or // will not get altered at all.

Sidenote

In the past Demand used to interpret paths as being "relative" by default. To be able to handle UMD modules natively this was changed lately. Therefore keep in mind that if you declare a relative path you will have to prefix it with any of ./ or ../.

As always resolving relative paths against base might not be desired and you would prefer or need a relative resolution demand provides three special dependencies:

Whenever you request demand, provide or path as a dependency of a module your modules definition wil get passed a localized version of it.

(function(global) {
	'use strict';

	function definition(demand, provide, path) {
		return function() {
			demand('dependency').then(
				function() {}
			);

			provide('module', function module() {
			})
		};
	}

	provide([ 'demand', 'provide', 'path' ], definition);
}(this));

If you load the above Module from e.g. the directory app/ and name it main.js it will get passed a localized version of demand and provide for the app/ context. So by demanding dependency you actually demand app/dependency and by providing module you really provide app/module.

Available plugins

Beside the above mentioned handlers demand offers a variety of plugins with different aims. Currently demand provides the following loadable plugins:

Cookie: store module cache states in cookies to exchange cache states with the server
LZString: compress/decompress localStorage content to safe space
SRI: adds sub-resource-integrity checks when loading modules

Plugins have to be loaded manually by simply demanding them. They can be configured via demand.configure just like the bundle handler mentioned above. cookie as well as lzstring use the same configuration theme while sriworks only slightly different:

(function(global) {
   'use strict';

   demand.configure({
   	modules: {
   		'/demand/plugin/cookie': {
   			// enable cookie plugin for modules
   			// starting with /app/
   			'/app/': true
   		},
   		'/demand/plugin/lzstring': {
   			// enable compression for all modules ...
   			'': true,
   			// ... but disable it for modules
   			// starting with a certain path
   			'/app/do/not/compress': false
   		},
   		'/demand/plugin/sri': {
   			'/nucleus/dom/element': { type: 'SHA-256', hash: 'pWpW0C5u/YafasONDfkNyRBA4ChXTsRMIk2CGi4wPaU=' }
   		}
   	}
   });
}(this));

The use-cases for lzstring as well as sri should be fairly obvious but cookiemost likely requires some explanation:

In some cases you might want to load (e.g.) a CSS resource via demand (directly from cache) if it has previously been cached and simply inline it server-side if it is not cached already. The cookie plugin will allow you to exchange cache-states between demandand your server to make this scenario possible.

More about handlers

demand comes with handlers for modules, components, legacyscripts, bundles, text, html, css and json. Handlers have four objectives:

provide an optional function named onPreRequest that modifies the final URL (e.g. add a file extension like .js) before requesting it via XHR/XDR
provide an optional function named onPostRequest that, if present, handles necessary conversion of the loaded source (e.g. CSS paths that are normally relative to the CSS-file path)
provide an optional function named onPreProcess
provide an optional process function that will handle DOM injection and final resolution of a module via an anonymous provide call

Handlers can, quite similar to require.js, be explicitly set for a certain module by prefixing the module path by [handler]!. The default handler, e.g., is module which will automatically be used when no other handler is explicitly specified.

I mentioned earlier that demand comes with handlers for modules, legacy JavaScript, bundles, text, CSS and JSON. This is technically not quite correct. As handlers are also modules the only handlers really built-in are module and bundle. All other handlers are automatically loaded on demand and, as they are modules as well, get cached in localStorage.

Handlers will automatically get loaded from demand's original location. So if you want to have a handler that is not present there you simply set your own pattern to change the URL to wherever you like. The default pattern is /demand/handler so if you, e.g., want a handler for mytype loaded from a custom location just create a pattern for /demand/handler/mytype via demand.configure.

All handler methods are called with their context set to the module's instance of Loader.

State of modules

Demand also provides means to get information of the state of modules. Similar to clear there is a list method attached to the global (or local) demand function.

// get a list of all handlers and their modules ...
	// ... regardless of state
	demand.list();

	// ... being currently loaded/resolved
	demand.list.pending();

	// ... that could not be loaded/resolved
	demand.list.rejected();

	// .. that where successfully loaded and resolved
	demand.list.resolved();

Removing loaded modules

If you need a possibility to remove an already loaded Module and its cache to force a reload (for, e.g., "hot reloading") Qoopido.demand provides a built-in demand.remove method that accepts a module path as argument. The code snippet

demand.remove('/nucleus/dom/element');

will remove any loaded version of this module from demand and will also clear the module's cache if it exists so that the next call of demand('/nucleus/dom/element') will fetch a fresh copy via XHR.

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

Qoopido.demand

Key features in detail

Compatibility

Requirements

External dependencies

Availability

Loading demand

Configuration

Usage

Controlling the cache

Demanding modules

Auto-bundling with genie

Providing inline modules

Developing loadable modules

Path resolution

Available plugins

More about handlers

State of modules

Removing loaded modules