@prantlf/amodro-trace

AMD module tracing for build processes.

Like the requirejs optimizer, but just traces a module ID for its nested dependencies, and the result is a data structure for that trace, instead of a fully optimized build.

It uses requirejs underneath to do the tracing, so things like loader plugins work and it has a full understanding of AMD APIs. It can also normalize the module contents so multiple define calls can be concatenated in a file.

Use the requirejs optimizer if you want a more complete build tool. Use this if you want more control over the build processes, and just want something to do the AMD bits.

This fork targets modern JavaScript code, which the original project fails parsing. The changes shouldn't break existing projects:

• Upgrade the parser to work with modern JavaScript.
• Pass only ESTree AST nodes to visitors. Not arrays or other objects.
• Ignore keys in AST nodes pointing to parent nodes. (Often needed and would cause a stack overflow.)
• Fix crashing with edge cases of usage of define and require statements.
• Support the simplified CJS syntax in when parsing a file with the single wrapping require statement.
• Expose parse, traverse, traverseBroad findDependencies and findCjsDependencies.

If you need an optimizer capable of processing modern JavaScript, have a look at @prantlf/r.js.

Use cases

Dependency tree for a module ID

Your build process uses a build dependency graph, like the one used by make. However, that tool does not understand module tracing. You can use this tool to figure out the graph for a given module ID. For fancier dependency graphing, node-madge may be a better fit.

amodro-trace starts with a single module ID, an AMD loader config, and traces the dependency tree that module ID. Example:

var amodroTrace = require('@prantlf/amodro-trace'),
    path = require('path');

amodroTrace(
  // The options for trace
  {
    // The root directory, usually the root of the web project, and what the
    // AMD baseUrl is relative to. Should be an absolute path.
    rootDir: path.join(__dirname, 'www'),

    // The module ID to trace.
    id: 'app'
  },
  // The AMD loader config to use.
  {
    baseUrl: 'lib',
    paths: {
      app: '../app'
    }
  }
).then(function(traceResult) {
  // The traceResult has this structure:
  traceResult = {
    traced: [
      { "id": "b", "path": "/full/path/to/www/lib/b.js", "dependents": ["a"] },
      { "id": "a", "path": "/full/path/to/www/lib/a.js", "deps": ["b"], "dependents": ["app/main"] },
      { "id": "app/main", "path": "/full/path/to/www/app/main.js", "deps": ["a"], "dependents": ["app"] }
      { "id": "app", "path": "/full/path/to/www/app.js", "deps": ["app/main"]  }
    ],

    // If parsing triggered warnings or errors, they will show here, as an array
    // of strings. These are treated as non-fatal, in that the file with the
    // issue is skipped (may just be invalid JS not meant to be fully traced),
    // but for best results it is best to investigate the messages that show
    // up here.
    warnings: [],
    errors: []
  };
}).catch(function(error) {
  console.error(error);
});

Content transforms after AMD normalization

You want a trace of the modules that may be used in a build layer, with the AMD calls normalized for file concatenation, but you want to do further work before combining them and generating the source map and minifying.

The call is similar to the simple dependency tree result, but ask amodro-trace to include the contents and provide a "write" transform that normalizes the AMD calls:

var amodroTrace = require('@prantlf/amodro-trace'),
    allWriteTransforms = require('@prantlf/amodro-trace/write/all'),
    path = require('path');

// Create the writeTransform function by passing options to be used by the
// write transform factories:
var writeTransform = allWriteTransforms({
  // See the write transforms section for options.
});

amodroTrace(
  // The options for trace
  {
    rootDir: path.join(__dirname, 'www'),
    id: 'app',

    includeContents: true,
    writeTransform: writeTransform
  },
  // The AMD loader config to use.
  {
    baseUrl: 'lib',
    paths: {
      app: '../app'
    }
  }
).then(function(traceResult) {
  // The traceResult has this structure:
  traceResult = {
    traced: [
      {
        "id": "b",
        "path": "/full/path/to/www/lib/b.js",
        "contents": "define('b',{\n  name: 'b'\n});\n",
        "dependents": [
          "a"
        ]
      },
      {
        "id": "a",
        "path": "/full/path/to/www/lib/a.js",
        "contents": "define('a',['b'], function(b) { return { name: 'a', b: b }; });",
        "deps": [
          "b"
        ],
        "dependents": [
          "app/main"
        ]
      },
      {
        "id": "app/main",
        "path": "/full/path/to/www/app/main.js",
        "contents": "define('app/main',['require','a'],{\n  console.log(require('a');\n});\n",
        "deps": [
          "a"
        ],
        "dependents": [
          "app"
        ]
      },
      {
        "id": "app",
        "path": "/full/path/to/www/app.js",
        "contents": "require.config({\n  baseUrl: 'lib',\n  paths: {\n    app: '../app'\n  }\n});\n\nrequire(['app/main']);\n\ndefine(\"app\", [],function(){});\n",
        "deps": [
          "app/main"
        ]
      }
    ],

    // If parsing triggered warnings or errors, they will show here, as an array
    // of strings. These are treated as non-fatal, in that the file with the
    // issue is skipped (may just be invalid JS not meant to be fully traced),
    // but for best results it is best to investigate the messages that show
    // up here.
    warnings: [],
    errors: []
  };
}).catch(function(error) {
  console.error(error);
});

Non-file inputs

Your build tool may not deal with files directly, maybe it builds up a list of files in a stream-backed objects. Gulp for example.

The fileExists and fileRead function options allow you to do this:

var amodroTrace = require('@prantlf/amodro-trace'),
    path = require('path');

// Build up an in-memory data structure for the files/modules involved.
// This example uses module IDs as the keys, but you could decide to use paths.
var fileMap = {
  main: 'require([\'a\'], function(a) {});',
  a: 'define([\'b\'], function(b) { return { name: \'a\', b: b }; });',
  b: 'define({\n  name: 'b'\n});\n'
};

amodroTrace(
  // The options for trace
  {
    // The root directory, usually the root of the web project, and what the
    // AMD baseUrl is relative to. Should be an absolute path.
    rootDir: path.join(__dirname, 'www'),

    // The module ID to trace.
    id: 'app',

    // You can ovrride the file existence checks by passing in a function for
    // fileExists. defaultExistst is the default exists function used by this
    // project's internals. A synchronous boolean result is expected to be
    // returned from this function. It should only return true for files, and
    // not directories.
    fileExists: function(defaultExists, id, filePath) {
      return fileMap.hasOwnProperty(id);
    },

    // You can override file reading by passing in a function for fileRead.
    // defaultRead is the default file reading function used by this project's
    // internals. You can call it if you want to delegate to that
    // functionality. A synchronous result is expected to be returned from this
    // function.
    fileRead: function(defaultRead, id, filePath) {
      return fileMap[id];
    }
  },
  // The AMD loader config to use.
  {
    baseUrl: 'lib',
    paths: {
      app: '../app'
    }
  }
).then(function(traceResult) {
  // See other examples for traceResult structure.
}).catch(function(error) {
  console.error(error);
});

Transform files to AMD before tracing

If you are using a transpiled language, or want to author in CommonJS (CJS) format but output to AMD, you can provide a read transform that can modify the contents of files after they are read but before they are traced.

Here is an example that uses the cjs read transform provided in this project (it just wraps CJS modules in AMD wrappers, it does not change module ID resolution rules):

var amodroTrace = require('@prantlf/amodro-trace'),
    cjsTransform = require('@prantlf/amodro-trace/read/cjs'),
    path = require('path');

amodroTrace(
  // The options for trace
  {
    // The root directory, usually the root of the web project, and what the
    // AMD baseUrl is relative to. Should be an absolute path.
    rootDir: path.join(__dirname, 'www'),

    // The module ID to trace.
    id: 'app',

    readTransform: function(id, url, contents) {
      return cjsTransform(url, contents);
    }
  },
  // The AMD loader config to use.
  {
    baseUrl: 'lib',
    paths: {
      app: '../app'
    }
  }
).then(function(traceResult) {
  // See other examples for traceResult structure.
}).catch(function(error) {
  console.error(error);
});

Install

This project runs in node/iojs, and is installed via npm:

npm install @prantlf/amodro-trace

API

amodro-trace

amodro-trace(options, loaderConfig);

Returns a Promise. The resolved value will be a result object that looks like this:

{
  "traced": [
    {
      "id": "b",
      "path": "/full/path/to/www/lib/b.js",
      "contents": "define('b',{\n  name: 'b'\n});\n",
      "dependents": [
        "a"
      ]
    },
    {
      "id": "a",
      "path": "/full/path/to/www/lib/a.js",
      "contents": "define('a',['b'], function(b) { return { name: 'a', b: b }; });",
      "deps": [
        "b"
      ],
      "dependents": [
        "app/main"
      ]
    },
    {
      "id": "app/main",
      "path": "/full/path/to/www/app/main.js",
      "contents": "define('app/main',['require','a'],{\n  console.log(require('a');\n});\n",
      "deps": [
        "a"
      ],
      "dependents": [
        "app"
      ]
    },
    {
      "id": "app",
      "path": "/full/path/to/www/app.js",
      "contents": "require.config({\n  baseUrl: 'lib',\n  paths: {\n    app: '../app'\n  }\n});\n\nrequire(['app/main']);\n\ndefine(\"app\", [],function(){});\n",
      "deps": [
        "app/main"
      ]
    }
  ],

  "warnings": [],
  "errors": []
}

The contents property for an entry is only included if the includeContents or writeTransform options are used. If keepLoader option is used, the result object will include a loader property.

The traced results are order by least dependent to more dependent. So, modules with no dependencies come first.

Each module entry also includes the normalized IDs for the dependencies in the deps property. If no dependencies, the deps property is not there. For files that have multiple named define()'d modules, their IDs with their dependencies will be listed with the otherIds property. Only top level define()s in the file are found, nested ones inside a UMD wrapper should not be traced.

Example result where "view1" was a built file containining a few other modules:

{
  "traced": [
    {
      "id": "view1",
      "path": "/full/path/to/view1.js",
      "deps": [
        "header",
        "content",
        "footer"
      ],
      "otherIds": {
        "inlay": {},
        "button": {},
        "header": {
          "deps": [
            "button"
          ]
        },
        "content": {
          "deps": [
            "inlay",
            "button"
          ]
        },
        "footer": {
          "deps": [
            "button"
          ]
        }
      }
    },
    {
      "id": "main",
      "path": "/full/path/to/main.js",
      "deps": [
        "view1"
      ]
    }
  ],

  "warnings": [],
  "errors": []
}

Eache module entry may also include a dependents property, which is the set of module IDs that statically specify the module as a dependency. It is only the direct dependents, not the dependents of those dependents.

loaderConfig is the AMD loader config that would be used by an AMD loader to load those modules at runtime. If you want to extract the loader config from an existing JS file, amodro-config can help with that.

If parsing triggered warnings or errors, they will show in the warnings or errors arrays respectively, as an array of strings. These are treated as non-fatal, in that the file with the issue is skipped (may just be invalid JS not meant to be fully traced), but for best results it is best to investigate the messages that show up here. If there are no warnings or errors, those properties will not show up in the trace result.

options

The following options

rootDir

String. The full path to the root of the project to be scanned. This is usually the top level directory of the project that is served to the web, and the reference directory for relative baseUrls in an AMD loader config.

id

String. the module ID to trace.

traced

If there was a previous traceResult that should be used as part of this new trace, then pass the traceResult.traced as this traced property. The items in the traced array will be used instead of loading and parsing the file contents in the current trace() call.

Notes:

• The traced array items should include the deps property. Only the id and deps properties are used from the traced array items.

findNestedDependencies

Boolean. Defaults to false. Normally require([]) calls inside a define()'d module are not traced, as they are usually meant to be dynamically loaded dependencies and are not static module dependencies.

However, for some tracing cases it is useful to include these dynamic dependencies. Setting this option to true will do that. It only captures require([]) calls that use string literals for dependency IDs. It cannot trace dependency IDs that are variables for JS expressions.

fileRead

Function. A function that synchronously returns the file contents for the given file path. Allows overriding where file contents come from, for instance, building up an in-memory map of file names and contents from a stream.

Arguments passed to this function:

function(defaultReadFunction, moduleName, filePath) {}

Where defaultReadFunction is the default read function used. You can call it with the filePath to get the contents via the normal file methods this module uses to get file contents.

fileExists

Function. If fileRead is provided, this function should be provided too. Determines if a file exists for the mechanism that reads files. A synchronous Boolean answer is expected. Signature is:

function(defaultExistsFunction, moduleName, filePath) {}

Where defaultExistsFunction is the default exists function used by the internals of this module.

readTransform

Function. A function that is used to transform the contents of the modules after the contents are read but before they are parsed for module APIs. The function will receive these arguments:

function(moduleName, filePath, contents) {}

and should synchronously return a string that will be used as the contents. If the readTransform does not want to alter the contents then it should just return the contents value passed in to it.

The readTransform function is run after the file has been read (or after fileRead has run), but before the text contents enter the loader for parsing and tracing. The result of the read transform is what is returned in the contents property for traced items if includeContents is set to true and no writeTransform is specified.

includeContents

Boolean. Set to true if the contents of the modules should be included in the output. The contents will be the contents after the readTransform function has run, if it is provided.

writeTransform

Function. When contents are added to the result, run this function to allow transforming the contents. See the write transforms section for example transforms. Setting this option automatically sets includeContents to be true. writeTransform should be a function with this signature:

function(context, moduleName, filePath, contents) {}

Where context is a loader context object created by the internal loader object used by amodro-trace. This object is mostly used by the write transforms included in this project, since they coordinate and deal with some of the results of the parsing, like what files needs AMD wrappers or normalization. It should be treated as an opaque object.

The function should synchronously return the value that should be used as the new value for contents. If the writeTransform does not want to alter the contents then it should just return the contents value passed in to it.

keepLoader

Boolean. Keep the loader instance and pass it in the return value. This is useful if transforms that depend on the instance's context will be used to transform the contents, and where writeTransform is not the right fit. For most uses though, writeTransform should be preferred over manually using the loader instance.

The traced result will include a loader property with the loader instance. You should call loader.discard() when you are done using it, to help clean up resources used by the loader.

If manually calling some transforms that would normally be called via writeTransform, you can use loader.getContext() to get the context object passed to those transforms. Example:

var amodroTrace = require('@prantlf/amodro-trace');

// Use the defines write transform manually.
var defineTransform = require('@prantlf/amodro-trace/write/defines')({});

amodroTrace({}, {}).then(function(traceResult) {
  var traced = traceResult.traced,
      loader = traceResult.loader,
      context = loader.getContext();

  // Iterate over all the traced items and modify their contents in some way.
  traced.forEach(function(item) {
    item.contents = defineTransform(context, item.id, item.path, item.contents);
  });

  // All done with the loader
  loader.discard();
}).catch(function(error) {
  console.error(error);
});

amodro-trace/config

This module helps extract or modify a require.config()/requirejs.config() config inside a JS file. The API methods on this module:

config.find

Finds the first requirejs/require call to require[js].config/require({}) in a file and returns the value as an object. Will not work with configs that use variable references outside of the config definition. In general, config calls that look more like JSON will work best.

var config = require('amodro-config').find(contents);

Aruguments to find:

• contents: String. File contents that might contain a config call.

Returns an Object with the config. Could be undefined if a config is not found.

config.modify

Modify the contents of a require.config/requirejs.config call and places the modifications bac in the contents. This call will LOSE any existing comments that are in the config string.

var config = require('amodro-config')
.modify(contents, function onConfig(currentConfig) {
  // This example just modifies the baseUrl.
  currentConfig.baseUrl = 'new/base';
  return currentConfig;
});

Arguments to modify:

• contents: String. File contents that may contain a config call.
• onConfig: Function. Function called when the first config call is found. It will be passed an Object which is the current config, and the onConfig function should return an Object to use as the new config that will be serialized into the contents, replacing the old config.

Returns a String the contents with the config changes applied.

amodro-trace/parse

This module helps to extract dependencies from a JS file, which is an AMD module - which is wrapped in a require or a define statement.

Methods accept either a string with the source content or an object with JavaScript AST. If you traverse the AST multiple times, you can improve performance by parsing the source just once by a parser producing output according to the ESTree Spec. For example, Meriyah:

var meriyah = require('meriyah');
var astRoot = meriyah.parse(fileContents, { module: true, next: true });

The method parse below using this parses is exposed for convenience.

The API methods on this module:

parse.parse

Parses the input JavaScript text and returns an AST of it. Expects a JavaScript content. The returned object can be used later in other calls, or to other module analysis.

var astRoot = require('@prantlf/amodro-trace/parse').parse(contents, options);

Arguments to parse:

• contents: String. File contents of an AMD module.
• options: Object. Optional. Options for the meriyah parser: Only ranges and loc properties are recognized. Also range will be recognised for compatibility woth esprima.

Returns an Object with the JavaScript AST build from the module contents.

parse.traverse

Walks the AST from the specified (root) node up to its leaves and calls the specified callback function with two arguments - the visited node and its parent node. Once the callback returns false, traversing stops.

require('@prantlf/amodro-trace/parse').traverse(rootNode, function (node, parent) {
  ...
});

Arguments to traverseBroad:

• node: Object. AST root node to start traversing with. Produced by the parse method.
• visitor: Function. Callback receiving nodes with its parents.

parse.traverseBroad

Walks the AST from the specified (root) node up to its leaves and calls the specified callback function with two arguments - the visited node and its parent node. Once the callback returns false, its children will be skipped and the traversing will continue with its next sibling.

require('@prantlf/amodro-trace/parse').traverseBroad(rootNode, function (node, parent) {
  ...
});

Arguments to traverseBroad:

• node: Object. AST root node to start traversing with. Produced by the parse method.
• visitor: Function. Callback receiving nodes with its parents.

parse.findDependencies

Finds all dependencies specified in the AMD module dependency array, or inside simplified CommonJS wrappers inside the module factory function. Expects a JavaScript AMD module wrapped in a requirejs/require/define call. The returned list of dependent modules and their formal parameters match in the correct code.

var dependencies = require('@prantlf/amodro-trace/parse').findDependencies(contents);

Arguments to findDependencies:

• contents: String or Object. File contents of an AMD module, or an AST root produced by the parse method.

Returns an object with two properties. The property "modules" is an array of dependent module paths as strings. The dependencies have not been normalized; they may be relative IDs. The property "params" is an array of formal parameter names as strings.

parse.findCjsDependencies

Finds only CommonJS dependencies, ones that are the form require('stringLiteral'). Expects a JavaScript module. The returned list of dependent modules and their formal parameters match in the correct code.

var dependencies = require('@prantlf/amodro-trace/parse').findCjsDependencies(contents);

Arguments to findCjsDependencies:

• contents: String or Object. File contents of a CommonJS module, or an AST root produced by the parse method.

Returns an object with two properties. The property "modules" is an array of dependent module paths as strings. The dependencies have not been normalized; they may be relative IDs. The property "params" is an array of formal parameter names as strings.

Read transforms

See the readTransform option for background on read transforms. This section describes read transforms provided by this project.

cjs

To use:

var cjsTransform = require('@prantlf/amodro-trace/read/cjs');

If the transform detects CommonJS usage without an AMD or UMD wrapper that includes an AMD branch, then the text will be wrapped in a define(function(require, exports, module){} wrapper.

Write transforms

See the writeTransform option for background on write transforms. This section describes the write transforms provided by this project. These transforms come from the write transforms that the requirejs optimizer would do to normalize scripts for concatenation.

They are listed in the order they are suggested to be applied. There is an @prantlf/amodro-trace/write/all module that chains all of these together in the correct order. If you need an example on how to chain a few different transforms together to just provide one writeTransform to amodro-trace, look at the source of @prantlf/amodro-trace/write/all.

All of these write transform modules export a function that can be called with an options object to generate the final function that should be passed to writeTransform. This allows one time options setup that applies to all scripts that are transformed to just happen once. This pattern is suggested in general for write transforms.

stubs

The stubs transform will replace a given set of module IDs with stub define() calls instead of the original contents of the modules. This is useful for modules that are not needed in full after a build and just need to be registered as being satisfied dependencies.

var stubs = require('@prantlf/amodro-trace/write/stubs');
var stubsTransform = stubs({
  // An array of exact module IDs to stub out.
  stubModules: ['text', 'a/b']
});

require('@prantlf/amodro-trace')({
  writeTransform: stubsTransform
});

defines

The defines transform will normalize define calls to include the module ID and parse out the dependencies so that Function.prototype.String is not needed at runtime to find dependencies. It will also add in some define calls for shimmed dependencies.

This transform is the one that normally is always needed. The other transforms could be avoided depending on project needs.

The only specific option for this transform is wrapShim. It provides wrapping of shim values similar to the requirejs optimizer option of the same name. Normally this should not be needed. Only set it to true for specific cases.

var defines = require('@prantlf/amodro-trace/write/defines');
var definesTransform = defines({
  wrapShim: true
});

require('@prantlf/amodro-trace')({
  writeTransform: definesTransform
});

packages

The packages transform will write out an adapter define() for a packages config main module value so that package config is not needed to map 'packageName' to 'packageName/mainModuleId'.

It does not have any transform-specific options.

var packages = require('@prantlf/amodro-trace/write/packages');
var packagesTransform = packages();

require('@prantlf/amodro-trace')({
  writeTransform: packagesTransform
});

requirejs optimizer differences

The feature set and config options are smaller since it has a narrower focus. If you feel like you are missing a feature from the requirejs optimizer, it usually can be met by creating a write transform to do what you want. While this project comes with some transforms, it does not support all the transforms that the requirejs optimizer can do. For example, this project's write transforms do not understand how to make namespace builds.