metalsmith-data-loader

Metalsmith plugin to add extra metadata from external files. Very similar to metalsmith-models with the difference that this allows you to load information from either a folder outside of the source or files within the source tree. Also, this one lets you use JSON or YAML files.

The "dependencies" badge says there's a potential problem with js-yaml and suggests that packages use .safeLoad(), which is exactly what this plugin does.

What It Does

When working with templates, you sometimes want to generate something like a table in Markdown from external data. You can do this with metalsmith-hbt-md but you need to get the extra data in your metadata. That can be accomplished by metalsmith-models as long as you are willing to separate your source data from the template that needs it.

This plugin differs from that approach, allowing you to have your Markdown files adjacent to the data that is inserted into the table. This plugin look at the data property (configurable) in your file's metadata, then replace that filename with the parsed contents of the file. This works better with an example.

table.md:

---
title: Just a test file to illustrate why the module is useful.
data: contacts.yaml
---

Here's a great list of contacts:

| First | Last | Email |
|-------|------|-------|
{{#data}}| {{first}} | {{last}} | {{email}} |
{{/data}

contacts.yaml (right next to the markdown):

-
    first: Tyler
    last: Akins
    email: [email protected]
-
    first: Jane
    last: Doe
    email: [email protected]

After parsing, the metalsmith file object for table.md would look like this:

---
title: Just a test file to illustrate why the module is useful.
data:
    -
        first: Tyler
        last: Akins
        email: [email protected]
    -
        first: Jane
        last: Doe
        email: [email protected]
---

Here's a great list of contacts:

| First | Last | Email |
|-------|------|-------|
{{#data}}| {{first}} | {{last}} | {{email}} |
{{/data}

This isn't limited to table generation. You could load metadata specific to a collection of pages. Maybe you have a site where different authors maintain different pages and you could point to a single source for the author's information.

If you prefer to work with code, there is an example repository set up that illustrates how the plugin functions. Please pay special attention to how the extra data is referenced in the templates.

Note: You can also use an array of filenames or an object whose values are all filenames. More information in the "Usage" section.

Installation

npm can do this for you.

npm install --save metalsmith-data-loader

Usage

Include this like you would include any other plugin. First, a CLI example that shows the default options. You don't need to specify any options unless you want to override their values.

{
    "plugins": {
        "metalsmith-data-loader": {
            "dataProperty": "data",
            "directory": "models/",
            "ignoreReadFailure": false,
            "match": "**/*",
            "matchOptions": {},
            "removeSource": false
        }
    }
}

And here is the JavaScript example. This includes brief descriptions of each of the options.

// Load this, just like other plugins.
var dataLoader = require("metalsmith-data-loader");

// Then in your list of plugins you use it.
.use(dataLoader())

// Alternately, you can specify options.  The values shown here are
// the defaults.
.use(dataLoader({
    // Property name to scan for files to include.
    dataProperty: "data",

    // Directory containing models.  This is relative to the working
    // directory.  It does not need to exist unless you start loading
    // files from here.
    directory: "models/",

    // Pattern of files to match in case you want to limit processing
    // to specific files.
    match: "**/*",

    // Options to ignore read failures, useful if you want to continue
    // processing even if the file is not found. When this is set to false,
    // the build will throw an exception when trying to load a file that
    // does not exist and is typically what people want.
    ignoreReadFailure: false,

    // Options for matching files.  See metalsmith-plugin-kit for
    // more information.
    matchOptions: {},

    // Flag indicating that the loaded data object should be removed
    // if it is found in the source.  Use this so your built project
    // doesn't include the metadata in the rendered output and another
    // copy that was consumed during the build.
    removeSource: false
})

This uses metalsmith-plugin-kit to match files. The .matchOptions object can be filled with options control how the matching behaves.

From here, you now need to specify the files to include. These examples all assume you didn't change dataProperty.

---
title: A single string loads one file
data: my-model.json
---

When loaded this way, the data metadata is replaced with the parsed contents of my-model.json.

---
title: An object holding a map of filenames.
data:
    users: users.json
    services: services.yaml
---

This lets you load multiple files and assign them to properties on the data object. One would access the information by using {{data.users}} or {{data.services}} in Mustache templates.

---
title: An array of files.  The filenames are replaced with their contents.
data:
    - file1.yaml
    - file2.yaml
---

The two files are loaded asynchronously and will replace the file1.yaml and file2.yaml. You would access these in Mustache by using {{data.0}} and {{data.1}} or you can iterate over {{#data}}.

Resolving Files

The combination of the directory configuration option and the data metadata property dictate which files are loaded. This table can help illustrate the relationship. In all of the examples, the directory configuration option is set to "models/" and the source file is always "src/path/file.md".

| Metadata Path | File to Load | Description | |-----------------------|--------------------------|-------------------------------------------------------------------| | file.yaml | src/path/file.yaml | Relative to source file. | | ../file.yaml | src/file.yaml | Relative to the source file. | | /other-path/file.yaml | src/other-path/file.yaml | Resolved from root of source folder. | | /../file.yaml | file.yaml | Can load things outside the source folder. | | !file.yaml | models/file.yaml | Resolved from the directory, not source folder. See the note! | | !../file.yaml | file.yaml | Can load items from outside the models directory. See the note! |

Note: When the metadata path starts with an exclamation point (!), you must use quotes in your YAML frontmatter. Here's a sample.

---
title: Sample file that loads a file from a models directory
data: "!file-from-models.json"
---

You can play with the example repository to get a better handle on how the plugin works.

API

metalsmith-data-loader

Params

• destination Object - Metalsmith Data Loader

Loads files and places their data structures in as file metadata.

Example

var dataLoader = require("metalsmith-data-loader");

// Create your Metalsmith instance and add this like other middleware.
metalsmith.use(dataLoader({
    // configuration goes here
}));

• metalsmith-data-loader
  • module.exports(options) ⇒ function ⏏
    • ~removeSource(sourceFile, modelFile, files, options)
    • ~resolveFile(metalsmith, sourceFile, modelFile, options) ⇒ string
    • ~loadModel(dataFile) ⇒ Promise.<*>
    • ~metalsmithFile : Object
    • ~metalsmithFileCollection : Object.<string, module:metalsmith-data-loader--module.exports~metalsmithFile>
    • ~options : Object

module.exports(options) ⇒ function ⏏

Factory to build middleware for Metalsmith.

Kind: Exported function
Params

• options options

module.exports~removeSource(sourceFile, modelFile, files, options)

Removes a file from the list of files to process. This only happens when it is included by another file through this data loader and when the option is enabled.

Kind: inner method of module.exports
Params

• sourceFile string - filename
• modelFile string - Not resolved to a full path
• files metalsmithFileCollection - Metalsmith files object
• options options

module.exports~resolveFile(metalsmith, sourceFile, modelFile, options) ⇒ string

Resolves a file path and checks to make sure it exists.

Paths can look like any of these:

model.json -> Relative to file ./model.json -> Relative to file dir/model.yaml -> Relative to file ../model.json -> Relative to file /model.yaml -> From root of source /dir/model.json -> From root of source !model.yaml -> From models folder (not in source) !dir/model.json -> From models folder (not in source)

Kind: inner method of module.exports
Params

• metalsmith module:metalsmith - Metalsmith instance.
• sourceFile string - Filename being processed.
• modelFile string - Reference inside the file's metadata.
• options options

module.exports~loadModel(dataFile) ⇒ Promise.<*>

Loads a file asynchronously and places metadata on the destination object.

Kind: inner method of module.exports
Returns: Promise.<*> - The loaded data.
Params

• dataFile string - Resolved filename to load.

module.exports~metalsmithFile : Object

Metalsmith file object.

Kind: inner typedef of module.exports
Properties

• contents Buffer
• mode string

module.exports~metalsmithFileCollection : Object.<string, module:metalsmith-data-loader--module.exports~metalsmithFile>

Metalsmith collection of file objects.

Kind: inner typedef of module.exports

module.exports~options : Object

Options for the middleware factory.

Kind: inner typedef of module.exports
See: https://github.com/fidian/metalsmith-plugin-kit
Properties

• dataProperty string - Name of property to use for loading models.
• directory string - Path for storing external modules.
• match module:metalsmith-plugin-kit~matchList - Files to match. Defaults to all files.
• matchOptions module:metalsmith-plugin-kit~matchOptions - Options controlling how to match files.
• removeSource boolean - When truthy, remove the model from the build result.

Development

This uses Jasmine, Istanbul and ESLint for tests.

# Install all of the dependencies
npm install

# Run the tests
npm run test

This plugin is licensed under the MIT License with an additional non-advertising clause. See the full license text for information.