@ndcode/jst

v0.1.5

Published

3 years ago

JavaScript Templates for HTML

Downloads

0High
0Medium
0Low

nick_d2

template HTML webpage dynamic server-side render

JavaScript Template system

An NDCODE project.

Overview

The jst package exports a single function jst(text, options) which parses the given text in a JavaScript dialect which can contain HTML-like constructs, then generates equivalent plain JavaScript code which when run, would generate the requested HTML. As well as the HTML generation, there can be normal JavaScript logic to influence the generated page or substitute into it.

The system is inspired by Pug (formerly Jade) templates, but emphasizing the JavaScript, for instance you do not need a - sign at the start of each line, and the parser automatically follows the JavaScript block scope. It is also inspired by JSX templates, but the embedded HTML is less verbose, there are no closing tags since HTML is bracketed by { } rather than by <tag>...</tag>.

Template syntax

HTML tags in templates

The plain JavaScript in the file is absolutely normal and is expected to use CommonJS conventions, for instance, require(...) is supported and so forth.

The embedded HTML uses a syntax similar to JavaScript function definitions, function(arg1, arg2, ...) {...} except that instead of the word "function" any valid HTML (or other) tag name is allowed, and instead of just argument names, HTML attributes of the form attrname or attrname=value are supported. In the attrname=value syntax, value is any valid JavaScript expression. No commas are required between attributes, so the parser has to automatically detect where one expression ends and another begins (similarly to how automatic semicolon insertion works in regular JavaScript).

The ( ) around the attributes is not necessary when there are no attributes. Thus a simple HTML file could be written as the following JavaScript template:

html(lang="en") {
  head {}
  body {}
}

and this would translate to the HTML file:

<html lang="en"><head></head><body></body></html>

For certain particular HTML tags such as img, no closing tag is generated in the output HTML. However, the empty { } must still be given in the template. This is partly for the parser's convenience, since it depends on recognizing the { } to distinguish between an HTML template and ordinary JavaScript code. It is also more uniform: there is no need to remember which tags are special.

Regular text in templates

Regular text is placed in the generated HTML by simply quoting it. That is, if a statement is seen consisting of only a JavaScript string or template string, the string is converted to HTML by escaping it (helping to guard against any HTML injection attacks), and then output as part of some enclosing template.

For example:

html(lang="en") {
  body {'Hello, world'}
}

This generates the text:

<html lang="en"><body>Hello, world</body></html>

If the text is multi-line, backquoted (template) strings are used, to allow embedded newlines. When the entire text is held in a variable, e.g. myvar, a template string such as ${myvar} should be used, to convert it into a standalone statement consisting of only a string constant, and hence into HTML.

Note that ordinary single- or double-quoted string constants should be used in preference to template strings where possible, since the constant strings will be HTML-escaped at template compile time rather than at template run-time.

Note that in ordinary HTML, certain tags are more sensitive to whitespace than others, according to complicated rules about inline block elements and so on. This is never an issue with JavaScript templates, we can use as much indenting and other whitespace as we want, and only quoted whitespace will be output.

Special tags in templates

The HTML script tag is treated specially, because it contains JavaScript, which is understood directly by the template parser. The JavaScript inside the script tag is minified (comments stripped, unnecessary braces and parentheses removed, semicolons added and so on), and then converted to a string constant, which will be copied out to the HTML file whenever the template is executed.

The HTML style tag is treated specially, because it contains CSS, which we handle by switching temporarily to another parser that understands CSS. We use a slightly modified version of the clean-css package to do this. The result is then minified and converted to a string constant, similar to script tags.

Note that the script and style tags cannot contain substitutions, in the way that ordinary JST code can. This is because (i) the entire contents of the special tag are minified and stringified at template compile time, so the content of the special tag is the same each time the page is generated, and (ii) it runs elsewhere, which doesn't have access to the template variables.

HTML class and id shorthand

The tag name can be followed by #name as shorthand for the HTML attribute id="name" or by .name as shorthand for the HTML attribute class="name" and these can be repeated as needed, the id attribute collects all #name separated by spaces and the class attribute collects all .name similarly. These must come before any ordinary attributes (before an opening parenthesis).

Parser limitations

Certain tag or attribute names present difficulty since they contain - signs or other characters invalid in JavaScript identifiers, or they may be reserved words in JavaScript. The parser copes with this quite well (most syntax errors can already be re-parsed as tag or attribute names), but in difficult cases it could be necessary to quote the tag and/or attribute names. For example, div.class-1.class-2 {} doesn't compile because "1." is a floating point number, for now we write it div.'class-1'.class-2 {} although we expect that this restriction can be lifted in a future version.

Also, the style parser for CSS code can be confused by tag, id or class names that end with -style. So we should be careful of code like this, div.my-style {...} since the div.my- prefix won't be seen until parsing is complete, hence the parser will switch to CSS parsing inside the braces. Therefore, quote it like div.'my-style' {...} although again, we expect that this restriction can be lifted in the future.

Another slight limitation of the current parser is that it is more permissive than normal in parsing regular JavaScript code, for instance commas are not required between function arguments, because an HTML template is basically parsed as a function call until the opening { is seen to identify it as a template. This can also likely be improved in a future version of the system.

Expression vs statement templates

HTML templates occuring at expression level will be completely rendered to a string, and the resulting string returned as the value of the expression.

HTML templates occurring at statement level are treated somewhat like quoted strings, in that the generated HTML will become part of an enclosing template.

Here is a complete example showing template substitution and HTML expressions:

let lang = 'en'
let name = 'John'
console.log(
  html(lang=lang) {
    body {`Hello, ${name}`}
  }
)

Running the above program will generate the output:

<html lang="en"><body>Hello, John</body></html>

Template output buffer

It is also possible to use statement-level HTML templates or strings to build up an output buffer, which can be used for whatever purpose. The output buffer must be called _out and must be a JavaScript array of string fragments. As each output fragment (such as an opening or closing tag or an embedded string) is generated, it will be sent to the output buffer by an _out.push(...) call.

If there is dynamically generated text in the template, then it will be esacped by the sequence .replace("&", "&").replace("<", "<"), this is chosen so that no external dependency such as the html-entities package is needed, and so that unnecessary escaping of characters such as ' or > is not done. Attributes are done similarly, except that " is replaced instead of <. We assume a UTF-8 environment, so there is really little need for HTML entities.

For example, consider a realistic template which we use on our demo server:

html(lang=lang) {
  head {
    link(rel="stylesheet" type="text/css" href="css/styles.css") {}
  }
  body {
    p {`Hello, ${name}`}
  }
}

This compiles to the following plain JavaScript code:

_out.push("<html lang=\"" + lang.toString().replace("&", "&amp;").replace("\"", "&quot;") + "\">");
{
  _out.push("<head>");
  _out.push("<link rel=\"stylesheet\" type=\"text/css\" href=\"css/styles.css\">");
  _out.push("</head>");
}
{
  _out.push("<body>");
  {
    _out.push("<p>");
    _out.push(`Hello, ${name}`.replace("&", "&amp;").replace("<", "&lt;"));
    _out.push("</p>");
  }
  _out.push("</body>");
}
_out.push("</html>");

If invoked as an expression, the same code will be generated, but wrapped in an Immediately-Invoked Function Expression (IIFE) that creates the _out buffer, executes the above, then returns the buffer concatenated into a single string.

Usage example

Suppose we want to try out the JST template processor. Returning to the example from the section "Expression vs Statement templates", we could create a file called mytemplate.jst containing the example JST code to print a simple page:

let lang = 'en'
let name = 'John'
console.log(
  html(lang=lang) {
    body {`Hello, ${name}`}
  }
)
```js
We could then create an example JavaScript program `example.js` to read,
compile and then execute the template, as follows:
```js
let fs = require('fs')
let jst = require('@ndcode/jst')

eval(jst(fs.readFileSync('mytemplate.jst')))

To run this, we would have to run the node interpreter at the command line:

node example.js

This produces the output described earlier:

<html lang="en"><body>Hello, John</body></html>

Command line preprocessor

Using the programmatic interface is overkill in many situations, e.g. if we just want to see the generated JavaScript code, or if we want to compile all pages of our website to JavaScript ahead of time by a Gruntfile or similar.

Hence, we provide the command line interface jst as a convenient way to convert JST to JavaScript. Using the command-line, the above example becomes:

jst <mytemplate.jst >mytemplate.js
node mytemplate.js

Inspecting the JavaScript file mytemplate.js shows how the HTML is generated:

let lang = 'en';
let name = 'John';
console.log((() => {
  let _out = [];
  _out.push("<html lang=\"" + lang.toString().replace("&", "&amp;").replace("\"", "&quot;") + "\">");
  {
    _out.push("<body>");
    _out.push(`Hello, ${name}`.replace("&", "&amp;").replace("<", "&lt;"));
    _out.push("</body>");
  }
  _out.push("</html>");
  return _out.join('');
})());

Various command line options are available for setting indentation and similar. There is also a --wrap option which adds some prologue and epilogue code, which is part of a larger system we have designed, for allowing templates to embed each other and so on. See the jst_server and jst_cache modules for more information. If the extra complexity isn't needed, simply omit --wrap.

JSTizer program

In development, we often want to refer to example code from the Web, e.g. if we want to incorporate the Bootstrap front-end framework or if we want to create a web-form or some other common web-development task. The example code is usually written in HTML. We can drop this straight into a JST project by JSTizing it.

See the jstize module for more information. Essentially the workflow is (i) get some HTML in a file *.html, it does not need to be a complete page, but must be a syntactically valid and complete HTML fragment, (ii) use the JSTizer to convert this into an equivalent *.jst file, (iii) use the JST system, by either the programmatic API or the command-line tool, to convert this into an equivalent *.js file, (iv) run the *.js file to recreate the original HTML. Of course, the *.jst template can be modified to add any needed extra logic.

GIT repository

The development version can be cloned, downloaded, or browsed with gitweb at: https://git.ndcode.org/public/jst.git

License

All of our NPM packages are MIT licensed, please see LICENSE in the repository. We also gratefully acknowledge the acorn team for the original MIT license of their JavaScript parser, which we've heavily modified to create the JST parser.

Contributions

The jst system is under active development (and is part of a larger project that is also under development) and thus the API is tentative. Please go ahead and incorporate the system into your project, or try out our example webserver built on the system, subject to the caution that the API could change. Please send us your experience and feedback, and let us know of improvements you make.

Contact: Nick Downing [email protected]