html-tools

A lightweight HTML tokenizer and parser which outputs to the HTMLjs object representation. Special hooks allow the syntax to be extended to parse an HTML-like template language like Spacebars.

HTMLTools.parseFragment("<div class=greeting>Hello<br>World</div>")

=> HTML.DIV({'class':'greeting'}, "Hello", HTML.BR(), "World"))

This package is used by the Spacebars compiler, which normally only runs at bundle time but can also be used at runtime on the client or server.

Invoking the Parser

HTMLTools.parseFragment(input, options) - Takes an input string or Scanner object and returns HTMLjs.

In the basic case, where no options are passed, parseFragment will consume the entire input (the full string or the rest of the Scanner).

The options are as follows:

getTemplateTag

This option extends the HTML parser to parse template tags such as {{foo}}.

getTemplateTag: function (scanner, templateTagPosition) { ... } - A function for the parser to call after every HTML token and at various positions within tags. If the function returns an instanceof HTMLTools.TemplateTag, it is inserted into the HTMLjs tree at the appropriate location. The constructor is HTMLTools.TemplateTag(props), where props is an object whose properties are copied to the TemplateTag instance. You can also call the constructor with no arguments and assign whatever properties you want, or you can subclass TemplateTag.

There are four possible outcomes when getTemplateTag is called:

• Not a template tag - Leave the scanner as is, and return null. A quick peek at the next character should bail to this case if the start of a template tag is not seen.
• Bad template tag - Call scanner.fatal, which aborts parsing completely. Once the beginning of a template tag is seen, getTemplateTag will generally want to commit, and either succeed or fail trying).
• Good template tag - Advance the scanner to the end of the template tag and return an HTMLTools.TemplateTag object.
• Comment tag - Advance the scanner and return null. For example, a Spacebars comment is {{! foo}}.

The templateTagPosition argument to getTemplateTag is one of:

• HTMLTools.TEMPLATE_TAG_POSITION.ELEMENT - At "element level," meaning somewhere an HTML tag could be.
• HTMLTools.TEMPLATE_TAG_POSITION.IN_START_TAG - Inside a start tag, as in <div {{foo}}>, where you might otherwise find name=value.
• HTMLTools.TEMPLATE_TAG_POSITION.IN_ATTRIBUTE - Inside the value of an HTML attribute, as in <div class={{foo}}>.
• HTMLTools.TEMPLATE_TAG_POSITION.IN_RCDATA - Inside a TEXTAREA or a block helper inside an attribute, where character references are allowed ("replaced character data") but not tags.
• HTMLTools.TEMPLATE_TAG_POSITION.IN_RAWTEXT - In a context where character references are not parsed, such as a script tag, style tag, or markdown helper.

It's completely normal for getTemplateTag to invoke HTMLTools.parseFragment recursively on the same scanner (see shouldStop). If it does so, the same value of getTemplateTag must be passed to the second invocation.

At the moment, template tags must begin with {. The parser does not try calling getTemplateTag for every character of an HTML document, only at token boundaries, and it knows to always end a token at {.

textMode

The textMode option, if present, causes the parser to parse text (such as the contents of a <textarea> tag or part of an attribute) instead of HTML. In a text mode, for example, the input "<" is not a parse error (because a bare < is allowed in a textarea or attribute).

The value of textMode must be one of:

• HTML.TEXTMODE.RCDATA - Interpret character references (the usual case)
• HTML.TEXTMODE.STRING - Don't interpret character references (the RAWTEXT case)

shouldStop

shouldStop: function (scanner) { ... } - A function that the parser invokes between tokens to check whether it should stop parsing. The function should return a boolean value.

The shouldStop function provides a way to put a "wall" in the input stream for the purpose of parsing HTML content embedded in a template tag. For example, take the template {{#if happy}}yay{{/if}}. The scanner will be advanced to the start of the word yay before parseFragment is called to parse the contents of the tag. (Note that the caller happens to be the getTemplateTag function of an enclosing parseFragment.) When parsing from yay, the shouldStop function is used to end the fragment at {{/if}}, which, like {{/blah}} or {{else}}, couldn't possibly be actual content that belongs in the fragment. Even if HTML tags are not closed, as in the malformed template {{#if foo}}<div>{{else}}, the fragment stops at the {{else}}, and the error is an unclosed <div> (before the parser notices the unclosed {{#if}}).

HTMLTools.Scanner class

To write getTemplateTag and shouldStop functions, you have to interface with the HTMLTools.Scanner class used by html-tools. It's a general class that could be used by any parser/lexer/tokenizer.

A Scanner has an immutable source document and a mutable pointer into the document.

• new Scanner(input) - constructs a Scanner with source string input
• scanner.input (read-only) - the entire source string
• scanner.pos (read/write) - the current index into the source string

Scanners provide these methods for convenience:

• scanner.rest() - input.slice(pos) (the rest of the document)
• scanner.peek() - input.charAt(pos) (the next character)
• scanner.isEOF() - true if pos is at or beyond the end of input
• scanner.fatal(msg) - throw an error indicating a problem at pos

Even though scanner.rest() performs a substring operation, it should be considered fast and O(1), because all known JavaScript runtimes in use have constant-time substring. It would be possible, but extremely clumsy, to avoid such a substring operation while performing the usual business of a parser, which is to try to match a regex anchored at a particular index.

Functions that take scanners generally have three possible outcomes:

• Success: Advance scanner.pos and return some truthy value
• Failure: Leave scanner.pos alone and return null
• Fatal: Throw an exception via scanner.fatal

It's particularly important that in the Failure case, the function restores the scanner to the state it found it. This makes it possible to immediately try another parse function when one fails and form alternations such as foo(scanner) || bar(scanner).

It's often easiest to avoid the Failure case altogether, writing parse functions that always succeed or throw. This requires less bookkeeping and leads to good error messages. A Failure case may be added if it is simple to check for up front and makes the function easier to use in an alternation. We say a function has "committed" or "will succeed or fail fatally trying" when it has reached a point where it must return a value or throw. Any parse function that has moved the scanner position and not remembered the original position is necessarily committed. Usually, committing is completely natural in the context of the language being parsed; for example, {{ in a template always starts a template tag or throws an error about a malformed template tag.

HTML Dialect

HTML has many dialects and potential degrees of permissiveness. We use the WHATWG syntax spec and are pretty strict, failing on any "parse error" cases, which basically means the input has to be valid "HTML5" (except for the template tags).

HTML syntax references:

• Human-readable syntax guide
• Tokenization state machine

The WHATWG parser without error recovery is strict compared to browsers (which will recover from almost anything), but lenient compared to the now-defunct XHTML spec (which required lowercase tag names and lots more escaping of special characters).

The following are examples of errors:

• A stray or unclosed < character
• An unknown character reference like &asdf;
• Self-closing tags like <div/> (except for BR, HR, INPUT, and other "void" elements)
• End tags for void elements (BR, HR, INPUT, etc.)
• Missing end tags, in most cases (e.g. missing </div>)

The following are permitted:

• Bare > characters
• Bare & that can't be confused with a character reference
• Uppercase or lowercase tag and attribute names (case insensitive)
• Unquoted and valueless attributes - <input type=checkbox checked>
• Most characters in attribute values - <img alt=x,y>
• Embedded SVG elements

Note: Currently you have to close your Ps, LIs, and other tags for which the spec allows the end tag to be omitted in many cases

Character References

This package contains a lookup table for all known named character references in HTML, of which there are over 2,000, from Á (capital A, acute accent) to ‌ (zero-width non-joiner), as well as code for interpreting numeric character entities like A.

Since character references are parsed into HTML.CharRef objects which contain both the raw and interpreted form, we never have to convert between the forms except at parse time.