The HTML parsing pipeline

3.1 Four things, not one

“Parsing HTML” sounds like one step. It is a short pipeline of distinct transformations, each with a different input and output, and conflating them is the source of most confusion about how pages load.

Figure 3.1 — Bytes become characters become tokens become a tree. Each arrow is a separate algorithm; the DOM at the end is the artifact everything else in the browser consumes.

3.2 Bytes are not characters

The network hands over bytes, and a byte is not a character until you know the encoding. text/html; charset=utf-8 in the response header is the browser's first clue; a byte-order mark at the start of the stream is another; a <meta charset> inside the document is a third. The last one creates a small chicken-and-egg problem — the declaration of how to decode the bytes is itself inside the bytes — which browsers solve by pre-scanning the first chunk for the charset before committing. The modern default, and the right answer, is UTF-8. Get this wrong and you get mojibake: the famous garbled characters that mean someone decoded text with the wrong alphabet.

This is why <meta charset="utf-8"> belongs as early in the document as possible, and why the bytes-versus-characters distinction is not pedantry: it is the first place a page can be silently wrong before a single tag has been recognized.

3.3 A detour through parsing theory

To see why the HTML parser is unusual, recall what an ordinary parser is. Parsing means turning a flat input into a structured tree according to a grammar. The work splits in two: a tokenizer (or lexer) groups characters into meaningful units — tokens — and a parser assembles tokens into a tree by matching grammar rules. A compiler does exactly this with source code: tokenize 2 + 3 into a number, an operator, a number; then build an expression tree.

Languages amenable to this have a context-free grammar: a finite set of rules that say how symbols combine, independent of surrounding context. CSS and JavaScript are close enough to this that their parsers can be generated mechanically from a grammar. The classic tools — Lex and Yacc, Flex and Bison — exist precisely to turn a grammar into a working parser. If HTML were context-free, the browser could do the same and this module would be short.

3.4 Why HTML breaks the rules

HTML is not context-free, and cannot be parsed by a generated parser, for three reasons that compound.

It is forgiving by design. You may omit the <html>, <head>, and <body> tags entirely and still get a correct document with all three elements — the parser inserts them. You may leave a <li> or a <p> unclosed and the parser closes it for you when the context demands. This forgiveness is the main reason HTML succeeded: it lowered the cost of authoring to near zero, and a generation of authors learned it by trial and error rather than by reading a grammar.

It must tolerate decades of broken pages. Browsers compete on rendering the existing web, and the existing web is full of malformed markup that nonetheless “worked” because some browser once guessed well. Any new parser has to make the same guesses, so the error handling is not an afterthought — it is a specification.

It can rewrite its own input. A <script> running mid-parse can call document.write() and inject new characters into the stream the parser is still reading. The source is not fixed during parsing; the act of parsing can change what is being parsed. No ordinary parser model survives that.

The resolution, hard-won, is that the HTML Standard now specifies one exact parsing algorithm — every state, every transition, every error-recovery action — so that all conformant browsers build the identical tree from identical bytes, valid or not. The forgiveness was made deterministic.

1990s–2000s: Each browser invents its own HTML error handling. The same broken page yields different trees in different browsers; “works in one, breaks in another” is the daily reality.
2008–2014: HTML5 specifies the parsing algorithm in full, including precise error recovery. Compatibility stops being folklore and becomes a written contract.
Today: The algorithm lives in the WHATWG HTML Standard as a living document. Every engine implements the same state machine; the tree is portable even when the markup is wrong.

HTML's parser is lenient on purpose — and the leniency is specified to the comma, so that being lenient is also being identical everywhere.

3.5 The tokenizer is a state machine

The first half of the parser is the tokenizer, and it is a finite state machine: it reads one character at a time, and what it does depends on which state it is in. In the data state, ordinary text accumulates into character tokens. A < moves it into a tag-open state; a letter after that starts a tag name; whitespace inside a tag begins the attribute machinery; a > emits the finished tag token and returns to the data state. The full specification defines roughly eighty states to handle comments, doctypes, character references, raw-text elements like <script>, and every edge case; the spine of it is small enough to draw.

Figure 3.2 — The spine of the tokenizer, radically simplified. The real machine has roughly eighty states, but the idea is here: one character at a time, the current state decides what happens, and tag and character tokens fall out as you go.

3.6 Tree construction and the art of fixing soup

Tokens are still flat. The second half of the parser — tree construction — consumes the token stream and builds the DOM, maintaining a stack of currently open elements and a notion of “insertion mode” that changes as it goes (before head, in head, in body, in table, and so on). A start-tag token pushes an element onto the stack and into the tree; an end-tag pops it. When the markup violates expectations — a stray </p> with no open paragraph, a <table> with text loose inside it, mis-nested bold and italic tags — the algorithm has named, specified recovery procedures with wonderful names like “foster parenting” and the “adoption agency algorithm.” The result is that a tag soup of broken markup still produces a clean, well-formed tree.

Consider a small, valid document and the tree it yields:

<!doctype html>
<html lang="en">
  <head><title>Demo Company</title></head>
  <body>
    <h1>Demo Company</h1>
    <p>Fake home page.</p>
  </body>
</html>

Figure 3.3 — The DOM. Element nodes (solid) and text nodes (dashed) form a tree rooted at the Document. Note what is not here: no angle brackets, no whitespace-only quirks of the source, no doctype as a child to style. The tree is a clean object model, not a copy of the text.

3.7 The DOM is the runtime, not the source

This is the conceptual payload of the module. The DOM is not your HTML file. It is a live object model built from your HTML file, and from that point on the file is irrelevant — everything operates on the tree. CSS selectors match against DOM nodes, not against source text. Layout reads the DOM. document.querySelector queries the DOM. When a script adds an element, it adds a DOM node, and there is no corresponding text anywhere; “view source” still shows the original bytes while the page on screen reflects a tree that has moved far beyond them.

The DOM also has children downstream of itself. The browser derives the accessibility tree — the structure a screen reader navigates — from the DOM, mapping elements and ARIA attributes to roles and names. It is a parallel tree computed from the same source, and it is how the blind user from the cast experiences the document the parser just built. A malformed tree, or a tree of meaningless <span>s, produces a malformed or meaningless accessibility tree.

You author text. The browser runs a tree. Confusing the two is the root of a surprising number of bugs.

3.8 Parsing is incremental — and interruptible

The parser does not wait for the whole document. It streams, building the tree from bytes as they arrive off the network, which is why pages can start displaying before they have fully downloaded. But the stream can be stalled, and by one thing in particular: a plain <script> with no async or defer attribute. Because that script might call document.write() and alter the very input the parser is consuming, the parser must stop, hand control to the JavaScript engine, let the script run to completion, and only then resume. If the script itself has to be fetched from the network first, parsing halts until it arrives.

To soften this, browsers run a preload scanner: a lightweight look-ahead that races down the raw bytes while the main parser is blocked, spotting src and href references and kicking off their downloads early. It is a speculative optimization layered on top of a fundamentally sequential algorithm. These two facts — that scripts block parsing and that a scanner mitigates it — set up the whole of Module 4, where a page reveals itself to be a tree of dependencies, and Module 6, where blocking gets explained through the event loop. Demo Company's critical-render-path page exists precisely to let you watch a synchronous script freeze the parser.

3.9 The lab: source versus tree

Open Demo Company's home page and do two things side by side. Use “View Source” to see the bytes the server sent — the input to this module. Then open DevTools' Elements panel to see the DOM — the output. On a clean page they look almost identical, which is the point of authoring valid markup. The lesson lands harder on a broken page: paste malformed HTML into a file — omit closing tags, nest things wrongly — load it, and compare. The Elements panel shows you the tree the parser repaired into existence, often with tags you never wrote, sitting exactly where the algorithm's recovery rules put them.

3.10 What you now have

The pipeline from bytes to DOM: decode bytes into characters under a known encoding; tokenize characters into tags, text, comments, and doctypes with a state machine; construct a tree from those tokens with a specified, error-tolerant algorithm that makes even broken markup yield an identical, well-formed tree across browsers. And the central idea: the DOM is the runtime artifact — not your source text but a live object model, parent to the accessibility tree, target of every selector and script, the thing the rest of the browser actually operates on.

But the DOM the parser builds is rarely complete on its own. It is full of references — to stylesheets, images, fonts, scripts, frames — each of which is another resource the browser must go and fetch. A page is not a file; it is a dependency tree. Module 4 is how the browser discovers, prioritizes, and loads everything the document points at, with Demo Company's resource pages adding one dependency type at a time.

The parser's output is a tree of nodes — and a list of things the page still needs. Next we go and get them.