The head and the machine-readable layer

Learning objectives

• Write a title that identifies its page out of context, and say who reads it
• Choose the meta names worth writing, and set the viewport without breaking zoom
• Tell the http-equiv pragmas that still work from the ones that never did, and say where caching belongs
• Place styles and scripts in the head and choose between defer, async and type="module"
• Order a head so that each element is read before something depends on it
• Write Open Graph tags that produce a good card, and schema.org data in JSON-LD that matches the page
• Explain what search engines and AI agents read, and which of their controls are standards and which are proposals

What the head is for

The course slides define metadata as "information about information": subject, author, ISBN, page count. The head is metadata about one document, and on the web it has many readers. A browser reads it to decode and lay out the page; a search engine to name and summarize it; a chat app to draw a preview card; an AI agent to answer a question about it; a screen reader announces part of it. Each takes what it understands and ignores the rest.

The head may contain only metadata content: title, meta, link, base, style, script, noscript and template. Lesson 02 showed what happens when anything else appears there: the parser closes the head early, and every element after the stray one lands in the body, where a crawler looking in the head will not find it. Lesson 02 also covered <meta charset="utf-8">, the viewport and the lang attribute; this lesson starts from there.

title: the page's name

The slides call <title> the most important head element, and it still is. It is the one element the head must contain, and the standard allows exactly one. It is plain text, an escapable raw text element: <b> inside it is just characters. It appears in more places than any other text on the page:

• the browser tab and the window's title bar;
• the default name of a bookmark, and the entry in history that people search when they type in the address bar;
• search results, usually as the headline of the result;
• a shared link's preview, when there is no og:title;
• screen readers, which typically announce it when a page loads, as the first thing a person hears.

That last use is why WCAG 2.4.2 Page Titled, at level A, requires a title that describes the page's topic or purpose. The HTML standard adds that a title should identify the document "even when used out of context," as in a history list.

Most specific first

The slides' bad titles still make the point: "My Homepage" and "Product X" say nothing out of context, and a title of forty words says too much. A good title is unique on the site and puts the most specific part first, then the wider context:

Order matters because titles get cut: tabs show the first few words, search results truncate by pixel width, and a screen reader user hears the start first. Twenty tabs that all begin "CSE 134B · HTML ·" look identical. The slides' limit of 20 to 30 characters is too strict now, but the principle holds. Google treats the title as a strong suggestion, and may rewrite a result's headline from the page's headings when the title is missing, repeated or misleading.

Single-page applications must set document.title on every route change. Otherwise every view has the same name in history and bookmarks, and a screen reader user gets no sign that anything changed.

meta: named metadata

<meta> is a void element with four uses, told apart by which of these attributes it has, and an element uses only one:

The slides make a point that is easy to forget: the name is arbitrary. Their example, <meta name="Favorite Sandwich" content="Turkey and Swiss">, is valid HTML that no program will ever read. A name matters only if some reader looks for it. The standard defines a few names, others are registered on the WHATWG wiki, and the rest are conventions of particular companies. Here are the ones worth writing, and one worth deleting.

description

<meta name="description"> is a one- or two-sentence summary of the page. Search engines may show it as the snippet under a result, when it describes the page better than a passage of the text, and social cards fall back to it. It is not a ranking signal: Google stated in 2009 that it does not use the description in ranking. There is no length limit, but long snippets are cut, so lead with what matters, and write a different one for each page.

viewport, done right

Lesson 02 gave the one correct value, and it does not change:

width=device-width lays the page out at the device's width instead of a pretend desktop of about 980 CSS pixels, and initial-scale=1 starts at no zoom. Never add user-scalable=no or maximum-scale=1, which tell a phone browser to refuse pinch zoom. Templates still include them, often to stop iOS zooming into small form fields, but people with low vision depend on zoom, and blocking it fails WCAG 1.4.4 Resize Text. Safari on iOS has ignored the setting since iOS 10; other mobile browsers obey it. Usable forms fixes the field zoom properly.

robots

<meta name="robots"> tells search engines what they may do with this page. The common values are noindex (leave the page out of results), nofollow (do not follow its links), and nosnippet. The default is to index and follow, so index, follow does nothing. A file with no head, such as a PDF, uses an X-Robots-Tag HTTP header.

noindex is not robots.txt. The file at the site's root controls crawling, which addresses a crawler may fetch; noindex controls indexing, whether a fetched page may appear in results. A page blocked in robots.txt is never fetched, so its noindex is never seen; Google says the page "must not be blocked by a robots.txt file" for noindex to work. To keep a page out of search, allow crawling and send noindex. To keep it private, put it behind a login: neither is access control.

color-scheme and theme-color

<meta name="color-scheme" content="light dark"> tells the browser, before any CSS arrives, that the page supports both schemes, so it can paint the first background and the form controls in the person's preferred one, with no white flash before a dark page's styles load. Every current browser supports it.

theme-color suggests a color for the browser's interface around the page, and is the one name that may take a media attribute, for separate light and dark colors. Support has narrowed: Chrome on Android uses it for the toolbar, Chrome on the desktop uses it only for installed web apps, Safari 26 stopped using it for its toolbars, and Firefox ignores it.

referrer, author, generator, and keywords

<meta name="referrer"> sets the referrer policy for every link and request on the page; Linking explains the values. author and generator, the software that produced the page, are harmless and lightly used.

keywords is the one to delete. It was stuffed with unrelated words for ranking from the start, and Google states it "is not used by Google Search, and it has no effect on indexing and ranking at all." The HTML standard notes that many search engines ignore it. The words that matter belong in the text.

http-equiv and its limits

The slides introduce http-equiv as a way to "set an arbitrary header for the browser to acknowledge," for authors who could not configure the server. Today it is a closed list. The HTML standard calls these pragma directives, notes that they are "almost entirely unrelated" to the HTTP headers they are named after, and defines seven:

The slides' example, x-ua-compatible, was for Internet Explorer, which is retired. Any value not in the list, such as cache-control or expires, does nothing.

Content Security Policy in a meta element

A Content Security Policy lists which sources of script, style and other resources the page may use, limiting what an injected script can do. It belongs in a response header, but a meta element in the head works, with limits. It applies only to what comes after it: in a test in Chrome, an inline script above a policy forbidding inline scripts ran, and one below it was blocked. And frame-ancestors, report-uri and sandbox are ignored in a meta policy. Put it early, and use the header for those directives.

refresh: a timed redirect

<meta http-equiv="refresh" content="5; url=/elsewhere"> loads another address after five seconds; without the URL, it reloads the same page. The slides call it "client pull," for banners that reload every so often. It still works, and with any delay above zero it is an accessibility failure: WCAG lists it under 2.2.1 Timing Adjustable, since the person cannot stop the timer, 2.2.4 Interruptions, and 3.2.5 Change on Request. A screen reader user halfway through a paragraph is sent back to the top.

A delay of zero is an instant redirect, which WCAG accepts, but a server redirect is faster and understood by every client; Google recommends one instead.

link and base, in outline

<link> declares a typed relationship between this document and another resource: its href is a URL, and its rel says what kind of relationship it is. The slides list over twenty rel values. Linking has a whole page on them, Relationships, so here is only the map:

<base href> changes the address that every relative URL in the document resolves against, and <base target> sets a default target for every link. There may be one, before any element with a URL. It is rarely needed and easy to get wrong; Destinations covers its rules.

style and script: what blocks and what waits

The head is read top to bottom, and the parser acts on many elements as it meets them. Source order is load order, and two kinds of element hold things up: stylesheets hold up rendering, and classic scripts hold up parsing. The Browser for Programmers follows the parser in detail; this section is about what to write.

Styles block rendering, on purpose

A <link rel="stylesheet"> or a <style> element in the head blocks rendering: the browser keeps parsing but paints nothing until the CSS has arrived, since painting unstyled and repainting would be worse. So styles go early, and should be few and small. A stylesheet whose media does not match, such as a print stylesheet, does not block.

Scripts: classic, defer, async and module

The slides warn that a script in the head is loaded "before the page unless directed otherwise by the (defer) attribute." A classic script stops the parser until it is downloaded and run, because it might call document.write. Three attributes change that:

• defer downloads the script while parsing continues, then runs it after the document is parsed, before the DOMContentLoaded event. Deferred scripts run in the order they appear. Use it for classic scripts that work with the page.
• async downloads while parsing continues and runs the script the moment it arrives, interrupting the parser, in whatever order the downloads finish. Use it only for scripts that depend on nothing and nothing depends on, such as some analytics.
• type="module" is deferred by default, inline or external, and fetches the module's imports along the way. It also runs in strict mode with its own scope. For new application code it is the default choice. async on a module makes it run as soon as it and its imports are ready.
• An inline classic script runs immediately where it is; defer and async have no effect on it.

Why "scripts at the bottom" changed

The old advice was to put scripts at the end of the body, so a blocking script would not hold up the content above it. A deferred or module script in the head does better: it is found early, downloads alongside the page, never blocks the parser, and runs when a script at the end of the body would, with the whole document available.

Scripts that are not scripts, and noscript

A script whose type the browser does not recognise is not run; it is a data block, such as the JSON-LD in the next section. Two other types are not run as JavaScript either, but the browser does process them. The head often holds a <script type="importmap">, which maps module names to URLs and must come before the modules that use it; <script type="speculationrules">, covered in Making clicks faster; and <script type="application/ld+json">, the structured data later in this lesson.

<noscript> holds what to use when scripting is off. In the head it may contain only link, style and meta, such as a stylesheet that shows content a script would have revealed. Scripts fail for more people than those who turn them off: on slow networks and behind filters.

Getting the head in order

The slides end their tour of the head with "Get your head straight": one title, the charset early, and an awareness that the order of the head affects how the page loads. Each ordering rule follows from something read top to bottom:

1. meta charset first. It must be within the first 1024 bytes, and an encoding change found later can make the browser start over.
2. viewport, and a meta CSP if there is one, next. A policy governs only what comes after it.
3. base, if any, before any element with a URL.
4. title. Early, so the tab is named as soon as the page starts to arrive.
5. preconnect and critical preloads, so that connections open while the rest of the head is read.
6. Stylesheets, then deferred and module scripts. A classic script cannot run until every stylesheet above it has loaded, and the parser waits with it, so avoid classic scripts here altogether.
7. Everything else: description, canonical, icons, Open Graph, JSON-LD. Crawlers read the whole head, so their order does not matter.

The order is close to the one computed by capo.js, a tool by Rick Viscomi that the slides recommend, which sorts head elements into eleven groups by their effect on loading. The slides also point to htmlhead.dev, a reference to head elements.

Social cards: Open Graph

Paste a link into a chat or a social network and a card appears: an image, a title, a line of description. The platform's crawler fetched the URL and built the card from tags in the head. Facebook introduced them in 2010 as the Open Graph protocol, and most platforms now read them.

Open Graph uses property, not name, because the protocol is built on RDFa, a way of embedding data in HTML attributes, and property is an RDFa attribute. Many crawlers accept either, but not all, so write property. The protocol requires four properties: og:title, og:type, og:image and og:url.

• og:image must be an absolute URL. The crawler fetches it separately, and a relative path is the most common reason a card has no image. Facebook recommends at least 1200 by 630 pixels, a ratio of about 1.91 to 1, and shows images smaller than 600 by 315 as a small thumbnail. Giving og:image:width and og:image:height lets a platform lay out the card before it has downloaded the image.
• og:image:alt describes the image; the protocol says a page with an og:image should have one. Platforms that show alt text use it.
• og:url is the page's canonical address, the same one as rel="canonical", so that shares of the same page with different tracking parameters count as one.
• og:title and og:description may differ from the title and description: a card has room for a shorter, more inviting line, and no need for the site name, which og:site_name supplies.

X's card tags

Twitter, now X, added its own tags, with name rather than property. X's documentation for them, as last published, says that twitter:title, twitter:description and twitter:image fall back to the matching Open Graph tags, so they need not be repeated. The one worth adding is twitter:card, the layout: summary_large_image for a wide image above the text, or summary for a small square one. Without it, X "may" render a small summary card. The same documentation limits images to 5 MB and does not accept SVG.

How cards are fetched, and how to debug them

A card crawler is not a browser. It fetches the HTML, reads the head, and, for most platforms, runs no JavaScript, so tags added by a script give a blank card. It also caches what it finds, often for days. Facebook's Sharing Debugger and LinkedIn's Post Inspector fetch a page the way their crawlers do and refresh the cache. The demo below draws an approximate card from the tags you enter; every platform draws, crops and truncates its own way.

Structured data: schema.org

Open Graph describes a page for one purpose, a preview. Structured data describes the things on a page, in enough detail for a program to use them: this is an event, it starts at 7 pm on 14 November, at this place, and tickets cost this much. Search engines use it for rich results, such as the stars, prices and cooking times shown under some results, and other programs use it to extract facts without guessing.

The vocabulary nearly everyone uses is schema.org, launched in 2011 by Google, Microsoft and Yahoo, with Yandex joining that year. It defines types, such as Event, Product, Recipe, Person and Organization, arranged in a hierarchy under Thing, each with properties. A vocabulary says what the data means; a format says how it is written down. schema.org can be written in three formats.

JSON-LD

JSON-LD, JSON for Linked Data, puts the data in a script element as a JSON object. The browser does not run it, since the type is not JavaScript, and nothing renders.

@context names the vocabulary and @type the type; the rest are that type's properties, and a value can be another typed object. Google recommends JSON-LD "if your site's setup allows it," while accepting all three formats. Its advantage is that it is separate from the markup: a template or a CMS can generate it from the same data that fills the page, and a redesign cannot break it. Its weakness is the same separation. Nothing connects the JSON to the visible text, so the two can drift apart, and a price in the JSON that differs from the price on the page is worse than no data at all. The robust approach is to generate both from one source. It must also be valid JSON: one trailing comma, which JavaScript would accept, makes the whole block unreadable.

Microdata

Microdata, defined in the HTML standard, writes the same vocabulary as attributes on the visible markup. The slides list its attributes: itemscope makes an element an item; itemtype names its type with a URL; itemprop marks a descendant as one of its properties; itemid gives the item a global identifier; and itemref pulls in properties from elsewhere on the page by id. Here is the same event:

A property takes its value from the element's text, or from an attribute: datetime on a time, href on a link, content on a meta, which may appear in the body for this purpose. The data is the visible content, so it cannot drift, but the markup is harder to read and breaks easily in a redesign.

The third format, RDFa, does the same with vocab, typeof and property attributes. It is the oldest and the most capable, and in practice its main survivor is Open Graph's property attribute. All three formats express the same data; choose one per page.

What search engines do with it now

Structured data makes a page eligible for rich results, not certain of them, and the list keeps shrinking. How-to results went in September 2023; FAQ results were limited to government and health sites in 2023 and stopped appearing on 7 May 2026; the sitelinks search box went in 2024, and course info, estimated salary and others in 2025. About two dozen remain, led by article, breadcrumb, event, product, recipe, review and video. Google says unused markup "does not cause problems," and other consumers may still read it.

The rule that does not change is that the data must describe the page. Google's guidelines say not to mark up content that is not visible to the user, "even if the information is accurate," and pages that do can lose rich results altogether. Google's Rich Results Test checks a page against Google's features, and the Schema Markup Validator checks it against schema.org itself.

Search engines and AI agents

The newest readers of the head are AI systems: crawlers that collect pages to train language models, search features that quote pages in generated answers, and agents that fetch a page because a person asked a question about it. They read the same things every other reader does: the title, the description, the headings, the landmarks, the structured data, and the text. Some agents that drive a browser read a page through its accessibility tree, the same structure a screen reader uses, which makes the semantic markup of lessons 05 to 08 directly useful to them.

Controlling AI crawlers

The control that works today is the oldest one, robots.txt, with the user-agent tokens each company documents. Most companies now run separate crawlers for different purposes, so a site can allow one and refuse another:

The details matter. OpenAI's GPTBot and Anthropic's ClaudeBot collect training data; OAI-SearchBot and Claude-SearchBot index pages for search answers. Fetches made because a person asked, by ChatGPT-User or Claude-User, are a third kind, and OpenAI says that for these "robots.txt rules may not apply." Google-Extended is not a crawler at all: it is a token that controls whether pages Google already crawls may be used for Gemini, and Google says it does not affect inclusion or ranking in Google Search. Nor does it keep a page out of Google's AI answers in Search, which use what Googlebot crawls. All of these rules are requests that well-behaved crawlers honour, not access control.

Proposals, not standards

llms.txt, proposed by Jeremy Howard in September 2024, is a Markdown file at a site's root that summarizes the site for language models. Some documentation sites use it; it is not a standard, and Google says no new machine-readable files, "AI text files," or markup are needed to appear in its AI features. Meta values such as noai and noimageai, introduced by DeviantArt in 2022 as an opt-out from AI training, are a convention, not part of any standard, and none of the crawlers above document them.

Google's advice for its AI features is the same as its advice for search: there are no special requirements, and the ordinary practices apply, including structured data that matches the visible text. This is the course's view too. The best machine-readable page is a well-structured page: a specific title, an honest description, one h1 and a sound heading outline, landmarks, real tables and lists, and, where the page is about a definite thing, structured data that says exactly what the visible page says.

A complete head, in order

Here is the head that puts the lesson together. Not every page needs every line: leave out the CSP if the server sends one, the preconnect if there is no third-party host, the JSON-LD if the page is not about a definite thing.

The inspector below reads its own head, which follows this pattern, and lists every element in it with the checks from this lesson. A second sample packs a dozen of the mistakes above into one head. You can also paste the head of any page from View Source.

Try it: audit and fix a head

Four exercises, each about fifteen minutes.

1. Titles out of context. Open five tabs from one site you use often and read only the tab titles. Rewrite any that do not identify the page with the specific part first. Then turn on your operating system's screen reader, load one of the pages, and note what is announced first.
2. A real head. Use View Source on a news article and on this page, and paste each head into the head inspector. List what each has and lacks: description, canonical, Open Graph, JSON-LD. For each script in the head, say whether it blocks the parser.
3. A card that works. For a page of your own, write the Open Graph tags and make a 1200 by 630 image for it. Check the result in the social card demo, then, once the page is online, in Facebook's Sharing Debugger. Explain in one sentence why a card generated by a client-side script would be blank.
4. The same data twice. Mark up a recipe, a product or an event on a page, first in JSON-LD and then in microdata. Run both through the Schema Markup Validator. Then change one visible value, such as a price, and say which of the two versions is now wrong, and how you would prevent it.

Summary

Next steps

Lesson 10, Extending HTML, is the last in the series. It turns back to the body and asks what to do when HTML's own vocabulary runs out: data-* attributes, and custom elements that name structure the standard does not. The links in the head lead into other sections: Relationships for every rel value, Performance for measuring what the head costs, and Accessibility for the readers that matter most.