QotNews
Hacker News, Reddit, Lobsters, and Tildes articles rendered in reader mode.

The HTML-in-Canvas API origin trial

Source: developer.chrome.com
118 points by andrewstuart a day ago on hackernews | 52 comments

Thomas Nattestad

Natalia Markoborodova

For years, web developers have had to make a tough architectural choice when building complex, highly-interactive visual applications on the web/ Should you lean on the DOM for its rich semantic features or render directly to the <canvas> element for low-level graphics performance?

With the new experimental HTML-in-Canvas API—available now in origin trial—you don't have to choose. This API lets you draw DOM content directly into a 2D canvas or a WebGL/WebGPU texture while keeping the UI interactable, accessible, and hooked up to your favorite browser features. By combining HTML with low-level graphics processing, you can create experiences that were previously impossible.

The DOM versus Canvas

To understand the power of this new API, it helps to look at the relative strengths of both the DOM and the Canvas.

The DOM is the staple of web UI. It offers text layout solutions out of the box, using semantically understood content to create rich interfaces. This lets users perform common operations across web pages seamlessly—things we often take for granted, like highlighting text to copy, or right-clicking an image to save it. The DOM also integrates with essential browser features: accessibility tools, translate, find-in-page, reader mode, extensions, dark mode, browser zoom, and autofill.

Canvas (and WebGL/WebGPU), on the other hand, allows for low-level access to drive a grid of pixels for highly advanced 2D and 3D graphics. Games and complex web apps (like Google Docs or Figma) require this performant, low-level access. Because the canvas is fundamentally a grid of pixels, supporting features like responsive text used to require complex custom UI logic, drastically increasing your bundle size. Crucially, all the powerful browser features integrated into the DOM break completely when the UI is trapped inside a static canvas pixel grid.

The advantages of bringing the DOM to Canvas

The HTML-in-Canvas API is the bridge that gives you the best of both worlds. By placing HTML inside the <canvas> element and synchronizing its transform, you ensure the content remains fully interactive, and that all browser integrations function automatically.

Here's what you get by letting the DOM handle your UI inside a <canvas> element:

  • Text layout and formatting: Simplified text layout and formatting, including multiline or bidirectional text with CSS styles applied.
  • Form controls: Expressive and easier to use form controls with extensive customization options.
  • Text selection, copy/paste, and right-click: Users can highlight text inside your 3D scenes, or right-click context menus natively.
  • Accessibility: Content rendered inside the canvas is exposed to the accessibility tree. Accessibility systems can parse the UI as they do normal HTML, and expose it to systems like screen readers.
  • Find-in-page: Users can use find-in-page (Ctrl/Cmd+F) to search for text, and the browser highlights it directly within your WebGL textures.
  • Indexability and AI agent interfaceable: Web crawlers and AI agents can seamlessly index and read the text rendered into your 2D and 3D scenes.
  • Extension integration: Browser extensions work natively. For example, a text-replacement extension automatically updates the text rendered on your 3D meshes.
  • DevTools integration: You can inspect your canvas content, including for WebGL/WebGPU UI elements directly in Chrome DevTools. Tweak a CSS style in the inspector, and watch it instantly update on the 3D texture!

High-level use cases

This API unlocks incredible potential across several domains:

  • Large canvas-based applications: Heavyweight web apps can now render complex application UI components natively into their canvas-driven workspaces, improving accessibility and reducing bundle weight.
  • 3D scenes and games: Marketing sites, immersive WebXR experiences, and web games can now place fully interactable web UI into 3D scenes—like a 3D book that uses real DOM text, or an in-game terminal that natively supports copying and pasting.

How to use the API

Using the API happens in three phases: Setting up your canvas, rendering into the canvas, and updating the CSS transform so the browser knows where the element physically sits on the screen.

Prerequisites

The HTML-in-Canvas API is in origin trial in Chrome 148 through 150. To test it on your site, use Chrome Canary 149 or later with the chrome://flags/#canvas-draw-element flag enabled. To enable the API for other users, register for the Origin Trial.

Step 1: Basic Canvas setup

First, add the layoutsubtree attribute to your <canvas> tag. This makes the browser aware of the content nested inside the canvas, preparing it to be displayed inside the canvas, and exposing it to accessibility trees.

<canvas id="canvas" style="width: 200px; height: 200px;" layoutsubtree>
  <div id="form_element">
    <label for="name">Name:</label> <input id="name" type="text">
  </div>
</canvas>

Size the canvas grid

To avoid blurriness of the rendered content, make sure to size the canvas grid to match the device scale factor.

const observer = new ResizeObserver(([entry]) => {
  const dpc = entry.devicePixelContentBoxSize;
  canvas.width = dpc ? dpc[0].inlineSize : Math.round(entry.contentRect.width * window.devicePixelRatio);
  canvas.height = dpc ? dpc[0].blockSize : Math.round(entry.contentRect.height * window.devicePixelRatio);
});

const supportsDevicePixelContentBox =
  typeof ResizeObserverEntry !== 'undefined' &&
  'devicePixelContentBoxSize' in ResizeObserverEntry.prototype;
const options = supportsDevicePixelContentBox ? { box: 'device-pixel-content-box' } : {};
observer.observe(canvas, options);

Step 2: Rendering

For a 2D context, use the drawElementImage method. Do this inside the paint event, which triggers whenever the element redraws—for example, during text highlighting or user input. It's crucial to update the element's CSS transform with the return value so interactivity continues to work.

const ctx = document.getElementById('canvas').getContext('2d');
const form_element = document.getElementById('form_element');
const canvas = document.getElementById('canvas');

canvas.onpaint = () => {
  ctx.reset();

  // Draw the form element at x:0, y:0
  let transform = ctx.drawElementImage(form_element, 0, 0);

  // Use the transform returned later on...
};

Render with WebGL

For WebGL, you use texElementImage2D. It functions similar to texImage2D, but takes the DOM element as the source.

canvas.onpaint = () => {
  if (gl.texElementImage2D) {
    gl.texElementImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, form_element);
  }
};

Render with WebGPU

WebGPU uses the copyElementImageToTexture method on the device queue, analogous to copyExternalImageToTexture:

canvas.onpaint = () => {
  root.device.queue.copyElementImageToTexture(
    valueElement,
    { texture: targetTexture }
  );
};

Step 3: Update the CSS transform

Now that you've rendered the element into the canvas, you have to update the browser on where it's located. This ensures spatial synchronization between the canvas and the DOM's layout. This is important so that the browser can correctly map the event zone—such where exactly the user clicks or hovers—with where the element is rendered.

For the 2D context case, apply the transform returned by the rendering call to the .style.transform property:

const ctx = document.getElementById('canvas').getContext('2d');
const form_element = document.getElementById('form_element');
const canvas = document.getElementById('canvas');

canvas.onpaint = () => {
  ctx.reset();
  // Draw the form element at x:0, y:0
  let transform = ctx.drawElementImage(form_element, 0, 0);

  // Sync the DOM location with the drawn location
  form_element.style.transform = transform.toString();
};

With WebGL or WebGPU, the on-screen location of an element depends on how the output texture is used by shader code, and can't be deduced from the canvas rendering context. However, if your shader program uses a typical model view projection to draw the texture, then you can use the new convenience function element.getElementTransform() to compute a transform that can be used in the same way as the return value from drawElementImage(). To facilitate this, you need to do the following:

  • Convert WebGL MVP Matrix to DOM Matrix.
  • Normalize the HTML element. HTML elements are sized in pixels (for example, 200px wide). WebGL, however, usually treats objects as "unit squares", for example, ranging from 0 to 1. If you don't normalize, your 200px button will look 200 times larger.
  • Map to the canvas viewport. This step is the "rescaling" phase. It stretches that unit-space math back out to match the actual pixel dimensions of your <canvas> element on the screen. It also flips the Y-axis, because in WebGL, up is positive, but in CSS, down is positive.
  • Calculate the final transform. Multiply the matrices in order: Viewport * MVP * Normalization. Combining them into one final transform produces a "map" that tells the browser exactly where that HTML element layer should sit to align with the 3D drawing.
  • Apply the transform to the HTML element. This moves the HTML element layer to sit directly on top of its rendered pixels. This ensures that when a user clicks a button or selects text, they're hitting the real HTML element.
if (canvas.getElementTransform) {
  // 1. Convert WebGL MVP Matrix to DOM Matrix
  const mvpDOM = new DOMMatrix(Array.from(htmlElementMVP));

  // 2. Normalize the HTML element (pixels -> 1x1 unit square)
  const width = targetHTMLElement.offsetWidth;
  const height = targetHTMLElement.offsetHeight;

  const cssToUnitSpace = new DOMMatrix()
    .scale(1 / width, -1 / height, 1) // Shrink to unit size and flip Y
    .translate(-width / 2, -height / 2); // Center the element

  // 3. Map to the canvas viewport
  const clipToCanvasViewport = new DOMMatrix()
    .translate(canvas.width / 2, canvas.height / 2) // Move origin to center
    .scale(canvas.width / 2, -canvas.height / 2, 1); // Stretch to canvas dimensions

  // 4. Multiply: (Clip -> Pixels) * (MVP) * (pixels -> unit square)
  const screenSpaceTransform = clipToCanvasViewport
      .multiply(mvpDOM)
      .multiply(cssToUnitSpace);

  // 5. Apply to the transform
  const computedTransform = canvas.getElementTransform(targetHTMLElement, screenSpaceTransform);
  if (computedTransform) {
    targetHTMLElement.style.transform = computedTransform.toString();
  }
}

Library and framework support

Some of the popular libraries have already shipped support for the HTML-in-Canvas feature.

Three.js

Updating matrixes manually can be tedious, which is why frameworks are already jumping on board. Three.js has experimental support using the new THREE.HTMLTexture:

const material = new THREE.MeshBasicMaterial();
material.map = new THREE.HTMLTexture(uiElement); // Pass the DOM element

const geometry = new THREE.BoxGeometry(1, 1, 1);
const mesh = new THREE.Mesh(geometry, material);
scene.add(mesh);

PlayCanvas

PlayCanvas also supports HTML-in-Canvas using their texture API:

// Wait for the 'paint' event to set the source
canvas.addEventListener('paint', () => {
    htmlTexture.setSource(htmlElement);
}, { once: true });
canvas.requestPaint();

// Keep up to date
canvas.addEventListener('paint', onPaintUpload);

const material = new pc.StandardMaterial();
material.diffuseMap = htmlTexture;
material.update();

Demos

Before trying out the demos, ensure your environment is properly configured.

There are several demos that serve as a reference for using the API. We are already seeing creative solutions from the community, ranging from translatable 3D books to UI elements that refract through glass shaders:

  • The 3D book: A WebGL-rendered 3D book that uses HTML layout for its pages. Users can swap fonts with CSS. Because it's DOM-based, built-in translation works instantly, and AI agents can extract the text with less complexity.
  • Interactive 3D UIs: A WebGPU jelly slider that refracts light based on an underlying 3D model, while still responding to standard HTML <input type="range"> step attributes.
  • Animated textures: A dynamic 3D billboard rendering an animated SVG pencil using the DOM directly into a WebGL texture without needing a custom animation loop.
  • Refractive overlays: An interactive typography layer distorted by a moving 3D cursor, yet fully selectable and searchable using find-in-page.

Check out the collection of demos created by the community. If you'd like your HTML-in-Canvas demo to be featured in this collection, create a pull request to add it.

Limitations

While powerful, the API has a few conscious limitations:

  • Cross-origin content: For security and privacy reasons, the API does not work with cross-origin iframe content.
  • Main thread scrolling: HTML-in-canvas is drawn with JavaScript, which means that scrolling and animations cannot update independently of JavaScript, like they can outside canvas. Developers should carefully consider the performance characteristics of putting scrolling content inside canvas versus having the entire canvas scroll.

Feedback

If you are experimenting with the HTML-in-Canvas API, we want to hear from you! You can sign up for the origin trial to enable the feature on your site while it's in the experimental phase to help us shape the API design. You can also file an issue to provide any feedback.

Resources