Skip to content
Documentation

Visual regression

Visual regression renders something, compares it to a stored image, and reports a difference. It is a genuinely useful instrument, and it is routinely asked to do a job it cannot do.

A screenshot diff is a change detector, not a correctness oracle. It reports that pixels moved. Whether that was right is a human judgement, every time.

Nothing on this page argues against having it. The argument is about how much, over what, and what it is allowed to be responsible for.

What it is good at

Section titled “What it is good at”

The class of change no assertion is ever written for:

  • a CSS change with a blast radius nobody predicted
  • a dependency upgrade that shifted a default
  • a token change reaching more places than intended
  • an element that vanished without anything throwing
  • a layout that breaks only at one viewport width

These are real, they are otherwise invisible, and they are why the level exists.

Why large suites go quiet

Section titled “Why large suites go quiet”

Not by decision. By arithmetic.

Once the design system is enforced, most visual change is intentional and broad: a token moves and three hundred snapshots move with it. Each one is a diff a human is asked to approve.

The cost per intended change grows with the size of the suite, while the value — finding the unintended one — does not. Somewhere past a few hundred snapshots, “approve all” becomes the ritual, and the suite has become decorative while continuing to appear in the pipeline and the metrics.

That is the failure to design against. It is not a discipline problem; it is what happens when the triage cost exceeds anyone’s available attention.

How to keep a small one honest

Section titled “How to keep a small one honest”

Snapshot components, not pages. A component snapshot has one owner and a small diff. A page snapshot changes whenever anything on the page changes, which is constantly, and it names nothing when it fails.

Snapshot the states that matter. Rest, error, empty, loading, and the two themes. Not every prop combination — combinatorics is how a suite gets to three hundred without anyone choosing it.

Mask the genuinely dynamic. Dates, avatars, generated ids, animations mid-flight. An unmasked clock is a daily false failure, and a suite with daily false failures is being trained to be ignored.

Pin the renderer. Same browser, same version, same fonts, in a container. Font rendering differences between a developer’s machine and CI produce diffs that are pure noise.

Review diffs like code. A snapshot update in a pull request is a change to the interface, and it deserves the same attention as a change to logic. An auto-approving bot removes the only step that made the level worth having.

Budget it. Decide the number of snapshots you are willing to triage in a week, and treat exceeding it as a signal rather than as a success.

What it must not be responsible for

Section titled “What it must not be responsible for”

Design correctness. Whether the design is good is a critique question, answered by a person looking at the real thing in context. A tolerance threshold has no opinion about whether a hierarchy reads.

Accessibility. A screenshot cannot see contrast ratios in computed styles, focus order, announcements or keyboard reachability. Use the scanner; it is cheaper and it actually answers the question.

Whether the journey works. The most common failure a real user meets is a screen that looks entirely correct and leads nowhere. That is a journey test, and no image comparison will ever notice it.

When it deserves more weight

Section titled “When it deserves more weight”

Be honest about which case you are in — in three of them, this level is not demoted at all:

  • The visual is the product. A charting library, a design tool, a document renderer, a game. Rendering fidelity is the behaviour here, not a presentation of it.
  • Brand compliance is contractual. Regulated disclosures, licensed marks, mandated layouts. The snapshot is evidence for an obligation, which is worth its cost for reasons unrelated to finding bugs.
  • A migration is in flight. Re-theming, a framework change, a design system rewrite. A broad net is the right temporary instrument for a change whose blast radius nobody can predict. Put a date on its deletion — this is the net that most often becomes permanent by accident.

Where it connects

Section titled “Where it connects”