Sharding Playwright visual tests across CI runners

This page is part of the Test Parallelization & Sharding in CI section of the CI/CD test gating guide. The problem here is throughput: a visual regression suite that takes twenty minutes on a single runner sits on the critical path of every pull request, and the only way to shorten it without deleting coverage is to run it on several runners at once.

Problem statement

Visual tests are slow in a way unit tests are not. Each one launches a real browser, navigates to a rendered component, waits for fonts and animations to settle, and captures and compares a screenshot. A design system with a few hundred component stories easily reaches 400–600 screenshots, and at roughly two seconds each — plus browser startup — the suite lands around twenty minutes on one runner.

Twenty minutes on the critical path is expensive in a way that compounds. It is twenty minutes before a required check goes green, so it is twenty minutes of context-switch latency on every PR, multiplied by every push that follows a review comment. Developers respond by batching changes into bigger PRs (which makes review worse) or by pushing to merge before the visual check finishes (which is exactly the gate this guide exists to prevent). Throwing a faster CPU at one runner barely helps, because the bottleneck is wall-clock latency of a sequential queue, not raw compute.

Root cause

The suite runs sequentially on a single machine, so its wall-clock time is the sum of every test’s runtime even though the tests are completely independent of one another. Visual tests parallelize almost perfectly — no shared state, no ordering constraints — so the fix is to spread the work across machines rather than time. Playwright supports this directly with --shard=index/total, which partitions the test files into disjoint slices; the remaining work is CI plumbing to fan those slices across a runner matrix and then recombine the per-slice results into one report and one suite-level verdict. Getting even slices matters, which is the concern of balancing test shards to reduce CI wall-clock; this page covers the mechanics of splitting and merging.

Sharding a sequential visual suite across four runners On the left, one runner executes all 400 tests sequentially in 20 minutes. An arrow labelled shard splits it into four parallel runners each running 100 tests in about 5 minutes and emitting a blob report. The four blob reports converge into a merge-reports step that produces one HTML report and a single pass or fail. 1 runner 400 tests · 20 min --shard shard 1/4 · 100 tests ~5 min → blob shard 2/4 · 100 tests ~5 min → blob shard 3/4 · 100 tests ~5 min → blob shard 4/4 · 100 tests ~5 min → blob merge-reports 1 HTML · 1 pass/fail ~5 min wall-clock

Minimal reproduction

The problem is easiest to see by timing the un-sharded suite on the runner CI actually uses. A single-job workflow captures the baseline number:

# .github/workflows/visual-single.yml — the slow starting point
name: visual (single runner)
on: pull_request

jobs:
  visual:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: time npx playwright test tests/visual

The time prefix reports the wall-clock cost of the whole suite on one machine:

Running 412 tests using 4 workers
  412 passed (19m 47s)

real    21m 3s

Nineteen minutes of tests plus browser install. Every reviewer round-trip pays that cost. The tests are independent, so almost all of that time is recoverable by running them on more than one runner.

Step-by-step fix

1. Switch shards to the blob reporter

A sharded run needs a report format that can be recombined afterwards. The blob reporter writes an opaque, mergeable artifact per shard; the HTML reporter cannot be merged because each HTML report is a self-contained snapshot of one shard. Select the reporter by environment so local runs still get HTML.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests/visual',
  fullyParallel: true,           // let each runner use all its cores within a shard
  reporter: process.env.CI ? 'blob' : 'html',
  expect: {
    toHaveScreenshot: { threshold: 0.2, maxDiffPixelRatio: 0.001 },
  },
});

What this does: on CI each shard emits a blob-report/ directory (a zipped, mergeable result), while local developers still get the browsable HTML report. fullyParallel: true ensures each runner also parallelizes across its own CPU cores, compounding with the cross-runner sharding.

2. Fan the suite out across a shard matrix

Add a GitHub Actions matrix over the shard index. Each matrix job runs playwright test --shard=index/total, so runner N executes only its slice. Upload each shard’s blob report as an artifact for the merge job.

# .github/workflows/visual.yml
name: visual
on: pull_request

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false                 # one failing shard must not cancel the others
      matrix:
        shard: [1, 2, 3, 4]            # keep in sync with the /4 total below
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run shard ${{ matrix.shard }} of 4
        run: npx playwright test tests/visual --shard=${{ matrix.shard }}/4
      - name: Upload blob report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report/
          retention-days: 1

What this does: four runners start at once, each running a quarter of the test files, so the test phase finishes in roughly a quarter of the time. fail-fast: false keeps every shard running even if one finds a regression, so the merged report reflects the whole suite rather than stopping at the first failure. if: ${{ !cancelled() }} uploads each blob even when that shard fails, which the merge job needs.

3. Merge the shard reports into one result

A separate job runs after the matrix, downloads all four blobs, and runs playwright merge-reports. This produces a single HTML report and — critically — a single exit code that represents the whole suite.

  merge-report:
    if: ${{ !cancelled() }}
    needs: [test]                      # wait for every shard to finish
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20, cache: npm }
      - run: npm ci
      - name: Download all blob reports
        uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*        # collect every shard's artifact
          merge-multiple: true
      - name: Merge into one HTML report
        run: npx playwright merge-reports --reporter=html ./all-blob-reports
      - name: Publish merged report
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/

What this does: merge-reports reads all four blobs and reconstructs one coherent report, reproducing the pass/fail totals as if the suite had run on a single machine. Because the merge job needs: [test], it only runs once every shard has reported, and a failure recorded in any blob surfaces in the merged result. This merge job is the natural place to hang the required status check described in making visual review a required status check, so the whole fan-out gates behind one green check.

4. Make sharding a required, single check

Point branch protection at the merge-report job, not the individual shards. The shards are implementation detail; the merged result is the gate. Requiring the merge job means a regression on any shard blocks the merge through one check name, and adding or removing shards never changes what branch protection references.

# Require only the merged result as the visual gate
gh api -X PATCH repos/acme/design-system/branches/main/protection \
  -H "Accept: application/vnd.github+json" \
  -f 'required_status_checks[strict]=true' \
  -f 'required_status_checks[checks][][context]=merge-report' \
  -F 'restrictions=null'

What this does: the four shard jobs stay off the required list, so you can rescale from four shards to eight by editing only the matrix — branch protection keeps requiring merge-report, whose name never changes. Threshold calibration for the diffs those shards run is covered in tolerance thresholds.

Verification

Two things must hold after sharding: every test still runs exactly once, and wall-clock actually dropped.

Coverage first — the merged report totals should equal the single-runner totals. Open the published playwright-report or read the summary:

npx playwright merge-reports --reporter=line ./all-blob-reports
  412 passed (5m 12s)

The count is 412, identical to the single-runner reproduction, so sharding split the work without dropping or double-counting any test. The longest shard, not the sum, now sets wall-clock. Confirm the fan-out timing in the Actions run:

test (1)        4m 51s
test (2)        5m 02s
test (3)        4m 44s
test (4)        5m 09s   ← slowest shard sets the floor
merge-report      38s

Wall-clock fell from ~21 minutes to roughly 5m 47s (slowest shard plus merge) — close to the theoretical 4× ceiling, with the gap explained by uneven shard weight. Tightening that gap is the subject of balancing test shards to reduce CI wall-clock.

Edge cases and caveats

  • Sharding splits files, not tests, so a fat spec skews the balance. --shard=index/total partitions the list of test files, not individual test cases. One 80-story spec file lands entirely on one shard, so four “equal” shards can still finish minutes apart. If wall-clock is dominated by one slow shard, split the large spec into smaller files or move to weight-based balancing rather than adding more shards.
  • Every shard must be able to read the baselines it owns. Sharding is deterministic by file, so a given spec always maps to the same shard index for a fixed total — but each runner still needs the committed baseline PNGs on disk for its slice. A shallow checkout that omits the snapshot directory, or storing baselines in Git LFS without fetching them, makes a shard fail with “missing snapshot” for tests it should have compared. Ensure the checkout includes the baselines on every matrix job.
  • fail-fast: false is mandatory, or you lose coverage on failure. GitHub’s default fail-fast: true cancels every other matrix job the moment one shard fails. For a visual gate that hides the majority of regressions — you would see the first shard’s failures and none of the rest. Always set fail-fast: false so all shards complete and the merged report is a full picture of what changed.

FAQ

How does Playwright --shard split tests across runners?

--shard=index/total splits the list of test files into total contiguous groups and runs only the group matching index, so shard 1/4 runs the first quarter of files and 4/4 the last. Sharding is at the file level, not the test level, so an even file count does not guarantee even runtime — a shard holding one slow spec can still dominate. Balance by weight rather than by count for uneven suites.

How do I combine reports from multiple Playwright shards?

Use the blob reporter on each shard, upload the blob-report directory as a CI artifact, then in a final job download every blob and run npx playwright merge-reports --reporter=html ./all-blob-reports. Blob is the only built-in reporter designed to be merged; merging HTML reports directly does not work because each is a self-contained snapshot of one shard.

Do visual snapshot baselines need to be available on every shard?

Yes. Each shard runs a disjoint set of files but every shard must be able to read the committed baseline PNGs for the tests it owns, so the baselines have to be checked out (or downloaded) on every runner. Because sharding is deterministic by file, a given spec always lands on the same shard index for a given total, so its baseline only needs to exist in the repository, not be copied between shards.