GitLab CI parallel matrix for cross-browser tests

This page is part of the GitLab CI Pipelines section, which covers gating merge requests on component and visual tests inside GitLab. Here we take a single visual-test job and fan it out across Chromium, Firefox, and WebKit using parallel:matrix, so three browser engines run concurrently instead of one after another — and each writes its own artifacts you can inspect independently.

Problem statement

Your visual tests run against one browser in CI. To catch rendering differences you also want Firefox and WebKit, but chaining three sequential commands in one job triples the wall-clock time and collapses three distinct results into a single pass/fail. When the job fails you cannot tell from the pipeline graph which engine regressed without scrolling the log, and all three browsers’ diff images land in the same artifact directory, overwriting each other.

You want three jobs that run in parallel, are named by browser, each keep their own JUnit report and diff images, and can be gated independently — without duplicating the job definition three times.

Root cause

A single job runs its script sequentially on one runner, so testing three browsers means three serial Playwright invocations sharing one artifact namespace and one exit code. GitLab’s parallel:matrix solves this by expanding one job definition into N instances, each with a different value of a variable such as BROWSER, scheduled concurrently on separate runners. The rendering differences you are hunting — the same ones catalogued in the cross-browser matrix guide — only surface when each engine is a first-class job with its own baseline and its own result.

One job definition fans out into three per-browser jobs A single visual-tests definition with parallel:matrix over BROWSER expands into three concurrent jobs named visual-tests chromium, firefox, and webkit, each producing its own JUnit report and diff artifacts. visual-tests parallel:matrix BROWSER [chromium] test-results-chromium/ [firefox] test-results-firefox/ [webkit] test-results-webkit/ three concurrent jobs, three artifact namespaces

Minimal reproduction

The sequential version below is what the matrix replaces — one job, three serial runs, one shared output directory:

# .gitlab-ci.yml — the anti-pattern: serial browsers, shared artifacts
visual-tests:
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  script:
    - npm ci
    - npx playwright test --project=chromium   # ~4 min
    - npx playwright test --project=firefox     # +4 min
    - npx playwright test --project=webkit      # +4 min → 12 min total
  artifacts:
    paths:
      - test-results/            # every browser overwrites this same folder

If WebKit fails, the pipeline graph shows a single red visual-tests node, the first two browsers’ diffs are gone (overwritten), and the whole job took as long as the sum of all three.

Step-by-step fix

1. Parameterize the Playwright command by an environment variable

Read a BROWSER variable in the project selector so one script targets exactly one engine. Define the three projects in playwright.config.ts first.

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests/visual',
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
    { name: 'firefox',  use: { ...devices['Desktop Firefox'] } },
    { name: 'webkit',   use: { ...devices['Desktop Safari'] } },
  ],
});
// package.json (scripts excerpt)
{
  "scripts": {
    "test:visual": "playwright test --project=$BROWSER"
  }
}

What this does: --project=$BROWSER selects a single Playwright project at runtime, so the exact same script runs Chromium, Firefox, or WebKit depending only on the BROWSER value the job is given.

2. Expand the job with parallel:matrix

Replace the serial script with a parallel:matrix block listing the three browsers. GitLab expands this into three concurrent job instances, each with BROWSER set to one value.

# .gitlab-ci.yml
stages:
  - test

visual-tests:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel:
    matrix:
      - BROWSER: [chromium, firefox, webkit]   # → 3 concurrent instances
  script:
    - npm ci
    - npm run test:visual                       # uses $BROWSER from the matrix

What this does: one job definition becomes three jobs that schedule in parallel on separate runners. Total wall-clock drops to roughly the slowest single browser instead of the sum of all three, and each instance carries its own exit code.

3. Scope artifacts and reports per browser

Interpolate $BROWSER into the output directory, the JUnit report name, and the artifact paths so each instance writes to a distinct location that never collides with its siblings.

visual-tests:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel:
    matrix:
      - BROWSER: [chromium, firefox, webkit]
  variables:
    # Per-instance output location, resolved from the matrix value
    PLAYWRIGHT_JUNIT_OUTPUT_NAME: "junit-$BROWSER.xml"
  script:
    - npm ci
    - npm run test:visual -- --output "test-results-$BROWSER"
  artifacts:
    when: always
    paths:
      - "test-results-$BROWSER/"        # e.g. test-results-webkit/
    reports:
      junit: "junit-$BROWSER.xml"       # e.g. junit-firefox.xml
    expire_in: 1 week

What this does: because every artifact path and report name embeds $BROWSER, the three instances upload to test-results-chromium/, test-results-firefox/, and test-results-webkit/ independently. A WebKit diff is downloadable without a Chromium run clobbering it, and each browser’s results feed the merge request’s Tests tab separately.

4. Read the generated job names and gate selectively

GitLab names each instance visual-tests: [value]. Use those names to keep Chromium as a hard gate while letting the other engines report without blocking during stabilization.

visual-tests:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  parallel:
    matrix:
      # Second variable adds ALLOW so each instance sets its own blocking policy
      - BROWSER: chromium
        REQUIRED: "true"
      - BROWSER: [firefox, webkit]
        REQUIRED: "false"
  script:
    - npm ci
    - npm run test:visual -- --output "test-results-$BROWSER"
  # Non-Chromium engines report but do not block while baselines settle
  allow_failure:
    exit_codes: 1
  rules:
    - if: '$REQUIRED == "false"'
      allow_failure: true
    - when: on_success

What this does: the matrix now expands into visual-tests: [chromium, true], visual-tests: [firefox, false], and visual-tests: [webkit, false]. The rules block flips allow_failure on for the non-required engines, so Firefox and WebKit surface diffs in the pipeline without red-gating the merge until you drop the REQUIRED: "false" rows.

Verification

Push the branch and open the pipeline. The single definition should have expanded into three named nodes running at once:

Pipeline #4821  ●  test stage
  ✓ visual-tests: [chromium]   3m 58s
  ✓ visual-tests: [firefox]    4m 11s
  ✗ visual-tests: [webkit]     4m 06s   ← engine identified from the name alone

Download artifacts per instance and confirm they are isolated:

$ ls
junit-chromium.xml   test-results-chromium/
junit-firefox.xml    test-results-firefox/
junit-webkit.xml     test-results-webkit/

The failing WebKit node names the engine directly in the pipeline graph, its diff images sit in test-results-webkit/, and the total stage duration is about four minutes — the slowest single browser — rather than the twelve minutes the serial version took.

Edge cases and caveats

  • Matrix variable values must be strings; GitLab does not coerce lists inside interpolation. BROWSER: [chromium, firefox, webkit] expands correctly, but referencing $BROWSER inside a path only ever holds a single value per instance. If you nest two variables, remember the job name lists them comma-separated ([chromium, true]), which changes the exact string a required-status-check rule must match.
  • Concurrent instances need enough runners or they queue. parallel:matrix only runs instances in parallel if runners are available; with a single runner the three jobs serialize and you lose the wall-clock benefit. Check the runner concurrency limit before assuming the matrix is faster, and consider sharding within each browser if a single engine’s suite is itself slow.
  • WebKit rendering differs enough to need its own baselines. Sharing one baseline set across engines produces false diffs from font hinting and sub-pixel anti-aliasing. Store baselines per project (Playwright already suffixes snapshots with the project name) and treat a new engine’s first run as baseline generation, not a gate.

FAQ

How does GitLab name the jobs generated by parallel:matrix?

GitLab appends the matrix variable values in square brackets to the base job name, for example visual-tests: [chromium], visual-tests: [firefox], and visual-tests: [webkit]. When a matrix has multiple variables the names contain each value comma-separated, such as visual-tests: [firefox, mobile]. These names appear in the pipeline graph, in the merge request widget, and in required-status-check rules, so a single failing browser is identifiable without opening any logs.

Why do my per-browser artifacts overwrite each other in a parallel:matrix job?

All matrix instances share the same base job definition, so if the artifact path is a fixed string like test-results/ every instance uploads to the same logical slot and only the last one is easy to find. Interpolate the matrix variable into the path — test-results-$BROWSER/ — and into the JUnit report name so each browser writes to a distinct location that you can download independently.

Should WebKit and Firefox failures block the merge or just warn?

That is a gating policy decision, not a matrix mechanic. Keep Chromium as a hard gate and, while you stabilize the others, mark the Firefox and WebKit instances allow_failure: true so they report without blocking. Once their baselines are stable, remove allow_failure so every browser in the matrix is a required check. Font and sub-pixel rendering differences between engines are the usual reason to stage this rollout.