GitLab CI Pipelines for Visual Regression #

A GitLab CI pipeline is where your component and visual regression tests stop being a local convenience and start being a merge gate. The .gitlab-ci.yml file is a declarative description of stages, jobs, caching, and artifacts — and getting its structure right is the difference between a suite that reliably blocks regressions in under ten minutes and one that either times out, re-downloads browsers on every job, or lets pixel drift slip through because the visual job was marked non-blocking by accident.

This page builds a production .gitlab-ci.yml step by step: ordered stages, a lockfile-keyed cache that survives across the browser fan-out, parallel:matrix for cross-browser coverage, failure-only artifacts for diff review, merge-request-scoped rules, and allow_failure tiers that decide exactly which jobs can block a merge. It is the parent guide for two focused walkthroughs — running Storybook visual tests in the pipeline and expanding the browser matrix — linked in context below.

Prerequisites #

• A GitLab project with CI/CD enabled and at least one available runner (shared SaaS runners or a self-hosted docker executor)
• A visual regression test runner in the repo: Playwright ≥ 1.40 (used throughout) or an equivalent that emits diff images to a known folder
• At least one accepted baseline committed, captured inside a container matching the CI image
• A package-lock.json (or yarn.lock / pnpm-lock.yaml) committed so the cache key can hash it
• Merge request pipelines enabled (Settings → Merge requests) so merge_request_event rules resolve

How a GitLab pipeline gates a merge #

Before writing YAML, it helps to see how the pieces connect. Stages impose order, needs relaxes that order into a graph, the cache flows install output into every downstream job, and the final gate stage decides the merge.

The strict browser jobs (allow_failure: false) can fail the pipeline and block the merge; the soft job (allow_failure: true) surfaces a warning without gating. Everything downstream depends on the cache the install job produced, which is why cache correctness is the first thing to get right.

Step-by-step implementation #

Step 1 — Declare the pipeline skeleton and image #

Start with a global image, an ordered stages list, and a default block that every job inherits. Pinning the Playwright image to an exact version keeps the runner’s font stack and browser binaries identical to the environment your baselines were captured in.

# .gitlab-ci.yml
# Pin the exact Playwright image: it ships the browsers and system fonts
# your baselines were captured against. Never use :latest for visual tests.
image: mcr.microsoft.com/playwright:v1.44.0-jammy

stages:
  - install       # resolve dependencies once, build the shared cache
  - unit          # fast component tests — fail early before slow visual jobs
  - visual        # cross-browser screenshot comparison
  - gate          # aggregate result that the merge request check reads

variables:
  # Keep the browser binaries inside the project dir so the cache can hold them.
  PLAYWRIGHT_BROWSERS_PATH: "$CI_PROJECT_DIR/.cache/ms-playwright"
  npm_config_cache: "$CI_PROJECT_DIR/.npm"

Verify: Commit the file and push. In the GitLab UI under CI/CD → Pipelines the pipeline should render four stage columns in order, even though no jobs exist yet — a syntax error in stages blocks the pipeline from being created at all.

Step 2 — Cache dependencies and browser binaries #

A cache keyed on the lockfile hash means the expensive install runs only when dependencies change. Point the cache paths at both the npm cache and the Playwright browser directory so the browser download survives into every downstream job instead of being repeated per browser.

# The install job owns cache population. Downstream jobs pull the same key.
install:
  stage: install
  cache:
    # Hashing the lockfile means the cache invalidates only on dependency change.
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
      - .cache/ms-playwright/
    policy: pull-push          # this job may write the cache
  script:
    - npm ci --cache .npm --prefer-offline
    - npx playwright install --with-deps chromium firefox webkit

Verify: Run the pipeline twice without changing package-lock.json. In the second run’s install job log, the cache section should report “Successfully extracted cache” and the playwright install step should print “browser is already installed” for each engine rather than downloading.

Step 3 — Run the fast unit tier first #

Component unit tests are cheap and catch the majority of regressions before a single screenshot is taken. Gate the pipeline on them early so a broken component fails in under a minute rather than after the whole visual matrix has spun up. This tier is the component testing layer that runs before anything visual.

unit:
  stage: unit
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
      - .cache/ms-playwright/
    policy: pull                # read-only: never mutate the shared cache
  needs: ["install"]            # start the moment install finishes (DAG)
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run test:unit -- --reporter=dot
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Verify: Open a merge request that breaks a component prop type. The unit job should turn red within roughly a minute, and because later stages depend on it, the visual jobs should never start.

Step 4 — Fan out browsers with parallel:matrix #

parallel:matrix expands one job definition into one child job per matrix entry, injecting the matrix variable into each child’s environment. This is the DRY way to cover Chromium, Firefox, and WebKit — the cross-browser matrix that drives most engine-specific pixel variance. The dedicated walkthrough on GitLab CI parallel:matrix for cross-browser tests goes deeper on multi-dimensional matrices.

visual:
  stage: visual
  needs: ["install"]            # does not wait for unit; both depend only on install
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - .npm/
      - .cache/ms-playwright/
    policy: pull
  parallel:
    matrix:
      # Each entry becomes its own job: visual: [chromium], visual: [firefox]...
      - BROWSER: [chromium, firefox, webkit]
  script:
    - npm ci --cache .npm --prefer-offline
    # $BROWSER is injected per child job by the matrix expansion.
    - npx playwright test --project=$BROWSER --reporter=line,junit
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Verify: The pipeline graph should show three parallel jobs named visual: [chromium], visual: [firefox], and visual: [webkit]. Each job log should print the browser it received via $BROWSER and run only that project.

Step 5 — Publish diff artifacts on failure #

When a screenshot comparison fails, reviewers need the actual/expected/diff PNGs Playwright writes to test-results/. Upload them only on_failure to keep green-pipeline storage costs down, and attach the JUnit report so GitLab renders per-test results inline in the merge request.

# Append this artifacts block to the `visual` job from Step 4.
  artifacts:
    when: on_failure           # only pay for storage when there is a diff to review
    name: "diffs-$BROWSER-$CI_COMMIT_SHORT_SHA"
    paths:
      - test-results/          # actual/expected/diff PNGs Playwright emits
      - playwright-report/
    reports:
      junit: results/junit.xml # surfaces per-test status in the MR widget
    expire_in: 14 days

Verify: Push a one-pixel colour change to a component. The failing visual: [chromium] job should expose a “Download artifacts” button; the archive must contain a *-diff.png under test-results/, and the merge request’s test summary should list the failed test name.

Step 6 — Gate the merge with allow_failure tiers #

The merge decision is the point of the pipeline. Split visual coverage into a strict tier that blocks and a soft tier that only warns. allow_failure: false (the default) lets a job fail the pipeline; allow_failure: true reports failure as a non-blocking warning. Pair this with a required pipeline check so strict failures actually prevent merge. Threshold tuning that decides what counts as a failure lives in pipeline gating thresholds.

# Strict tier: engines with stable rendering block the merge.
visual:blocking:
  extends: .visual_base        # reuse script/cache via a hidden template job
  parallel:
    matrix:
      - BROWSER: [chromium, firefox]
  allow_failure: false         # a diff here fails the pipeline and blocks merge

# Soft tier: WebKit's font hinting is flakier — surface it without gating.
visual:advisory:
  extends: .visual_base
  parallel:
    matrix:
      - BROWSER: [webkit]
  allow_failure: true          # failure shows an orange warning, does not block

# Final aggregate the merge request required check points at.
gate:
  stage: gate
  needs: ["visual:blocking"]   # green only if every blocking job passed
  script:
    - echo "All blocking visual tiers passed — merge is permitted."
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Verify: In Settings → Merge requests, enable “Pipelines must succeed”. Open a merge request where a WebKit-only diff appears: the visual:advisory job shows an orange warning but the merge button stays enabled. Introduce a Chromium diff and the merge button becomes disabled until the pipeline is green.

Configuration reference #

Common pitfalls #

1. Caching browsers to a path GitLab cannot persist. Playwright installs browsers to ~/.cache/ms-playwright by default, which lives outside $CI_PROJECT_DIR. GitLab CI only caches paths under the build directory, so the cache silently holds nothing and every job re-downloads three browsers. Set PLAYWRIGHT_BROWSERS_PATH to a project-relative directory and add that exact path to cache:paths.

2. Using pull-push cache policy on every job. If all jobs write the cache, parallel matrix jobs race to upload it and clobber each other, and the final stored cache is nondeterministic. Only the install job should use pull-push; every downstream job must use policy: pull so it reads the install artifact without mutating it.

3. Forgetting rules and running visual jobs on every branch push. Without an if: $CI_PIPELINE_SOURCE == "merge_request_event" rule, the full cross-browser matrix runs on every commit to every branch, burning runner minutes on work that gates nothing. Scope the expensive jobs to merge request pipelines.

4. Assuming allow_failure: true still blocks the merge. A soft-tier job that fails shows an orange warning that is easy to read as “blocking”. It is not — the pipeline is still considered successful and, with “Pipelines must succeed” enabled, the merge proceeds. Only allow_failure: false jobs gate. Keep anything you actually rely on in the strict tier.

5. Relying on stage order alone and paying for idle runners. With only stages and no needs, the visual stage waits for the entire unit stage to finish even though it only depends on install. On wide matrices this adds minutes of idle time. Add needs: ["install"] to let visual jobs start as soon as their real dependency completes.

Integration point #

This pipeline is the GitLab-specific execution surface for the wider CI/CD gating strategy. The allow_failure tiers here decide whether a diff blocks a merge, but what counts as a diff is set by pipeline gating thresholds — coordinate the two so a job marked allow_failure: false is also running against a threshold strict enough to be worth blocking on. As the browser matrix and test count grow, the single-job-per-browser fan-out shown here becomes the seam where test parallelization and sharding plugs in, splitting each browser’s specs across multiple runners.

For the two most common job payloads, see the focused walkthroughs: running Storybook visual tests in GitLab CI covers building and serving a static Storybook inside the job, and GitLab CI parallel:matrix for cross-browser tests expands the single-axis matrix here into multi-dimensional browser-by-viewport coverage. The Storybook interaction tests and accessibility checks are the kinds of assertions these jobs actually run.

FAQ #

How do I run cross-browser visual tests in a single GitLab CI job definition? #

Use parallel:matrix under a single job. GitLab expands each matrix entry into a separate child job and injects the matrix variable into that job’s environment, so one job block defining BROWSER: [chromium, firefox, webkit] produces three jobs that each run the same script with a different browser value. This keeps the YAML DRY while fanning the work out across runners.

How does allow_failure control merge gating in GitLab CI? #

allow_failure: false (the default) means a failed job fails the pipeline and, when the pipeline is a required check, blocks the merge. allow_failure: true lets the job report a failure without failing the pipeline — it shows an orange warning icon instead. Use false for strict tiers that must never regress and true for soft tiers you want visible but non-blocking.

Why is my GitLab CI cache not restoring Playwright browsers between jobs? #

The most common cause is that the Playwright browser download path is outside the project directory, and GitLab CI only caches paths under the build directory. Set PLAYWRIGHT_BROWSERS_PATH to a project-relative folder such as $CI_PROJECT_DIR/.cache/ms-playwright and add that path to the cache block. Also confirm the cache key matches across jobs so they share one cache entry.

Should visual regression jobs use needs or stages in GitLab CI? #

Use stages for the coarse ordering (install before test before gate) and needs to build a directed acyclic graph that lets a job start the moment its specific dependencies finish rather than waiting for the whole previous stage. Adding needs: to the visual jobs so they start as soon as the install job completes can cut several minutes off wall-clock time on wide matrices.

Related #

• CI/CD Test Gating & Pipeline Integration — parent guide covering pipeline gating across GitHub Actions, GitLab, and Bitbucket
• Pipeline Gating Thresholds — set the diff thresholds and required checks that these allow_failure tiers enforce
• Test Parallelization & Sharding — split each browser’s specs across runners once the matrix outgrows one job per engine
• Running Storybook Visual Tests in GitLab CI — build and serve a static Storybook inside the pipeline before capturing screenshots
• GitLab CI parallel:matrix for Cross-Browser Tests — expand the single-axis browser matrix into multi-dimensional coverage