Pipeline Gating Thresholds & Required Checks

A gate is the moment a pipeline stops describing what happened and starts deciding what is allowed to happen next. This page sits under the CI/CD test gating guide and covers the mechanics of that decision: how a test runner’s exit code becomes a red or green check, how you map visual-diff tolerance thresholds onto hard-fail and soft-warn tiers, and how each of GitHub, GitLab, and Bitbucket enforces that a check must be green before a branch can merge.

Getting the gate wrong is expensive in both directions. Too strict and every anti-aliasing wobble blocks a release, training your team to click “merge anyway” until the gate is noise. Too loose and a genuine regression slips through a check that was never actually required. The goal is a small number of gates that block only what should block, backed by the platform’s branch-protection primitives so “merge anyway” is not an option.

Prerequisites

How CI turns a process into a verdict

Every gate ultimately reduces to one integer: the exit code of the last command in the job. By POSIX convention, 0 means success and any value from 1 to 255 means failure. CI runners inherit this directly — the runner executes your script, reads $?, and reports the job as passed or failed accordingly.

How an exit code becomes a merge decision A test command emits an exit code. Exit code zero flows to a passing job, a green status check, and an allowed merge. A non-zero exit code flows to a failing job, a red status check, and a blocked merge when the check is required. Test command playwright test exit code? Job passed green check 0 Job failed red check ≠ 0 Merge allowed gate open Merge blocked if required check

Two failure modes silently break this chain, and both are common enough to check for first:

  • Swallowed exit codes. A step written as npx playwright test || true or one that ends with an unconditional exit 0 will always report success, even when assertions fail. The gate looks present but never fires.
  • Detached background processes. If your test job starts a dev server with npm run dev & and never waits on the test command’s exit code, the shell may return the background job’s status instead. Always run the test command in the foreground and let it own the exit code.

The rest of this page assumes a runner that faithfully returns its exit code. The work is deciding which non-zero outcomes should actually stop a merge.

Step-by-step implementation

Step 1 — Confirm the exit code propagates

Before wiring any branch protection, prove that a failing test actually turns the job red. A gate built on a job that never fails is worse than no gate — it manufactures false confidence.

# .github/workflows/visual.yml — the gating job, stripped to essentials
name: Visual Regression
on: [pull_request]

jobs:
  visual-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      # No `|| true`, no trailing `exit 0`. The runner reads Playwright's code.
      - name: Run visual tests
        run: npx playwright test --project=chromium --reporter=github
# Local sanity check — this is exactly what the runner evaluates
npx playwright test --project=chromium
echo "exit code was: $?"   # expect non-zero when a snapshot diff exceeds tolerance

Verify: Introduce a deliberate one-pixel colour change, push to a branch, and confirm the visual-tests job shows a red X in the PR’s checks list. Then run echo $? locally after the same test and confirm it prints a non-zero number.

Step 2 — Map diff thresholds onto gate tiers

Not every diff deserves to block a merge. Use the tolerance thresholds you already calibrated to sort outcomes into three tiers, then run each tier as its own job so the pipeline can gate them independently.

// gating-tiers.ts — single source of truth for what blocks vs. warns
export const gatingTiers = {
  /** Design-token primitives: any drift blocks the merge. */
  hardFail:  { maxDiffPixelRatio: 0.01, blocking: true,  label: 'visual/primitives' },
  /** Interactive components: block, but with a slightly looser tolerance. */
  standard:  { maxDiffPixelRatio: 0.03, blocking: true,  label: 'visual/components' },
  /** Composite marketing pages: warn only, review asynchronously. */
  softWarn:  { maxDiffPixelRatio: 0.06, blocking: false, label: 'visual/pages' },
} as const;
# Two jobs: one required (hard-fail), one advisory (soft-warn).
jobs:
  visual-blocking:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright test --grep @primitive --grep @component

  visual-advisory:
    runs-on: ubuntu-latest
    continue-on-error: true        # never turns the workflow red
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright test --grep @page

Verify: Trigger a diff that only affects a @page-tagged test. The visual-advisory job should show as neutral/failed without turning the overall PR merge-blocking, while a @primitive diff should hard-fail visual-blocking.

Step 3 — Make the blocking job a GitHub required status check

continue-on-error and job success are only advisory until the branch itself refuses to merge without them. On GitHub that enforcement lives in a branch protection rule (or the newer ruleset), configured against the exact check name.

# Add the visual-blocking job as a required status check via the REST API.
# The context name must match the job/check name GitHub reports on the head SHA.
gh api \
  --method PUT \
  -H "Accept: application/vnd.github+json" \
  /repos/acme/design-system/branches/main/protection \
  -f "required_status_checks[strict]=true" \
  -f "required_status_checks[checks][][context]=visual-blocking" \
  -f "required_status_checks[checks][][context]=UI Tests" \
  -F "enforce_admins=true" \
  -F "required_pull_request_reviews[required_approving_review_count]=1" \
  -F "restrictions=null"
# Equivalent with a repository ruleset (newer, org-shareable) — required_status_checks rule
# .github/rulesets are managed in the UI, but the rule shape is:
#   type: required_status_checks
#   parameters:
#     strict_required_status_checks_policy: true
#     required_status_checks:
#       - context: visual-blocking
#       - context: "UI Tests"        # e.g. the Chromatic check

strict: true additionally requires the branch to be up to date with main before merging, which prevents a green check that was computed against a stale base. The UI Tests context above is the check Chromatic publishes; requiring it is covered in depth in making visual review a required status check.

Verify: Open a PR whose visual-blocking job is red. The merge button should read “Required statuses must pass” and stay disabled. Re-run the job green and confirm the button unlocks.

Step 4 — Configure GitLab approval rules and allow_failure

GitLab splits the same concern across two mechanisms. The pipeline decides pass/fail per job, allow_failure marks a job as non-blocking, and merge request approval rules plus “pipelines must succeed” enforce the gate at the MR level.

# .gitlab-ci.yml
stages: [test]

visual:blocking:
  stage: test
  script:
    - npm ci
    - npx playwright test --grep @primitive --grep @component
  # default allow_failure: false — a red job blocks the pipeline

visual:advisory:
  stage: test
  script:
    - npm ci
    - npx playwright test --grep @page
  allow_failure: true          # job can go red without failing the pipeline
# Enforce "Pipelines must succeed" on the project's merge settings via the API,
# so a failed pipeline (any non-allow_failure job) blocks the merge.
curl --request PUT \
  --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.com/api/v4/projects/acme%2Fdesign-system?only_allow_merge_if_pipeline_succeeds=true&only_allow_merge_if_all_discussions_are_resolved=true"

Layer a merge request approval rule on top so a human must sign off on any accepted visual change. Approval rules are configured under Settings → Merge requests → Approval rules, requiring at least one approval from the design-system code owners.

Verify: Push a branch where visual:blocking fails. The MR should show “Merge blocked: pipeline must succeed” while visual:advisory going red leaves the merge available.

Step 5 — Require passing builds in Bitbucket

Bitbucket Pipelines reports each step’s status back to the pull request as a build result. Branch restrictions then enforce a minimum number of passing builds before merge — the equivalent of a required check.

# bitbucket-pipelines.yml
pipelines:
  pull-requests:
    '**':
      - step:
          name: Visual regression (blocking)
          image: mcr.microsoft.com/playwright:v1.44.0-jammy
          script:
            - npm ci
            - npx playwright test --grep @primitive --grep @component
      - step:
          name: Visual regression (advisory)
          image: mcr.microsoft.com/playwright:v1.44.0-jammy
          script:
            - npm ci
            - npx playwright test --grep @page || true   # advisory: never fails the build
# Add a "minimum passing builds" merge check to the main branch restriction via the API.
curl -X POST -u "$BB_USER:$BB_APP_PASSWORD" \
  -H "Content-Type: application/json" \
  "https://api.bitbucket.org/2.0/repositories/acme/design-system/branch-restrictions" \
  -d '{
        "kind": "require_passing_builds_to_merge",
        "pattern": "main",
        "value": 1
      }'

The advisory step uses || true deliberately so it never counts against the passing-builds total, while the blocking step’s real exit code does. Set value to the number of blocking steps you expect to pass.

Verify: Open a PR where the blocking step fails. Bitbucket should display “At least 1 successful build required” and refuse the merge, while the advisory step’s status has no effect on the button.

Configuration reference

Setting Platform Type Default Effect
continue-on-error GitHub Actions boolean false When true, a failed job is reported but does not fail the workflow — the soft-warn tier
required_status_checks.checks[].context GitHub branch protection string — Exact check name that must be green before merge
strict GitHub branch protection boolean false Require the branch to be up to date with the base before the check counts
enforce_admins GitHub branch protection boolean false Apply the required checks to admins too, closing the “merge anyway” loophole
allow_failure GitLab CI job boolean false When true, a red job does not fail the pipeline (soft-warn)
only_allow_merge_if_pipeline_succeeds GitLab project boolean false Block merge unless the latest pipeline succeeded
require_passing_builds_to_merge Bitbucket branch restriction int — Minimum number of successful builds required before merge
--exit-zero-on-changes Chromatic CLI boolean false When false, any unreviewed diff exits non-zero and blocks the check
maxDiffPixelRatio Playwright number (0–1) undefined Fraction of pixels allowed to differ before the test — and therefore the gate — fails

Common pitfalls

1. Requiring a check whose name does not match what CI reports. GitHub matches required status checks by exact context string. If your workflow job is named visual-tests (chromium) but the branch rule requires visual-tests, the rule matches nothing and the merge is never blocked — yet the UI shows the check as “passing”, so the misconfiguration is invisible until a regression ships. Copy the context name from an actual check run, including any matrix suffix.

2. Leaving enforce_admins off. A required check that admins can bypass is a suggestion, not a gate. During an incident someone with admin rights will “merge anyway”, and the exception quietly becomes the norm. Enable enforce_admins (or the ruleset’s bypass-list equivalent) unless you have a deliberate, documented break-glass process.

3. Confusing “pipeline succeeded” with “the right jobs ran”. A GitLab MR can merge on a green pipeline that skipped the visual job entirely because a rules: or only: condition excluded it on that branch. “Pipelines must succeed” only checks the jobs that ran. Make the gating job unconditional on merge requests, or add it as a code-owner-enforced required job.

4. Marking a diff job allow_failure and then forgetting it exists. A soft-warn tier only works if someone reads the warning. An allow_failure: true visual job that no one reviews is dead weight — it consumes runner minutes and produces artifacts nobody opens. Route advisory results to a PR comment or a dashboard so the soft tier stays actionable, or promote it to blocking.

5. Copying a local threshold straight into CI. Local runs on macOS CoreText and CI runs on Linux software rendering produce different sub-pixel output. A maxDiffPixelRatio that is perfect locally will flap in CI. Store thresholds in a version-controlled config and let CI select the per-engine row that matches its baselines, as covered in setting failure thresholds for visual diffs in CI.

Integration point

Gating thresholds are the decision layer that sits on top of everything else in this pipeline. The tolerance thresholds guide supplies the raw pass/fail values; this page decides which of those failures actually stops a merge. Upstream, the GitHub Actions workflows section defines the jobs whose names you register as required checks, and when those jobs are split across many runners with test parallelization and sharding, you must gate on the merged result rather than any single shard — otherwise a green shard can mask a red one.

Two deeper walkthroughs continue from here: setting failure thresholds for visual diffs in CI covers the per-engine threshold values and how to keep them stable across runners, and making visual review a required status check covers wiring a Chromatic or Percy approval into branch protection so an unreviewed UI change cannot merge.

FAQ

How does CI decide whether a test job passed or failed?

CI reads the exit code of the job’s final command. Exit code 0 means success; any non-zero code marks the job failed. Test runners like Playwright, Jest, and Vitest exit non-zero when an assertion fails, so the gate works automatically as long as no wrapping command swallows the code with a trailing || true or an unconditional exit 0.

What is the difference between a hard-fail and a soft-warn gate?

A hard-fail gate exits non-zero and is a required status check, so it blocks the merge. A soft-warn gate surfaces information (a comment, an annotation, a non-required check) without blocking. In GitHub it is a non-required check or a continue-on-error job, in GitLab an allow_failure: true job, and in Bitbucket a build that is not counted toward the minimum-passing-builds merge check.

How do I make a visual review a required status check on GitHub?

Add the check’s exact name — for example the Chromatic UI Tests context, or your workflow job name — to the required status checks list in a branch protection rule or repository ruleset for the default branch. GitHub then disables the merge button until that context reports success on the head commit. See making visual review a required status check for the full flow.

Should visual diffs use the same failure threshold in CI as locally?

No. CI runners render with different GPU drivers and font stacks than developer laptops, so CI usually needs slightly looser per-engine thresholds. Keep the threshold values in a version-controlled config so both environments read the same source of truth, but allow CI to select the browser-specific row that matches the baselines stored for that runner.