Making visual review a required status check

This page is part of the Pipeline Gating Thresholds & Required Checks section of the CI/CD test gating guide. The previous page tuned a visual diff job so its result is trustworthy; this one closes the last gap — a trustworthy job that reports its result but is not wired into branch protection, so pull requests merge straight past it.

Problem statement

The visual regression workflow runs on every pull request. It even goes red when a regression slips in. And yet features ship with broken UI, because the red check never stopped anyone from clicking merge.

The disconnect is that a running job and a required job are two unrelated facts. GitHub, GitLab, and Bitbucket all execute your pipeline eagerly, but by default they treat the result as advisory. The merge button stays enabled whether the visual check is green, red, or still running. A reviewer who is focused on the code diff sees a small red cross next to a check they do not recognize, assumes it is a flaky lint job, and merges. Nothing in the platform prevents it.

This is especially common with hosted visual tools like Chromatic, whose result arrives as a separate commit status (UI Tests) named nothing like your workflow file — so even teams that require their own jobs forget to require the one that actually gates visual diffs.

Root cause

Branch protection is opt-in per check. A status check only blocks a merge if its exact context name is listed under the protected branch’s required status checks, and by default that list is empty. The job you configured back in setting failure thresholds for visual diffs in CI reports a status, but reporting a status and requiring it are separate settings. Until the check’s name is added to branch protection — and the name has to match character-for-character, including the third-party UI Tests and UI Review contexts Chromatic posts — the platform has no instruction to block on it.

Reported status versus required status check Two lanes. Top lane: a pull request receives a red visual status, but because branch protection does not list it, the merge button is enabled and the merge proceeds. Bottom lane: the same red status, but branch protection lists it as required, so the merge button is disabled until the check passes. Not required → merge slips through Visual job reports RED Branch protection list is empty Merge ENABLED regression ships Required → merge blocked Visual job reports RED Branch protection lists UI Tests Merge BLOCKED until check is green

Minimal reproduction

Open a pull request whose only change breaks a snapshot, and watch it merge anyway. With the visual workflow running but no branch protection configured, the PR’s check summary looks like this:

Some checks were not successful
  ✓  build            Successful in 41s
  ✓  unit-tests       Successful in 1m 12s
  ✘  visual           Failing in 58s          ← the regression is caught here
  ✓  UI Tests         (Chromatic)  — 3 changes must be accepted

This branch has no conflicts with the base branch
[ Merge pull request ]   ← button is green and clickable

The visual job did its job. The Chromatic UI Tests status is red with unaccepted changes. And the merge button is still enabled, because nothing told the repository that either result matters. Click it and the regression is on the main branch.

Step-by-step fix

1. Read the exact check context names

Branch protection matches on the literal check name. Guessing it is the number one cause of a “required” check that silently never applies. Pull the real names from a recent commit on the PR branch.

# List every check context GitHub recorded for the PR head commit
gh api repos/acme/design-system/commits/$(gh pr view 812 --json headRefOid -q .headRefOid)/check-runs \
  --jq '.check_runs[].name'

# And the legacy commit statuses (this is where Chromatic posts UI Tests / UI Review)
gh api repos/acme/design-system/commits/$(gh pr view 812 --json headRefOid -q .headRefOid)/statuses \
  --jq '.[].context' | sort -u

What this does: it prints the exact strings — for example visual, UI Tests, and UI Review — that you must copy verbatim into branch protection. A trailing space or a job-name change breaks the requirement without any warning.

2. Require the checks on the protected branch

Add the visual job and the Chromatic contexts to the protected branch. Through the UI: Settings → Branches → Add branch ruleset (or Edit protection) → Require status checks to pass before merging, then type each name. To make it reproducible, do it through the API instead.

# Require the visual job AND Chromatic's UI Tests context on main.
# strict:true also forces the branch to be up to date before merging.
gh api -X PATCH repos/acme/design-system/branches/main/protection \
  -H "Accept: application/vnd.github+json" \
  -f 'required_status_checks[strict]=true' \
  -f 'required_status_checks[checks][][context]=visual' \
  -f 'required_status_checks[checks][][context]=UI Tests' \
  -F 'enforce_admins=true' \
  -F 'required_pull_request_reviews[required_approving_review_count]=1' \
  -F 'restrictions=null'

What this does: visual and UI Tests are now blocking. A pull request cannot merge into main until both report success, and enforce_admins=true means even repository admins are held to the rule. Add -f 'required_status_checks[checks][][context]=UI Review' when you also want Chromatic’s human sign-off recorded before merge.

3. Stop a skipped job from blocking every merge

A required check that never runs stays permanently pending and blocks all merges. This bites teams who gate the visual job behind a path filter — PRs that touch no components never run it, so the required check never reports. Fix it by making the check always report a conclusion.

# .github/workflows/visual.yml
name: visual            # this workflow's job name becomes the check context
on: pull_request

jobs:
  visual:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      # Decide inside the job — never gate the whole job with a top-level path filter,
      # or the required check will hang "pending" on docs-only PRs.
      - id: changed
        run: |
          if git diff --name-only origin/${{ github.base_ref }}... | grep -qE '\.(tsx|css|scss)$'; then
            echo "run=true" >> "$GITHUB_OUTPUT"
          else
            echo "run=false" >> "$GITHUB_OUTPUT"
          fi
      - if: steps.changed.outputs.run == 'true'
        run: |
          npm ci
          npx playwright install --with-deps chromium
          npx playwright test tests/visual
      - if: steps.changed.outputs.run == 'false'
        run: echo "No UI changes — visual check passes trivially."

What this does: the visual job always runs and always finishes with a conclusion (success), so the required check is never stuck pending. The expensive Playwright steps are skipped on non-UI PRs via step-level if, not by skipping the whole job — which is what would otherwise leave the required check hanging.

4. Apply the GitLab and Bitbucket equivalents

The same gate exists under different names on other platforms.

# GitLab: .gitlab-ci.yml — a job WITHOUT allow_failure blocks the MR merge
# when "Pipelines must succeed" is enabled in Settings → Merge requests.
visual:
  stage: test
  image: mcr.microsoft.com/playwright:v1.48.0-jammy
  script:
    - npm ci
    - npx playwright test tests/visual
  allow_failure: false          # false is the default; being explicit is the gate
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

In the GitLab project settings, enable Settings → Merge requests → Pipelines must succeed, and add a Coverage/Approval rule if you also want a required reviewer. On Bitbucket, the equivalent is a branch restriction:

Repository settings → Branch restrictions → Add restriction on `main`:
  ☑ Require a minimum number of approvals: 1
  ☑ Require passing builds before merging: 1

What this does: GitLab’s “Pipelines must succeed” plus a non-allow_failure job makes the visual job block the merge, and Bitbucket’s “Require passing builds” does the same for its pipeline result. All three platforms end at the same guarantee — a failed visual result disables merge.

Verification

Confirm the gate actually blocks. Re-open the reproduction PR and query its mergeability after protection is in place.

gh pr view 812 --json mergeStateStatus,statusCheckRollup \
  -q '{state: .mergeStateStatus, checks: [.statusCheckRollup[] | {name, conclusion}]}'
{
  "state": "BLOCKED",
  "checks": [
    { "name": "build",     "conclusion": "SUCCESS" },
    { "name": "unit-tests","conclusion": "SUCCESS" },
    { "name": "visual",    "conclusion": "FAILURE" },
    { "name": "UI Tests",  "conclusion": "FAILURE" }
  ]
}

mergeStateStatus is now BLOCKED and the merge button in the UI is disabled with the message “Required statuses must pass before merging.” Accept the intended snapshot changes (or fix the regression), let visual and UI Tests go green, and the state flips to CLEAN — the merge button re-enables. A gate that reports BLOCKED while the visual check is red and CLEAN once it is green is correctly enforced.

Edge cases and caveats

  • Renaming the job silently disables the requirement. Branch protection stores the check name as a plain string. If you rename the workflow job from visual to visual-regression, the old required check will sit pending forever (blocking every PR) while the new one is not required at all (gating nothing). Update the required-checks list in the same commit that renames a job, and prefer GitHub rulesets, which surface “expected but never reported” checks.
  • Third-party contexts only appear after they have run once. GitHub’s required-checks picker only lists check names it has already observed. A brand-new Chromatic integration will not offer UI Tests in the dropdown until the first build posts it — run one PR through, then add the context. The gh api approach in step 2 sidesteps this because it accepts arbitrary context strings.
  • Forked-PR runs may not post third-party statuses. For security, some CI providers and Chromatic project settings withhold tokens on pull requests from forks, so the visual status never posts and a required check hangs. Decide deliberately: either require contributors to branch within the repository, or use a pull_request_target-based flow with reviewed, least-privilege token exposure so the status still reports on fork PRs.

FAQ

Why does my PR merge even though the visual test job exists?

Running a job and requiring a job are different things. Unless the job’s check context is listed under required status checks in branch protection, GitHub treats it as informational and allows the merge regardless of its result. Add the exact check name to the protected branch’s required checks so a red or missing result blocks the merge button.

What are the Chromatic UI Tests and UI Review status checks?

Chromatic posts two commit statuses to your PR. UI Tests is the visual regression result — it turns red when snapshots change and no one has accepted them. UI Review is the collaborative sign-off state. Add UI Tests as a required check to block on visual regressions, and add UI Review when you also want a human approval recorded before merge.

How do I require a check that is skipped on some PRs?

A required check that never runs stays pending forever and blocks every merge. If you gate the job with a path filter, add a companion job with the same check name that always runs and reports success when the real job is skipped, or drive the skip with conditional steps inside an always-running job so the check always reports a conclusion.