GitLab Well Architected
Cookbook navigation
Skills / diagnose-failed-pipeline

Diagnose a failed pipeline

On this page
CI/CDTestingTroubleshooting

A failed pipeline doesn't always mean the latest code change caused it. A passing retry might point to an intermittent test, a runner problem, or a service that recovered. Give this Skill the pipeline URL in a GitLab Duo, Claude, or Codex session with GitLab access. The agent reads the job attempts and logs, compares them with a relevant successful run when one exists, and recommends a check to narrow down the cause.

The agent gathers the GitLab records itself. Terminal-based sessions need an authenticated glab client; a custom Flow can use its GitLab tools. The Skill instructs the agent to keep the investigation read-only, including leaving retries and pipeline starts for a separate request. Review the diagnosis before asking for a repair.

Add the Skill to an agent

Download the package from this page and keep the directory together. For GitLab Duo CLI or a custom GitLab Flow, place it in the project's skills/ directory:

skills/
└── diagnose-failed-pipeline/
    ├── README.md
    ├── SKILL.md
    ├── references/
    ├── templates/
    └── examples/

GitLab's Agent Skills feature requires Premium or Ultimate; project Skills in Duo CLI also need version 8.73.0 or later. Start a new agent conversation after adding the package, then enter /skills in Duo CLI to check that diagnose-failed-pipeline is listed. For a custom Flow, commit the package and configure its Skills input as shown below before starting a new Session.

Run it from GitLab Duo CLI

Open the target project's checkout after authenticating glab, then start an interactive Duo session:

glab duo cli

Give Duo the pipeline URL with the Skill's slash command:

/diagnose-failed-pipeline https://gitlab.example.com/group/project/-/pipelines/12345

You can also run it headlessly:

glab duo cli run --goal "Use diagnose-failed-pipeline to diagnose https://gitlab.example.com/group/project/-/pipelines/12345 and recommend the next check."

The agent uses the GitLab access available in the session to gather the pipeline details and logs. When it uses glab, authentication must cover the host in the pipeline URL.

Use it from Claude or Codex

Open the target project in a Claude or Codex session that can run the authenticated glab CLI. If you've kept the package at the path above, you can point the agent straight to it without relying on automatic discovery:

Read skills/diagnose-failed-pipeline/SKILL.md and follow its instructions to diagnose https://gitlab.example.com/group/project/-/pipelines/12345. Recommend the next check without changing the project.

The Skill tells the agent which references to read and which glab ci and glab api commands to run; you don't need to run those commands yourself.

If the agent reports that glab auth status fails for the pipeline's GitLab host, authenticate once with glab auth login, then ask it to continue. You don't need to export a token or collect the logs yourself.

Follow the failure before the diff

Once activated, the agent reads the GitLab access and CI context guide and chooses the access already available in its session. GitLab Duo Agent Platform sessions use pipeline tools; terminal-based agents use authenticated glab commands.

It then reads the failure-signals guide and gathers all the attempts for each job so it can compare retries and runners. Before opening the diff, it looks for a successful run that can help explain the failure. If retries are involved, it also uses the retries and flakes guide.

That order gives the diff a specific failure to explain instead of treating every changed line as a possible cause. When no successful run exists, the Skill says so and uses the merge-request or parent diff only to understand what changed.

Read the diagnosis

The agent returns a diagnosis report with linked jobs, the first meaningful error, retry and runner comparisons, relevant changes, a likely explanation, remaining unknowns, and the next check.

Start with the job links and comparison run. A useful report should let you tell what was observed and what still needs testing:

What the report shows How to judge it
A failed attempt followed by a pass Check whether the commit, runner, inputs, and environment were comparable before calling the test flaky.
Several unrelated jobs failing on one runner Look for a shared preparation error and a comparison with another runner before recommending source changes.
No comparable successful pipeline Expect the report to say so. A parent or merge-request diff shows what changed, not that the earlier code passed.
A suspected code regression Look for a changed file or behavior that explains the actual error, plus a check that could confirm or disprove it.

If the report ends at exit code 1, omits retries, or blames the latest diff without connecting it to a log error, ask the agent to revisit those records. The next check should name what to compare and how each possible result would change the diagnosis.

The runner-capacity example shows how several job failures can point to a shared runner problem, and why that finding doesn't justify a source-code change. The unrelated-change example shows how to proceed when the project has no successful run to compare with and the only changed file can't explain the observed failure.

Use it from a custom Flow

A custom Flow uses the same diagnosis method with GitLab tools instead of glab in the runtime. The following configuration follows GitLab 19.4, including the tools for downstream pipelines. Configure its Agent component before starting a Session.

Pass the project's Skills context to the Agent component:

inputs:
  - from: "context:inputs.workspace_agent_skills"
    as: "workspace_agent_skills"
    optional: true

The Flow's Agent component also needs get_pipeline_failing_jobs, get_job_logs, and the read-only gitlab_api_get tool. Add get_failing_bridge_jobs and get_downstream_pipelines when your CI configuration launches downstream pipelines. The specialized tools trace the failure through the jobs and retrieve their logs. The API tool provides the broader context: complete retry history, pipelines to compare, related merge requests, and commit comparisons. The Skill uses those tools directly and doesn't expect glab inside the Flow runtime.

Package contents

Browse every file without leaving the page. Directories can contain scripts, references, templates, examples, assets, or anything else the author includes.

Files8
  • examples2
  • references3
  • templates1
SKILL.mdmarkdown · 59 lines
---
name: diagnose-failed-pipeline
description: Diagnose failed or intermittent GitLab pipelines from GitLab Duo CLI or another AI coding session with GitLab access. Use when a user asks why a pipeline or job failed, whether a passing retry indicates a flaky failure, or which check should come next before changing code.
metadata:
  slash-command: enabled
---

# Diagnose a failed GitLab pipeline

Trace the failed jobs first, compare them with a relevant successful run, and only connect the failure to a code change when the observations support that conclusion.

Resolve every path below relative to this Skill's directory.

## Work from the current agent session

A pipeline URL or ID is normally the only run-specific input the user needs to provide. Gather the rest yourself through the access already available to the agent:

1. Read [the GitLab access and CI context guide](references/gitlab-ci.md).
2. If the session exposes GitLab Duo Agent Platform pipeline tools, use `get_pipeline_failing_jobs` and `get_job_logs`. Use `get_failing_bridge_jobs` and `get_downstream_pipelines` when the failure crosses into a downstream pipeline. Use `gitlab_api_get` for all job attempts, pipeline history, merge requests, and repository comparisons when the specialized tools don't return them.
3. Otherwise, when the session has shell access, run `glab auth status` for the pipeline's GitLab host. If authentication succeeds, use the `glab` commands in `references/gitlab-ci.md` yourself.
4. If neither access path is available, explain that the session needs GitLab pipeline access or an authenticated `glab` client. Don't ask the user to export tokens, run collection scripts, or copy logs that the agent can retrieve after authentication is available.

Keep collection read-only. Don't retry jobs, start pipelines, modify CI configuration, or edit source code while diagnosing the failure.

## Collect what happened

For each failure group, collect:

- The pipeline source, ref, commit, timestamps, and related merge request when one exists.
- Any failed trigger job and downstream pipeline that owns the actual failed jobs.
- Every attempt for each logical job, including attempts that later passed.
- The job name, stage, URL, status, duration, attempt order, and runner.
- The first meaningful error in the trace, along with enough surrounding output to understand it.
- Other jobs with the same error signature.
- The nearest useful successful pipeline when one exists.
- The exact repository change range that could explain the failure.

Keep these two comparisons distinct:

- A successful pipeline is a behavioral baseline: it shows that comparable code and CI conditions completed successfully.
- A commit, merge-request, or parent diff is a change comparison: it shows what changed, but doesn't show that either side passed.

## Work from the failure toward the change

1. Group attempts by logical job name and stage, then group jobs by their first meaningful error rather than their final status message. One root failure can create several later failures.
2. Read [the failure-signals guide](references/failure-signals.md), then compare each group across retries, runners, and the nearest useful successful run.
3. If the pipeline includes retries, inconsistent outcomes, or a suspected flaky test, read [the retries and flakes guide](references/retries-and-flakes.md).
4. Classify each group using the categories in `references/failure-signals.md`. Keep a group inconclusive when more than one explanation still fits.
5. Inspect the relevant diff after the failure pattern is clear. Check whether GitLab reports an incomplete comparison, and name a changed file or behavior only when it can explain the observed error.
6. Choose the smallest next check that can separate the leading explanation from the remaining alternatives.

## Write the result

Read [the diagnosis report template](templates/diagnosis-report.md), then use all of its sections for the final response. Keep observed results separate from interpretation, include GitLab links when they are available, and state what would confirm or disprove the working diagnosis.

When several unrelated jobs fail on the same runner or during the same preparation step, read [the runner-capacity example](examples/runner-capacity.md). When no successful pipeline exists or the changed files don't explain the failure, read [the unrelated-change example](examples/unrelated-change.md). Use the examples' reasoning and report structure, not their conclusions.

Stop after the diagnosis and recommended next check. Modify source code only when the user explicitly asks for a repair after reviewing the diagnosis.

Source file

File