An intervention plan becomes useful when the people responsible have practised it under representative conditions. Before granting Act with guardrails authority, agree who can stop the workflow, revoke its credentials, suspend a release, preserve records, assess impact, approve recovery, and communicate with affected owners. Make sure they can reach the systems and people they'll need during an incident.
The following GitLab.com operations cover revoking an affected credential, canceling identified pipelines, preserving incident records, and restoring a reviewed repository or policy revision. Practise them in a private project before an incident so responders know what each operation does. During an incident, coordinate with the owners of connected systems, who will need to stop work already accepted outside GitLab and revoke external credentials. Deployment recovery follows later in this section.
| Standard example control | Implementation steps |
|---|---|
| Tested kill switch | Revoke access, cancel pipelines, and recover a running release. |
| Trace impact | Capture the affected work and preserve a time-ordered record. |
| Monitor consequential actions | Plan audit coverage, preserve credential and deployment activity, and configure audit event streaming where available. An incident snapshot alone doesn't provide continuous monitoring. |
Trace deployment and response records
Complete the audit plan and assign audit and response owners. Configure audit event streaming where available, plus credential and deployment activity, with retention and access controls.
Trace one ordinary deployment and one harmless response exercise across the records. Record missing events or identities as specific integration work. Coverage across systems can only be shown by following a real event.
Set GSS_GROUP_ID to the group's numeric ID to read recent group audit events:
glab api --hostname gitlab.com "groups/${GSS_GROUP_ID}/audit_events?per_page=20" | jq '[.[] | {created_at, event_name, author_name}]'
An incident-time API export can't substitute for monitoring that was never configured. Retain the configured streams and retention, plus one deployment and one response exercise traced across them.
Prepare the response session
Start from a trusted responder's glab session, separate from the suspected identity. Confirm the identity before setting the non-secret identifiers. Record the UTC incident window, the default branch SHA observed before intervention, and the known-good revision selected by the repository or policy owner:
glab api --hostname gitlab.com user | jq '{id, username}'
export GSS_APP_ID='<numeric-application-project-id>'
export GSS_GROUP_ID='<numeric-group-id>'
export GSS_DEFAULT_BRANCH='main'
export GSS_INCIDENT_AFTER='<incident-start-in-UTC-YYYY-MM-DDTHH:MM:SSZ>'
export GSS_INCIDENT_BEFORE='<incident-end-in-UTC-YYYY-MM-DDTHH:MM:SSZ>'
export GSS_KNOWN_GOOD_SHA='<full-reviewed-commit-sha>'
Retrieving all users' project audit events requires Maintainer or greater access. Token administration depends on the token type and namespace, so use an Owner or other identity permitted by the project access token or group access token documentation. Keep token values out of the incident record. Record the token's type, numeric ID, name, namespace, scopes, bot user and expiry instead.
Before changing state, capture the branch tip and active pipeline inventory. Save the complete responses in the approved incident-record location. The filtered output gives the operator a focused view:
glab api --hostname gitlab.com \
"projects/${GSS_APP_ID}/repository/branches/${GSS_DEFAULT_BRANCH}" \
| tee branch-before.json \
| jq '{name, protected, commit_id: .commit.id}'
glab api --hostname gitlab.com --paginate \
"projects/${GSS_APP_ID}/pipelines?per_page=100" \
| tee pipelines-before.json \
| jq --slurp '[.[][] | select(.status != "success" and .status != "failed" and
.status != "canceled" and .status != "skipped") |
{id, status, ref, sha, source, web_url}]'
glab api --hostname gitlab.com --paginate \
"projects/${GSS_APP_ID}/pipelines?source=parent_pipeline&per_page=100" \
| tee child-pipelines-before.json \
| jq --slurp '[.[][] | select(.status != "success" and .status != "failed" and
.status != "canceled" and .status != "skipped") |
{id, status, ref, sha, source, web_url}]'
Treat created, waiting_for_resource, preparing, waiting_for_callback, pending, running, canceling, manual, and scheduled as work that still needs a decision. The ordinary project listing omits child pipelines, so the second query explicitly selects same-project child pipelines with source=parent_pipeline. The pipelines API documents this behavior, the current statuses and pagination. Follow bridge and trigger relationships into downstream, policy, verifier, or deployment projects and repeat both queries there, because one project query can't establish that the whole workflow has stopped.
Revoke the affected access
Set the numeric token ID so the request targets exactly the token under investigation, then choose the endpoint matching its type:
Revoke a project or group access token
export GSS_TOKEN_ID='<numeric-token-id>'
# Project access token:
glab api --hostname gitlab.com --include --method DELETE \
"projects/${GSS_APP_ID}/access_tokens/${GSS_TOKEN_ID}"
# Group access token:
glab api --hostname gitlab.com --include --method DELETE \
"groups/${GSS_GROUP_ID}/access_tokens/${GSS_TOKEN_ID}"
Confirm token revocation
A successful revoke returns HTTP 204. Send only the request for the token's actual type. The project and group APIs describe these operations. Then read the exact token back from the corresponding collection and require the matching ID to report revoked: true and active: false:
glab api --hostname gitlab.com --paginate \
"projects/${GSS_APP_ID}/access_tokens?state=inactive&per_page=100" \
| jq --slurp --argjson id "$GSS_TOKEN_ID" \
'([.[][] | select(.id == $id) |
{id, name, user_id, scopes, expires_at, active, revoked}]) as $match |
if ($match | length) == 1 then $match
else error("expected exactly one inactive token with the selected ID") end'
For a group token, substitute groups/${GSS_GROUP_ID} for projects/${GSS_APP_ID} and retain state=inactive. The slurped filter searches every returned page. Require exactly one matching result with the expected ID. An empty result or 404 is unresolved until the responder checks the inactive filter, pagination, correct namespace, token type, permissions, retention behavior, and the original revocation response. A timeout during revocation has an unknown outcome: read the token state before retrying.
The metadata readback proves that GitLab considers that token inactive. It doesn't prove that a process has discarded a cached secret, that an already-running job has stopped, or that an external service has ended an established session. Revoke or rotate other credential types through their own owner-approved procedures, then verify each affected system separately.
For a group service account, use the owning group's token endpoint. Its personal access token is distinct from a group access token. In the trusted Owner session, set the service account's owning group, user ID and exact token ID, then revoke and read it back:
export GSS_SERVICE_ACCOUNT_GROUP_ID='<owning-group-id>'
export GSS_SERVICE_ACCOUNT_ID='<service-account-user-id>'
export GSS_TOKEN_ID='<affected-personal-access-token-id>'
GSS_SERVICE_TOKEN_BASE="groups/${GSS_SERVICE_ACCOUNT_GROUP_ID}/service_accounts/${GSS_SERVICE_ACCOUNT_ID}/personal_access_tokens"
glab api --hostname gitlab.com --include --method DELETE \
"${GSS_SERVICE_TOKEN_BASE}/${GSS_TOKEN_ID}"
glab api --hostname gitlab.com --paginate \
"${GSS_SERVICE_TOKEN_BASE}?state=inactive&per_page=100" \
| jq --slurp --exit-status --argjson id "$GSS_TOKEN_ID" \
--argjson user "$GSS_SERVICE_ACCOUNT_ID" \
'[.[][] | select(.id == $id)] |
if length == 1 and .[0].user_id == $user and
.[0].revoked == true and .[0].active == false
then .[0] | {id, user_id, name, scopes, expires_at, active, revoked}
else error("selected service-account token is not confirmed closed") end'
Require DELETE 204 and the matching inactive record. If the record is absent, keep the result unresolved until its namespace, permissions and retention are reconciled. GitLab documents both operations in the service accounts API. Close the affected credential before issuing a fresh recovery credential, and retain both numeric IDs to show that recovery didn't reuse the contained token.
Cancel identified pipelines and confirm their state
For each pipeline the response owner decides must stop, record its project, ID, ref, SHA, trigger identity and downstream relationships, then cancel it:
Cancel a pipeline and read back its state
export GSS_PIPELINE_ID='<numeric-pipeline-id>'
glab api --hostname gitlab.com --include --method POST \
"projects/${GSS_APP_ID}/pipelines/${GSS_PIPELINE_ID}/cancel"
glab api --hostname gitlab.com \
"projects/${GSS_APP_ID}/pipelines/${GSS_PIPELINE_ID}" \
| jq '{id, status, ref, sha, source, web_url, finished_at}'
Wait for the selected pipeline's readback to reach canceled. HTTP 200 from the cancel endpoint alone is insufficient because GitLab documents that the endpoint returns 200 regardless of the pipeline's state. This result applies only to that pipeline. Inventory same-project child pipelines explicitly, follow parent and downstream links, and repeat the decision and state check for each affected pipeline. If the endpoint returns 429, retain the Retry-After value and follow the documented rate limit. If a job has already delivered a package, deployment, message, or external change, canceling its pipeline doesn't undo that result.
Preserve a time-ordered record
Retrieve project audit events for the incident window. GitLab requires Maintainer or greater access to retrieve all users' events; a Developer sees only events based on their own actions. The API window is UTC:
Pull project audit events for a window
glab api --hostname gitlab.com --paginate \
"projects/${GSS_APP_ID}/audit_events?created_after=${GSS_INCIDENT_AFTER}&created_before=${GSS_INCIDENT_BEFORE}&per_page=100" \
| tee audit-events.json \
| jq --slurp '[.[][] | {id, created_at, author_id, author_name,
target_id, target_type, target_details, event_name, details}]'
The audit events API provides one part of the record rather than a complete execution history. Preserve relevant commits, merge requests, pipeline and job records, job logs and artifacts, deployments, approvals, policy revisions, and branch readbacks. Add identity-provider, cloud, agent, service, and external-system records where the workflow crossed those boundaries. Keep observed facts, inferences, missing data, and unresolved targets distinct. Preserve raw records under the organization's retention and access controls; deleting a GitLab pipeline also deletes related logs and artifacts, so deletion isn't a containment step.
Restore a reviewed repository or policy revision
First compare the selected known-good revision with the current default branch and review the returned commits and diffs:
glab api --hostname gitlab.com \
"projects/${GSS_APP_ID}/repository/compare?from=${GSS_KNOWN_GOOD_SHA}&to=${GSS_DEFAULT_BRANCH}&straight=true" \
| tee recovery-compare.json \
| jq '{compare_timeout, compare_same_ref, commits:
[.commits[] | {id, title, authored_date}], diffs:
[.diffs[] | {old_path, new_path, new_file, deleted_file, renamed_file,
collapsed, too_large, diff}]}'
The complete response remains in recovery-compare.json, while the terminal view includes each returned patch in diff. If compare_timeout is true, or a diff is collapsed or too large, the repository comparison API warns that the returned diff can be incomplete. Review repository history and narrower comparisons before approving recovery. Keep the protected default branch intact rather than resetting or force-pushing it.
Inspect the working clone before preparing recovery:
git rev-parse --show-toplevel
git remote get-url origin
git status --short
Confirm that the repository root and origin URL identify the intended project and that git status --short returns no entries. Stop if the clone is dirty or points to another repository. Then prepare a recovery branch from the current default branch. The subshell stops on the first failed command:
(
set -e
git fetch origin "$GSS_DEFAULT_BRANCH"
git fetch origin "$GSS_KNOWN_GOOD_SHA"
git cat-file -e "${GSS_KNOWN_GOOD_SHA}^{commit}"
git switch --create incident-recovery "origin/${GSS_DEFAULT_BRANCH}"
git restore --source "$GSS_KNOWN_GOOD_SHA" -- \
'<reviewed-application-or-policy-file>'
git diff --check
git diff -- '<reviewed-application-or-policy-file>'
git add '<reviewed-application-or-policy-file>'
git diff --cached --check
git diff --cached
)
The cat-file check needs to succeed before the repository or policy owner reviews the staged diff. If the server doesn't allow fetching an arbitrary SHA, fetch a branch or tag that contains the approved revision and repeat the reachability check. If any preparation command fails, don't run the commit block. After the human reviewer accepts the exact staged change, commit it and push the recovery branch for a merge request through the normal protected-branch path:
git commit -m 'Restore reviewed configuration after incident'
git push --set-upstream origin incident-recovery
Rehearse these commands in a private practice project before adopting them as an incident runbook.
If the unsafe change is a self-contained commit, the owner can instead revert that commit on the recovery branch. Don't restore a whole tree merely because one file is known bad. For a security policy project, preserve its separate ownership and require its configured review; restoring file bytes without effective policy propagation isn't recovery.
After review and merge, read the default branch and the pipelines for that exact revision:
export GSS_RECOVERY_SHA="$(
glab api --hostname gitlab.com \
"projects/${GSS_APP_ID}/repository/branches/${GSS_DEFAULT_BRANCH}" \
| jq --exit-status --raw-output '.commit.id'
)"
glab api --hostname gitlab.com --paginate \
"projects/${GSS_APP_ID}/pipelines?sha=${GSS_RECOVERY_SHA}&per_page=100" \
| jq --slurp '[.[][] | {id, status, ref, sha, source, web_url}]'
Confirm that the branch SHA, effective policy or configuration, successful security result, and intended application behavior match the recovery decision. If the merge didn't start the expected pipeline, diagnose its workflow rules, CI configuration, policy scope, runner eligibility, and project settings; an absent check can't support a recovery claim. Repeat the active-pipeline and token inventories, and retain the recovery merge request, exact source and target SHAs, pipeline and report IDs, and selected result fields. A clean scan supports recovery of the tested source and configuration; it doesn't establish that a deployed environment received that artifact.
If the recovery branch already exists, stop and either reconcile its exact SHA and contents or choose a fresh incident-specific name. A push denial can be the intended protected-branch behavior. In that case, open the merge request from an authorized source branch rather than weakening protection. A conflict, stale base, failed pipeline, missing report, or policy-propagation delay leaves recovery incomplete. Preserve the failed attempt, correct the cause, and repeat the selected state and security readbacks.
Choose rollback or forward recovery based on the state you can establish. Rolling back an application won't invalidate an exposed credential, and rotating a credential won't remove a malicious source change. A controls change that admitted unsafe work may require both restoring the rule and examining everything admitted while it was weak. Verify service health, the original security condition, deployed artifact identity, external sessions, and affected access before declaring full recovery.
Practise the response with the people who would perform it during an incident, using a harmless running pipeline and a short-lived credential. Confirm that they can stop current work, identify its effects, and restore a reviewed change. Include deployment suspension, external-system containment, and communications in the service's response plan, because the repository recovery commands can't perform those actions for you.
Recover a running release
For the local deployment example, follow Cancel work, close its credential, and restore a verified release while version B is still running, before the standalone rollback and cleanup. This walks through canceling current work, revoking the affected credential, stopping the current release, and deploying retained version A with a new approval and recovery credential.
Use the protected environment's allowed deployer to stop it. Administrative access alone doesn't establish that permission. After a cancellation, check the selected pipeline and job until they reach their terminal state. If a request times out, inspect the same IDs before retrying a write.
Verify the recovered artifact's identity and service health before resuming work. Check external clients, downstream services, active sessions and affected data with their owners as well. Stopping a GitLab pipeline or local container cannot recall output another system has already accepted.