Use this procedure to rehearse how a team stops in-flight GitLab work, closes the credential behind a deployed release, and restores a previously approved version. Start only after completing the connected release through a healthy B deployment. Keep the A and B acquisition records and the healthy B runtime because they establish the starting state. The rehearsal connects one running GitLab job to the deployed service, cancels that exact work, closes the release credential, removes B, and deploys retained A with a fresh signing decision.
Use a group service account as the release operator, as described in the release setup, and retain its owning group ID, user ID and token ID for revocation. The release project must be private. Its local-release-lab environment must allow deployment only by that service account, and the application identity must not be an allowed deployer. A trusted human responder controls the local Docker host, revokes the affected service-account token, and uses a newly issued short-lived token for the same service account to stop and recover the release. This credential change doesn't change the deployer identity or add another environment grant. An Owner can administer the project, but protection of this environment doesn't imply that the Owner can stop it: the driver uses the configured service-account deployer for stop and deploy operations. GitLab documents the deployment-only access model under protected environments.
The result applies to one GitLab pipeline and one local container with no published ports. It doesn't end external client sessions, recall output already sent elsewhere, assess impact, switch production traffic, or complete incident communications. A connected service needs those systems in its response inventory as well.
Prepare exact identities
Use the absolute guide and release checkout paths established during connected release. Keep tokens out of command arguments and shell history. The paths below reuse release_driver.py, release_tools.py, the retained A and B acquisitions, and the promote function defined in Create the release identity and sign A. Continue in the same Bash session used for the release. If you opened a new session, set and check the non-secret values below, then copy only the promote() function definition from that subsection into it; don't rerun its key-generation or initial-promotion commands. The commands below supply the recovery key explicitly.
set -euo pipefail
python3 -c 'import sys; sys.exit(0 if sys.version_info >= (3, 12) else "Python 3.12 or later is required")'
: "${GSS_GUIDE_CHECKOUT:?Follow examples/README.md#get-the-example-files first}"
export GSS_RELEASE_CHECKOUT='<absolute-path-to-private-release-project-clone>'
test -f "$GSS_GUIDE_CHECKOUT/chapters/01-prepare.md"
test -d "$GSS_GUIDE_CHECKOUT/examples"
export GSS_DRIVER="$GSS_GUIDE_CHECKOUT/examples/connected-release/release_driver.py"
export GSS_TOOLS="$GSS_GUIDE_CHECKOUT/examples/connected-release/release_tools.py"
export GSS_RESPONSE_CI="$GSS_GUIDE_CHECKOUT/examples/connected-response/.gitlab-ci.yml"
export GSS_RELEASE_RECORD_DIR='<absolute-path-to-release-records-through-b>'
[[ "$GSS_GUIDE_CHECKOUT" = /* && "$GSS_RELEASE_CHECKOUT" = /* && "$GSS_RELEASE_RECORD_DIR" = /* ]]
GSS_RELEASE_CHECKOUT="$(python3 -c 'import os,sys; print(os.path.realpath(sys.argv[1]))' "$GSS_RELEASE_CHECKOUT")"
GSS_RELEASE_RECORD_DIR="$(python3 -c 'import os,sys; print(os.path.realpath(sys.argv[1]))' "$GSS_RELEASE_RECORD_DIR")"
python3 - <<'PY'
import os
from pathlib import Path
guide = Path(os.environ['GSS_GUIDE_CHECKOUT']).resolve()
release = Path(os.environ['GSS_RELEASE_CHECKOUT']).resolve()
record = Path(os.environ['GSS_RELEASE_RECORD_DIR']).resolve()
if guide == release or guide in release.parents or release in guide.parents:
raise SystemExit('Guide files and release clone need separate directories')
if any(record == checkout or checkout in record.parents for checkout in (guide, release)):
raise SystemExit('Release records must be outside both checkouts')
if not release.is_dir() or not record.is_dir():
raise SystemExit('Retain the release clone and A/B records before response')
PY
export GSS_RESPONSE_RECORD_DIR="$(mktemp -d)"
chmod 700 "$GSS_RESPONSE_RECORD_DIR"
test -f "$GSS_RELEASE_RECORD_DIR/a/operator-inputs.json"
test -f "$GSS_RELEASE_RECORD_DIR/b/operator-inputs.json"
test -f "$GSS_RELEASE_RECORD_DIR/b/promotion.json"
test -f "$GSS_RELEASE_RECORD_DIR/b/runtime.json"
export GSS_PROJECT_ID='<private-release-project-id>'
export GSS_PROJECT_PATH='<namespace/private-release-project>'
export GSS_OWNER_ID='<trusted-owner-user-id>'
export GSS_RELEASE_OPERATOR_ID='<release-service-account-user-id>'
export GSS_ENVIRONMENT_ID='<local-release-lab-environment-id>'
export GSS_B_SHA='<full-build-b-source-sha>'
export GSS_RESPONSE_BRANCH='response-rehearsal'
export GSS_RESPONSE_MARKER='bounded response task ready'
test "$(glab api --hostname gitlab.com user | jq -er '.id')" \
= "$GSS_OWNER_ID"
GSS_RELEASE_PROJECT="$(glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID")"
test "$(jq -er '.path_with_namespace' <<<"$GSS_RELEASE_PROJECT")" \
= "$GSS_PROJECT_PATH"
export GSS_RELEASE_REMOTE_URL="$(jq -er '.ssh_url_to_repo' \
<<<"$GSS_RELEASE_PROJECT")"
for GSS_RETAINED_INPUTS in \
"$GSS_RELEASE_RECORD_DIR/a/operator-inputs.json" \
"$GSS_RELEASE_RECORD_DIR/b/operator-inputs.json"; do
test "$(jq -er '.project.id' "$GSS_RETAINED_INPUTS")" = "$GSS_PROJECT_ID"
test "$(jq -er '.project.path' "$GSS_RETAINED_INPUTS")" = "$GSS_PROJECT_PATH"
done
test "$(jq -er '.version' \
"$GSS_RELEASE_RECORD_DIR/a/operator-inputs.json")" = known-good-a
test "$(jq -er '.version' \
"$GSS_RELEASE_RECORD_DIR/b/operator-inputs.json")" = candidate-b
GSS_RETAINED_A_SHA="$(jq -er '.source_sha' \
"$GSS_RELEASE_RECORD_DIR/a/operator-inputs.json")"
GSS_RETAINED_B_SHA="$(jq -er '.source_sha' \
"$GSS_RELEASE_RECORD_DIR/b/operator-inputs.json")"
[[ "$GSS_RETAINED_A_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$GSS_RETAINED_B_SHA" =~ ^[0-9a-f]{40}$ ]]
test "$GSS_RETAINED_A_SHA" != "$GSS_RETAINED_B_SHA"
test "$GSS_RETAINED_B_SHA" = "$GSS_B_SHA"
GSS_PROTECTED_ENVIRONMENT="$(glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/protected_environments/local-release-lab")"
jq -e --argjson deployer "$GSS_RELEASE_OPERATOR_ID" \
'(.deploy_access_levels | length) == 1 and
.deploy_access_levels[0].user_id == $deployer' \
<<<"$GSS_PROTECTED_ENVIRONMENT" >/dev/null
test "$(glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/environments/$GSS_ENVIRONMENT_ID" \
| jq -er '.name')" = local-release-lab
python3 "$GSS_DRIVER" inspect-runtime \
--receipt "$GSS_RELEASE_RECORD_DIR/b/runtime.json" \
--docker-context desktop-linux \
--journal "$GSS_RESPONSE_RECORD_DIR/inspect-healthy-b.jsonl"
Compare the inspection with B's exact mounted digest, version, and healthy response. If the receipt or container is absent, stop: recovery needs an established starting state rather than a guessed one.
Start one harmless task
Select the clean, authenticated clone from connected release. Require its exact filesystem root, configured origin, GitLab project path and ID, and empty worktree before creating the temporary response branch. Copy the fixed CI file without changing its image, marker, duration, tag, or rule:
cd "$GSS_RELEASE_CHECKOUT"
test "$(git rev-parse --show-toplevel)" = "$GSS_RELEASE_CHECKOUT"
test "$(git remote get-url origin)" = "$GSS_RELEASE_REMOTE_URL"
test -z "$(git status --short)"
test "$(jq -er '.id' <<<"$GSS_RELEASE_PROJECT")" = "$GSS_PROJECT_ID"
git fetch origin main
GSS_RETAINED_B_SHA="$(jq -er '.source_sha' \
"$GSS_RELEASE_RECORD_DIR/b/operator-inputs.json")"
[[ "$GSS_RETAINED_B_SHA" =~ ^[0-9a-f]{40}$ ]]
test "$GSS_RETAINED_B_SHA" = "$GSS_B_SHA"
test "$(git rev-parse origin/main)" = "$GSS_B_SHA"
git switch --create "$GSS_RESPONSE_BRANCH" origin/main
cp "$GSS_RESPONSE_CI" .gitlab-ci.yml
git diff --check
git diff -- .gitlab-ci.yml
git add .gitlab-ci.yml
git diff --cached --check
git diff --cached
git commit -m 'Test bounded response cancellation'
export GSS_RESPONSE_SHA="$(git rev-parse HEAD)"
git push --set-upstream origin "$GSS_RESPONSE_BRANCH"
Watch the exact branch and SHA through one 90-second read-only window. Continue only when there is one pipeline, its sole response_wait job, both states are running, and there is exactly one readiness marker. An empty early trace means the same job isn't ready yet; it isn't a reason to create another job or weaken the marker check. Never use trace contents as a shell command:
set -euo pipefail
GSS_READY_DEADLINE=$((SECONDS + 90))
GSS_RESPONSE_PIPELINE_ID=""
GSS_RESPONSE_JOB_ID=""
while test "$SECONDS" -lt "$GSS_READY_DEADLINE"; do
GSS_PIPELINES="$(glab api --hostname gitlab.com --paginate \
"projects/$GSS_PROJECT_ID/pipelines?sha=$GSS_RESPONSE_SHA&ref=$GSS_RESPONSE_BRANCH&source=push&per_page=100" \
| jq -s 'add')"
test "$(jq 'length' <<<"$GSS_PIPELINES")" -le 1
if test "$(jq 'length' <<<"$GSS_PIPELINES")" -eq 1; then
GSS_OBSERVED_PIPELINE_ID="$(jq -er '.[0].id' <<<"$GSS_PIPELINES")"
if test -n "$GSS_RESPONSE_PIPELINE_ID"; then
test "$GSS_OBSERVED_PIPELINE_ID" = "$GSS_RESPONSE_PIPELINE_ID"
else
export GSS_RESPONSE_PIPELINE_ID="$GSS_OBSERVED_PIPELINE_ID"
fi
GSS_PIPELINE="$(glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/pipelines/$GSS_RESPONSE_PIPELINE_ID")"
GSS_JOBS="$(glab api --hostname gitlab.com --paginate \
"projects/$GSS_PROJECT_ID/pipelines/$GSS_RESPONSE_PIPELINE_ID/jobs?include_retried=true&per_page=100" \
| jq -s 'add')"
test "$(jq 'length' <<<"$GSS_JOBS")" -le 1
if test "$(jq -r '.status' <<<"$GSS_PIPELINE")" = running \
&& test "$(jq 'length' <<<"$GSS_JOBS")" -eq 1 \
&& test "$(jq -r '.[0].status' <<<"$GSS_JOBS")" = running; then
GSS_OBSERVED_JOB_ID="$(jq -er \
'.[0] | select(.name == "response_wait") | .id' <<<"$GSS_JOBS")"
if test -n "$GSS_RESPONSE_JOB_ID"; then
test "$GSS_OBSERVED_JOB_ID" = "$GSS_RESPONSE_JOB_ID"
else
export GSS_RESPONSE_JOB_ID="$GSS_OBSERVED_JOB_ID"
fi
glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/jobs/$GSS_RESPONSE_JOB_ID/trace" \
>"$GSS_RESPONSE_RECORD_DIR/response-trace.txt"
GSS_MARKER_COUNT="$(python3 - \
"$GSS_RESPONSE_RECORD_DIR/response-trace.txt" \
"$GSS_RESPONSE_MARKER" <<'PY'
from pathlib import Path
import sys
print(Path(sys.argv[1]).read_bytes().count(sys.argv[2].encode()))
PY
)"
test "$GSS_MARKER_COUNT" -le 1
if test "$GSS_MARKER_COUNT" -eq 1; then
test "$SECONDS" -lt "$GSS_READY_DEADLINE"
break
fi
elif jq -e '.status == "success" or .status == "failed" or
.status == "canceled" or .status == "skipped"' \
<<<"$GSS_PIPELINE" >/dev/null; then
printf 'Selected pipeline became terminal before readiness.\n' >&2
exit 1
fi
fi
sleep 2
done
test -n "$GSS_RESPONSE_PIPELINE_ID"
test -n "$GSS_RESPONSE_JOB_ID"
test "${GSS_MARKER_COUNT:-0}" -eq 1
Immediately repeat the pipeline and job reads and require both still running, the same IDs, branch, and SHA. The marker proves that this harmless task began; it doesn't prove that another service hasn't already accepted output.
GSS_PIPELINE_BEFORE="$(glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/pipelines/$GSS_RESPONSE_PIPELINE_ID")"
GSS_JOB_BEFORE="$(glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/jobs/$GSS_RESPONSE_JOB_ID")"
jq -e --argjson id "$GSS_RESPONSE_PIPELINE_ID" \
--arg sha "$GSS_RESPONSE_SHA" --arg ref "$GSS_RESPONSE_BRANCH" \
'.id == $id and .sha == $sha and .ref == $ref and .status == "running"' \
<<<"$GSS_PIPELINE_BEFORE" >/dev/null
jq -e --argjson id "$GSS_RESPONSE_JOB_ID" \
--arg sha "$GSS_RESPONSE_SHA" \
'.id == $id and .commit.id == $sha and .status == "running"' \
<<<"$GSS_JOB_BEFORE" >/dev/null
Cancel once and reconcile the result
Send one cancellation request for the recorded pipeline ID. GitLab's pipeline cancellation endpoint returns a pipeline object, but that response only acknowledges the request. The terminal readback shows whether the pipeline and job actually stopped:
glab api --hostname gitlab.com --method POST \
"projects/$GSS_PROJECT_ID/pipelines/$GSS_RESPONSE_PIPELINE_ID/cancel" \
>"$GSS_RESPONSE_RECORD_DIR/cancel-response.json"
for attempt in $(seq 1 60); do
GSS_PIPELINE="$(glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/pipelines/$GSS_RESPONSE_PIPELINE_ID")"
GSS_JOB="$(glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/jobs/$GSS_RESPONSE_JOB_ID")"
if test "$(jq -r '.status' <<<"$GSS_PIPELINE")" = canceled \
&& test "$(jq -r '.status' <<<"$GSS_JOB")" = canceled; then
break
fi
test "$attempt" -lt 60
sleep 2
done
glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/pipelines/$GSS_RESPONSE_PIPELINE_ID/bridges" \
| jq -e 'length == 0'
If the read times out while either state is canceling, preserve the request and last read. Read the same IDs again; don't send a second cancellation for the same request. A canceled pipeline doesn't undo B or end a session outside GitLab.
Contain B and suspend the environment
Revoke the B credential and confirm its inactive readback using the service-account token procedure. Close it before issuing a recovery credential. Then remove only the runtime named in B's receipt:
python3 "$GSS_DRIVER" inspect-runtime \
--receipt "$GSS_RELEASE_RECORD_DIR/b/runtime.json" \
--docker-context desktop-linux \
--journal "$GSS_RESPONSE_RECORD_DIR/inspect-b-before-removal.jsonl"
python3 "$GSS_DRIVER" remove-runtime \
--receipt "$GSS_RELEASE_RECORD_DIR/b/runtime.json" \
--docker-context desktop-linux \
--journal "$GSS_RESPONSE_RECORD_DIR/remove-b.jsonl"
After the Owner has revoked the affected token and confirmed its inactive readback, use the service accounts API to create a fresh, short-lived token for the same release service account. Record the replacement token ID and expiry alongside GSS_RELEASE_OPERATOR_ID. Expose its value only as GITLAB_OPERATOR_TOKEN to the driver. Confirm /user returns GSS_RELEASE_OPERATOR_ID, then stop the exact environment:
test "$(GITLAB_TOKEN="$GITLAB_OPERATOR_TOKEN" \
glab api --hostname gitlab.com user | jq -er '.id')" \
= "$GSS_RELEASE_OPERATOR_ID"
for GSS_RETAINED_INPUTS in \
"$GSS_RELEASE_RECORD_DIR/a/operator-inputs.json" \
"$GSS_RELEASE_RECORD_DIR/b/operator-inputs.json"; do
test "$(jq -er '.project.id' "$GSS_RETAINED_INPUTS")" = "$GSS_PROJECT_ID"
test "$(jq -er '.project.path' "$GSS_RETAINED_INPUTS")" = "$GSS_PROJECT_PATH"
done
python3 "$GSS_DRIVER" stop-environment \
--project-id "$GSS_PROJECT_ID" \
--environment-id "$GSS_ENVIRONMENT_ID" \
--operator-id "$GSS_RELEASE_OPERATOR_ID" \
--environment local-release-lab \
--operator-token-env GITLAB_OPERATOR_TOKEN \
--journal "$GSS_RESPONSE_RECORD_DIR/stop-after-b.jsonl"
The driver repeats the /user check with the exact token it will use, then binds the environment ID and name and reads it back as stopped. A mismatched ID, authentication or transport failure, or 403 stops the operation. Keep the protected-environment rule unchanged and use the same configured service-account deployer with its replacement token.
Recover A with a fresh decision
Generate a new recovery key, communicate its fingerprint through the release approval channel, and sign a new recovery-a promotion over the retained A inputs. Don't reuse B's key or A's earlier approval:
mkdir -p "$GSS_RESPONSE_RECORD_DIR/recovery-keys"
chmod 700 "$GSS_RESPONSE_RECORD_DIR/recovery-keys"
python3 "$GSS_TOOLS" keygen \
--private-key "$GSS_RESPONSE_RECORD_DIR/recovery-keys/private.pem" \
--public-key "$GSS_RESPONSE_RECORD_DIR/recovery-keys/public.pem" \
| tee "$GSS_RESPONSE_RECORD_DIR/recovery-keygen.json"
export GSS_RECOVERY_FINGERPRINT="$(jq -er '.fingerprint_sha256' \
"$GSS_RESPONSE_RECORD_DIR/recovery-keygen.json")"
test "$GSS_RECOVERY_FINGERPRINT" != "$(jq -er \
'.signer_fingerprint_sha256' \
"$GSS_RELEASE_RECORD_DIR/b/promotion.json")"
promote "$GSS_RELEASE_RECORD_DIR/a/operator-inputs.json" recovery-a \
"$GSS_RESPONSE_RECORD_DIR/recovery-a.json" "$GSS_RESPONSE_RECORD_DIR/recovery-a.sig" \
"$GSS_RESPONSE_RECORD_DIR/recovery-keys/private.pem" \
"$GSS_RESPONSE_RECORD_DIR/recovery-keys/public.pem"
python3 "$GSS_DRIVER" deploy \
--operator-id "$GSS_RELEASE_OPERATOR_ID" \
--inputs "$GSS_RELEASE_RECORD_DIR/a/operator-inputs.json" \
--attestation "$GSS_RESPONSE_RECORD_DIR/recovery-a.json" \
--signature "$GSS_RESPONSE_RECORD_DIR/recovery-a.sig" \
--public-key "$GSS_RESPONSE_RECORD_DIR/recovery-keys/public.pem" \
--trusted-fingerprint "$GSS_RECOVERY_FINGERPRINT" \
--release-id recovery-a --environment local-release-lab \
--receipt "$GSS_RESPONSE_RECORD_DIR/recovery-runtime.json" \
--operator-token-env GITLAB_OPERATOR_TOKEN \
--docker-context desktop-linux \
--journal "$GSS_RESPONSE_RECORD_DIR/deploy-recovery-a.jsonl"
python3 "$GSS_DRIVER" inspect-runtime \
--receipt "$GSS_RESPONSE_RECORD_DIR/recovery-runtime.json" \
--docker-context desktop-linux \
--journal "$GSS_RESPONSE_RECORD_DIR/inspect-recovery-a.jsonl"
Confirm that the new successful deployment record contains A's exact source SHA. Then check the recovered runtime's mounted digest, health response, and version against A. This is a new authorization and deployment; it doesn't rewrite B's record.
Remove the temporary response resources
Remove the recovered runtime, stop the environment through the release service account, then have the trusted Owner revoke that replacement token and confirm its inactive readback. Delete the temporary response branch only after retaining the trace and terminal state.
python3 "$GSS_DRIVER" remove-runtime \
--receipt "$GSS_RESPONSE_RECORD_DIR/recovery-runtime.json" \
--docker-context desktop-linux \
--journal "$GSS_RESPONSE_RECORD_DIR/remove-recovery-a.jsonl"
python3 "$GSS_DRIVER" stop-environment \
--project-id "$GSS_PROJECT_ID" \
--environment-id "$GSS_ENVIRONMENT_ID" \
--operator-id "$GSS_RELEASE_OPERATOR_ID" \
--environment local-release-lab \
--operator-token-env GITLAB_OPERATOR_TOKEN \
--journal "$GSS_RESPONSE_RECORD_DIR/stop-after-a.jsonl"
GSS_REMOTE_RESPONSE_SHA="$(glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/repository/branches/$GSS_RESPONSE_BRANCH" \
| jq -er '.commit.id')"
test "$GSS_REMOTE_RESPONSE_SHA" = "$GSS_RESPONSE_SHA"
git push origin --delete "$GSS_RESPONSE_BRANCH"
GSS_BRANCH_ABSENCE="$(mktemp)"
if glab api --hostname gitlab.com \
"projects/$GSS_PROJECT_ID/repository/branches/$GSS_RESPONSE_BRANCH" \
>"$GSS_BRANCH_ABSENCE" 2>&1; then
printf 'Response branch still exists.\n' >&2
exit 1
fi
grep -F '404 Branch Not Found' "$GSS_BRANCH_ABSENCE" >/dev/null
rm -f "$GSS_BRANCH_ABSENCE"
Finish by confirming that the branch is absent; the selected pipeline and job remain canceled; the bridges list remains empty; B and recovered A deployments remain successful historical records; the environment is stopped; both service-account tokens are inactive and revoked; the protected deployer list is restored; and neither runtime receipt names a live container. Destroy both private signing keys while retaining the public keys, fingerprints, signed decisions, API records, trace, deployment readbacks, and runtime proofs required by your retention policy.
Follow the main release procedure's Close the exercise steps for the two direct memberships and protected-environment deployer entry created during the connected release. Remove only those exact temporary grants, then read direct and effective membership plus protected-environment access back. Preserve every preexisting member and deployer. "Restored" means those recorded temporary entries are absent and the earlier entries are unchanged; it doesn't mean replacing the access list with an assumed baseline.
For a real service, closure also depends on the owners of external clients, credentials, queues, agents, and downstream deployments to confirm their own sessions and work have ended. Mark each unobserved system unresolved rather than inferring closure from this GitLab and local-runtime result.