A release can pass its tests and still fail in production. Two pipelines might change the same environment at once, a rebuild might produce different bytes from the version tested in staging, or the previous application might no longer understand the database after a migration. Recovery gets harder when the team has to discover those conditions during an incident.
Design the release path around a known artifact, controlled changes to the target, and a recovery action that fits the state of the service. Start with one application whose delivery path the team understands. You can improve its deployment records, sequencing, and recovery without replacing its deployment tool or introducing approvals for every change.
Start with an ordinary release and a failed one
Follow a recent change from the accepted commit through its build, registry, staging checks, production deployment, and health monitoring. Then follow a failed release or a recovery exercise. Include actions outside the pipeline: a database migration run from a terminal, a configuration change in a cloud console, or an operator restarting a deployment controller can affect the result as much as the application package.
Use those examples to choose the first improvement. If nobody can identify the running artifact, establish that connection before adding more approval stages. If an older pipeline has overwritten a newer deployment, fix sequencing. If the previous version can't be retrieved, address retention before promising a faster rollback.
Agree with the service owner what recovery needs to achieve. A read-only reporting service might tolerate an hour of reduced availability; an order-processing service may need to preserve every accepted order while service is restored. Those differences influence deployment size, observation time, data protection, and who needs to be available. Choose a recovery target that the team can rehearse, including the time to notice a problem and make a decision.
Promote the artifact you tested
Build the deployable application once from the accepted source revision, publish it, and promote that same artifact through the environments used to qualify it. Deployments should select an existing artifact instead of rebuilding the application for each environment. Rebuilding the same commit can pick up different dependencies or build tools, so passing staging tests doesn't establish what a later production rebuild will do. GitLab's container build guidance describes this risk even when an image tag contains a Git SHA.
For containers, select the image by its digest, using a reference shaped like registry.example.com/payments/api@sha256:<digest>. A tag such as release-42 is convenient for people, but the recorded digest identifies the image content. Keep the association between that digest, the application commit, the build pipeline, and the tests that qualified it. GitLab's pipeline security guidance explains digest-based image references. For a packaged application, use a versioned package and verify its checksum before deployment; configure the publishing process so an existing release isn't silently replaced.
Keep environment configuration separate where the application supports it. The same image can use a staging database in staging and a production database in production. Record the configuration revision alongside the artifact without copying secrets into the release record. If a frontend embeds an environment-specific endpoint at build time, either move that configuration to runtime or treat each resulting bundle as a distinct artifact that needs its own qualification. Calling both bundles the same release doesn't make their contents identical.
Retain the artifacts needed for the service's recovery window, including deployment packages and configuration. GitLab's latest-successful-pipeline retention doesn't preserve every earlier release: older job artifacts can expire when later pipelines run on the same ref. Set retention deliberately and check registry cleanup rules as well. During a rehearsal, retrieve an earlier artifact with the credentials available to the recovery job, after newer releases have been published. A digest establishes identity, not continued availability.
Know what went into the build
The application commit doesn't describe everything a build consumes. Packages, base images, pipeline components, and downloaded tools can change the result. Whether a dependency was selected by a developer, an update bot, or an agent, the team needs to know where it came from and which version actually entered the build.
Maintained lockfiles, digest-pinned images, and versioned build tooling make those choices reviewable. For new dependencies, a package source the organization controls or approves can provide a place to examine unfamiliar packages before builds execute their installation scripts. That doesn't make an accepted package permanently safe: new advisories and changes in the application can still require another assessment. Pipeline security guidance explains how to keep build inputs identifiable.
Build provenance records how an artifact was produced. Signing that record can help a release process check that it came from the expected builder and refers to the artifact being deployed. It doesn't prove the software behaves correctly or contains no vulnerabilities. The release decision still needs the relevant tests and security results. If the record is missing, doesn't match, or names an unexpected builder, the deployment process should stop for investigation rather than treating the signature alone as approval.
This also helps during recovery. When an unwanted package enters a build, maintainers need to identify the releases that contain it, not merely the projects that declare it. Connecting the build inputs to the published artifact and its deployments narrows that investigation.
Keep deployment history ready for an investigation
Give long-lived targets stable environment names, such as production/eu, and set environment:deployment_tier to production for production targets. Put the environment declaration on the job that performs the deployment. GitLab then has a deployment history that operators can follow back to the job and commit. For a separate job that only checks an environment, use environment:action: verify so the check doesn't create another deployment. The environment guidance covers these settings.
Connect each deployment to the artifact digest or package checksum and the configuration it used, through a release manifest or linked job output. When a separate deployment project drives the change, distinguish its configuration commit from the application's source commit. Otherwise, the commit shown in the pipeline can be mistaken for the application version now serving requests.
During diagnosis, compare that record with the version reported by the running service. A Git tag identifies source, while a deployment record describes an attempt to change an environment; neither substitutes for checking the live result. Make the deployment job wait for the target's rollout result instead of reporting success as soon as a remote tool accepts the request.
If an external tool owns deployment, report its start and final status through the Deployments API. External deployment tracking doesn't give GitLab control over that tool's authorization, approvals, sequencing, or rollback. Keep those controls in the system that actually makes the change, and link its operation details so an on-call engineer can investigate without reconstructing events across unrelated logs.
Serialize production changes and handle waiting work
Give jobs that change the same production target a shared resource_group, such as payments-production. Builds and tests can continue in parallel while those deployment jobs run one at a time. Resource groups are scoped to a project, so using the same name in several projects doesn't create a shared lock. If several repositories change one service or database, coordinate them through a common deployment project or a lock owned by the deployment system.
Keep the lock for the operation that needs protection. A job that submits an asynchronous rollout and immediately exits leaves the target changing after the job has released its lock. Similarly, locking a migration job and a deployment job separately allows another pipeline to run between them. Where the whole sequence needs exclusive access, keep it under one coordinating job or pipeline. GitLab's downstream-pipeline example holds the resource group on a trigger job with strategy: mirror until the downstream pipeline finishes. Avoid having the child wait for the same lock its parent holds.
Choose what should happen to queued releases. The default resource-group mode, unordered, doesn't guarantee order. oldest_first processes pipelines in ascending pipeline-ID order; newest_ready_first prioritizes the newest job that's ready and requires idempotent deployment jobs. The process mode is changed through the resource-group API. For independent application releases, prioritizing the newest qualified candidate can reduce a stale backlog. Where releases depend on an earlier migration or intermediate state, preserve that dependency in the deployment sequence instead of skipping ahead.
Enable Prevent outdated deployment jobs to stop superseded deployment work from replacing a later deployment. Its age comparison uses job start time, not commit time, so delayed manual jobs deserve particular attention. It isn't a guarantee that the newest source commit will always win. The deployment-safety guidance explains this behavior.
Exercise the combination with two candidates: delay one, deploy the other, and attempt to start the delayed job. Check the resulting running version and deployment history. Repeat with a recovery candidate. Sequencing ordinary releases and allowing an intentional return to an earlier artifact are separate choices, and both need to work under the project's settings.
Use human approval where it changes the decision
Match approval to the risk and the information someone can assess. A small, well-tested application change with a rehearsed recovery path may be suitable for automatic deployment. A destructive data change, an unusual traffic cutover, or a release during a dependency incident may benefit from someone checking conditions immediately before it proceeds.
Give that person a decision they can make: which artifact and target, what changed, relevant test results, current service health, and whether the proposed recovery still works. Asking someone to click through every successful pipeline without this context adds delay without much judgment.
A manual job provides a deliberate start, but when: manual alone isn't independent approval or a restriction to named release operators. If it should block pipeline progress, set allow_failure: false for that job. The default differs depending on whether when: manual appears inside or outside rules; an optional manual job can leave the pipeline successful without a deployment. See manual-job behavior.
Protected environments can restrict deployers, and deployment approvals can enforce a review before deployment.1 Approval doesn't start the deployment job; someone still has to run it. Decide who does that and how urgent work reaches an available approver. When those features aren't available, you can still record a human review before manually starting deployment. Keeping the reviewer separate from the deployer depends on your operating process and existing permissions.
Review the route to production as well as the button in GitLab. A person who can change the deployment script or use the target credentials directly may have another way to change the service. Keep deployment execution and credentials appropriately restricted; runner separation by workload trust helps when release jobs currently share access with unreviewed work.
Choose recovery for the state the release changed
Decide the likely recovery action while preparing the release. Redeploying the previous application version remains an option only while it's compatible with the data, configuration, dependencies, and messages now in use.
| What happened | Recovery option | What to establish first |
|---|---|---|
| The correct deployment failed during a transient infrastructure problem | Retry the intended deployment | Find out which changes already happened and whether repeating the operation is safe. |
| The new application is faulty and the prior version still supports current state | Redeploy the retained earlier artifact | Check artifact availability, configuration, and compatibility with data written since the release. |
| Returning to the old application would break the changed schema or message format | Roll forward with a corrective release | Build and qualify the fix, and reduce user impact while it progresses. |
| The release damaged or removed data | Repair data or restore from a suitable recovery point | Understand writes made since that point, acceptable loss, reconciliation, and who can authorize the operation. |
A retry repeats an action; it doesn't undo it. If a job lost its connection after applying a migration, retrying may encounter a partly changed system. Make deployment steps recognize completed work where possible, and stop for investigation when the state is ambiguous. Cancelling a pipeline also doesn't reverse changes already made to the target.
GitLab's rollback creates a new deployment associated with the earlier commit and reruns deployment jobs. It doesn't automatically rerun earlier build or preparation jobs. Keep the deployment script capable of retrieving the intended retained artifact, and arrange any other prerequisites. For infrastructure changes, reassess a plan against current state rather than blindly reusing a plan from before later changes. The rollback documentation explains which jobs GitLab runs.
Check Allow job retries for rollback deployments alongside outdated-job protection. Retries are allowed by default; if disabled, recovery needs a new pipeline at the earlier commit. Design that pipeline to select the previously qualified artifact, not silently rebuild it. Whichever route you choose, exercise it with the credentials, deployment tooling, and approval rules operators will actually have.
Keep earlier versions compatible during the recovery window
Separate a schema's destructive cleanup from the release that first uses its replacement. For example, when replacing full_name with given_name and family_name, add the new fields first and arrange for old and new application versions to coexist while the data is populated. Check that writes from either version remain usable by the other during the recovery window. Once the new path has been exercised and the old application is no longer a recovery candidate, remove the obsolete field in a later change.
This costs temporary compatibility code and extra testing, but it preserves the option to redeploy the earlier application. A reverse migration that recreates a dropped column won't recover its former contents. Queued messages, serialized cache entries, and API clients can create the same problem even without a database migration.
If a change can't preserve compatibility, prepare a forward fix or a data-recovery plan before the release. Rehearse restoration using representative data in an isolated environment, because restoring over live data can discard legitimate writes. Include how to reconcile external effects such as accepted payments or sent notifications. An application rollback won't reverse those effects.
Detect a bad release before widening its impact
Choose health signals that reflect what the service does. A running process doesn't tell you whether a checkout service can accept an order correctly. Combine rollout readiness with a small set of user-facing checks and operating signals: transaction success, error rate, latency, queue growth, or a relevant correctness check.
For a change that can reach a limited population first, use a canary or an incremental rollout. Decide the initial exposure, observation period, and conditions for continuing before starting. For example, send a small share of traffic to a new checkout version, exercise a synthetic purchase that won't charge a real customer, and compare failed checkouts and latency with the previous version. Low traffic may require a longer observation period; a quiet dashboard isn't convincing if the changed path hasn't been used.
Keep recovery practical for the rollout method. A blue-green deployment can make traffic switching quick, but both versions may still share a database. A canary limits exposure only if the changed background workers, migrations, and shared dependencies don't affect everyone at once. Include those effects when choosing the first release group.
Consider automated recovery when the failure signal is reliable and the recovery action is known to be safe. Start with a narrow case, such as a failed readiness check before any traffic is switched. Broader automation needs protection against repeated deployment-and-rollback cycles and a clear handoff when it can't restore health. Route an actionable alert to the person responsible for the service; a red pipeline that nobody sees won't shorten an outage.
Rehearse the failure path, then expand
Introduce the approach on the representative application before moving a wider portfolio. Establish artifact identity and retention, connect deployment history to the running version, add sequencing, and then exercise recovery. This lets the team distinguish a problem in its deployment tool from a problem in the new controls.
Practice more than the successful rollback button. Exercises could include an older release waiting behind a newer one, a partially completed deployment, an unavailable registry, expired credentials, an absent approver, and an earlier application reading data produced by the new version. Start with the failures most likely to delay recovery for this service. Destructive data exercises need an isolated copy; production exercises need an agreed impact limit and someone who can stop them.
Ask someone other than the author to carry out the recovery using the normal instructions. Time the interval from the first detectable symptom to restored user-facing behavior. Include finding the correct artifact, getting authorization, obtaining credentials, applying the change, and confirming service health. A rehearsal that succeeds only with the author's local files or administrator access has identified work to do before wider adoption.
Make responsibilities easy to find. The application team owns compatibility and release-health checks; the platform team typically owns shared deployment tooling, runner availability, and registry retention. Agree who can stop a rollout, choose recovery, authorize data restoration, and communicate service status. One person may hold several responsibilities in a small team, but the on-call engineer still needs a backup when that person isn't available.
Expand to another application with a different dependency or state model once the first team's ordinary releases and recovery exercises are understandable. Keep service-specific health checks and compatibility decisions with that service rather than hiding them inside a shared pipeline component.
Judge the result by restored service
Compare several releases and rehearsals with the starting examples. Look for fewer overlapping changes, reliable identification of the running artifact, shorter time to detect a failed release, and recovery that meets the service's needs. Include the time people spend waiting for approval or resolving a blocked deployment, since those delays affect both routine delivery and urgent fixes.
When recovery misses its target, improve the step that consumed the time. Missing artifacts call for retention work; slow decisions call for clearer ownership or better health signals; incompatible data calls for a different release sequence. Add those improvements to normal engineering work and repeat the relevant exercise after changing deployment tooling, credentials, or schema practices.
Further Reading
Notes
-
Protected environments and deployment approvals are available in Premium and Ultimate. They provide deployment-access restrictions and required approvals; a manual start by itself doesn't provide the same enforcement. ↩
- Generate build provenance to record how an artifact was produced, with GitLab's supported configurations and limits.
- Sign and verify build artifacts with Sigstore for pipeline examples, signing identities, and a link to the Self-Managed setup.
- Set deployment ordering and lock scope when overlapping pipelines can change the same target, including downstream-pipeline examples.
- Configure outdated-job and rollback-retry behavior before exercising delayed releases and intentional recovery.
- Configure deployment approvals when production changes need enforced review and a clear handoff to the deployer.
- Review artifact retention when recovery depends on earlier pipeline outputs.
- Retry or roll back a deployment to understand the available actions and the earlier jobs they don't rerun.
- Connect an external deployment tool when another system deploys the application and GitLab needs an accurate history.