Use these questions to find where delivery is hard to predict, access is broader than a job needs, or recovery depends on an untested assumption. Start with a project and a recent change you understand. Add examples with different build tools, runner needs, or release paths where they could lead to a different answer.
Compare the intended process with pipeline history, job configuration, authentication logs, and deployment records. If an answer is uncertain, investigate it. For an improvement you choose to make, record the current problem, who will take it forward, and what result would show it worked. Give exposed credentials and unintended production access early attention.
Shared pipelines that teams can maintain
- Can a team understand why its pipeline differs from the shared setup? Compare a few common variants. Different build tools or release needs can explain a variation; copied fixes and unexplained drift suggest shared maintenance would help.
- Can consumers find an owner and get help when shared pipeline code fails? Look for a support route, supported releases, and a deprecation plan that reaches the teams using it.
- Do configuration mistakes fail early and explain what needs correcting? Try an invalid input and check whether types, allowed values, defaults, and descriptions make the required correction clear.
- Does a component behave correctly after it is included in a real project? Check job names, global settings, conditions, and outputs in the merged configuration, including conflicts with other components.
- Can a team choose and test an upgrade before it affects production delivery? Check the referenced version, upgrade notes, and representative consumer tests. A moving reference needs a deliberate decision about automatic updates.
- Can consumers on each GitLab instance obtain the components and dependencies they need? Where mirroring is required, check that someone maintains the mirror and that an upgrade reaches it.
- Do sensitive pipeline changes receive appropriate review? Use the review controls available to your team, including required Code Owner approval where configured.1
If teams are maintaining copies or struggling to upgrade shared jobs, adopting and maintaining reusable pipelines offers a way to start with one common task and grow from there.
Jobs that can reach only appropriate resources
- Can untrusted code reach signing credentials or production systems? Follow a fork or unprotected-branch job through its runner, host permissions, secrets, caches, and network access. A gap in any one of these can undermine separation elsewhere.
- Would requesting a deployment runner's tags give an ordinary project access to that runner? Check runner scope, protected status, and whether it accepts untagged jobs. Tags describe job matching; they don't establish trust by themselves.
- Do merge request pipelines receive only the protected resources you intend? Check the project setting, protection of both branches, their project, and the triggering user's access. Also review what unprotected resources a fork pipeline receives when run in its parent project.
- Can one job leave data or processes that affect a later job? Check executor behavior, instance reuse, caches, and cleanup. Separate runner registrations on one host don't establish separate execution environments.
- Are Shell and privileged-container workloads contained according to the host access they receive? Look for dedicated infrastructure, restricted networks, and an appropriate disposal strategy.
- Can only authorized operators change runner configuration or use its authentication token? Check file access, token handling, rotation, and host patching alongside job permissions.
Where access is broader than intended, use runner separation by workload trust to trace the jobs and decide which pools, networks, and credentials need to change.
Capacity that supports the delivery path
- Do jobs start and complete within the time your teams need, including during peak demand? Compare queue time, provisioning time, utilization, and failures before deciding whether more runners will help.
- Are idle capacity and maximum concurrency appropriate for both cost and reliability? Check CPU, memory, disk, instance limits, and the effect of a burst of work. More concurrent jobs can increase contention instead of improving throughput.
- Is there a clear owner for runner maintenance and upgrades? Confirm how Runner compatibility, executor changes, and host cleanup are handled. If Docker Machine is still in use, identify the next step in moving to a supported autoscaling executor.
Feedback that helps developers move forward
- Does a common mistake reach its author early enough to avoid a long wait? Look at when the failure is reported, not just total pipeline duration.
- Do jobs wait only for work they need? Check scheduling dependencies and the artifacts each job actually consumes before moving or parallelizing work.
- Are required checks still present after reducing duplicate or irrelevant jobs? Compare merge requests, shared-file changes, schedules, and release paths, including a known failing change.
- Can someone understand and repeat a failed check without searching an entire log? Check the test report, error details, and reproduction command, including whether failed tests make the job fail.
- Are intermittent failures being repaired rather than repeatedly rerun until green? Look for an owner and a repair plan where a test is temporarily excluded from required checks.
Improving pipeline feedback without weakening checks helps distinguish capacity problems from slow preparation, unnecessary waits, and hard-to-understand failures. Use it to choose an improvement and compare the result.
Access that ends with the work
- Does each pipeline call use only the authority required for its target action? Follow access to registries, cloud services, secret managers, deployment targets, and downstream projects. Investigate personal tokens or static keys that no longer have a clear purpose.
- Does the pipeline work with the expected access under its actual triggering identities? Compare developer, schedule-owner, manual, retry, and downstream paths.
CI_JOB_TOKENdepends on the triggering user's permissions. - Is cross-project job token access restricted to intended consumers? Check the receiving project's allowlist and fine-grained permissions, including nested and future projects covered by group entries. Compare authentication logs with expected use.
- Would a token from an unrelated job be rejected by the external service? Check audience and the supported identity and execution-context claims in the provider's trust policy, then test a denied case as well as the allowed one.
- Do temporary credentials expire within the time the work needs? Check the job timeout behind the ID token and any separate provider session lifetime. Longer jobs can mean longer exposure.
- Can a job retrieve only the secrets it needs, with a way to revoke or rotate them? Check provider permissions as well as the pipeline request. GitLab's external secrets integrations and direct OIDC authentication have different tier requirements.2
For persistent keys or tokens that are ready to be replaced, replacing long-lived pipeline credentials covers choosing an alternative, testing its access, and removing the old credential without losing a delivery path.
Production changes you can explain
- Can you follow both an ordinary change and an urgent fix from commit to production? Include branch rules, reviews, pipeline creation, artifact publication, approvals, and external deployment tools.
- Can you connect the running artifact with its commit and deployment result? Check environment declarations,
deployment_tier, job and user records, and API records for external deployments. A tag or release record alone won't answer this. - Are deployers and approvers limited to the intended people? Test protected-environment access where you use it,3 and check that someone knows to start the deployment after approval.
- Can overlapping pipelines deploy safely? Check serialization through
resource_groupand what happens to an older job after a newer deployment. Include jobs that have waited for manual approval. - Do deployment credentials stay within the intended environment and team? Check variable scope and inherited membership, particularly where a separate deployment project holds production access.
- Does a freeze actually stop the deployment path? Test the job behavior when
CI_DEPLOY_FREEZEis present, and check how any authorized urgent exception is reviewed and recorded.
Recovery that restores service
- Can an earlier version be deployed with the artifacts it needs? GitLab rollback reruns deployment jobs, so check retained build outputs and any plans or other jobs that must be regenerated.
- Is there a usable recovery path when outdated-job protection or rollback retry settings block the usual action? Practice the alternative, including a new pipeline at the earlier commit where required.
- Would the earlier application version still work with the current data, configuration, flags, and dependencies? Identify where a migration makes a forward fix safer than rollback.
- Has a recovery exercise restored the functions users rely on within the required time? Check the application and data, record the elapsed time, and follow up on failures. A successful deployment job alone doesn't establish service recovery.
- If recovery is automatic, does the intended failure trigger it and can it still complete? Check the retained artifacts, credentials, and supporting services the recovery job depends on.
Designing deployment and recovery practices addresses both production changes and recovery. It helps you connect the artifact, authorization, deployment record, and service checks instead of treating a successful deployment job as the whole outcome.
Further Reading
Notes
-
Code Owners is available in Premium and Ultimate; required approval also needs the relevant protected-branch configuration. ↩
-
GitLab's external secrets integrations are available in Premium and Ultimate. OIDC ID tokens are available on all tiers for direct authentication to a compatible provider. ↩
-
Protected environments and deployment approvals are available in Premium and Ultimate. ↩
- Maintain and test reusable pipeline components when shared configuration or upgrades are difficult to manage.
- Review runner security and merge request pipeline access when job trust and execution resources don't align.
- Restrict CI/CD job tokens and configure OIDC trust when jobs have unnecessary or persistent access.
- Monitor runner performance when queues or failures delay delivery.
- Configure deployment safety when approval, concurrency, or production access needs attention.
- Understand GitLab deployment rollback when building or repairing the recovery path.