Your CI/CD Pipeline Is Not Slow. Your Release Process Is Waiting

A pull request is merged at 11:20. The pipeline finishes at 11:31. Production gets the change the next afternoon.

The eleven-minute pipeline is not the bottleneck.

The change is waiting for someone to copy a version number into another job. Then it waits for a message in a release channel. Then it waits for the one person who knows which values differ in production. The deploy itself takes six minutes, but the release takes more than a day.

This is a common shape of CI/CD failure: the automated part is visible, while the waiting happens between the automated parts.

The business experiences that waiting as a delayed fix, a missed launch window or another day of operating an expensive workaround. Engineers experience it as context switching and release anxiety. The pipeline can be green in both cases.

The symptom is not “slow CI”

Teams usually notice one of these:

  • a small change misses the day’s release window;
  • production deploys depend on one engineer being online;
  • staging is automatic, but production is a different procedure;
  • the same commit is rebuilt for each environment;
  • an approval means “ask around until somebody says yes”;
  • rollback exists as a document but has not been run recently;
  • several changes are bundled because releasing each one feels expensive.

It is tempting to respond by buying faster runners, caching more dependencies or splitting the test suite. Those can be useful improvements. They do not fix a release that spends most of its life outside the runner.

The first useful distinction is:

Interval What it tells you
Commit to verified artifact How efficiently the code is built and tested
Verified artifact to production start How long the release waits
Production start to healthy How safely and predictably the system deploys
Failure detected to service restored Whether recovery is an operation or an investigation

Measure these intervals separately. A single “pipeline duration” hides the queue you need to remove.

CI/CD becomes a queue one handoff at a time

Manual steps rarely arrive as a deliberate design. Each one solves a local problem.

Production needs different configuration, so somebody edits a value before deploying. A previous release caused an incident, so a senior engineer must approve every change. Two deploys collided, so the team announces releases in chat. Rollback failed once, so people now prepare a custom command before each launch.

Every step sounds reasonable on its own. Together they produce a system whose safety depends on memory, availability and coordination.

The test is simple: could another qualified engineer release the same artifact using only the repository and the deployment system?

If the answer is no, the missing part is not more CI. It is an explicit release contract.

What a release contract contains

A useful release contract answers five questions before a change reaches production.

1. What exactly are we deploying?

Build an immutable artifact once and identify it by a version or digest. Promote that same artifact through environments.

Rebuilding for production creates a new object after testing has finished. Even if the source commit is the same, dependencies, base images or build inputs may have moved. “Build once, promote many” keeps the thing that passed verification identical to the thing being released.

2. What evidence allows it to proceed?

An approval should answer a named risk, not act as a general request for confidence.

Useful gates include:

  • the required automated checks passed;
  • the database change is backward-compatible;
  • the change is inside an agreed maintenance window;
  • error rate and latency remain within a defined range during a canary;
  • a regulated change received approval from the required role.

“A senior person clicked approve” is not evidence unless the reviewer knows what condition they are confirming. Modern delivery systems can keep necessary approvals inside the deployment history instead of scattering them across chat. For example, GitHub environments support protection rules, reviewers and environment-scoped secrets.

3. How do we know the deploy is healthy?

A job returning exit code zero only proves that the deployment command completed. It does not prove that the new version is serving useful traffic.

Health needs signals from the workload:

  • readiness succeeds;
  • the expected number of instances becomes available;
  • error rate does not cross its threshold;
  • a critical user path passes;
  • queues are draining rather than accumulating;
  • the rollout completes before its deadline.

On Kubernetes, the Deployment controller exposes rollout state and can report a stalled rollout through progressDeadlineSeconds. The pipeline still has to observe that state and decide what to do; Kubernetes reports a failed progression but does not invent your recovery policy for you. The Kubernetes Deployment documentation describes rollout status, revision history and rollback behavior.

4. What happens when it is not healthy?

“We can roll back” is incomplete.

A rollback needs:

  • a known previous artifact;
  • retained deployment history;
  • compatible data changes;
  • a command or automated action that has been exercised;
  • a health check that also validates the restored version.

Database migrations are where optimistic rollback plans usually break. Prefer expand-and-contract changes: introduce a backward-compatible schema, deploy code that can work with both versions, move the data or traffic, and remove the old path later. Application rollback should not require reversing a destructive schema change during an incident.

5. Who may start or stop a production release?

Access should be broad enough that the team is not waiting for a single operator and narrow enough that production is not an untracked shell session.

The deployment system should provide identity, history and controlled credentials. Environment secrets should be released to the job only after its protection rules pass. Concurrent releases to the same environment should be prevented or deliberately queued; GitHub Actions concurrency controls are one implementation of that pattern.

Fix the queue without rebuilding the platform

Do not start with a new CI product. Start with timestamps.

Week 1: draw the real path

Choose five recent production changes. For each one, record:

  1. merge time;
  2. artifact-ready time;
  3. production start time;
  4. healthy-in-production time;
  5. every human action between those points.

Do not document the intended process. Document what actually happened.

Mark each manual action as one of:

  • decision — a person evaluates risk;
  • translation — a person copies information between systems;
  • execution — a person runs a known command;
  • recovery — a person diagnoses or reverses a failure.

Keep real decisions. Automate translation and repeatable execution first. Recovery work tells you which missing signals or rollback primitives to build next.

Week 2: make the artifact promotable

Create one artifact per commit. Store its digest. Make staging and production consume the digest as an input rather than rebuilding from the branch.

Move environment differences into versioned configuration or environment-scoped variables. A production deploy should not depend on editing a file that was already tested elsewhere.

Week 3: turn the runbook into a job

Take the production commands from the runbook and encode them in the deployment workflow.

This does not mean removing every approval. It means the approval starts a deterministic job instead of authorizing a person to improvise. The workflow should:

  1. acquire the environment deployment lock;
  2. select the verified artifact;
  3. apply the versioned configuration;
  4. wait for rollout health;
  5. record the deployed version;
  6. stop or roll back when the health contract fails.

Week 4: rehearse failure

Deploy a version that fails a safe health check in a non-production environment. Confirm that the pipeline detects it. Run the rollback. Measure restoration time.

Then test the uncomfortable cases:

  • the new instance never becomes ready;
  • the artifact cannot be pulled;
  • the deployment exceeds quota;
  • a migration is only partly complete;
  • monitoring is unavailable during the release;
  • two releases are requested at once.

The goal is not to prove that failure cannot happen. It is to make the response predictable before failure has an audience.

What to measure after the change

DORA’s delivery metrics give a useful frame: change lead time, deployment frequency, failed deployment recovery time, change fail rate and reliability. DORA’s current history of the model explains how these measures evolved and what they represent: A history of DORA’s software delivery metrics.

For the specific problem in this article, start with:

  • median time from verified artifact to production start;
  • 90th percentile time from merge to healthy production;
  • percentage of releases requiring an unrecorded manual command;
  • number of people able to run a normal release;
  • percentage of rollbacks completed through the standard workflow;
  • time from a failed health signal to restored service.

Do not set a target such as “deploy ten times per day” without a product reason. The objective is not more pipeline activity. It is the ability to move a small change when the business needs it, without assembling a temporary release team.

Some friction belongs in the system

Not every manual gate is waste.

A production database deletion, a regulated change or an irreversible customer migration may deserve an explicit human decision. The improvement is to make the decision specific, informed and recorded.

Good friction protects a named risk. Bad friction exists because the system cannot establish its own state.

If a reviewer is checking test results, compatibility and rollout health, put that evidence next to the approval. If a person is copying a tag, changing a namespace and running the same command every Thursday, that is not governance. It is an unfinished deployment job.

The outcome to aim for

A healthy CI/CD system does not make production casual. It makes production legible.

The team knows which artifact is running, why it was allowed through, whether it is healthy and how to restore the previous state. A release can still require judgment, but it no longer requires archaeology.

When a pipeline finishes in eleven minutes and the release happens the next day, optimize the day first.