GitOps Is Great Until You Actually Need to Operate Something

I like being able to review an infrastructure change before it happens. I like knowing what changed, who approved it, and how to reproduce it. Git is useful for all of that.

I get less enthusiastic when the same workflow is expected to handle every operational action, including the ones that need to stop a failure already in progress. A good configuration history does not answer every question an operator faces during an incident.

Desired state and immediate control are different jobs

Consider a scheduled process that has started producing duplicate work. This is a hypothetical example, but the operational question is specific: how do we stop another run from starting while we work out what happened?

Disabling its schedule may be the right immediate action. That action still needs authorization, a defined scope and an audit trail. Waiting for a routine pull request can extend the damage if the normal review path is slower than the process we are trying to contain.

Now add a controller that keeps applying the configuration from Git. An operator changes the live schedule, sees it disabled and moves on. If reconciliation restores the previous configuration, the operator's intervention may not last. The record in Git is consistent while the people operating the system have a different understanding of its state.

The precise behavior depends on the controller and its settings. Argo CD documents automatic self-healing as an option, rather than the default response to every live change. Its documentation also describes additional considerations for multi-source applications and applications managed by ApplicationSets. “We use GitOps” is not enough information to predict whether an emergency edit will stick.

Find the controller that can reverse the action

Before designing an emergency button, identify what owns the value behind it. Does the schedule come from a repository, a generated application, an operator or another service? Can more than one controller write it? Which level must change for the intervention to remain in effect?

That matters because stopping reconciliation and stopping work are separate actions. Flux exposes suspension on a Kustomization to pause its reconciliation. That does not, by itself, establish that the workloads it previously applied have stopped.

The same distinction appears inside the scheduler. Suspending a Kubernetes CronJob does not affect Jobs it has already started. If an existing execution is doing the damage, disabling future launches is only part of containment. Whether to drain, cancel or reconcile that execution needs its own decision and supported control.

A screen that says “paused” should make that scope clear. Paused reconciliation, paused scheduling and a stopped execution should not share one reassuring status badge.

Design the exception before the incident

I want a documented way to suspend reconciliation or apply an approved temporary state. It should identify who can act, which resource they can change, what evidence they must capture and who can restore normal operation. Giving every operator unrestricted cluster access is not the only way to provide a timely intervention.

For the duplicate-work example, a useful procedure would identify the owning controller, prevent the relevant configuration from being restored, disable further launches and read back both states. It would then separately inspect work that was already running. The exact commands depend on the installed controllers and authorization model; a generic emergency command pasted from another system is not a runbook.

Record the incident reference, the operator, the previous state, the temporary state and the observation that confirmed it. Include a review time and a named restoration owner. Temporary exceptions need a visible place in the operating picture so they do not survive through several handoffs unnoticed.

An expiry also needs thought. Automatically restoring a dangerous schedule because an hour has passed may restart the original failure. A timer can trigger a review or escalation; automatic restoration needs conditions that make it safe. The service's behavior should determine that choice.

Restoration is another operational change

Getting the permanent configuration reviewed is not the last step. Check what the controller will apply before resuming it, what work is still active and what the scheduler considers missed. Kubernetes, for example, documents catch-up behavior when a suspended CronJob is resumed without a starting deadline. Re-enabling a schedule deserves more attention than clearing an incident checkbox.

Connect the temporary action to the lasting change, confirm the intended revision and observe the system after reconciliation resumes. Preserve the record even when the permanent result looks identical to the temporary fix. It explains why the system spent time outside its ordinary operating path.

This is related to automating a workflow without resolving its decision boundaries. A pipeline can move a change efficiently while nobody has decided who is allowed to contain a failure or accept the risk of restarting it.

I want GitOps to make the system easier to understand and recover. If a team can describe its normal deployment path in detail but cannot explain how a responsible intervention survives reconciliation, the operating model needs more work.