After an Emergency Production Fix, Who's Responsible for Syncing the Changes Back to Git?

During production incident response, on-call engineers often make manual configuration changes to quickly restore service. The service comes back up — but a critical question surfaces: how does that temporary change get synced back to the repository, and who owns that task? This question, raised by a Skyportal team member on Reddit, cuts to the heart of a frequently overlooked yet crucial aspect of DevOps and SRE practice: the closed-loop governance of configuration drift.

Service Restoration Doesn't Mean the Incident Is Over

A typical incident scenario plays out like this: service goes down, the on-call engineer manually tweaks a configuration to get things running again. The immediate fire is out — but the deployed environment's configuration has now diverged from what's in the Git repository. Should the incident ticket be closed at this point?

The original poster's position is clear: service restoration and remaining work should be treated as separate concerns. Restoring service is damage control — the immediate goal. Getting configuration back in sync with code is the follow-up work that prevents recurrence and keeps technical debt from accumulating. If you close the incident the moment service recovers, that manual change is likely to stay in production indefinitely, never making it into version control.

This kind of "configuration drift" is a hidden operational risk. When the deployed configuration diverges from the repository, the next repository-based deployment could silently overwrite that life-saving manual change — causing the failure to reappear, often at the worst possible moment.

Configuration Drift refers to the phenomenon where an environment's actual configuration gradually diverges from its "desired state" (typically the version declared in the code repository) over time. In IaC (Infrastructure as Code) and GitOps practices, the Git repository is treated as the Single Source of Truth, and any manual change that bypasses version control creates a crack between the two. The danger of configuration drift lies in its cumulative and stealthy nature: a single manual change may seem trivial, but after several accumulate, production can be dramatically different from what's in the repo — and the team loses accurate visibility into what's actually running in production. When a disaster recovery (DR) drill or blue-green deployment triggers a full redeployment, all manual changes that never made it back to Git are silently overwritten. This is often the root cause of repeating failure cycles. At the tooling level, configuration management tools like Puppet, Chef, and Ansible, as well as GitOps controllers like Flux and ArgoCD, have built-in reconciliation mechanisms that periodically detect and correct drift — but only if the team has first written the correct configuration back to Git.

What Incident Records Should Actually Capture

The author argues that incident documentation shouldn't just say "service restored" — it should contain enough context for the next person to put together an accurate PR. Specifically, three categories of information should be recorded:

• What changed: the specific configuration values that were modified and their new values
• What the original values were: for comparison, traceability, and as a basis for rollback
• How recovery was verified: the checks used to confirm service actually came back up

These three points seem basic, yet they're the most commonly missing elements in post-incident reviews. Without the original values, whoever writes the PR later can't determine the true intent of the change. Without verification steps, there's no way to confirm the fix was genuinely effective rather than just masking symptoms temporarily. Clear incident documentation is fundamentally about converting the tacit knowledge in an on-call engineer's head into explicit, reusable team assets.

The Real Hard Problem: Ownership

The part the author admits uncertainty about is precisely the most valuable part of the discussion — after the on-call shift hands off, who actually completes the PR?

Two reasonable but competing perspectives exist:

View 1: The Engineer Who Made the Emergency Change Should Close the Loop

The on-call engineer who made the change has the most context about why the change was made, what alternatives were tried, and what the edge cases were. Having them write the PR best preserves the original intent and minimizes information loss during handoff. The downside: after their shift ends, these engineers may have already gone off-duty or rotated out. Forcing them to clean up afterwards disrupts their rest and doesn't support a sustainable on-call culture long-term.

View 2: The Service Owner Should Take It Over

The long-term owner of a service has a better grasp of the overall codebase structure, coding conventions, and long-term direction. Having them translate the temporary fix into a properly structured commit tends to produce higher-quality output. The downside: they weren't on the scene, and if the incident documentation isn't thorough enough, they can easily produce a PR that misses the original intent.

The author identifies the core issue: "A ticket can track this work, but someone still has to own it." Processes and tools can carry information — they can't substitute for clear accountability. A follow-up task with no named owner is the most likely item to fall through the cracks in a busy team, ultimately going unresolved.

---

This ownership dilemma has a direct institutional parallel in the SRE world: Google's Postmortem culture requires that every post-incident action item have one and only one named owner with a due date — not a vague assignment to a team. Statistically, action items without a named owner are essentially equivalent to items that will never be completed. Similarly, the ITIL framework explicitly separates Incident Management from Problem Management: incident management aims to restore service as quickly as possible, while problem management aims to identify root causes and implement permanent fixes. Syncing configuration drift back to Git is fundamentally a problem management concern — it should be tracked in a separate problem ticket after the incident ticket closes, bound to a named owner. This institutional separation is a key design pattern that mature engineering organizations use to avoid "accountability vacuums."

Practical Recommendations Distilled from This Discussion

Building on the thinking in the original post, a more complete handling framework might look like this:

1. Two-phase incident lifecycle: Rather than closing immediately after service restoration, transition to a "pending sync" state with explicit remaining work documented.
2. Require the three-element record: What changed, the original values, and verification method — as a prerequisite for closing the incident ticket.
3. Designate a named owner: Whether it's the original on-call engineer or the service owner, put it in writing at handoff time who is responsible for the PR. Eliminate the accountability vacuum.
4. Set a completion deadline: Give the sync work an SLA to prevent configuration drift from persisting indefinitely.

It's worth noting that there's no universal answer here. Team size, on-call structure, and service complexity all influence the optimal choice. Smaller teams may find it more efficient to default to having the person who made the change close the loop. More mature, large-scale engineering organizations are more likely to rely on clear ownership structures and automation (such as continuous reconciliation in configuration-as-code pipelines) as a safety net.

Ultimately, the question of who owns the responsibility for syncing emergency fixes back to Git isn't a test of tooling — it's a test of a team's accountability culture and process design. Who pressed the button on that temporary change matters, but what matters more is whether someone is willing — and explicitly required — to take ownership of closing the loop.