Incident Commander
A production incident response role originating from Google's Site Reliability Engineering (SRE) practices. The Incident Commander coordinates resources, drives decision-making, and maintains communication cadence during large-scale incidents. A defining characteristic of the role is the explicit separation of responsibility between the person investigating the incident and the person authorized to make decisions.
Timeline (last 90 days)
The Incident Commander role originated from Google's SRE framework and is one of the core designs for large-scale production incident response
The Incident Commander is not necessarily the most technically skilled person, but rather a role specifically responsible for coordinating resources, advancing decisions, and maintaining communication cadence, explicitly separating 'who is investigating' from 'who makes the call'
The difficulty of introducing the Incident Commander role into ML incident scenarios lies in the fact that the 'recovery' standard for model quality incidents is often ambiguous, lacking a clear definition of recovery
Decouple decision authority from investigative authority: the Incident Commander holds the decision authority to run or roll back, while each team only provides diagnostic conclusions without making their own decisions
All Facts (4)
The Incident Commander role originated from Google's SRE framework and is one of the core designs for large-scale production incident response
50%UnverifiedThe Incident Commander is not necessarily the most technically skilled person, but rather a role specifically responsible for coordinating resources, advancing decisions, and maintaining communication cadence, explicitly separating 'who is investigating' from 'who makes the call'
50%UnverifiedThe difficulty of introducing the Incident Commander role into ML incident scenarios lies in the fact that the 'recovery' standard for model quality incidents is often ambiguous, lacking a clear definition of recovery
50%UnverifiedDecouple decision authority from investigative authority: the Incident Commander holds the decision authority to run or roll back, while each team only provides diagnostic conclusions without making their own decisions
50%