Model Drafts On-Call Commands, Read-Only Validation Keeps It Safe

Title

Model Drafts On-Call Commands, Read-Only Validation Keeps It Safe

Meta Description

The first commands in a runbook carry the highest risk. This article shows how to define command blocks in YAML, validate read-only behavior with Python, and replay to ensure they still match the alert context—building a pipeline that separates model drafting from safety validation and can be integrated into CI.

Article Content

The most dangerous part of a runbook is often not the long prose but the first few commands typed during troubleshooting. A single wrong parameter can turn one response into a second incident. This article introduces a pipeline that separates "model drafting" from "safety validation": define command blocks in YAML, build a read-only validator in about 120 lines of Python, then replay to ensure the commands still match the alert context. The whole process can be integrated into CI, ready before the next shift.

Why the first commands carry the highest risk

Based on a review of public runbooks, four failure patterns recur, and none of them involve a lack of knowledge:

• Commands that look like reads but actually write: for example, curl -X POST https://api.internal/health/refresh returns JSON, reads like a probe, but quietly enqueues a job at 3 a.m.
• Leftover mitigations: someone added a restart command during an incident because it helped at the time, but never demoted it to the escalation section afterward.
• Pipes that execute: for formatting convenience, commands end with | sh or xargs ... kubectl delete pod, causing the runbook to change system state.
• Outdated parameter flags: the command was correct for an old API version, so it fails in a way that looks like a brand-new symptom.

So the solution is a three-stage gate: draft, validate, replay. Anyone can do stage one; stages two and three are what make a runbook safe to hand to a tired engineer.

The "read" vs. "write" boundary for kubectl is not always in the verb. The kubectl get pods -o wide and kubectl logs mentioned in the article are公认 read-only subcommands, but kubectl apply, kubectl patch, and kubectl rollout restart all change workload state; more subtly, kubectl exec—which does not directly modify objects itself but can run arbitrary processes inside a container—is therefore grouped with delete and scale in the deny list by the validator. Understanding this helps explain why the linter adopts a dual strategy of "allowlist + deny tokens": the allowlist determines which binaries and subcommands may appear, while the deny tokens catch write operations wrapped in pipes, variables, or subshells.

The risk of | sh and xargs comes from secondary execution by an interpreter. A command ending in kubectl get ... | sh looks on its static text like it only invokes kubectl, but what actually executes is the shell on the right side of the pipe, whose contents are entirely unconstrained by the allowlist. The same is true of xargs kubectl delete pod: xargs assembles upstream output into a new command line, causing write verbs to appear in arguments that the linter cannot easily identify by position. Therefore the validator directly rejects | sh, | bash, xargs, eval, and redirections like > / and >>, excluding "executing" pipes from the runbook's first commands altogether.

A Runbook pattern that separates drafting from execution

The pattern matters more than the model, because the pattern determines what a machine can check. Here is a recommended starting version:

service: checkout-api
owner: payments-oncall

alerts:
  - name: CheckoutApiHighLatency
    signal: p99_latency_ms > 900 for 10m
    severity: page
  - name: CheckoutApiErrorBudgetBurn
    signal: burn_rate_1h > 14.4
    severity: page

first_commands: |
  date -u
  kubectl get pods -n checkout -l app=checkout-api -o wide
  kubectl logs -n checkout deploy/checkout-api --tail=80 --since=15m
  aws cloudwatch get-metric-statistics --namespace AWS/ELB \
    --metric-name TargetResponseTime --period 60 --statistics p99

escalation:
  - after: 10m
    if: error_rate still above 2%
    to: payments-oncall-secondary
  - after: 25m
    if: no mitigation applied
    to: payments-incident-commander

freeze:
  declared_by: incident-commander
  scope: [deploys, schema_migrations, feature_flags]
  max_duration: 60m
  unfreeze_requires:
    - two consecutive green 5xx SLO windows
    - follow_up_ticket_id present
    - secondary_oncall_ack within 15m

A few deliberate choices: first_commands is a single block string, making it easy to diff and statically check as a whole; escalation is structured data described with after, if, and to fields, so the on-call person can scan it like a table; freeze is also data, which leads to the mistake most runbooks make.

Freeze rules should be state, not a sentence

"Freeze deploys during an incident" is not a rule; it is a mood. Record the freeze as an object with a declarer, scope, validity period, and explicit unfreeze conditions, so a five-line script can answer "are we frozen right now" without asking a human. Each unfreeze_requires entry must be computable—a green SLO window, a ticket ID, an acknowledgment timestamp. If a condition can't be computed, it's just a wish, and wishes are exactly what people argue about at 4:10 a.m.

The validator: fifty lines of code that say "no"

The validator is deliberately boring. It allows specific binaries and subcommands, directly rejects write verbs, rejects piping to a shell, and fails closed on anything it doesn't recognize.

#!/usr/bin/env python3
"""runbook_lint.py - prove the first_commands block is read-only."""
import re
import shlex
import sys
from pathlib import Path

import yaml

ALLOWED_SUBCOMMANDS = {
    "kubectl": {"get", "describe", "logs", "top", "events", "explain", "version", "api-resources"},
    "aws": {"describe", "list", "get"},
    "gcloud": {"describe", "list", "get-iam-policy"},
}

PLAIN_READ_ONLY = {"date", "hostname", "uptime", "dig", "ss", "df", "free"}

DENY_TOKENS = (
    " delete ", " apply ", " patch ", " scale ", " rollout restart ", " rollout undo ",
    " exec ", " cordon ", " drain ", " taint ", " edit ",
)

DENY_SHELL = ("| sh", "| bash", "|sh", "|bash", "xargs ", "eval ", "> /", ">>")


def split_block(block: str):
    out = []
    for raw in block.strip().splitlines():
        line = raw.strip().lstrip("-").strip()
        if not line or line.startswith("#"):
            continue
        for part in re.split(r"&&|\|\|", line):
            if part.strip():
                out.append(part.strip())
    return out


def violations(command: str):
    hits = []
    padded = f" {command} "
    for token in DENY_TOKENS:
        if token in padded:
            hits.append(f"write verb {token.strip()!r}")
    for token in DENY_SHELL:
        if token in command:
            hits.append(f"pipe or redirect {token!r}")
    try:
        argv = shlex.split(command)
    except ValueError as exc:
        return hits + [f"unparsable command: {exc}"]
    if not argv:
        return hits
    binary = Path(argv[0]).name
    if binary in PLAIN_READ_ONLY:
        return hits
    allowed = ALLOWED_SUBCOMMANDS.get(binary)
    if allowed is None:
        return hits + [f"unknown binary {binary!r}: allowlist it or drop the command"]
    sub = argv[1] if len(argv) > 1 else ""
    if not any(sub.startswith(prefix) for prefix in allowed):
        hits.append(f"{binary} {sub!r} is not an allowlisted read-only subcommand")
    return hits


def main(path: str) -> int:
    doc = yaml.safe_load(Path(path).read_text())
    block = doc.get("first_commands", "")
    if not isinstance(block, str) or not block.strip():
        print(f"FAIL {path}: no first_commands block")
        return 1
    failures = 0
    for command in split_block(block):
        for hit in violations(command):
            failures += 1
            print(f"FAIL {command!r}: {hit}")
    print(f"{'PASS' if failures == 0 else 'FAIL'} {path}: {failures} violation(s)")
    return 0 if failures == 0 else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "runbook.yaml"))

Run it locally, then against all runbooks in CI:

python -m pip install pyyaml
python runbook_lint.py runbooks/checkout-api.yaml
for f in runbooks/*.yaml; do python runbook_lint.py "$f" || exit 1; done

The exit code is the contract. A failing check blocks the merge, meaning nobody can quietly slip a mutating operation onto the page path in a rush.

Why use shlex.split instead of simple string matching. Command blocks may contain quoted arguments, escaped spaces, or inline environment variables; splitting on spaces would mis-split arguments like --since=15m and would misjudge delete inside quotes. shlex.split parses according to shell lexical rules, giving the validator an argv close to what would actually execute; when parsing fails it returns unparsable command and fails closed, preventing malformed commands from slipping past the check. After obtaining argv, the validator first looks at the executable name (Path(argv[0]).name, which handles absolute paths like /usr/local/bin/kubectl), then checks whether the subcommand prefix falls within ALLOWED_SUBCOMMANDS.

"Fail-closed" is the safety premise of this design. For binaries not in the allowlist, the validator does not let them through by default but directly reports unknown binary and counts it as a violation. This means adding a new tool requires explicitly adding it to the allowlist, at the cost of a manual review, with the benefit that unknown commands cannot quietly pass just because "no deny token matched." The exit code plays the role of a contract here: main returns 1 when violations exist, and CI uses that to block merges, making the static check an unavoidable gate on the page path rather than a mere advisory hint.

Replay the command block with recorded payloads

Commands that pass the static check may still be incorrect for today's cluster, so add a replay stage with fixtures. Capture sanitized alert payloads and the output of each read-only command, then assert that the runbook still produces the observations it claims.

1. Store fixtures/CheckoutApiHighLatency/alert.json, pods.txt, and logs.txt next to the runbook.
2. Run each command against a dry-run kubeconfig or a recorded namespace, never against production.
3. Assert that the fields the runbook reasons about—pods_ready, error_rate, p99_latency_ms—actually exist in the parsed output.
4. Designate an owner for fixture refresh, because stale fixtures are the same defect as a stale runbook.

Does this sound like a lot of ceremony for five commands? It's about the effort of one post-incident action item, and it's reusable across every page you own.

The core of fixture replay is "assertable fields." Replay does not re-execute real queries; instead it stores sanitized alert payloads (alert.json) and command outputs (pods.txt, logs.txt) as fixed samples, then asserts that the observations the runbook depends on—pods_ready, error_rate, p99_latency_ms—can indeed be parsed from those samples. The value is in catching drift where "the command syntax is still valid, but it no longer produces the expected output for the current cluster or current API version"—for example, a parameter flag renamed across versions while the command still passes the linter but leaves the on-call person staring at empty results.

A dry-run kubeconfig and a recorded namespace are the safety boundary for replay. The replay stage is explicitly required never to run against production, but to use a kubeconfig pointing at recorded data or an isolated namespace. This completely separates "verifying whether the runbook is correct" from "executing commands on the real system," avoiding the risk of the validation behavior itself introducing changes. Fixtures need a designated refresh owner, because stale samples create the same false sense of safety as a stale runbook; refresh frequency should track the change cadence of alert signals or service interfaces.

Where free model access and free servers actually help

Two parts of this loop are tedious and low-judgment enough to delegate: drafting first_commands from alert payloads and service READMEs, and summarizing long output into three observed facts during an incident.

Disclosure: This article was prepared as part of promoting the MonkeyCode product.

MonkeyCode is an open-source project whose operators say they offer free model access—currently advertised as a free quota of 10 million tokens—and a free server option, enough to run this loop end-to-end on day one without a paid account. Verify the current terms before relying on any specific numbers, because free-tier numbers change, and treat every capability claim as something to re-check rather than something I can commit to on their behalf.

Specifically, the free tier suits three tasks in this workflow:

• Drafting: Send the alert name, signal expression, and the list of read-only tools the service exposes, and ask for a command block in the pattern above.
• Nightly static checks: Run the linter against runbooks/*.yaml on a free server, open an issue when commands start failing, and let drift surface on Tuesday instead of at 3 a.m.
• Incident note summaries: Paste text you have already read; keep the process environment free of write credentials and long-lived cloud tokens.

The rule I won't bend: model output never touches a human until it passes the linter. The value is a faster first draft, not authority, and your on-call person should still read every command before running it. If you want to rehearse the drafting part before connecting anything to production, free model access and the free server option are enough to close the loop.

Decision table: what to delegate

Task	Hand to a model?	Gate before merge
Draft first_commands from payload + README	Yes	runbook_lint.py passes, human reads every line
Choose escalation thresholds	No	SLO owner decides, runbook records the numbers
Summarize a 40-line output dump during an incident	Yes	Read-only text input, no credentials in the environment
Rewrite unfreeze_requires	No	Incident commander owns the criteria
Generate a follow-up ticket skeleton	Yes	Human signs the ticket ID into the runbook

Limitations and who should not use this method

A deny list is not a sandbox. The linter can catch known write verbs and unknown binaries; it cannot understand that in a custom CLI, status means "apply pending changes." Read-only also does not mean safe to paste—get secrets output is read-only but still a credential leak, so exclude such commands from the block altogether. If your team doesn't have a review habit, this automation mostly produces confident-looking green checkmarks.

Skip this method if your runbooks don't have machine-checkable command blocks, if your environment is air-gapped so the hosted server option is unavailable, or if you need formal security guarantees rather than guardrails. The technique is a speed bump for humans who are awake at the wrong time.

The difference between a deny list and a sandbox is whether it can constrain semantics. A sandbox limits what a process can do through kernel-level isolation, while a deny list only pattern-matches command text and cannot understand whether a custom CLI's status subcommand implies "apply pending changes." Therefore the article explicitly says this is a "guardrail, not a formal security guarantee": it can block known write verbs, unknown binaries, and executing pipes, but it cannot prove that a command is absolutely read-only in every environment. Teams that need formal security guarantees should rely on permission controls, read-only credentials, and network isolation, not on the linter alone.

Read-only and safe-to-paste are two independent dimensions. kubectl get secrets -o yaml is entirely read-only, yet it outputs credential contents; putting it in the first commands is equivalent to leaking keys into the incident channel. Similarly, logs and environment variable output may also contain tokens. Therefore, beyond read-only behavior, a content rule is needed: commands involving secrets, Secret objects, or dumps of environment variables containing credentials should be excluded from the command block altogether, rather than expecting the validator to judge whether output is sensitive. This boundary also explains why the incident summary task requires "read-only text input, with no write credentials or long-lived cloud tokens in the environment."

A twenty-minute adoption checklist

1. Move the first five commands of one page into first_commands as a single block.
2. Copy runbook_lint.py, install PyYAML, and fix anything it flags.
3. Convert freeze sentences into freeze objects with a validity period and computable unfreeze criteria.
4. Add the lint invocation to CI and make it a required check for runbook changes.
5. Capture one set of fixtures from a real, sanitized alert and add the replay step.
6. Optionally, point the drafting prompt at free model access and diff its output against the reviewed block.

What would your runbook look like if every command had to pass the linter before anyone gets paged? Probably shorter, certainly more readable, and less likely to become the next incident.

One-line Summary

Define first commands in YAML and use a read-only validator plus fixture replay to form a three-stage gate: draft, validate, replay.

Paragraph Summary

The article points out that the riskiest part of a runbook is the first commands typed during troubleshooting, with four recurring failure patterns: looks-like-a-read-but-writes, leftover mitigations, piped execution, and outdated parameter flags. It proposes a three-stage gate: define first_commands in YAML as a single diffable block, enforce read-only with a ~50-line Python validator based on an allowlist and deny tokens that fails closed, then replay sanitized alert fixtures to assert the commands still produce the expected observations. Freeze rules are modeled as an object with declarer, scope, validity period, and computable unfreeze conditions rather than a slogan. The article also provides a delegation decision table, three tasks suited to free models and servers, and emphasizes that this is a guardrail rather than a security guarantee, and that read-only-but-sensitive commands like get secrets must be excluded.