Runbook Template (Markdown) for On-Call and Operations

# Runbook: {{Alert or operation name}}

| | |
|---|---|
| **Service** | {{Service name}} |
| **Owner** | {{Team or name}} |
| **On-call channel** | {{Channel or paging policy}} |
| **Severity if unresolved** | {{SEV1 / SEV2 / SEV3}} |
| **Last tested** | {{YYYY-MM-DD}} by {{Name}} |
| **Next review** | {{YYYY-MM-DD}} |

## When to use this

*The alert name, the symptom, the threshold. Paste the alert text so searching for it finds this page.*

- **Alert:** `{{alert_name}}`
- **Symptom:** {{What the user or the dashboard shows}}
- **Do not use if:** {{When another runbook applies — link}}

## Impact

- **Who is affected:** {{...}}
- **What breaks:** {{...}}
- **Time budget:** {{How long before this becomes a SEV1}}

## Before you start

- [ ] Access to {{system, role, VPN}}
- [ ] Tools: {{kubectl, psql, cloud console}}
- [ ] Dashboards open: {{link}}
- [ ] Change freeze or maintenance window checked: {{link}}

## Triage

*Edit the labels so the flowchart matches your service.*

```mermaid
flowchart TD
    A[Alert fires] --> B{Is the service up?}
    B -- No --> C[Go to step 1: restart and check logs]
    B -- Yes --> D{Error rate above threshold?}
    D -- Yes --> E[Go to step 3: roll back the last change]
    D -- No --> F[Check for a false alarm and tune the alert]
    C --> G{Recovered?}
    G -- Yes --> H[Verify and close]
    G -- No --> I[Escalate]
    E --> G
```

## Procedure

### Step 1. {{Check the current state}}

```bash
{{command}}
```

**Expected:** `{{expected output}}`
**If different:** {{what it means and what to do}}

### Step 2. {{Mitigate}}

```bash
{{command}}
```

**Expected:** `{{expected output}}`
**If different:** {{...}}

> Risky step: {{what can go wrong}}. Rollback: see "Rollback" below.

### Step 3. {{Next action}}

```bash
{{command}}
```

**Expected:** `{{...}}`
**If different:** {{...}}

## Verify

- [ ] {{Metric or health check is back to normal: link to dashboard}}
- [ ] {{A test request succeeds: command}}
- [ ] {{No new alerts for N minutes}}

## Rollback

1. {{How to undo step 2}}
2. {{How to undo step 3}}
3. {{Confirm the previous state}}

## Escalation

| Condition | Who | How |
|---|---|---|
| No progress after {{N}} minutes | {{Secondary on-call}} | {{Paging policy or phone}} |
| Data loss suspected | {{Incident commander}} | {{Channel}} |
| Customer-visible outage | {{Communications owner}} | {{Status page, template link}} |

## Communication

- Update {{channel}} when you start, every {{N}} minutes, and when resolved.
- Status page: {{link and who updates it}}

## Known problems

| Symptom | Cause | Fix |
|---|---|---|
| {{...}} | {{...}} | {{...}} |

## Background (read after the incident)

{{How the system works, why the steps work, links to architecture pages.}}

## After the incident

- [ ] Fix anything in this runbook that was wrong or unclear
- [ ] Write a postmortem if the incident met the criteria: {{link}}

## Revision history

| Date | Author | Change |
|---|---|---|
| {{YYYY-MM-DD}} | {{Name}} | First version |