How Site Reliability Engineers Capture Outages & Write Post-Mortems Offline

It's 3:14 AM. PagerDuty blasts your phone awake. Production API error rate spikes to 84%.

In the heat of a major infrastructure outage, every second counts. You are tailing Kubernetes pod logs, running kubectl commands, checking Datadog graphs, and communicating in an emergency Slack war room.

The last thing you need is a bloated documentation tool that takes 6 seconds to open or locks up because of network latency.

Site Reliability Engineers (SREs) and DevOps leads rely on Benchpad as an offline-first, zero-latency incident response scratchpad.

---

Benchpad SRE Incident Response View

---

1. Real-Time Incident War Room Scratchpad

During an active outage, your scratchpad acts as your single source of truth for raw terminal outputs, timeline events, and immediate remediation steps:

🚨 INCIDENT-2026-0730: API Gateway 500 Spike

Severity: SEV-1 | Incident Commander: Dev (SRE Lead) | Status: Mitigated

---

⏱️ Outage Timeline (UTC)

• 03:14 — PagerDuty alert triggered: ApiGatewayHighErrorRate (>5% 5xx errors).
• 03:18 — Incident room created. Identified database connection pool exhaustion.
• 03:24 — Traced issue to unindexed query in POST /pads/:userId/:padId/updates.
• 03:31 — Applied emergency hotfix index in production Postgres cluster.
• 03:35 — Connection pool stabilized. Error rate returned to 0.01%.

---

📟 Raw Diagnostic Terminal Logs

bash

Check connection pool state on DB replica

psql -h pg-prod-01.internal -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

Output captured at 03:22 UTC:

count | state
-------+-------- 198 | active <-- Connection pool maxed at 200 connections 2 | idle

---

## 2. Structured Incident Post-Mortem Template

Once the outage is resolved, convert your war room notes into a clean, blameless **Incident Post-Mortem** ready for leadership review:

# 📝 Post-Mortem: DB Connection Pool Exhaustion (INC-0730)

## 📌 Executive Summary
On July 30 at 03:14 UTC, our API gateway experienced a 21-minute disruption causing 500 status codes for ~4.2% of active users. The root cause was an unindexed lookup on `PadUpdate.seq` during high-concurrency background sync log compaction.

---

## 🛠️ Action Items & Prevention Tasks

| Task | Owner | Priority | Status | Target Date |
| :--- | :--- | :---: | :---: | :---: |
| Add composite index `(userId, padId, seq)` to Prisma schema | @Dev | P0 | ✅ Completed | July 30 |
| Implement PgBouncer connection pool max cap threshold | @DevOps | P1 | 🔄 In Progress | Aug 02 |
| Add Datadog alert for DB active connection pool > 75% | @SRE | P1 | ⏳ Scheduled | Aug 04 |

---

![Benchpad Incident Post Mortem Action Items Table](/blog/incident-post-mortem-and-bug-triage-scratchpad-screenshot-2.webp)

---

## 3. Collapsible Diagnostic Evidence Sections

Keep long terminal logs, stack trace outputs, and config diffs cleanly organized using Benchpad's **Collapsible Callout Sections** (`/callout`):

> **▶ Raw Stack Trace (Click to Expand)**
> ```
> Error: PrismaClientKnownRequestError: Timed out fetching a new connection from the pool.
>   at PrismaClientInitializationError.extractRawError (client/runtime.js:142:11)
>   at async PadEventsService.publishUpdate (pad-events.service.ts:88:24)
> ```

---

## Why SREs Trust Benchpad

1. **Zero Cloud Dependency**: If AWS or your cloud provider experiences an outage affecting external docs, Benchpad runs locally on your machine without relying on external servers.
2. **Instant Code Block Copying**: Copy diagnostic shell scripts or database queries straight to your terminal with one click.
3. **Encrypted Local Storage**: Infrastructure URLs, SQL queries, and internal pod names stay private on your hardware.

---

## The On-Call SRE 3-Pad Layout

- **Pad 1: `Active Incident War Room`** → Current outage log, raw terminal outputs, live timeline.
- **Pad 2: `Post-Mortems & RFCs`** → Incident post-mortems, action item tables, blameless reviews.
- **Pad 3: `On-Call Cheat Sheet`** → Kubernetes flags, database connection strings, rollback commands.

*[Equip your on-call team with Benchpad →](/)*