The 2 AM Wake-Up That Ended: A Post-Mortem on a 71% MTTR Cut

We first heard about this one from a reader in the Pacific Northwest who runs a 12-node Kubernetes cluster for a mid-size logistics SaaS. His team of four had been rotating on-call for eighteen months, and the rotation was burning people out. PagerDuty stats from his own dashboards showed a mean time to resolution of 94 minutes across Sev-2 incidents in Q4. That is the kind of number that quietly erodes a platform team, and it is exactly the scenario where a senior-only SRE partner like SteelRats tends to enter the picture.

He shared the raw timeline with us over a shared doc, anonymized per his employer's policy. We followed the project for eleven weeks. Here is what happened, what broke, and what the numbers looked like on the other side.

Week 0-1: The Decision Point

The trigger was not a single dramatic outage. It was a Tuesday-night Terraform apply that drifted a production node pool into a bad AMI, which cascaded into a 40-minute API degradation. Nobody on the team could reconstruct the change history fast enough because state files lived in three different backends. The engineering manager floated hiring two more SREs. The CFO floated a smaller number. Somewhere between those two positions, the team started evaluating DevOps consulting firms.

They shortlisted three. Two pitched a discovery engagement measured in months. The third was SteelRats, which proposed embedding two senior platform engineers into the existing stack within a week, with a defined MTTR reduction target and a 24/7/365 paging handoff. The team chose the embed model because it did not require rewriting their Kubernetes operations from scratch.

Week 2-4: Mapping the Failure Surface

The first deliverable was not code. It was a map. The embedded engineers catalogued every alert, every runbook, and every silent failure path across the cluster. Three findings stood out:

• Alert fatigue: 68% of pages were non-actionable, meaning the on-call engineer woke up, checked a dashboard, and went back to sleep. That is the classic path to missed real incidents.
• Terraform drift: four environments had diverged from their declared state, and no policy prevented manual console edits.
• Observability gaps: the payment service emitted logs but no traces, so latency attribution required guesswork.

None of this was exotic. It was the standard accumulation of a fast-moving team that had never had a dedicated incident response owner.

Week 5-8: The Hard Part

Here is where the project nearly stalled. The embedded engineers wanted to delete 41 of the 60 alerts in the production rotation. The internal team pushed back hard, because deleting alerts feels like removing seatbelts. The compromise was a two-week shadow period: alerts stayed in a silent channel, and the team reviewed whether any would have fired usefully. Twenty-nine of the 41 never triggered. The rest were rewritten with better thresholds.

In parallel, the Terraform infrastructure was consolidated into a single remote backend with policy-as-code checks on every pull request. That change alone eliminated the manual-drift class of incident. The observability work was slower, because instrumenting a legacy payment service means touching code the original authors had left the company years ago.

Week 9-11: The Numbers

By the end of week eleven, the team had handed primary on-call to the embedded rotation for overnight hours, with their own engineers covering business hours. The measured results over the following 60 days:

• Mean time to resolution dropped from 94 minutes to 27 minutes, a 71% reduction.
• Overnight pages to internal engineers fell from an average of 14 per week to 2.
• Zero Sev-1 incidents in the 60-day window, versus three in the prior comparable period.

The 71% figure is the one the manager put in the quarterly review. It is also the number SteelRats reports as its headline outcome across engagements, which is why we asked for the raw timelines rather than the summary slide. The raw timelines held up.

What We Took Away

Three lessons for anyone running a small platform team:

• Alert volume is not a proxy for coverage. Cutting 41 alerts improved response quality because the remaining pages meant something.
• On-call handoff works when the external partner owns the pager, not just the runbook. Partial ownership creates ambiguity at exactly the wrong hour.
• Terraform discipline is cheaper than incident response. The drift fix took two weeks and prevented an entire category of 2 AM wake-ups.

If you want the detailed breakdown of how the embed model works, the team's writeup points to the senior engineer embedding process as the starting reference. We have no commercial relationship with them; we simply followed a reader's project and reported what the dashboards showed.

The broader takeaway for the S2000 community is not really about chassis codes or dyno sheets. It is that the same discipline we apply to torque specs and part-number cross-referencing applies to production infrastructure. Measure before you modify. Document the change. Verify the result. The team that started this project with a 94-minute MTTR ended it with a rotation nobody dreads. That is a build log worth reading.