Practice area · built
Optimizing and automating the incident & problem management workflow with seven tested AI skills.
A working severity matrix, escalation rules, status-update templates, and a root-cause process built from years running on-call programs and high-severity response — encoded so Claude can run the workflow the way a real ops team would.
See it run
One incident, seven skills, start to finish.
Every step applies the actual tested skill logic — the severity matrix, escalation rules, and templates built and iterated earlier in this project — walked through end to end on one incident. Press play to watch it move through the workflow.
"Run all" streams the whole workflow. "Step through" reveals one skill's output at a time, at your own pace.
The skills
Seven skills. One workflow. Each one hands off to the next.
Built and tested individually, in the order a real incident actually moves through them. Each one's actual source is viewable below — not a description of what it does, the real instructions.
-
01
Triage & classification
Classifies severity (Sev0–5) from a real CEN escalation matrix, and says exactly who gets paged, how fast, and whether a bridge opens.
Why this skill exists
Severity is the first decision in an incident, and it's a load-bearing one — it determines who gets paged, how fast, whether a bridge opens, and how often people hear updates. Getting it wrong in either direction has real cost: over-classify and you wake up executives for something minor; under-classify and a real crisis sits unescalated while an SLA penalty clock runs. This skill exists to make that first call consistently, using a fixed framework rather than a fresh judgment call each time.
● Full skill source is encrypted. Unlock at the foot of this page to read it. -
02
Status updates
Drafts consistent updates for four audiences at once — status page, exec email, support briefing, eng Slack — from one set of facts.
Why this skill exists
During an incident, the same facts need to reach very different audiences, and it's easy for the versions to drift apart — a Slack message says one thing, the status page says another, and support ends up telling a customer something leadership never confirmed. This skill exists to produce all the needed updates from one consistent set of facts, each shaped for its audience, so nothing contradicts across channels.
● Full skill source is encrypted. Unlock at the foot of this page to read it. -
03
Diagnosis engine
Live troubleshooting during an active incident — ranks hypotheses by recent-change correlation and cost to check, and isn't afraid to recommend a rollback before the cause is fully confirmed.
Why this skill exists
RCA is slow and rigorous because it has to survive scrutiny months later. This is the opposite situation: someone is mid-incident, impact is ongoing, and every minute spent being thorough is a minute of continued customer pain. The job here isn't to be right and defensible — it's to be fast and useful. A ranked guess that leads to a good next check beats a comprehensive analysis that arrives after the incident would have self-resolved.
Two consequences of that: - **Bias toward action over certainty.** If a fast, low-risk mitigation exists (rollback, restart, failover, scale up, disable a feature flag) that plausibly helps, recommend it even before the cause is confirmed. Restoring service doesn't require knowing why it broke. - **Don't demand the evidentiary rigor RCA demands.** Working hypotheses, hunches ranked by plausibility, and "let's check the cheap thing first" are the right tools here — save the evidence ledger and alternative-hypothesis rigor for the RCA that happens after this is over.
● Full skill source is encrypted. Unlock at the foot of this page to read it. -
04
Resolution & RFO
Confirms a fix actually held before calling it resolved, then produces the Reason For Outage handoff — an initial assessment, not a finished investigation.
Why this skill exists
"It looks fixed" and "it's actually resolved" are different claims, and collapsing them is how incidents reopen — a metric dips back to normal for five minutes and everyone declares victory, then it regresses an hour later with no one watching anymore. This skill separates **verifying** the fix actually held from **documenting** it, and produces the RFO that hands a real root-cause investigation off to the postmortem rather than leaving it to fade once the pressure's off.
● Full skill source is encrypted. Unlock at the foot of this page to read it. -
05
Problem record & priority
Logs the underlying problem and sets investigation priority — inheriting real severity when an incident caused it, estimating carefully when it didn't.
Why this skill exists
Not every open question from an incident's RFO deserves the same urgency. A rare bug with minor impact and a recurring failure mode that could eventually make the news are both "unresolved," but they don't belong in the same queue position. This skill logs the problem clearly and gives it a priority that reflects both how bad it would be if it recurs and how likely that actually is — so investigation effort goes where it matters most, rather than to whatever was logged most recently.
● Full skill source is encrypted. Unlock at the foot of this page to read it. -
06
Root cause analysis
Five Whys and Ishikawa analysis converging on a stated root cause, plus typed corrective and preventative actions — not just a guess with a rollback attached.
Why this skill exists
An RFO is a fast, initial assessment made under pressure — useful for handoff, not meant to be the final word. This skill does the slower, evidence-backed version: testing that initial assessment against actual logs, timestamps, and alternative explanations rather than assuming it holds up, and turning "we think it was the migration" into either a validated conclusion or a revised one. The output needs to survive scrutiny from people who weren't in the incident and hold up months later when someone asks "why did we conclude that."
If this incident already has an RFO or a Problem Record, use them as the starting point — carry over the incident summary, timeline basics, and initial assessment rather than re-asking for what's already been established. Treat the RFO's assessment as a hypothesis to confirm or revise, not a settled fact.
● Full skill source is encrypted. Unlock at the foot of this page to read it. -
07
Known error record
Turns a confirmed cause and workaround into something searchable next time — so the same failure isn't reinvestigated from scratch.
Why this skill exists
An RCA answers "why did this happen." A Known Error Record answers a different, more urgent question mid-incident: "have we seen this before, and what do we do about it right now?" Without one, the same failure mode can hit again and get treated as brand new — reinvestigated from scratch — even though someone already found the cause and a working stopgap. This skill turns a completed (or in-progress) RCA into something searchable and immediately actionable, and keeps it alive until the permanent fix actually ships.
If a Root Cause and CAPA already exist from incident-root-cause-analysis, pull from them directly rather than re-deriving anything — this record's job is to make that knowledge findable and usable, not to redo the investigation.
● Full skill source is encrypted. Unlock at the foot of this page to read it.
Beyond this demo
Where these skills sit in a real incident stack.
Signals become alerts, alerts get correlated into one incident, someone gets paged, and the work gets recorded in a system of record. The skills slot into that existing flow rather than replacing it. Conceptual architecture — not a live integration on this page.
Metrics, logs and synthetic checks fire raw alerts. High volume, high noise — one bad deploy can throw hundreds.
Automated{ alertname, instance, labels{service,region},
value, firing_since, source } × ~200
Hundreds of alerts collapse into one incident with a probable blast radius. This is the layer that makes the signal usable — by a human or a model.
Automated{ incident_id, member_alerts[200], services[],
blast_radius, first_seen, confidence }
Severity set against the CEN matrix; engagement and bridge decisions follow from it. Paging fires automatically at Sev0–2.
AI proposes · human confirms{ sev: 1, scope: all_customers, impact: 30%,
engage[AOD,TDO,EDO], tcb: open, page_policy }
Ranked hypotheses with the cheapest next check. Reading telemetry is automatic; executing a rollback is not.
Human decides on mitigation{ ranked[{cause, evidence, next_check}],
recommended_action, reversible: true }
Four audiences drafted from one set of facts. Internal posts can auto-publish; anything customer-facing shouldn't.
AI drafts · human approves before publish{ stage: investigating, audiences[4],
next_update_at, publish_state: pending_approval }
Recovery confirmed against live metrics before the incident closes; RFO written into the ticket as the initial assessment.
AI drafts · human closes the incident{ start, end, duration, repair_type: rollback,
verified_for: 15m, rfo_initial_assessment }
Priority inherited from the incident severity; postmortem flagged mandatory at Sev0–1.
Automated from incident record{ problem_id, from_incident, sev_inherited: 1,
priority: P1, postmortem: mandatory }
Five Whys and Ishikawa over collected evidence; CAPA items become tracked tickets with owners.
Human owns the causal conclusion{ timeline[], five_whys[], ishikawa{},
root_cause, capa[{type,owner,due}] }
Symptoms and workaround made searchable, so the next occurrence is recognised instead of reinvestigated.
AI drafts · human validates the workaround{ ke_id, symptom_signature, workaround_steps[],
permanent_fix_status, recurrences[] }
On what runs itself and what doesn't
The rule I'd apply: automate anything reversible and observable, gate anything that reaches a customer, spends money, or asserts a conclusion. Alert correlation and record-keeping are safe to automate — they're reversible and auditable. Publishing to a status page, executing a rollback, and declaring a root cause are not, because a confident wrong answer there costs more than the time saved. That line will sit differently at every company; the point is to draw it deliberately rather than let it default either way.
- Automated runs without a human in the path
- Gated AI produces, a human approves before it lands
- Human-led AI assists, the person decides
Failure mode
What happens when the AI is the thing that's down.
An AI-assisted incident process is most likely to be unavailable precisely when it's needed, because incidents are when infrastructure is degraded. Any serious design has to answer this before it answers anything else — and the answer is not “add a backup model.”
The governing rule
The AI layer is never load-bearing. The process artifacts exist independently of it — the severity matrix is a table, the status updates are templates, the RFO is a form, the escalation path is a documented rota. The model accelerates filling them in. It does not own them. If the runbook assumes AI, the runbook fails.
Nothing time-bound sits in the model's control path. Paging fires from PagerDuty's own escalation rules. Alert correlation runs in BigPanda. Ticket creation is a workflow. If the model were the thing deciding to page, an AI outage during a Sev0 would mean nobody gets paged — the worst possible coupling. AI proposes and drafts; deterministic systems act.
Normal
AI-assisted
Full capability. Skills read live telemetry, draft artifacts, rank hypotheses, and write to the system of record. Humans approve anything gated.
Requires: model API, MCP server, retrieval index, source tools
Degraded
Deterministic fallback
No model in the path. The severity matrix becomes a static lookup table, status updates become fill-in templates, the RFO and RCA become structured forms, and the known error database is searched with plain keyword search. Slower, no drafting, no ranked hypotheses — but every decision the process needs is still makeable.
Requires: ITSM tool and published documents only
Manual
Human-run from documented procedure
The ITSM tool itself is unavailable. Incident command runs on a bridge, severity is called from a printed or offline copy of the CEN matrix, comms go out by phone and email from a distribution list, and the timeline is kept by a scribe for later back-entry.
Requires: a phone, an offline copy, and people who have practised this
What makes this actually work
- Assume correlated failure. The MCP server, retrieval index, and knowledge base are usually your own infrastructure — the same infrastructure having the incident. Host the fallback path somewhere the outage can't reach it, and keep an offline copy of the severity matrix and escalation rota. A knowledge base that lives inside the failed region is not a fallback.
- Degradation must be automatic and obvious. Health-check the AI path and fail over without waiting for someone to notice. Signal the current tier plainly in the interface — a responder should never have to wonder whether what they're reading was drafted or is just slow to arrive.
- Slow is worse than down. A model taking forty seconds during a Sev0 is more damaging than one that's cleanly unavailable, because people wait on it. Set an aggressive timeout and fail to the deterministic tier rather than degrading gracefully into a stall.
- A local model is optional, not the answer. It can cover a provider outage where your own environment is healthy. But it's weaker at exactly the judgment work you'd want it for — and a confidently wrong severity classification is worse than none, because someone acts on it. If you run one, restrict it to drafting and summarizing, never to classification or diagnosis.
- The manual override exists at all times, not just during outages. A responder must always be able to overrule a severity call, a hypothesis, or a draft. The lever isn't a disaster switch — it's the normal state of a system where humans are accountable.
- Practise the fallback. A degraded mode discovered during a Sev1 is not a degraded mode. Run it deliberately — a scheduled game day where the AI path is switched off and the desk works a real ticket without it — the same way you'd test a DR plan.
Key process indicators
What gets measured across the incident lifecycle.
Incident metrics are mostly intervals along a single timeline, not independent numbers. Each one measures the gap between two events, and each gap has a different owner and a different fix — which is why an aggregate “MTTR” is close to useless for deciding what to improve. Break the clock apart and the bottleneck becomes obvious.
occurs alert
fires someone
owns it right people
engaged customers
told fix
begins impact
stops service
verified
Every segment has a different remedy. A long TTD is a monitoring gap. A long TTA is a paging or on-call gap. A long TTE means severity was called late or the escalation matrix isn't being followed. A long TTN is a communications process problem, not an engineering one. A long TTAct usually means diagnosis stalled. Averaging them into one number hides all five.
Detection
- Time to detect (TTD) fault occurring to alert firing
- Detection source split monitoring vs. user-reported vs. customer-reported — the single most revealing incident metric you can track
- False positive / alert noise rate alerts that needed no action
- Correlation ratio raw alerts collapsed per incident
Response
- Time to acknowledge (TTA) alert firing to a human owning it
- Time to escalate (TTE) against the CEN engagement SLA for the severity
- Engagement SLA attainment by severity — 15 min at Sev0/1, 30 min at Sev2
- Time to action (TTAct) engagement to first corrective action — where diagnosis time lives
- Bridge time TCB / MCB duration
Communication
- Time to notify (TTN) impact beginning to first customer-facing update
- Update cadence adherence did promised update times get met
- Notification accuracy updates later corrected or retracted
- Internal vs external notification gap how long customers waited after staff knew
Restoration
- Time to mitigate (TTM) impact beginning to impact stopping — the number customers actually feel
- Time to restore (TTR) through verified, sustained recovery
- Restoration goal attainment against the CEN target for the severity
- Reopen / regression rate incidents that came back after being called resolved
- Severity reclassification rate how often the initial call was wrong, in either direction
Problem management
- Incident-to-problem linkage rate share of incidents tied to a problem record — low linkage means causes aren't being pursued
- Time from incident closure to problem record the handoff either happens while it's fresh or not at all
- Postmortem completion rate for Sev0–1 where it's mandatory, this should be 100%
- Time to RCA completion by problem priority
- Problem backlog and aging by P0–P5
Prevention & outcome
- Repeat incident rate the real outcome measure — problem management exists to move this
- CAPA completion rate and overdue actions past target date
- Corrective vs preventative split all-corrective means symptoms are being fixed, not causes
- Known error coverage KEDB entries with a valid workaround
- KEDB recurrence count a known error hit repeatedly is a permanent fix overdue
- Proactive vs reactive problem ratio problems found from trend analysis rather than from an outage
Read them in pairs, never singly
As on the service desk, most of these mislead alone — and the failure mode here is more expensive, because incident metrics get reported to executives who make staffing and investment decisions from them.
- TTM × detection source — a falling TTM looks like the response got faster. If user-reported detection is rising at the same time, monitoring is degrading and you're just getting quicker at handling incidents customers found for you.
- TTD × alert noise rate — detection time improves quickly if you lower thresholds, at the cost of alert fatigue that eventually raises TTA. The two move against each other.
- TTM × reopen rate — mitigating faster while reopens rise means incidents are being called resolved before the fix held.
- TTN × notification accuracy — notifying faster is only better if the updates don't have to be corrected later. A retracted status update costs more trust than a slower accurate one.
- Repeat incident rate × CAPA completion — the honest test of whether problem management is working. RCAs produced without actions completed is documentation, not prevention.
- Severity distribution × reclassification rate — a drop in Sev1 volume looks like stability, unless severities are simply being called lower than they should be.
Full skill sources
The complete instructions are encrypted.
The rationale for each skill is open above. The full operational detail — the matrices, thresholds, output formats and judgement rules — is AES-256 encrypted and decrypted in your browser only. The ciphertext is all that exists in the page source or the repository.
Recruiters and hiring teams: get in touch and I'll share access.
Hiring for TPM or incident management roles
I'd like to talk about how this kind of work applies to your team.
ahlevin@hotmail.comlinkedin.com/in/alanlevin
Want this running for your team
Interested in these skills — or something like them — supporting how your team actually operates. Let's talk about what that would take.
ahlevin@hotmail.comlinkedin.com/in/alanlevin