A shared response to uncertainty
Our proposed incident doctrine gives teams a repeatable sequence: declare the incident, establish command, identify impact, stabilize service, and reassess after each intervention.
Incident Readiness & Operational Resilience
In development · Pilot design
Train before production does it for you.
Inspired by emergency-room response protocols and training such as ACLS, Emberside is developing practical incident-response training for engineering teams.
A proposed five-day team residency: face repeated outages, rotate through every response role, and improve the organization between incidents.
Help shape the first pilot cohorts. Curriculum, delivery, and assessment are being developed.
- 01Respond
A working system fails. Establish command and restore service safely.
- 02Debrief
Reconstruct decisions. Find what helped and what made the response difficult.
- 03Improve
Repair the process, access, documentation, and handoffs.
- 04Repeat
Rotate roles. Face a different failure. Apply what you learned.
Each response becomes the starting point for the next.
The inspiration: emergency medicine
From the emergency room to the incident bridge.
Train the human response as deliberately as you train the technical skills.
Advanced Cardiovascular Life Support (ACLS) brings clinical protocols together with team training. The American Heart Association’s course includes realistic clinical scenarios, hands-on practice, team dynamics, debriefing, and skills testing.
That combination is the starting point for Emberside’s program: a shared way to respond, roles people have practised, and repeated opportunities to make decisions under pressure before facing a real incident.
American Heart Association: ACLS course approach →Make the response explicit
Practise named roles and closed-loop communication: assign a task, acknowledge it, carry it out, and report back. Command keeps the shared picture while responders investigate.
Practise, debrief, repeat
Use simulation to observe decisions, surface coordination failures, and give structured feedback. Rotate roles and try again, with practical assessment of demonstrated capability.
Software outages are not medical emergencies. The connection is how teams work under uncertainty, cognitive load, and time pressure. This is an independent software-incident training concept, not a medical course or an ACLS-equivalent qualification; no affiliation with or endorsement by the American Heart Association is claimed.
The coordination gap
Knowing the stack is only part of the job.
Knowing Kubernetes is not the same thing as knowing how to lead twelve people through a production failure at 2 AM.
Technical onboarding can leave incident coordination to experience, confidence, or whoever happens to be available. Access gaps, unclear ownership, stale runbooks, and conflicting changes then surface when the stakes are already high.
Slides and exams can build knowledge. Tabletops can examine plans. A GameDay can expose a technical weakness. Our focus is sustained practice that connects those lessons: repeated response, feedback, organizational repair, and another attempt.
Confidence during a Sev1 should be an outcome of preparation. Protocol, repetition, role clarity, and team design belong in that preparation.
Why this program
Practise the response. Improve the organization.
The aim is a team that can respond together, even when its usual incident hero is unavailable.
Incident simulations, command courses, and GameDays already exist. Emberside’s proposed emphasis is an intensive, instructor-led residency that brings command doctrine, human factors, technical response, organizational repair, and practical assessment into one continuous learning experience.
The same cohort works in the same fictional company across five days, preferably on-site. The incidents change; the team carries its decisions and improvements forward. Here is why that design matters.
Find the gap. Fix it. Test the fix.
An access failure or unclear escalation path becomes work for the team between incidents. The next exercise tests whether the repair helped. That connects the debrief to an observable change in how the organization operates.
Build depth across the team.
Mandatory role rotation gives everyone practice in command, communication, coordination, and technical response. Individual role records are intended to reveal gaps that a successful team recovery could otherwise hide.
Rehearse the difficult human decisions.
Surprise outages in a working lab require participants to switch context, challenge a confident diagnosis, control competing changes, and protect responders from interruption. Instructors can observe whether the agreed response actually holds up under pressure.
Leave with a clearer next step.
The proposed practical assessment and evidence package connect observed behavior to further practice and remediation: which roles need development, which procedures failed, and what leadership should address. The aim is actionable readiness evidence.
Why commit five days?
The working format makes room to apply feedback, change roles, repair the operating model, and face another incident. It is designed for teams that need sustained practice across those responsibilities. The curriculum focuses on vendor-neutral response skills that participants can take back to their own tooling.
Choose the depth your team needs.
Five days is a significant commitment of staff time. If your immediate need is an introduction to incident command, a specific technical exercise, or frequent short drills, a focused course or simulation platform may be the better fit. A pilot conversation should establish whether the residency’s depth is useful for your team.
This is the design we are validating through pilots. These elements are not claimed to be exclusive to Emberside, and comparative performance has not yet been demonstrated.
How the simulation would work
Fictional company. Working systems. Real decisions.
Participants join a deliberately imperfect organization inside an isolated training environment.
The proposed lab brings together services, monitoring, deployment tools, chat, tickets, access controls, and runbooks. Failures happen within the lab and interrupt other work, creating the context switch that an announced exercise can miss.
Early access, documentation, and ownership gaps each serve a learning objective. Participants improve the organization between incidents. The approach is vendor-neutral: the focus is how the team operates across tools.
Make decisions in motion
Declare, delegate, investigate, communicate, mitigate, and recover while the situation changes.
Give everyone a job
Responders practise their roles. Observers track a specific behavior, such as fixation, escalation, or uncoordinated changes.
Examine the system
Discuss what happened, why it made sense at the time, and what to change. No blame, humiliation, or personality judgments.
Flagship concept · In development
Production Incident Command Residency
Five days to practise the response—and improve the system around it.
The working format is approximately eight participants with two instructors, preferably on-site. Remote delivery is also being explored.
The design combines two surprise incidents per day with short skills stations, structured debriefs, and time to fix the weaknesses uncovered. Timing, cohort size, and delivery will be validated through pilots.
- Day 1
Expose the gaps
Establish command, identify impact, and discover what gets in the way.
- Day 2
Control the response
Assign work clearly, stabilize service, and coordinate changes.
- Day 3
Work with human factors
Challenge assumptions, manage attention, and protect responders.
- Day 4
Navigate cascading failures
Manage dependencies, escalate, and recover deliberately.
- Day 5
Lead the response
Rehearse with less instructor support, then attempt a practical assessment.
Monday is allowed to be ugly. The aim for Friday is a team that establishes command, controls changes, and communicates with less instructor support. That is a training objective, not a published result.
Rotate through the response
Your strongest debugger has to put the keyboard down.
The proposed curriculum rotates every participant through command, coordination, communication, and technical response.
The natural communicator also investigates. The quiet engineer gets a turn at command. A role record tracks practice and demonstrated competencies, so nobody disappears into their most comfortable job.
- Incident Commander
- Maintain the shared picture, set priorities, and coordinate the response.
- Deputy Incident Commander
- Support command, manage resources, and prepare a clear handoff.
- Technical / Operations Lead
- Coordinate investigation, mitigation, and safe technical changes.
- Communications / Liaison
- Translate uncertain technical information into useful stakeholder updates.
- Scribe / Timeline Keeper
- Capture decisions, interventions, results, and the incident timeline.
- Technical Responder / SME
- Test hypotheses, carry out assigned work, and report what changed.
Between incidents: short skills stations would rehearse a command handoff, a stakeholder update from messy information, review of simultaneous change proposals, and evidence that could disprove a favored hypothesis.
Measure observable behavior
Show what changed between attempts.
The pilot should produce more than “the course was fun.”
We intend to compare early and later exercises against defined behaviors, recording scenario difficulty and instructor assistance alongside timing. Faster is only useful if the response is also safe and effective.
Command & impact
Time to declaration, clear command, assigned roles, and identified customer impact.
Coordination & safety
Duplicated tasks, conflicting changes, closed-loop communication, and escalation.
Reasoning & recovery
Reassessment after interventions, abandoned hypotheses, safe mitigation, and stabilization.
Communication & continuity
Stakeholder-update accuracy, handoff quality, and timeline completeness.
The measurement framework is in development. No pilot results, guaranteed reduction in outages, or MTTR improvement are claimed.
Practical assessment · Proposed
Can you command the response?
Customer transactions are failing. You are the Incident Commander. Begin.
The assessment would examine how a candidate organizes the people and decisions needed to recover service safely.
The candidate need not personally discover the root cause. The proposed rubric covers command, impact, delegation, situational awareness, hypothesis management, mitigation, change control, escalation, communication, and handoff.
Scenarios are intended to respond to the candidate’s decisions. Unsafe changes, worsening impact, and failure to seek needed expertise matter alongside successful recovery.
This is a proposed practical assessment, not an established certification or accredited qualification. The rubric and assessor consistency need validation.
Regulated & high-consequence organizations
Make the exercise useful. Keep the evidence.
Designed to support organizations that need to train responders, exercise incident processes, document results, and demonstrate operational resilience.
A pilot could suit engineering leaders, SRE and platform teams, and operational resilience teams managing response across several groups.
A proposed evidence package
- Participants, attendance, roles performed, and competencies exercised.
- Scenario coverage, exercise methodology, observations, and assessment results.
- Performance trends, deviations from procedure, and organizational weaknesses.
- Remediation recommendations and a management summary.
For example, Bank of Canada guidance addresses role training, testing, and records of results and corrective actions. Exact obligations vary. Scope and evidence would be agreed with your risk team; participation does not establish regulatory compliance.
Bank of Canada: operational risk and incident response guidance (PDF) →Practitioner-led
A process on paper is only the beginning.
Brandon Gould · Founder & Principal Consultant
At a previous employer, Brandon worked in an environment where production outages recurred and formal incident management was largely absent. He volunteered to build the response process: declaration and escalation, command structure, stakeholder updates, task ownership, runbooks, documentation, and protection for critical responders.
About two months later, a production incident put that process under pressure. The experience reinforced the gap between designing a process and watching people execute it while systems are failing.
This training direction grows out of that experience and a question explored in the accompanying article.
Read: Why Doesn't DevOps Have an ACLS?The origin of the concept · Published on LinkedIn
Pilot conversations
Help shape the first cohorts.
Tell us where incident response gets difficult for your team.
We are validating the curriculum and delivery model with potential pilot customers. A conversation helps establish fit, scope, timing, and what evidence would be useful to your organization.
Start with a little context
Your company, approximate team size, preferred delivery format, and biggest coordination challenge are enough to begin. Please leave out credentials, customer information, and sensitive incident details.
The five-day residency is the flagship concept. Shorter leadership exercises, responder fundamentals, Incident Commander intensives, and recurring drills are also being explored.
Pilot inquiry contact is being configured.