Your Plan Is Not a Tested Plan

Complete on Paper, Broken in the Field

Read enough post-incident reviews from industrial organizations and a pattern appears that has very little to do with the attacker. The organization had an incident response plan. It was complete in the sense an auditor would recognize: roles were assigned, contacts were listed, recovery steps were written out. And when the plan was activated for the first time, during a real event, it did not work. People could not find their own role assignments, or found them and did not recognize the role. The contact list had been accurate once, years earlier, before two reorganizations and a vendor change. The manual operating procedures the plan relied on existed as a sentence in the document and nowhere else, so the operators who were supposed to carry them out had never seen them. Each of these failures is mundane, and that is the point. None of them required a sophisticated adversary. All of them were sitting in the plan from the day it was approved, waiting for the first real activation to expose them. A plan that has never been activated is a theory about how your organization will behave under pressure. Most organizations have one. Very few have a tested one, and the distance between those two states is where real incidents turn into disasters. The worst OT cyber events on record tend to share this feature: the plan existed and failed when it was finally used.

The Three Gaps Only Testing Finds

When a plan is run for the first time, the failures it produces fall into three recognizable kinds, and it helps to name them because each one calls for a different fix. The first is a procedure gap: the plan describes a step that does not work as written. The plan says the affected network segment will be isolated within 15 minutes, and when someone actually does it, with the real switches, the real change process, and the real people who have to approve it, it takes 90. Nothing in the document was wrong in principle. The timing was simply never measured. The second is a knowledge gap: the person assigned to step seven did not know step seven was their responsibility, because nobody ever told them. Their name or their job title was typed into a table during a planning meeting they did not attend. The third is a resource gap: the plan calls for a recovery tool, a spare workstation, a license key, or a vendor engineer, and at 2 a.m. on a holiday weekend that resource is not available. What makes these three gaps dangerous is that they are invisible to review. You can read a plan ten times and never see that isolation takes 90 minutes, that the person named in step seven is unaware of it, or that the recovery laptop is locked in an office nobody can reach overnight. None of these gaps surface until the plan runs. All of them surface the first time it does. The only choice you have is whether that first time is a drill or a real incident.

Regulators Read the Same Reports

The people who write regulations read those post-incident reviews too, and the same conclusion has made its way into the rules: a documented plan is no longer enough on its own, and for some sectors testing is mandatory. NERC CIP-008-6, the incident reporting and response planning standard for the North American bulk electric system, requires covered entities to test their cyber security incident response plans at least once every 15 calendar months. The TSA security directives for pipeline and LNG operators require the cybersecurity incident response plan to be tested every year. For organizations those rules cover, testing is a compliance obligation with evidence that an auditor will ask to see, and Chapter 8 reads both requirements closely. Outside those sectors, NIST SP 800-82r3, the US guide to OT security, includes guidance on testing incident response plans, but it is voluntary guidance rather than a requirement, and many other industries follow sector-specific or voluntary frameworks of their own. It is tempting to treat these rules as the reason to test. They are better understood as a symptom. The rules exist because the same finding kept coming back from real events: complete on paper, broken in the field. If you work in a sector with no testing mandate at all, the three gaps described above are still sitting in your plan, and your adversary is not going to check which framework you follow before finding them for you.

Two Different Tools

Once you accept that the plan has to be run, the next question is how. Ask a team what they ran the last time they held a scenario session, and most will say "a drill." Many of them will be wrong, and the confusion is not just a matter of vocabulary. The two words describe different tools that find different problems, and using the wrong one brings the wrong people into the room and produces the wrong findings. A drill: a single-function practice that tests one specific procedure and nothing else. How long does it take to switch a given controller to manual operation? Can the on-call engineer reach the right contact list in under five minutes? Can last month's backup actually be restored? A drill is narrow and technical, it does not require coordination across teams, and it ends with a clear answer. The best comparison is a fire exit test: one thing, timed, and at the end you know whether it works. An exercise: a simulated event that tests decision-making across teams under pressure. It brings in multiple roles at once, typically operations, security, management, and legal, and it forces them to work through ambiguity together, with incomplete information and competing priorities. An exercise is not checking whether individual procedures are correct. It is checking whether the organization can function as a unit when the procedures run out. That is why the finding from a failed exercise is rarely "the procedure failed." Far more often it is "the team did not know who was in charge."

The Difference, Side by Side

Laid next to each other, the two formats differ on every dimension that matters when you plan a session. A drill is narrow, focused, and technical, and it is short enough that you can run one at the start of a shift. An exercise is broad and organizational, it takes a block of several hours out of the calendars of people who are hard to get in one room, and what it produces is a different class of finding altogether. Read the table below row by row and notice that the last row, the output, follows directly from the first four. A session scoped to one procedure, staffed by the technical team, and run for half an hour can only ever tell you about a procedure or a timing problem. A session scoped to a full incident, staffed across functions, and run for an afternoon is the only kind that can surface a coordination or escalation problem. Neither format can find what the other finds, which is exactly why mixing them up costs you the result you were looking for.

Picking the Right Tool

The choice between the two starts with a single question about your objective: are you verifying that a procedure works, or are you testing whether your people can coordinate under pressure? If it is the first, run a drill. Good drill candidates are the procedures your plan depends on mechanically: a backup restore, a manual switchover on a specific process unit, or activating the communication tree, which is the escalation contact list your plan relies on to get the right people moving. Each of these has a yes-or-no outcome and a time you can write down and compare against what the plan claims. If the objective is the second, run an exercise, built around a realistic OT situation such as ransomware on the historian, a loss of SCADA visibility, or an unrecognized workstation appearing on the control network. These scenarios do not have one right procedure; they have a series of decisions, handoffs, and judgment calls that involve several teams at once. A mature incident response program uses both formats, scheduled at different points in the year, and treats neither one as a substitute for the other. The drills keep the individual mechanisms honest. The exercises test whether those mechanisms add up to an organization that can respond. A site that only drills will have fast procedures and nobody sure who decides. A site that only runs exercises will have confident decision-makers relying on steps that have never been timed. How the two are balanced across a year is the subject of Chapter 11.

Choose Before You Schedule

The practical habit to take from this chapter is simple, and it comes before anything goes on a calendar: decide what you are testing, then let that decision choose the format for you. Write the objective down in one sentence. If the sentence names a procedure and a time ("restore the engineering workstation from last week's image within four hours"), you are planning a drill, and the invitation list, duration, and expected findings all follow from that. If the sentence names a decision or a handoff ("decide whether to isolate the plant from the corporate network and notify the right external parties"), you are planning an exercise, and the participants must include whoever actually holds that authority, even if their calendar makes scheduling harder. Doing this before you schedule protects you from the most common failure in exercise programs, which is not a bad scenario but a mismatch between the question and the people in the room. It also makes your findings easier to act on, because a drill finding is usually a procedure to rewrite and a time to improve, while an exercise finding is usually a role, an authority, or an escalation path to clarify. Either way, every session should end with gaps written down and owned. A plan you have never run is a plan you have never read in the way that matters. Finding the gaps in a drill costs you an hour. Finding them during a real incident costs far more.

What's Next

Before you can test a plan, it has to be the right plan, and most incident response plans used at industrial sites were written from an IT template. Chapter 2 walks through the standard six-phase incident response lifecycle and shows exactly where a physical process overrides the IT defaults, then sets out the elements an OT plan must contain that no IT template includes.

Reflect

• When was your incident response plan last run, rather than read, and which of the three gaps (procedure, knowledge, or resource) did that run expose?
• Pick the step in your plan with a stated time limit. Has anyone ever timed it with real equipment and real people?
• Think of the last scenario session your team held. Based on its scope, participants, and findings, was it really a drill or really an exercise?
• Who at your site holds the authority to stop production during a cyber incident, and when did that person last take part in an exercise?
• If you had to write one sentence describing the objective of your next session, what would it say, and which format does that sentence point to?