The 3 Key Meetings to Hold Once Disaster Strikes

NOTE: This originally appeared on this date at Quest Software's ToadWorld, on the expert blog "John Weathington's Quest for Compliance". The link to the actual ToadWorld article is at the bottom.

Today, September 11, 2008, is the 7th anniversary of one of the worst tragedies to appear on American soil.

Disasters are No Fun

I was consulting at Silicon Graphics at the time, and struggling through a difficult personal time as my wife’s father was in the hospital battling life for the last time. He had Muscular Dystrophy and an attack of Influenza was too much for his muscles to handle. Several family members were staying at the house to help us through the event. When I came downstairs, everybody was crowded around the television and nobody was saying a word. When I first saw the footage of the large airliner crashing into the tall New York tower, my mind could not process what was going on. Even after family members explained what had happened, I just could not believe it.

Later that day at Silicon Graphics, nobody really talked about it. If it was discussed at all, it was a very brief conversation, and then we went about our business of the day. It was almost as if nothing happened.

Now we’ve had seven years to reflect, to learn, and to grow as a nation, however the somberness of the event still hovers.

Disasters are painful experiences. I remember when I was a young programmer, I was working as the chief ( and only ) software engineer for a small startup. We were doing C programming against an Informix database on a Solaris workstation. Under my home directory was a temp directory where I did a lot of experimentation. Although I had a personal account on the system, I was the only programmer, so I always just logged in as root.

One day at the close of a successful experiment, I issued the following command: “rm –r *”. For those of you who don’t know Unix, these 7 simple characters will delete everything in the current directly, and everything in every directory below the current directory. The command would have been innocuous, if it weren’t for the fact that I wasn’t actually in my temp directory – I was one level up in my home directory. Once again for the people who are not familiar with Unix, this is a very unforgiving language. There is absolutely no “undo”, so in a split second 2 years of work were gone without a trace.

How to Characterize a Disaster

Before we learn how to move on from a disaster, let’s characterize it so we know what we’re dealing with. My definition of a disaster is this. It’s a risk event that hasn’t been previously identified ( otherwise known as an “unknown unknown” ), that carries an extremely high degree of impact.

If you’ll remember in Beyond Compliance – Understanding Risk, we modeled risk as having at least the following three characteristics: probability, detectability, and impact. I’d like to add the characteristic of “Predictability” to further illustrate and characterize the disaster ( this isn’t a bad idea for all your risks ). If we quantify and normalize these characteristics on a scale from 1 to 100, then here’s an example of what a disaster would look like:

• Probability: 01
• Detectability: 100
• Impact: 95
• Predictability: 01

This is a very unlikely event that hits with very little warning ( i.e. low predictability ) and causes a severe amount of impact to both schedule and budget. You would have to characterize this after the fact, as by definition it did not even hit your radar in your initial risk analysis.

A proper risk management plan would cover this risk under “Management Reserve” because it is an unknown unknown. In other words, we don’t know what we don’t know, but we know we don’t know it, so we account for it. The problem however, is that a prudent management reserve will not allow for a catastrophic event. When disaster strikes, it will wipe out your management reserve, your contingent reserve, and pretty much blow your risk management plan away.

Dealing with Disaster – Hold 3 Key Meetings

So, what do you do?

When a disaster strikes, I want you to immediately hold 3 meetings:

Meeting # 1 : No Tears Allowed From This Point Forward

You must understand that you cannot move forward until all the negative energy is out of the project team. Realize the disaster for what it is – painful. Don’t try to ignore it, just bring everything out in the open. This meeting’s primary purpose is for everybody to vent. Talk about how serious the impacts are. Talk about all the wasted time. Get it all out. Then at the resolution of the meeting you must be serious about leaving the negative behind.

Meeting # 2 : Undo Brainstorm

In an Undo Brainstorm, you gather your best resources together, and brainstorm on ways to “undo” what has happened. If there’s even an ounce of complaining, you’re not ready for this meeting yet – Redo Meeting # 1.

You will need a little luck; however your chances these days aren’t that bad. Operating systems today are built with some pretty cool undo features, and organizations routinely back up even non-critical data. If you think you lost all your data, check with your system administrators to see if they have a file backup.  Brainstorm, and capture any and all ideas that come up. You will be surprised.

Meeting # 3: We Can Rebuild It

If the Undo Brainstorm doesn’t work in your favor, you will simply need to move on. Keep the positive energy up, and go to work on rebuilding. Project managers, your plan is shot – start over. Don’t try to salvage the old plan – this is actually an opportunity to improve on the execution imperfections that happened along the way up until this point. The same goes for the developers – start over. You know where you need to end up, so start thinking about the smartest way to get there.

Here’s an important thing that I want you to remember at this point. It will not take you the same amount of time to rebuild the plan or the solution. In fact, things will be rebuilt much quicker, and much better. Most of the time in a development effort is spent in thought work, and readjusting from failed attempts. That’s all behind you now, so you can now go straight to the solution.

September 11, 2001 was an awful day in American history, but we’ve moved on as a nation. You will learn valuable lessons from the disasters that pop up on your compliance projects, but you will never forget what happened. These experiences will always stay with you. The difference between the companies that recover, and the companies that sink has to do with attitude and action. Three simple meetings can gracefully move you on your path to recovery.

...read the article on ToadWorld