Black Box Data Store: Lessons Learned from the NTSB

US Airways flight 1549 teaches us that improbable events actually do occur sometimes. When the NTSB goes to investigate, the airplane’s black boxes will prove vital in the determination of cause. We can leverage this concept to fortify our chances of surviving a serious investigation. In this article I introduce design considerations for what I call the Black Box Data Store, the important data you need to prove your innocence in an investigation.

NOTE: This originally appeared on this date at Quest Software's ToadWorld, on the expert blog "John Weathington's Quest for Compliance". The link to the actual ToadWorld article is at the bottom.

By now, I’m sure you’ve heard about the Hudson River landing of US Airways flight 1549. What a remarkable story of an unbelievably low probability risk event actually occurring, and a heroic contingent effort that mitigated a potential disaster. The popular theory on causation is that a flock of birds flew into both engines, causing them both to fail. But how do we know for sure that’s what happened?

When the dust settles, I’d be really surprised if this wasn’t the case. I’m not trying to pitch a conspiracy theory here. My point is that we won’t know the definitive answer until the NTSB (National Transportation Safety Board) completes its investigation. And fortunately, they’re well prepared. Every airplane is fitted with two “black boxes” at the tail of the plane that records vital information for the investigators: the cockpit voice recorder and the flight data recorder.

Now the Hudson River case may seem obvious, and maybe they have enough ancillary evidence to make a decisive conclusion on the bird theory, however maybe not. In fact, in a lot of cases the NTSB investigates, these black boxes are all they really have.

These boxes serve a specific purpose; they support NTSB investigations. The NTSB at some point realized that it would be a smart idea to collect a lot of data on every flight of every plane. The large majority of this data is never used. However, in the unlikely event of a water landing (can you tell I’ve had my share of plane trips), or any other unusual (and usually unfortunate) happening, they have the data that they need to determine what happened.

Your company is in the same position when it comes to compliance. The intelligent architect of a compliance program will design a component of the data system that collects data in anticipation of an investigation. If you follow my work, you know that what I’m suggesting is a contingent control for the risk event of an investigation.

In most cases your controls are dealing with an unfavorable risk to an objective. In this case however, the objective itself is the efficient compliance program, and the risk is that you’ll be audited and subsequently investigated.  The effects of the investigation are probably detrimental to the company (i.e. penalties and fines), so to deal with the effect before the risk event occurs, is what I call a contingent control. Assuming your company is doing the right things, lack of data to prove this can kill you in an investigation. So the approach is to be prepared with lots of good data. And in honor of the aviation system just described, the component of the compliance data system that supports this control is what I call the Black Box Data Store.

The Black Box Data Store is simple in concept, but powerful in function. It can connect to any component of your compliance data system, including your source systems, compliance operational data store, compliance enterprise data warehouse, or even your compliance data marts. Think of it as being downstream from your core compliance data system. When designing your Black Box Data Store, keep the following design principles in mind:

Design Principle #1: Assume Your Company Will Be Investigated

Don’t assume that since an investigation is unlikely, you don’t have to put effort into preparing for it. The incident that just happened with US Airways had a probability of less than a million to one. You must approach this with the attitude that you will be investigated, and design the system appropriately.

Design Principle #2: Collect More Data than You Need

I’ve been on several data warehouse analysis sessions, where the user is mandated to provide an exhaustive list of data points that they want brought from the source system to the data warehouse. This may be appropriate for some strategic purposes, but for a compliance data system it’s the wrong approach. More often than not, during an investigation, you will discover the usefulness of data points that never served a purpose prior to the investigation. You definitely want this data available. Trying to pull in data after the fact is always painful and in some cases impossible.

Design Principle #3: Plan for Design Iterations and Mock Investigations

As with most designs, you will not get it right the first time. To think you will is arrogant, and will cost your company dearly. As with the Hudson River landing, you only get one shot to get it right, so you need to practice several times. Assemble a team of coders, designers, experts on the regulation or standard, and auditors. Build the system you think is appropriate, and have a mock investigation. Do a thorough post-mortem and go back to the drawing board. Plan on at least three iterations, and more is better.

Investigations are a good thing when the NTSB is trying to get to the bottom of an airplane crash, but not so good when your company is the one being investigated. You can help your company be prepared for an investigation with a Black Box Data Store. Assume your company will be investigated, collect more data than you need, and conduct as many design iterations and mock investigations it takes to get it right. You probably won’t get to the point where your company is being investigated; however, you’ll be happy you have a Black Box Data Store if you ever do.

...read the article on ToadWorld