Hard-Boiled Compliance

Policy Data Management in 3 Stages

NOTE: This originally appeared on this date at Quest Software's ToadWorld, on the expert blog "John Weathington's Quest for Compliance". The link to the actual ToadWorld article is at the bottom.

I’m in the middle of doing a proactive audit program reinforcement effort (unbelievably, they do exist ), and I just completed overhauling the organization’s policies, so since that’s top of the brain for me today, I’d like to talk a little about policy management.

In my mind, a good policy is the glue that holds governance, risk, and compliance practices together, and crystallizes the process to an auditable format. As a data professional, once again you can really add value here to help your business users stay in control of their policies.

What is a Good Policy?

To put it into perspective, let’s take a look at where policies fit into the governance, risk, and compliance domain. If you’ll remember, the basic relationship between strategic goals, risk, and compliance is this. Strategic goals help the company obtain its strategic objectives. Governance is making sure the strategic objectives are being accomplished. These objectives have risks, which are uncertainties about what lies in the future, which will prevent your company from attaining its strategic objectives. Risks can also come from other parties (e.g. the US Government) that are concerned about the effects your strategic goals might have on its concerns (e.g. the investing public). To mitigate these risks, controls are put in place, and making sure these controls are being followed is compliance.

Now, think of a good policy (emphasis on good) as a document that clearly spells out your governance, risks, and compliance implementation in clear and unambiguous language, with step by step procedures that integrate the tasks necessary to accomplish both the strategic goals and the compliance activities. This is a document worth keeping track of!

Your company will have its own way of constructing policy, which you should become familiar with. Ask one of your business users to show you a policy document so you can study it. Pay attention to the section headings, and the way things are organized and itemized. As stated, not all policy formats are the same, but I’d like to extract some common section headings to set the stage for our policy management architecture.

• Purpose / Introduction. As you might expect, there’s usually a section that introduces the reader to what the policy is all about.
• Revision History. As its name implies, this is a record of all the changes that have been made to the policy, who did it, and why.
• Policy. The policy itself should be a general statement about what your company’s position is on certain details. It calls out your company’s goals, highlights the risks, and clearly spells out how those risks will be controlled. If the control is imposed (i.e. from a regulation), the source of the control should be available here. For instance there might be a policy that states, “It is our policy that expenses over $5000 shall be approved by a Vice President.” This is obviously a policy that controls for some sort of risk, which should be spelled out in the policy as well.
• Definitions. A good policy document will have definitions defined.
• Procedures. These are step by step instructions for how a process should execute.
• Evidence. Your company’s policy format might have an Evidence section, which outlines what sort of evidence or documentation you should expect to see as a result of this procedure.

Evolving the Policy Management System

When designing and / or implementing any sort of architecture like this, I’m a big proponent of starting with the simplest thing possible that can add value, and evolving from there. Consistent with that philosophy, I’d like to suggest these 3 stages to the effective design and architecture of your policy management system.

Stage 1 – Document Store

The first stage should be a simple document store. Create a simple database table that looks like this:

• POLICIES
  • POLICY_ID – Unique identifier
  • POLICY_DATE – Date policy was uploaded
  • POLICY_NAME – Name of the policy, should include version number
  • POLICY_DOCUMENT – A scanned image of the policy

This should be a piece of cake, and it will immediately add value to your business users. Anytime a policy is created, it’s scanned into the database. Do not try to over-scope this, as it’s an easy trap to fall into. Just keep it simple, and deliver it quickly.

Stage 2 – Structured Policy Management

In stage two, you evolve into a database that contains the text of the policy, and handles change management a little better. Remember, this is an evolution, so the original table still remains. The scanned policy is still very valuable –why get rid of it?

The new table will have more fields, based on the format of your company’s policies. For instance, you will have a PURPOSE field and a REVISION_HISTORY group of fields. Consider indexing the PURPOSE field for searchability, as this is a good place to find keywords about the policy. For REVISION_HISTORY, make sure you separate out who did the update, when they did it, and why they did it ( i.e. VERSION_NUMBER, REVISION_DATE, REVISION_WHO, REVISION_DETAILS  ). The POLICY_NAME should now stay the same through different versions of the policy, so you can quickly query policies. You may consider a CURRENT_FLAG to indicate the latest version of the policy. Another strategy is to keep revision histories in a separate table.

POLICY, PROCEDURE, DEFINITION, and EVIDENCE fields should be broken out into separate detail tables. They will contain itemized lists that you want to capture in separate detail table rows.

Stage 3 – Integrated Policy Management

This is where you really start integrating your policy management system into your larger compliance data system architecture. In the spirit of evolution, I suggest you take baby steps within this 3rd stage of development, taking small wins where you can.

Consider these ideas for integrating your policy management system:

• There’s a great leverage point between your procedures and your Automated Process Auditing system. Your policies set the stage for assertions which can be tracked during an audit. In the example above the policy was “It is our policy that expenses over $5000 shall be approved by a Vice President.” You could model this into a test that’s tracked through the execution of this procedure.
• You can abstract the people in the equation (i.e. person making revision to policy) into roles, and tie the actual person making the change to your HR management system. If you’ll remember in How To Keep Dead People Out of the Database we discussed some ways of avoiding an embarrassing situation where the auditor discovers people making changes to the database that shouldn’t be (i.e. somebody who has left the company, or worse yet deceased!) This is a good place to tie into that architecture.
• Definitions are a good place to start a data governance effort. Build a data dictionary of terms, and empower the users with good data governance strategies. You can then leverage your data dictionary to all areas of the compliance data system, not just policy management.

These are just some ideas. I’m sure once you get started, you’ll come up with a lot more ways to leverage your new component of the compliance data system. Good policy management is another great way for your business to demonstrate effective control over processes and procedures to an auditor.

Take some time today to understand your company’s policy format, and build a simple database like the one I described in Stage 1. It shouldn’t take long, and it will add a lot of value to your businesses compliance operation.

...read the article on ToadWorld

People Make Mistakes

NOTE: This originally appeared on this date at Quest Software's ToadWorld, on the expert blog "John Weathington's Quest for Compliance". The link to the actual ToadWorld article is at the bottom.

Everybody makes mistakes, even seasoned professionals. If you’ve ever tried to learn a computer language, I’m sure you can relate to what I’m talking about. Even if you have 20 years experience, it’s quite uncommon to just crank out code that works perfectly the first time. This doesn’t mean you don’t know how to program – it just means that even the best of us get it wrong sometimes.

I made a mistake myself, just today. I accidentally posted something to a forum board, in a place I wasn’t supposed to be posting. The rules are very clear, and I’ve been posting to this forum for a very long time, however today I wasn’t paying attention and stepped on the board moderator’s toes.

Whenever the human touch is involved, this is a risk we need to deal with. In Japanese, the term Poka-Yoke means “mistake-proofing”, and people in quality circles use this term to describe the actions taken to make sure errors are either avoided ( the ideal situation ), or made immediately apparent ( not as great, but still better than letting the error go unnoticed ).

When compliance is concerned, human error is a big issue. An inappropriate post in a forum board is pretty innocuous, but something like an “additional” zero in the wrong place can really wreak havoc. As data professionals, we need to be aware of how we can keep our business partners from making these kinds of mistakes.

An interesting architecture to consider is a “mistake database.” Now, nobody likes their mistakes thrown back in their face, but we’re all grown-ups here, and there’s a greater goal at stake other than protecting somebody’s ego. The idea of the mistake database is to capture all the mistakes people make, so we can analyze the data to make improvements for the future.

Records in the mistake database can be started with an automated process, however need to be finished by human intervention. For example, you could have an automated reconciliation control in place that would be upstream from your mistake database. Each time the control failed, a mistake would be recorded in the mistake database. In addition, people should be allowed to manually enter their mistakes as they are discovered.

At the time the mistake is discovered, at a minimum you should record:

• What the mistake is ( i.e. description of the mistake )
• Who made the mistake
• When the mistake was made ( date and time )
• What process was being performed when the mistake was made

The more additional information you collect about the mistake, the better. You will want to brainstorm on different factors surrounding the mistake, so you can identify possible patterns associated with the mistake. For example, capturing the department of the employee may give you the opportunity to highlight departments that have a high probability for making mistakes.

Shortly after the mistake is recorded, the following information needs to be collected by the person that made the mistake:

• What caused the mistake to be made
• What could prevent the mistake from happening in the future
• What will be done to correct the mistake

Although it seems somewhat like an issue tracking system, it is not. Do not make the mistake ( pun intended ) of mixing the two together. This system’s sole purpose, is for mistake analysis. That’s why we’re not concerned that the mistake was corrected, only in what is planned. This will give the analysis team insight into possible corrective and / or preventive measures to consider.

Periodically, a review should be done on this database by a Poka-Yoke team – a team responsible for error proofing the process or situation. This takes a combination of analysis and innovation skills. In practical terms, it is not required for these skills to be terribly acute. The data in most cases will speak for itself. If you see the same mistake being made, something needs to be done about it, and the best advice on correcting the problem will come straight from the people making the mistakes. There are occasions however, where either trends and / or causation are hard to spot, or you realize the trend but it’s hard to come up with a solution to the problem. In these cases, you may need to defer to the quality people ( e.g.  Six Sigma Black Belts ).

As long as humans are involved mistakes are going to be made. It’s your company’s option to allow the mistakes to continue, or to actively do something about preventing them. This article has given you some advice on how to assist your company in that direction. Start a simple mistake database today, and you’ll uncover obvious fixes in no time.

...read the article on ToadWorld

The 3 Key Meetings to Hold Once Disaster Strikes

NOTE: This originally appeared on this date at Quest Software's ToadWorld, on the expert blog "John Weathington's Quest for Compliance". The link to the actual ToadWorld article is at the bottom.

Today, September 11, 2008, is the 7th anniversary of one of the worst tragedies to appear on American soil.

Disasters are No Fun

I was consulting at Silicon Graphics at the time, and struggling through a difficult personal time as my wife’s father was in the hospital battling life for the last time. He had Muscular Dystrophy and an attack of Influenza was too much for his muscles to handle. Several family members were staying at the house to help us through the event. When I came downstairs, everybody was crowded around the television and nobody was saying a word. When I first saw the footage of the large airliner crashing into the tall New York tower, my mind could not process what was going on. Even after family members explained what had happened, I just could not believe it.

Later that day at Silicon Graphics, nobody really talked about it. If it was discussed at all, it was a very brief conversation, and then we went about our business of the day. It was almost as if nothing happened.

Now we’ve had seven years to reflect, to learn, and to grow as a nation, however the somberness of the event still hovers.

Disasters are painful experiences. I remember when I was a young programmer, I was working as the chief ( and only ) software engineer for a small startup. We were doing C programming against an Informix database on a Solaris workstation. Under my home directory was a temp directory where I did a lot of experimentation. Although I had a personal account on the system, I was the only programmer, so I always just logged in as root.

One day at the close of a successful experiment, I issued the following command: “rm –r *”. For those of you who don’t know Unix, these 7 simple characters will delete everything in the current directly, and everything in every directory below the current directory. The command would have been innocuous, if it weren’t for the fact that I wasn’t actually in my temp directory – I was one level up in my home directory. Once again for the people who are not familiar with Unix, this is a very unforgiving language. There is absolutely no “undo”, so in a split second 2 years of work were gone without a trace.

How to Characterize a Disaster

Before we learn how to move on from a disaster, let’s characterize it so we know what we’re dealing with. My definition of a disaster is this. It’s a risk event that hasn’t been previously identified ( otherwise known as an “unknown unknown” ), that carries an extremely high degree of impact.

If you’ll remember in Beyond Compliance – Understanding Risk, we modeled risk as having at least the following three characteristics: probability, detectability, and impact. I’d like to add the characteristic of “Predictability” to further illustrate and characterize the disaster ( this isn’t a bad idea for all your risks ). If we quantify and normalize these characteristics on a scale from 1 to 100, then here’s an example of what a disaster would look like:

• Probability: 01
• Detectability: 100
• Impact: 95
• Predictability: 01

This is a very unlikely event that hits with very little warning ( i.e. low predictability ) and causes a severe amount of impact to both schedule and budget. You would have to characterize this after the fact, as by definition it did not even hit your radar in your initial risk analysis.

A proper risk management plan would cover this risk under “Management Reserve” because it is an unknown unknown. In other words, we don’t know what we don’t know, but we know we don’t know it, so we account for it. The problem however, is that a prudent management reserve will not allow for a catastrophic event. When disaster strikes, it will wipe out your management reserve, your contingent reserve, and pretty much blow your risk management plan away.

Dealing with Disaster – Hold 3 Key Meetings

So, what do you do?

When a disaster strikes, I want you to immediately hold 3 meetings:

Meeting # 1 : No Tears Allowed From This Point Forward

You must understand that you cannot move forward until all the negative energy is out of the project team. Realize the disaster for what it is – painful. Don’t try to ignore it, just bring everything out in the open. This meeting’s primary purpose is for everybody to vent. Talk about how serious the impacts are. Talk about all the wasted time. Get it all out. Then at the resolution of the meeting you must be serious about leaving the negative behind.

Meeting # 2 : Undo Brainstorm

In an Undo Brainstorm, you gather your best resources together, and brainstorm on ways to “undo” what has happened. If there’s even an ounce of complaining, you’re not ready for this meeting yet – Redo Meeting # 1.

You will need a little luck; however your chances these days aren’t that bad. Operating systems today are built with some pretty cool undo features, and organizations routinely back up even non-critical data. If you think you lost all your data, check with your system administrators to see if they have a file backup.  Brainstorm, and capture any and all ideas that come up. You will be surprised.

Meeting # 3: We Can Rebuild It

If the Undo Brainstorm doesn’t work in your favor, you will simply need to move on. Keep the positive energy up, and go to work on rebuilding. Project managers, your plan is shot – start over. Don’t try to salvage the old plan – this is actually an opportunity to improve on the execution imperfections that happened along the way up until this point. The same goes for the developers – start over. You know where you need to end up, so start thinking about the smartest way to get there.

Here’s an important thing that I want you to remember at this point. It will not take you the same amount of time to rebuild the plan or the solution. In fact, things will be rebuilt much quicker, and much better. Most of the time in a development effort is spent in thought work, and readjusting from failed attempts. That’s all behind you now, so you can now go straight to the solution.

September 11, 2001 was an awful day in American history, but we’ve moved on as a nation. You will learn valuable lessons from the disasters that pop up on your compliance projects, but you will never forget what happened. These experiences will always stay with you. The difference between the companies that recover, and the companies that sink has to do with attitude and action. Three simple meetings can gracefully move you on your path to recovery.

...read the article on ToadWorld

Flawless Compliance Newsletter - September 2008

The September 2008 issue of Flawless Compliance is available, with Center Stage article, Out-Of-Control Internal Controls, Is COSO's Monitoring Guidance Really Necessary? Plus, Governor Sarah Palin brings a breath of cool Alaska air to the GOP campaign, what to look for in a Business Analyst that will translate your compliance requirements, A Clinton cronie gets sued and gets paid, and lessons learned from a brave kitten.

... read the September 2008 issue of Flawless Compliance

The September 2008 issue of Flawless Compliance is available, with Center Stage article, Out-Of-Control Internal Controls, Is COSO's Monitoring Guidance Really Necessary? Plus, Governor Sarah Palin brings a breath of cool Alaska air to the GOP campaign, what to look for in a Business Analyst that will translate your compliance requirements, A Clinton cronie gets sued and gets paid, and lessons learned from a brave kitten.

... read the September 2008 issue of Flawless Compliance

Getting Risk Right - How Probable is Your Probability?

NOTE: This originally appeared on this date at Quest Software's ToadWorld, on the expert blog "John Weathington's Quest for Compliance". The link to the actual ToadWorld article is at the bottom.

What’s the probability and impact of inaccurate values in your risk database? In other words, have you done a risk analysis on your risk analysis?

I know this sounds circular, and I am being a little facetious here, but this should be a real concern for you. What happens if your company decides a risk has a 10% probability when it really has a 90% probability? The whole purpose for modeling risk, is to prepare yourself for the unknown, and you cannot be properly prepared if your data isn’t accurate. It’s like trying to drive your car across the country on one tank of gas. Your intentions are in the right place, but your assumptions will get you in trouble. This is the same thing that can happen at your company, so we’ll spend some time today on how you can help your company get it right.

The Problem with Bad Probabilities

Last week in Beyond Compliance – Understanding Risk, we started with some architecture for modeling risk, and identified these properties of risk:

• Risk Name – Name the risk ( i.e. fraud )
• Probability – The probability that the risk will occur.
• Impact –What will happen if the risk occurs.
• Detectability – How easy it is to detect the risk.

You can of course expand on this, however it’s important to remember these three properties about risk, as they actually serve a purpose over and above qualitative composition. Among other places, these properties are used in Six Sigma when constructing an FMEA ( Failure Modes and Effect Analysis ). An FMEA is one of Six Sigma’s primary ways of characterizing and analyzing the different types of risk that could show up in a process or project.

To do the analysis, all three of these properties ( probability, impact, and detectability ) need to be quantified ( converted to a numerical representation ), and normalized ( set on the same scale ). That’s why in last week’s discussion, I suggested a numerical representation of all three numbers ( indicated by the _GUAGE suffix ), and further suggested that all three hold a range between 1 and 100.

The reason why these numbers need to be quantified and normalized, is because to complete the FMEA, you will need to multiply all three of these numbers together, to arrive at a Risk Priority Number ( RPN ). The RPN will then help separate high risks from low risks.

So with this in mind, can you imagine what would happen to your RPN if your inputs are wrong? Obviously, your RPN would be greatly skewed, and the whole analysis would be worthless. Unfortunately, this happens a lot in the real world. We’re going to make sure it doesn’t happen to you.

Out of the three properties, in my experience probability is the hardest to get accurate. Impact is largely subjective so there’s no real “wrong” answer, and detectability is pretty easy to discern. Probability however, must be calibrated as there is a “true” answer, and if you guess at it, you could be wrong.

The Financial Risk Data Mart Revisited

Previously, in How To Help Your Company with Financial Risk, I introduced the concept of a Financial Risk Data Mart. Today, we’re going to leverage this concept to get our risk probability as accurate as we can. You can’t do this with everything, but when you can it’s a very useful strategy.

To make this work, let’s profile a risk:

• Risk Name : Risk of Inaccurate Data Entry
• Risk Description : The risk that an inexperienced data processor will enter the wrong value
• Control Measure: Redundancy – We will have two different people enter the exact same data. Anytime there is a discrepancy between the two entries, we have an exception.
• Probability : ???
• Impact : Minor ( Score: 10 out of 100 ). Exceptions will be highlighted, and correct data values will be re-input.
• Detectability : Very High ( Score: 10 out of 100 ). With the control in place, this is very easy to detect ( note that when the detectability is high, the score is low ).

As you can see, with the control in place, this is a pretty low risk item. However, we’ll use it for illustrative purposes.

We will use our risk data mart to inform our probability metric here. You can use the same data mart with all your risks, by simply making risk a dimension.

The metrics that you will capture in your fact table will be observations per day ( # of data entry points ) and exceptions per day ( # of times the data points did not match up between the two data processors ). You can then find the exception percentage for that day by dividing the number of exceptions by the number of observations. For instance, if there were 1000 data entry points entered on a day, and 35 had conflicting entries between the two data processors, then the exception percentage would be 3.5 ( [ 35 / 1000 ] * 100 ). This exception percentage will inform the overall probability for your risk.

Using the Exception Percentage to Get Risk Probability

The mistake most people would make at this point is to directly translate 3.5% to the risk probability and call it a day. As you can tell by my language, this is the wrong approach. This is called a point estimate, and it’s is never correct.

Now, if you said you would collect 30 days of data and take an average, then I’d say you’re getting much closer to the answer. The only problem with this approach is that one day can really mess up your average. Let’s say you had 29 days where the value was between 3 and 5, and then one day for some strange reason you recorded a 95! Your average could be 6 or 7 which probably isn’t an accurate risk probability.

This is actually a trick question. The best answer will involve more than an average. To characterize anything like this, you need some measure of central tendency ( i.e. average ), some measure of spread ( i.e. standard deviation ), and some idea of the distribution’s shape ( i.e. normal, logarithmic, etc. ).

For central tendency, I would suggest taking the mean ( average ), median ( 50% percentile ), and mode ( most common value ) and compare them. If they’re all about the same, you’re in good shape – go with the mean. For spread, as long as you’re looking at 30 days, I think standard deviation is okay unless your distribution shape is not normal. The best way to get your distribution shape is to just plot the values, and look at it. If it looks like a bell curve, this is a normal distribution, and once again you’re safe using the mean and standard deviation. If not, consult a statistician or a Six Sigma Black Belt.

A Lot of Work for One Value

I know it seems like a lot of work just to get one value, but it’s absolutely necessary. If your risk probabilities are inaccurate, that could invalidate all the rest of your assumptions. Trying to get probabilities correct can be tricky, but you can leverage the power of your risk data mart to get as close as possible. Take some time today to re-evaluate your highest risks.

...read the article on ToadWorld