The Fallacy of Complete and Accurate Risk Quantification

Rich,

I think that you’‘re actually upset about is the outcomes of models they used and how they were used to make decisions, not probability theory, nor the validity of making statements in a quantitative or qualitative nature - and these are independent subjects to address.

Now I vehemently disagree with your core argument that:

1.)  Financial Risk (variance from expected return) is the same as Operational Risk (Frequency & Impact of Threat Event).  There is a etymological problem we English speakers are now forcing upon the world that blurs the two - but they are completely different problems that need different ways to address them.

2.)  Accuracy is unattainable (precision I would agree with). 
Of course, accuracy is within the context of the quality of the decision the existing uncertainty causes.  So accuracy is certainly attainable, but It seems to me that it’s a case by case basis, based on the decision makers subjective willingness to act -  not something you can say across the board.

@Rich…
- First off, to the point of your rant.  AMEN brother, is what I say.
- Second off, I’‘ve been preaching this for years and no one listens… sadly.

But to address your entire article, and it’s a good one indeed, well-put… the bottom line on everything is greed.  It goes back to what drives human nature - GREED.  Greed is what had us here in the 20’‘s, 70’‘s, 80’‘s, late 90’s and now today, and it’‘ll keep driving us to the brink of extinction like this repeatedly over the course of history.  Let’s not kid ourselves… people are greedy and no matter how we regulate, no matter how we try ... no one is above greed.  I know I’‘m not - as virtuous as I try and be .  What can we do but take all the failed CEOs to the middle of Wall Street, hang ‘‘em high, and try to make sure no one forgets?

I am starting to wonder if it still isn’‘t a good time to invest in gold, because despite how high it is - stuff like this really makes one seriously wonder if it isn’‘t going to get a lot worse; driving the value of gold even higher. Seriously, how in the hell can it get better?

Dennis Groves

Good write-up Rich!

I don’‘t know the American regulatory environment well enough (I am an Aussie) - but it looks as though a large part of the problem is a lack of enforcement of existing regulations.

The American taxpayer is now up for $900 billion - all because executives who received multi-million dollar bonuses deliberately engaged in promiscuous financing. Companies that survived the Great Depression have been killed - without even a proper recession to blame it on.

Will they pay back their multi-million dollar salaries and bonuses?

In 2003/4 Warren Buffet warned everyone of the financial "weapons of mass destruction" being used around the world.  Now that it al blows up… people are acting as though it is a surprise!

I think the mistake that most organisations make is treating Risk Management as a "science" when in fact it is an "art". What we are all trying to do is to predict what might happen in the future (both risk and opportunity) and then put systems in place to either mitigate the potential threats or to seize the opportunity. Is it any surprise that we find that trying to preduct uncertainty is well…...... UNCERTAIN - hardly an earth shattering realisation.

At best Risk Management is a best guess about what might happen in the future. Hopefully given our modelling, common sense and experience it is an educated best guess, but at the end of the day it is still our best guess about what might happen.

I often find in organisation where complex metrics and methodology is used that people and decision makers in particular tend to treat Risk Management as a science with mathematical certainty. The complex metrics aid this opinion.

As risk managers we often forget that our methodologies are based upon assumptions which even if backed by a large amount of data are still assumptions that are trying to predict future uncertainty. The assumptions tend to be forgotton amongst complex mathematicall formulae.

Now don’‘t get me wrong I am not saying that risk metrics are wrong, but what I am saying that they are an aid to assist professional judgement not to replace it.

Is it the best we have? Sure - but we shouldn’‘t forget that assumptions may be wrong and that the future may unearth a different threat that we wouldn’‘t even consider on our radar today.

For example in an engineering context harmonic reasonance was a danger that wasn’‘t even considered prior to the Washington bridge collapse, and I doubt if any Dutch company had cartoons of Mohammed on their risk registers but many suffered huge sales losses from the middle east due to their appearance in a Dutch newspaper.

Anyway at the end of the day we don’‘t manage any risks we only manage people’s (including ours and our CEO’‘s) perception of risk!!!

And don’‘t forget - our organisations have only one risk - the decisons they make.

Good luck because as Napoleon said "I would rather have a lucky risk manager (general) than a good one"

[...] usually try to stay far away from politics and current events, but my friend Rich has put up a blog post blaming the credit crisis on quantitative analysis, and then positing that because the economy [...]

>>

I don’t see how anyone can justify voting for a platform of less regulation and reduced government oversight. Now that we own AIG and a few other companies, it seems that’s just a good way to socialize big business. It didn’t work in the 80’s, and it isn’t working now. I support free markets, but damn, we need better regulation and oversight. I’m tired of paying for big business’s big mistakes and people pretending that this time it was just a mistake and it won’t happen again if we just get the government out of the way and lower corporate taxes. Enough of the fracking corporate welfare!

<<

I think you miss the point in a way that most do.  Deregulation only works if the people taking the risk suffer the consequences.  Now, we are socializing the risk but the rewards are still in private hands.  The government should get out of the way, period.  The answer isn’‘t to over regulate on the front to prevent this downside, the solution is to clear out and let people fall off a cliff if they make bad decisions. 

If we don’‘t do that, we get where we are today.  We don’‘t have independant boards, we don’‘t have investor transparency, we don’‘t have people taking sensible risks, all because as investors, we hope someone else will look out for our interests.  Collectively, we are complacent as a socieity, and here is the result. 

Also, this excessive risk taking by business is largely due to the Government holding down intertest rates too low for too long, forcing these financial firms to look for creative ways to make money.  Also, our lovely housing boom/bust has also been caused by the same irresponsible fiscal policy, teamed with the Government "encouraging" lenders to offer loans to people who weren’‘t really qualified in the first place…  and that brings us to Fannie and Freddie, which we also own. 

So the moral of the story is: stop incenting negative behavior and it goes away.

Sub-text: stop trying to save people from themselves and we’‘ll all be happier.

“Which leads to my message of the day. Quantified risk is no more nor less valuable or effective than qualified risk.” I do information risk assessments for a living. There is value in attempting to quantify information security risk and business executives are beginning to demand this. 4 “highs”, 20 “mediums” and 65 “lows” is not valuable information to a decision maker that needs to manage a budget, determine how best to use his resources, while trying to achieve the company’s goals. I think it is irresponsible to poo-poo an emerging discipline within our industry because of the failures or shortcomings within the financial industry.

[...] The recent “midwest wind storm” combined with some crazy work activities has hindered my ability to get in some blog postings. I took a few minutes this morning to quickly peruse some blogs and stumbled across this posting over at securosis. [...]

I think the question you hinted to but didn’‘t actually pose in this article is "how do we correctly manage risk?" and the answer is accountability.

This is not fun to do and so it is not really done.

You also say that you support free markets but want more regulation. These are not contradictory.

Free markets work because mistakes and bad judgements are punished in the cruelest way - your company goes under and you lose your job and all you have invested.

Since it is now possible that the people who own a company are not the same people who make the decisions of the company we have regulations and controls in place so that the people who own the company can watch those that run the company. Essentially annual statements etc. These give those that would feel the pain of a bad decision (the shareholders/owners) the ability to monitor the company and, if need be, force their will on the company by firing the top management. This makes the workers accountable to the shareholders.

The problem with a bank/financial institution is that the government can’‘t allow the institution to be punished (by closing down and hence making shareholders lose their money) because that would shake the markets.

So, the government becomes a stakeholder (not shareholder) in the business because it will have to help out if there is an issue. And, as such, should have regulations in place to make sure that it would not need to jump in and sort things out.

Taking this to Information Security - I am working on an idea I call "Shareholder-centric Security". ;)

Basically a Shareholder could be anyone and is defined as a person who has the most to lose if something bad happens. Information Security advise all shareholders on what the threats are and together they work out what the risks are. The shareholder then accepts the risk and takes steps to mitigate it.

Getting back to "accountability" - there is a natural tendency to understate risk basically because the compensating controls will then be cheaper. There has to be accountability in that shareholder will *want* to get a good idea of the risks so that they can cover themselves properly.

@Chris,

Just because the business wants numbers, and you can make up numbers, doesn’‘t mean they are accurate or reflect the real risk. You are irresponsible if you provide a purely quantified assessment that leads to a poor risk decision.

Not all risk can be quantified. Please respond to my core points if you think that I am wrong that a quantified assessment is both just as prone to error as a qualified assessment, and in many cases can lead to worse decisions.

@Rich – Luckily, I leverage a risk methodology that breaks risk into elements that I can numerically represent based off my experience, the data I have available, and with input from other subject matter experts. In addition, the same methodology accounts for my confidence (or lack there of) in what you refer to as “made up numbers”. There will always be an element of uncertainty with risk. 2006 and 2007 were expected to be some of the worst years on record for hurricanes in the US – and there were no major hurricanes – do we write that off to “made up numbers” as well.

If the business wants numbers, then we should strive to meet their needs and show value – not bury our head and admit defeat. How I articulate a risk scenario is probably more important then the risk being represented because that decision maker knows there is an element of uncertainty and yet a level of reasonableness behind it. And guess what? The decision maker can agree or not agree with my findings. I have had some state the risk is not enough but very few that though the risk was more then what was being articulated.

I understand your frustration and skepticism, but please understand that information security risk quantification is occurring, it is wanted by businesses, it can facilitate cost benefit analysis in terms of risk vs. cost to mitigate, it is not wild guessing or “made up” numbers, and it can result in better decision making. Finally, I do not work for an information security / risk management vendor – I work for a company that understands risk (financial services industry) and embraces these concepts for treating operational risk exposures (information security risks) like product risk.

What the world would be like if we used qualitative labels for everything that costs money:

Loaf of bread A: LOW RISK, cost unknown until you get to the register
Loaf of bread B: HIGH RISK, cost unknown until you get to the register
Loaf of bread C: MEDIUM RISK, cost unknown until you get to the register

Thoughts?