Use THEIR data to tell YOUR story

By Mike Rothman

I’m in the air (literally) on the way to Metricon 6; so I’m thinking a lot about metrics, quantification, and the like. Of course most of the discussion at Metricon will focus on how practitioners can build metrics programs to make their security programs more efficient, maybe more effective, and certainly more substantiated (with data, as opposed to faith). Justifiably so – to mature the practice of security we need to quantify it better.

But I can’t pass up the opportunity to poke a bit at the type of quantification that comes from the vendor community. Surveys and analyses which always end up building a business case for security products and services. The latest masterpiece from the king of vendor-sponsored quantification, Larry Ponemon, is the 2nd annual cost of cyber-crime survey – sponsored by HP/ArcSight. To be clear, I’m not picking (too much) on Dr. Larry, but I wanted to put the data he presents in the report (PDF) in the proper context and talk briefly about how a typical end user should use reports like this.

First of all, Ponemon interviewed 50 end users to derive his data. It’s been a long time since I’ve done the math to determine statistical significance, but I can’t imagine that a sample size of 50 qualifies. When you look at some of the results, his findings are all over the map. The high level sound bites include a median annualized cost of $5.9 million from “cyber crime,” whatever that means. The range of annualized losses goes from $1.5 to $36.5 million. That’s a pretty wide range, eh?

His numbers are up fairly dramatically from last year, which plays into the story that things are bad and getting worse. Unsurprisingly, that’s good for generating FUD (Fear, Uncertainty, and Doubt). And that’s what we need to keep in mind about these surveys. Being right is less important than telling a good story, but we’ll get to that.

Let’s contrast that against Verizon Business’s 2011 DBIR, which used 761 data points from their own data, data from the US Secret Service, and additional data from Dutch law enforcement as a supplement. 761 vs 50. I’m no mathematician, but which data set sounds more robust and representative of the overall population to you?

Even better is one of Larry’s other findings, which I include in its entirety because it must be seen to be believed.

The most costly cyber crimes are those caused by malicious code, denial of service, stolen or hijacked devices and malicious insiders. These account for more than 90 percent of all cyber crime costs per organization on an annual basis. Mitigation of such attacks requires enabling technologies such as SIEM and enterprise GRC solutions.

Really? Mitigation of malicious code attacks requires SIEM and GRC? Maybe I’m splitting hairs here, but this kind of absolute statement make me nuts. The words matter. I understand the game. Ponemon needs to create some urgency for ArcSight’s prospects to justify the report, so throw a little love at SIEM and GRC. Rock on. Yeah, the cynic is in the house.

This statement is then justified by some data that says surveyed customers using SIEM lost on average 25% less than those without SIEM. Those folks with SIEM were able to detect faster and contain more effectively. Which is true in my experience. But only if the company makes a significant and ongoing investment. Right – to the tune of millions of dollars. I wonder if any of those 50 companies had, let’s say, a failed SIEM implementation? Were they counted in the SIEM bucket?

Again, let’s not confuse correctness of the data with the story you need to tell to do your job. That’s the value of these reports. They provide data, that is not your own, allowing you to tell a story internally. Lord knows our organizations want to see hard costs, showing real losses, to justify continued spending on security. This is the same message I deliver with our Data Breaches presentation. The data doesn’t matter – the story does.

A key skill for any management position is the ability to tell a story. In the security business, our stories must paint a picture of what can happen if the organization takes its eyes off the ball. If the money is spent elsewhere and the flanks are left unprotected. Understand that your VP of Sales is telling his/her story, about how further investment in sales is important. VPs of manufacturing tell stories about the need to upgrade equipment in the factories, and so on and so forth. So your story needs to be good.

Not all of us are graced with a breach to create instant urgency for continued security investment. Though if you believe Ponemon’s data, fewer and fewer escape unscathed each year. So you need to create your own story – preferably leveraging another organization’s pain rather than your own. In this case, the empirical correctness of the data isn’t important. It’s how the data allows you to make the points you need.

No Related Posts

I’m a big fan of Red Team, or security assurance, activities. The bad guys are testing your defenses every day, so there is a good amount of data that can be gleaned by breaking your own protections. I also understand how it’s a little hard to deal with when trying to spin data to make a point. But that’s how the world works and it’s how other folks competing for the same resources do things. So we can watch and wait for our budgets to get pulled year after year, or we can be more proactive about “selling” the need for security internally.

@Bill, solid point about taking any conclusions with a grain of salt. Like anything else, if there is a data point that substantiates your “story,” use it. If there are lessons that you can apply to your business, use it. If those breach stats help you create urgency within your own organization (without being chicken little), then use it.

@mort, I know. Had to call that one out, since it’s just such a non-sequitur. And it’s the only mitigation mentioned. That’s what makes it stick out like a sore thumb.


By Mike Rothman

Arghhhhhh! I can’t believe they said that siem/grc is a mitigation. I can just see the conversation now between me and an auditor:

Auditor: We found this vulnerability in our tech assessment

Me: Oh yes, we are aware of that. We couldn’t patch it, so instead we took measures to mitigate attacks:

Auditor: Great! What did you do? Did you firewall the ports? chroot the service? Reconfigure things to prevent the compromise

Me (Proudly): I deployed SIEM and a GRC product

Auditor (to CEO/Audit Committee: We have a problem…..


By David Mortman

Excellent insights. I would also like to point out that while the Verizon DBIR may have 761 incidents, it’s conclusions must also be taken with a grain of salt. See Ben Tomhave’s comments at where Ben concludes: “Hence… marginal utility. Interesting. Informative. Not conclusive. Not comprehensive. Not complete. But, potentially useful, if used carefully.”

By Bill Frank

Mike, its a good point although I’ll admit I feel dirty every time I do it.

What do you think about ‘Red Team’ exercises as a way of generating breach data? Our business continuity team run regular simulations with all levels of management, and use external assessments of performance to justify continued spend by the business. They get the funding they want while our security services are struggling. The problem I can see is that we don’t have an industry for ‘assessing’ security response, so its harder to compare results with our peers.

By Bluesheep

If you like to leave comments, and aren’t a spammer, register for the site and email us at and we’ll turn off moderation for your account.