Analysing Aberdeen Group Application Security Study

I had a comment on my post early security engagement in projects - critical or a waste providing a link to a study. It asked for my thoughts on a December 2010 Aberdeen Group study titled "Security and the Software Development Lifecycle: Secure at the Source". The study has a headline: "Top Performers Invest More Annually in Their Application Security Initiatives, but Realize a Higher Return by Identifying and Remediating More Vulnerabilities Prior to Deployment". The key conclusion is that companies practising security at the source saw "a very strong 4.0-times return on their annual investment". I was asked. so here are my thoughts:

Wrong conclusions?
A four times ROI sounds impressive, but as always you need dig a little deeper. The natural assumption and the one that will make it into many management Powerpoint deck's is that securing at the source or early engagement in security leads to 4 times return on investment.

However this conclusion is not what is presented by their own data. The last two columns highlighting the differences between security at source and the industry average have been added by me:

Assessing the Business Value Derived from Application Security Secure at the Source Industry Average Difference Difference %
Application vulnerabilities identified and remediated prior to deployment 83.90% 81.70% 2.20%
Application security-related incidents experienced in the last 12 months 6.9 6.3 0.6 10%
Annual cost of application security initiatives ($K) (includes all related costs for people, process, and technologies) $620 $330 $290 88%
Return on annual investment from application security initiatives 4 3.8 0.2 5%

My analysis of these numbers is that despite spending 88% more the secure at source companies only achieved a 5% greater return on investment. They also only remediated 2.2% more vulnerabilities prior to deployment despite all that early involvement but did experience a 10% reduction in application security incidents. The latter sounds impressive until you consider they say the average application security incident cost for participants in this study is $300,000. So the secure at the source crowd source spent $290,000 more to save $180,000 (6.3 incidents rather than 6.9 incidents @ $300,000 per incident).

Now this is all their numbers, I didn't make any of it up, just looking at the differences highlights that you often need a comparison to add context and draw real insights. So while the 4 x ROI makes a nice headline, a lot of people may conclude that actually the data shows for nearly double the cost there was very little additional benefit to secure at the source. In fact it was a strategy to lose $110,000 each year. Now try selling that one to management!

So the headline "Top Performers Invest More Annually in Their Application Security Initiatives, but Realize a Higher Return by Identifying and Remediating More Vulnerabilities Prior to Deployment" is technically correct it just misses any comparison by how much. Also the point that you may actually be better off with the industry average of all three methods rather than secure at the source to the tune of over $110,000 a year. The study may just suffer from what even more mature industries like medicine face - if you start with a conclusion you want to prove, there is a temptation to do that regardless of what the numbers say. Aberdeen group at least has to be commended for presenting the actual numbers.

It is listed that the Industry average figures here are from the "Aberdeen study on Securing Your Applications: Three Ways to Play benchmark study". Therefore this presumably averages the results of those that secure at source (42 from 150), Find and Fix (pen test and fix only), Defend and Defer (web app firewalls etc only). It would have been nice to compare just these scores across the three different approaches rather than an average of all three but you have to pay Aberdeen $399 to get access to the other two reports! If that comparison showed a much bigger difference then the argument for secure at source would be more compelling. Why Aberdeen didn't provide this is either for selling these reports individually or that the numbers really add a lot more than using the industry average figures.

Some other thoughts:

Sample size:
They have a sample of 150 companies, 42 who use secure at source (or early engagement as I phrase it). The companies have on average 6800 end-users. This is not bad but suggests they are not very large, while this does not invalidate the conclusions by itself it would be nice to have a comparison to large multinationals with a 100,000 or higher employees, especially as these types of companies are more likely to have made investments in enterprise level controls as I discuss.

Soft numbers
Their number for quantifying the business value is:

Value = Total application security costs avoided / (Total application security costs + Total application security costs not avoided)

For the numerator even they say: “these may be difficult to come by, and imprecise at best”

Even for the denominator I would like to know whether these are survey type estimates or numbers that are rigorously measured and reliable. When they say figures like the average cost for an actual application security incident for participants in this study was $300,000. I ask because finding a 150 companies that measure these types of numbers so accurately would be extremely impressive.

Focus on web applications
Due to OWASP there is a great focus on web application security but my experience is that there are many projects especially in large companies are not developing web applications – especially from scratch. There are infrastructure projects, updates to platforms, adding new functionality to legacy systems, virtualization and technology refresh projects, even updating existing web apps without changing any security. I would love to see a study that looks at the ROI of early and constant security engagement in these projects. There maybe a good argument that all new web application development projects should have secure at the source, whether it applies to all projects is another matter.

I have to thank the commenter for linking the study, it is exactly what I asked for. I hope in time our industry will have multiple independent studies performed by academics into matters such as this, performed with rigour and sufficient sample sizes and based on reliable numbers consistently gathered by companies as part of their business as usual systems, so that confident conclusions can be drawn. Until then, studies like this are a good start, and I like their conclusions, a lot of their recommendations make sense, is aligned to the axiom and is what we in security continue to preach. The only problem is their own numbers don't back them up!

No comments:

Post a Comment


Written by