As with all New Year seasons, everyone chimes in with predictions for the year to come, along with retrospectives of what the previous year brought. While not strictly a seasonal occurrence, many such writings/articles/declarations/rants/etc. contain rather outlandish predictions and assertions which are meant to be shocking at worst, and visionary at best.
Recently, I’ve seen several pieces which fall into this category, and would like to toss my own two cents on the pile. The one that sort of kicked off my initiative to write this was a posting by Adriel Desautels which appeared on the Snosoft blog and on the pentesting mailing list. The post asserts that vulnerability scanners don’t work. The point is made that vulnerability scanning is not an effective tool because the core pieces fall on the tail end of vulnerability research and the scanners themselves aren’t accurate. On the accuracy part, the claim is made that his best case experience with scanning tools is 30% accuracy (that’s obviously a guesstimate, as no hard data is provided). Adriel’s conclusion is that the best replacement for a vulnerability scanner is a well-trained penetration testing team which conducts its own research.
Of course, as with many outlandish claims, I disagree. Going down the list, I’d have to say first that of course vulnerability scanners have a huge amount of value, and definitely have their place. His estimate of 30% accuracy I suspect to be completely made up, and will continue to until I see some sort of data to back it up. Also, no reference point is given on that number. Is it 30% of all vulnerabilities that ever have been or ever will be known about a system? What is 100% then? Is this counting false positives and negatives (or just one of them, or neither)?
My experience with Nessus, Saint, nCircle, and others has shown the whole class of tools to be hugely useful, so long as you look at them in the right context. What’s important to keep in mind is exactly the title of this post. All tools have their place, and all needs have a set of tools which address them best. Vulnerability scanners are generally useful in two roles. The first is in a one-time look at a network. I use Nessus whenever I assess a client’s network for the first time. That’s because in addition to pointing out vulnerabilities, it gathers tons of data. What’s best is that it does this automatically. I could use a collection of 20 or more tools to enumerate hosts and poll their various services for data, but that’s a waste of time when a vulnerability scanner will do all of it for me. While the vulnerability scanner is doing its thing, I’m free to do a walk of the premises, talk to staff, or take a nap. All of these are much better uses of my time than running individual tools manually. When the initial scan is done, I can then take closer looks at specific hosts and services with more specific tools if need be. My time at a client site is usually quite limited, so it’s important for me to make the best use of it.
The other case where a vulnerability scanner is very useful is where a whole network needs to be monitored for change. Because the vulnerability scanner is pretty consistent in what it does and very broad in what it covers, and because the whole process is automated, scans can be done in intervals very easily. Data sets can then be compared over time to show trends in vulnerability and give hard data about where the most vulnerability probably is. I want to key in on the fact that I say “probably” in the last sentence for a reason. Vulnerability scanners are plenty of useful, but what they aren’t is perfect. Adriel’s article is pretty much centered around the main weakness of any automated security tool, which is that they can’t see everything, and a good deal of security is in the soft/squishy part (the people and organization). Additionally, he makes the point of the lag time between vulnerability discovery and detection by scanner, but we’ll get back to that. Backing up a bit, the reason you can’t use a vulnerability scanner to identify where vulnerability in your network definitely is, is because of a lack of comprehensiveness of the tool, and the sheer complexity of vulnerability.
Vulnerability scanners can’t tell you that your policy is terrible or that the structure of your network is poor. They can’t tell you that Frank in accounting took home all of your company’s customer data, and they can’t even begin to detect your lack of visibility into the traffic flowing across your WAN. What they can do for you (and me), however, is catch low-hanging vulnerabilities and report them in an automated manner and allow us to use our time handling other tasks. As with all tools in all industries, you can’t expect a tool meant for one task to be the end-all, be-all solution to something as conceptually large and complex as “security.”
To address the issue of lag time between vulnerability discovery and detection by a vulnerability scanner, I believe this is a moot point. This lag exists in all tools and solutions in one form or another, and where one tool or solution might have a lower lag than others, it can’t be considered comprehensive. Specifically, Adriel suggests that teams of security professionals replace vulnerability scanners functionally in organizations. This is just plan silly for a variety of reasons. First is cost. an internal team of security professionals is simply out of the question financially for most organizations. Contracting out the service just once is also expensive compared to even the most expensive vulnerability scanners (Nessus goes for $1200/year/scanner, nCircle is upwards of $30k for the initial installation). Getting a team of specialists to do even the basics of what a vulnerability scanner can do is a waste.
However, sticking to the title of this article, I believe that all tools and solutions have their place and that there is a proper process to attack any problem. For security, you first need to perform gap analysis. In the early stages of reaching a secure state, tools like vulnerability scanners are extremely useful, because they allow you to efficiently identify and address the most numerous and obvious flaws. A security professional’s role here shouldn’t require much direct interaction with the systems at all. Automated tools will churn up enough information to get an idea of where things stand, and help to identify major problems. This is your typical Vulnerability Assessment; a broad process which should take a shallow look at the whole organization, identify major and core issues, and develop a plan for action. This sets the current state and outlines a goal state.
The middle parts of the process require different tools for different reasons. The vulnerability scanner is still very useful here for the tracking of remediation and detection of new minor issues, but gives way to higher-end planning and consulting which aims to set up controls which proactively secure the network, and do so in intelligently redundant layers. The latter parts of a security ramp-up are where the vulnerability scanner becomes a minor player in that it is used to catch smaller issues which fall through the more proactive steps put in place to take care of issues before they cause vulnerability. Also, this phase of the process is where penetration testing becomes relevant. As is stated in this rather controversial prediction by Brian Chess, penetration testing should be used as testing is in the scientific process. That is, penetration testing should be used to test a theory. The theory should be something along the lines of: “security control ABC should stand up to XYZ types of attack, and those attacks should trip some sort of alarm when a certain point is reached.” In other words, a security control is designed and put in place earlier in the process, then needs to be stress tested to prove that it is working as expected. This is exactly how we’ve approached penetration testing (especially since we make it very distinct from our vulnerability assessment service) since day one here at White Badger.
So to sum up, early in the security ramp-up process (where most organizations haven’t even started), automated tools like vulnerability scanners have a huge amount of value because they allow for a large amount of data to be collected and acted on in a very efficient manner. As the process goes on, the more automated tools give way to more specialized tools used to assess certain specific hosts/services/vulnerabilities. At the end of the process, penetration testing is performed, and that is almost entirely manual. The best (in my opinion, the best are the most realistic, regardless of scale) penetration tests will combine attack methods from all different angles to properly estimate the success rate of exploiting a vulnerability.
Vulnerability scanners aren’t worthless. That’s like saying that a table saw is worthless because all woodworking can be done with a screwdriver and a utility knife. The worth of a tool is directly proportional to its cost and benefit. Vulnerability scanners generally have a low cost relative to their closest alternatives, and a high payoff so long as the expectation is reasonable. They have their place in the security process and won’t be replaced functionally by an adjacent tool any time soon.
Static screen shot from an animated presentation illustrating Iran’s cyber security incompetence
A Cost-Based Analysis of User Effort in Security
This article does a fantastic job of quantifying the somewhat nebulous idea of why end users generally tend to make decisions about security that seem poor. Looking at the cost analysis comparing the price of end user time to the actual losses, it’s clear that many of the basic traditional wisdoms surrounding daily web usage are a huge hassle with little payoff.
In other words, the average user would spend far more time attempting to maintain strong passwords and check every URL and SSL certificate than he/she is likely to lose by failing to do so. The article goes much deeper into the math leading to those conclusions, but it’s not necessary to whip out a calculator to know he’s right.
In my opinion, this solidifies the fact that much of the burden of end user protection is shifted up the chain. While one user may only see a few pennies per year of realized risk, the organizations serving all of those end users will see substantially more loss, as it is all aggregated. By the same token, organizations serving end users are in a position to much more economically deploy countermeasures, and have access to more data which allows them to do so effectively.
The other angle on this is that the more transparent security controls become to end users, the more effective they’ll be. This paper covers directly the effort needed to validate SSL and read URLs. Other controls like virus scanning at the perimeter, IPS, and well-implemented least-access principal (while not without issues of their own) are considerably less likely to require end user effort to be effective, and therefore be a better answer in that regard than any amount of awareness training.