This article on physorg.com was posted a little while ago and gives an excellent example of how important it is to stay paranoid (or develop a healthy sense of it). In short, a photograph of physical keys can be used to duplicate them. All that is required is a shot showing all of the details, which can be snapped at quite a distance with the right lens. While it definitely is more of a threat to residential-grade stuff than Medeco government/military-grade locks, it's a great example that threats are everywhere. It's important to always assume every part of your organization is vulnerable to some extent, and to plan for it. Doing so will ensure that you are considerably more prepared for the time when (not if) something fails.
Recently in Security Articles Category
You might have noticed our new web site and its central flash animation. At the conclusion of each round of frames, we declare the following:
Compliance is just the beginning!
Know your enemy. Know your weaknesses. Have a plan.
Behind these seemingly simple statements lies a lot of thought. Firstly, we talk about compliance. By definition, compliance means that you comply with standards. These standards are set up as a bare minimum operating level for any given industry so that all the players meet some standard set of rules and can work together based on them. With cars, it's state inspection. With food preparation, it's health inspection. With any structure, it's building code. In every one of these, the bare minimum is almost always just that, and the gross majority strive to be better. If your car only barely passes inspection, it's likely not very safe or efficient. If your food was cooked in a kitchen that got the lowest allowable score on a health inspection, there's a good chance that you'll be sick in the near future. If your house only meets building code minimums, it likely won't hold up very well in a wind storm.
So, given that compliance is just the bare minimum, and that the bare minimum is not something you should be aiming for, why is it that so much effort is spent in the financial industry on being compliant? Almost all of the security breaches in recent memory and likely in to the future have been and will be at organizations compliant with security requirements. Compliance is a minimum, and the minimum is never good enough when you're dealing with other people's money. Striving for compliance is like trying to come in last place.
Real security should be approached just like all other parts of the business. You need to have a metric, you need to measure it, and you need to manage it. In security, the metric is risk, and it is measured against cost and the risk mitigated. That's the theory anyways. In reality, it's so much more than just cost vs reward. Something we try and make clear to our customers is that there is a balanced security level for every organization, system, and situation. It is reached when security reaches a level where it complements all other parts of the business and is maintainable.
In the end, your goal should be security, not compliance. Compliance is a byproduct of good security practices and good corporate stewardship.
Many months ago, we started looking around for a platform on which to build our new Persistence™ service. We were looking for something that had scanning appliances which connect back over a secure tunnel (SSL/SSH, certificate-based) and have no listening ports. With this scheme, there would be no listening ports, and no obvious way to attack the boxes we deploy to client networks. These appliances would also need to have multiple interfaces (VLAN and physical), because the appliances aren't exactly cheap, and I think it's just dumb to have redundant hardware sucking electricity where it's not necessary.
So, as many in the IT and security fields know, it's bad practice to have any one device touching more than one network. The exceptions to this rule are usually security devices like firewalls and IDS sensors, all of which need to be specially hardened as any one vulnerability would go against the point of having multiple networks to separate groups of devices. In the process of evaluating devices from all the top vendors in the arena, I set up a test network on which to run each of the platforms. As part of this testing, I wanted to be sure that straddling a firewall and connecting to several network segments wouldn't be an issue. As it turns out, one of the boxes I tested had an issue. (As I'm under NDAs with pretty much every company I dealt with, I'm not going to comment on which product had the issue. I have been told that a patch is on its way and will hit in several weeks.)
The device in which I found an issue did meet all of my criteria. It had almost zero footprint on the network. It communicated only in encrypted tunnels, which were outbound from the appliance only. It had no listening ports. However, what I did find is that the device responded to ICMP pings (standard echo request/reply). I can see where this functionality would be useful for the end users, but from the standpoint of super-hardening, it should be disabled, or at least have an option to be disabled.
The specific flaw I found was that the OS's kernel didn't perform proper source checking on packets. For example, if interface 1 is on 10.1.1.0/24, and interface 2 is on 10.2.2.0/24, it should drop packets claiming to come from 10.2.2.2 which arrive on interface 1, and vice versa. Failing to do this, the device I was testing happily replies back out the other interface. Spoofing packets from both directions allows us to set up an ICMP tunnel, over which we can move just about anything. Unless the backbone is set up to detect excessive ICMP traffic (monitor port + IDS, or switch-based IDS), or if the traffic needs to pass over standard security devices (an upstream firewall/IDS/IPS/router), this would be pretty difficult to track down. Most organizations I've been in do not have the infrastructure to watch for this kind of tunneling when it is used specifically to bypass the firewall.
This attack isn't exactly l33t or hardcore, it's quite basic. It's unfortunate that some of the basics do get missed when most attention is paid to higher level attacks and other threats, but it does happen quite often. Fortunately, the vendor had an excellent response in terms of time and quality, and hopefully will give me many discounts in the future for doing some QA work for them ;-).
Paranoia
Every good security professional has a healthy respect for the unknown, which usually is tagged as paranoia. An appropriate amount of paranoia in the right area leads to being quite effective in securing the network you're in charge of. The key is education. This doesn't mean that you need a degree in watching over your shoulder, but you should know what to be afraid of, and how to make sure you know if/when trouble is happening.
Listening to a recent Pauldotcom podcast, the point was made that most people don't really get serious about securing their code or systems until they've personally been attacked. The other side of this is that actually doing some of the attacking and seeing it work firsthand usually has the same effect. As such, it's important that all IT professionals get that experience some time in their careers.
I find it very interesting how human psychology impacts the information security industry. To me, it seems very straightforward. You have something of value, bad people want it, you need to protect yourself. There's nothing outlandish or new there, that concept has existed since the beginning of life. But somehow, despite our advanced intelligence and civilized society, we all seem to have a default naiveté when it comes to information and IT security. I suppose this effect also occurs elsewhere, such as when a person doesn't regularly wear a seat belt until he/she has been in a serious car accident.
Given the above, it's not difficult to understand why FUD-style (fear, uncertainty, doubt) marketing is necessary and does work. Placing someone in a scared mode is an incredibly effective method for switching them from actively opposing security to actively embracing it.
To be an effective information security professional, what you need is a balanced, aware, and fact-driven sense of paranoia. A proper balance is struck when your paranoia drives you have a constant need for security while not blinding you to the reality of your environment. As such, you need to be incredibly aware of the organization you are working within. While security is important, it needs to coexist with functionality, usability, and visibility, while also making the political, budgetary, and regulatory parts of the equation work. Because paranoia is a 100% emotional response, it's important to keep it properly in check. All decisions need to be passed through a filter of reasonability and facts, so that you don't spend all of your resources defending unevenly against the threat that scares you the most.
I think all good security professionals possess the right kind of paranoia that drives them to stay effective and not become stagnant or complacent. I personally think every IT professional would do well to acquire a bit of the paranoia. While this usually comes as a result of a negative experience, it can also be had by way of a class with hands-on hacking. However you get it, your effectiveness and career will benefit greatly, as will the security of everything you touch.
Compartmentalization
Anyone who knows me will be able to attest to the fact that I have at least a touch of the OCD. Most of those people will probably tell you it's more than just a touch. In any case, I like things to be neat and orderly. When looking at networks and security, this usually translates into splitting up an otherwise monolithic network into smaller, more manageable chunks.
If you look at the design of a submarine, the vessel is divided into many different compartments, usually grouping like functions into a single physical space. Between all of these sections there is a thick barrier with heavy doors which can be locked down at a moment's notice. Why is that? Well, if they spring a leak somewhere, the Navy prefers that the entire crew doesn't drown.
The same principal should be applied to most business networks. Splitting things up along the lines of security, functionality, and physical location can yield a much more secure and manageable network. If it's done right, it will be transparent to everyday business, but be a serious barrier to anyone attacking the network. It will also make monitoring of network traffic much easier, as all traffic traveling between segments will be traveling across a device which should be able to do some accounting and reporting.
In my personal experience, I've seen absolutely enormous (>6000 hosts) networks set up as a single, switched subnet. The more average case usually involves 100 or so devices total, and sometimes multiple locations. Whatever the size or configuration of the network (aside from the really small <10 devices networks, of course), there is usually some split that can be made to improve it. At the very least, I push customers to move administrative interfaces of all devices that have them to a different network. Every network I've ever run in to has at least one device with a web or telnet administrative interface on its internal network. Most IT managers never think that the secretary or the accounting guy will ever want to or be able to do anything with those interfaces. The issue is, that if an attacker manages to get inside, those administrative interfaces are up for grabs along with everything else. Furthermore, if someone on the inside stumbles across one which isn't properly secured, malice isn't required to cause some serious down time.
When performing a vulnerability assessment on your organization and network, it is very typical to consider only the inbound attacks (even those coming from the inside). However, it is critical to consider what would happen should an attack be successful. What if every other layer of defense failed to block or even notify you of an attack. What keeps an attacker from moving around within your network and further compromising your organization once on the inside? Well, a properly segmented network can help there. If traffic between segments of your network are limited only to what is needed, it is considerably less likely that an attacker will be able to attack or move any data between segments. Furthermore, if all segments connect only through a firewall with an IDS/IPS, there is a much higher chance of catching the attacker traversing segments.
The down side of segmentation is that it requires a fair amount of work up front, and it forces you to know everything about your network. The latter isn't really a down side as I see it. But it does mean extra work, and extra work is generally considered a bad thing in the IT world. As network devices go, midrange UTM-type firewalls with 8 interfaces or more aren't very expensive ($3000-4000) when compared to other network devices, but they will provide you with more visibility and security than pretty much anything else.
In pretty much every assessment I've done, and a good deal in day-to-day life, I see a disparity between physical security and information security. While I could ramble endlessly about monolithic networks and their evils (especially the irony in locking servers in a secure room, while leaving access to the same network open in a totally unsecured room), I'm going to talk today about cameras.
Closed Circuit TV (CCTV) systems installed in most banks and other facilities are there to catch thieves with guns and ski masks. They can and do serve other purposes, like accounting for everyone entering/exiting a building, and watching accident-prone areas, but for the most part, they're only there to try and get a shot of a robber's face good enough to put on the evening news.
In this day and age, while there is still a significant amount of robbery at gunpoint, there are much more costly thefts and intrusions that need to be watched for by CCTV systems. I can't tell you how many times I've been in some financial institution (as a customer, not doing work for them) and been left alone in a room with a network-connected PC with full access to the rear of the machine. Furthermore, because of the camera coverage in the building being geared heavily towards the lobby, I was unwatched.
Given this level of access, one could easily use a pocketable USB device (a USB hacksaw, for example) that would steal credentials and leave an agent capable of stealing even more data, all without the victim organization knowing.
According to this article about bank robbery, the average bank robbery costs about $25,000 when it is all said and done (including turnover, lost time, etc.). The average ID theft usually ends up costing between $90 and $305 per record, according to this page. This can vary wildly based on how high- or low-profile the incident is. If we were wanting to make it an average, let's say that 5,000 records were lost (that's a very conservative size for a small bank or credit union). That would be between $450,000 and $1,525,000 in total cost for that breach. A bit of a difference, huh? Now, imagine the public backlash at an organization that just "let" someone walk in and take data, versus the relative empathy the organization would receive because of losses due to gun-wielding robber.
So, given the absolutely enormous difference in the financial damage able to be done between the two attacks, why do financial organizations not seem to take the more costly attack seriously? Well, the big reason is lack of knowledge. Despite cybercrime and hacking being buzzwords in today's society, most of the defensive effort goes into antivirus and perimeter security. My experience in performing security assessments for these organizations says that they rarely have a thorough and consistent approach to security. A secondary reason is inertia. It's the same reason that banks still have large vaults, despite their most valuable items being in the server room. It's just that financial institutions have a long history of needing physical security to stop intruders with guns, jackhammers, torches, and crowbars, but have relatively little history in dealing with thieves bent on stealing data. Displaying a large number of cameras in the front lobby can often deter an attacker before anything happens.
Given all of the above, what should your cameras be looking at? Well, here's a quick list:
- All major entrances to the building. You need to be able to account for everyone entering and exiting your facility in order to narrow a suspect list given a breach.
- Everywhere that customers normally go. This is not limited to the lobby. There should be camera coverage of all side offices, conference rooms, and other places customers are taken regularly. Furthermore, once these places are defined, customers should be kept out of all other areas.
- Areas surrounding the building. For attacks that happen wirelessly (and shame on any financial institution employing wireless to begin with), attackers are likely to sit in parking lots or on the sides of streets to do so.
- Last, but not least, areas with servers and infrastructure devices should be covered. These devices are the core of everything you do, and having physical access grants you a ton of opportunity to steal data.
Additionally, physical considerations need to be made for all computers and network jacks in the areas where customers are allowed. Computers need to be hidden or turned away from places where customers sit, preventing them from accessing the ports where USB devices or keyloggers could be installed. Network jacks should be completely disabled (unplugged from the switch, not just disabled in configuration), or placed on a separate switch for a guest network.
Of course, many of the attacks mentioned above (except the ones with guns, naturally) can be mitigated using good policy and configuration. For example, the USB hacksaw attack is ineffective if autoplay is disabled. It is also considerably less effective if the account currently logged in is running under limited permissions. USB-based attacks are even less effective if the ports are disabled altogether. Other, more common-sense solutions for these problems include simply not allowing customers to be alone in offices. In my experience, the person who was assisting me kept having to leave in order to go to the copier/printer which was located down the hall.
Visibility
You very simply can't fix what you can't see. This is something that nearly everyone in the IT field struggles with at some point. This is largely due to the lack of proper tools being used to manage the network (see part 1 about consistency). This is also usually the fault of network structure.
Data flows around a network can be difficult to observe and control except on the edges, where there are devices like firewalls and routers which are made for this exact purpose. Monolithic (or flat) networks are particularly bad in this case because of the lack of borders between parts of the network which should be separate. When dealing with such a network, you only have a few options if you are interested in watching the traffic flowing internally. The first is to have a large number of span/monitor/mirror ports which copy the traffic from the ports to which your network devices are connected. Unfortunately, this breaks down rather quickly when the number of used ports and the bandwidth used gets to be too high. The second option is to use Cisco's NetFlow (or similar). Of the networks I've seen, many do run Cisco switches, so this is a viable option. For those running lesser switches, you're pretty much stuck with mirroring ports, or with nothing. Of course, there is always the option of installing a physical tap for every single port, but that's pretty well unreasonable for the average organization.
In my opinion, the best way to increase security and visibility simultaneously is to separate your monolithic networks into smaller, more manageable subnets. Generally, these subnest should be cut along the lines of security, physical, and logical lines. Which mix of these will work best for your organization is something you'll need to consult a security professional to find out. In any case, once the network has been divided, the only way data should flow between any two subnets is through a firewall. Most modern firewall appliances come with the tools necessary to watch and manage the data flowing through them. Some of the better ones will provide tools for visualization and reporting to make understanding these flows easier and more efficient. Honestly, I can't get enough when it comes to tools which crunch otherwise boring data into neat little pictures, so long as detail is available when I want it.
Another incredibly helpful pile of data that your network generates is log data. The problem is that logs are tedious and boring. That's why there are log analyzers like Splunk which give you a nice web interface and search/index functionality which helps turn your log data into something useful. One of the best uses I've heard of for a log analyzer is anomaly reporting. Basically, you run your network for a while and let the analyzer collect log entries. After enough has been accumulated, you go through and mark all the normal data as being so. Once you have tagged the data which you expect to see, you can very easily filter for what you don't expect. This effectively sifts out errors and other unexpected messages from a sea of information.
Of course, doing any of the above assumes that you have time in the day to make it happen. Unfortunately, it seems to be incompatible with IT departments which work behind the curve of break/fix.
Consistency
Most organizations don't start out with the resources or immediate need to properly roll out imaged workstations or manage their networks using high-level tools. What happens more often than not is that the network is a collection of very different PCs, all of which have been built and configured individually. As a result, there are as many variations in loadout as there are workstations. As the organization grows, this practice is scaled to meet the need, and results in an IT department which works mainly in fire-extinguishing mode.
There is a certain point at which more consistent control over the network is an absolute must. The problem is that, in my experience, many medium-sized networks are being run without the proper tools. Management tools, strict domain policy, and imaging are all incredibly useful when combatting sprawl and growing pains.
Consistent network management not only makes IT staff more productive, but also allows for self-auditing and quick response to security events. Patching becomes easier (and can be more comprehensive than just WSUS or Windows Update), which allows the very common issue of third-party software vulnerabilities to be addressed in a more elegant and effective way.
Let's take a look at the basic functions that these tools need to help accomplish in order to be useful. The first is enumeration, cataloging, and inventory control. Basically, you need to know what each device is, does, and has on it. By the same token, you need to know what is on your network that shouldn't be. In addition to a simple understanding of the devices on your network, a good management system will keep track of when devices were purchased, what kind of warranty/service plan they have, and what their serial numbers are. Having all of this information in one place is the first step towards getting your network under control to the point where securing it becomes an easy task.
The second tool in your arsenal should be a package/file management tool. In order to properly manage software updates, it's important to know the loadout of each machine, to be able to verify that ONLY the software you approve is installed, to be able to perform updates to your software, and to be able to remove unauthorized installs. Many of the issues I see in security assessments come down to good patch management. I'd estimate that at least half of the vulnerabilities are a result of old software which has updates available. For those thinking that Windows Update or WSUS will cover all of these issues, you are sorely mistaken. Many of the vulnerabilities I find in assessments aren't in Microsoft products, simply because they do get automatically updated. More often than not, it's old antivirus or backup software which runs at system privileges that provides a way in.
Overall, being consistent usually comes down to putting a bit of initial effort in to a structure, then being disciplined enough to follow it. Automation of the boring, repetitive tasks makes this worlds easier. Manually going to every device in your organization and checking for updates is very tedious and gets frustrating rather quickly for the high-IQ types who usually make up an IT department. This leads to mistakes being made and eventually to the practice being missed or avoided.
Find some good tools, become intimately familiar with their innerworkings and behavior, then use them as an extension of yourself in confidently controlling your network. Get things to be consistent across the organization, then you can begin to manage one set of common machines instead of many individuals.
In doing assessments for various organizations (many of which are in the financial sector), there are a number of common issues that come up. Flat networks, lack of good patching practice, and vulnerable (and unfixable) legacy systems almost always make an appearance, but usually it boils down to a few basic problems. In a series of weekly posts, I will go over these basic issues I see again and again.
In the first part of this article, I wrote about how the term "Penetration Test" is very much misused and has had its meaning distorted by a variety of different players in the industry. So, how do I answer the title of this article?
I'll use, as a base, the definition that the FFIEC publishes:
Penetration Tests. A penetration test subjects a system to the real-world attacks selected and conducted by the testing personnel. The benefit of a penetration test is that it identifies the extent to which a system can be compromised before the attack is identified and assesses the response mechanism's effectiveness. Because a penetration test seldom is a comprehensive test of the system's security, it should be combined with other monitoring to validate the effectiveness of the security process.
- FFIEC Information Security IT Examination Handbook, p. 89
The key points that make a penetration test are as follows:
- The testing involves actually exploiting the found vulnerabilities. The point of performing a penetration test is to know for certain what an attacker can definitely do. It is not, on the other hand, supposed to guarantee what an attacker cannot do, only what might be difficult to pull off given a limited time scope. If exploiting found vulnerabilities is not part of the assessment, it cannot honestly be called a penetration test.
- The test is performed manually. While some automated tools may be used in the initial scanning and cleanup, the exploitation should be performed, step-by-step by the tester. Compromising a system can put it in a fragile state, which no automated scanning software can handle properly. Furthermore, stealth is usually part of the tester's goal, and automated tools tend to generate more noise than what is acceptable.
- Penetration testing is not comprehensive. No testing is 100% comprehensive, but penetration testing is usually set up in a way that limits the scope and kinds of attacks that can be performed. Also, time limitations restrict the number of systems that can be thoroughly attacked, as well as the types of attacks (i.e. brute force attacks can take weeks, which usually fall outside the scope of a normal engagement).
- The report generated at the end of the engagement should be written by a person. Only a person can properly express in writing the potential impact of the findings given your organization. The report should include lots of technical details, which will likely be generated by the software that was used, but the main part of the report should be written by the tester him/herself.
What we've seen being passed off as "penetration tests" do not come even close to qualifying for any of the above statements. What usually is being sold is actually a vulnerability scan, which is effectively an automated scan of your network. The reports are generated by the software being used, and delivered without any human interaction. While this service has its place, it is certainly not a penetration test, and considerably less useful than a well-done vulnerability assessment. Unless your IT department has a deep understanding of the types, potential impact, and probable false-positive nature of the results, a vulnerability scan is nearly worthless. Their best role is in keeping up with new issues over time. Since such services are usually cheap, running them on a regular basis shows change and verifies remediation efforts. Once a baseline has been established with a vulnerability assessment, using vulnerability scanning to monitor progress and new issues can be very effective, but starting with a vulnerability scan can hurt more than help.
When planning a proper penetration test, the very first thing that needs to be decided is what scenario should be played out. This should include three distinct parts that get specified before any other planning is done. The first is origin. This defines where the attacker is coming from. Usually, this is broken down to inside vs outside, and with vs without privileges. Using a combination of these four options, the penetration test can be used to simulate a highly privileged insider, a totally independent outsider, or anything in between. The next part is scope. The scope should cover not only the systems that should be part of the test, but also the types of attack, times of day, and people involved. The last part is the goal. Different attackers have different goals once they have broken in to your systems. Usually, this can be narrowed down to system control (and resource utilization), information extrusion, and service disruption. The goal can also be very much more specific, such as performing some action once a system has been compromised.
In general, penetration tests are very expensive and only appropriate once an organization has remediated known issues and desires to test critical systems under real-world attack scenarios. Until that point, there should be a cycle of vulnerability assessment and remediation.
