Virtualization security - what is the fuss?

This is a bit of a companion piece to my Cloud computing security article. Just the other day I was amazed at the fear and trepidation that colleagues viewed putting everything on "one" machine i.e virtulization and the doom and gloom it meant for security. I mean seriously? Are we still here? Virtualization should now be viewed as a mature technology that can save you a lot of money and increase your scalability and speed to market. Again don't believe the FUD just do the basics well and you will be fine. Here is how....


The real question is why treat virtualized systems any different from physical systems. 95% of the right thing to do for security (oh yeah those things you never did anyway?) still apply in virtual systems. Yes there are a couple of extra security controls you may want to have in place but they are hardly going to be your highest priority from a risk and return perspective.

Stick to this mantra when it comes to virtulization security: apply the same best practice security principles - just treat the hypervisor and the guest machines as just another server/desktop. Your implementation steps and technologies maybe a bit different and some processes will be different but the principles do not change.

I am going to examine the basic security requirements you should have in place using a high level STRIDE threat model:

Spoofing - attacker is able to login under another users account to either the guest OS or the hypervisor gaining unauthorized access.

Authentication and authorization is still what is required to mitigate this. For the guest OS's just make sure they are part of the domain or however else you manage authentication and authorization and apply the group policies or other security policies for the server type i.e. domain controller is a still a domain controller, member server is still a member server etc etc. The Hypervisor also needs authentication and authorization if you are using VMWare 4 and above simply use Windows integrated authentication.

Tampering - attacker is able to modify a system or data file on the guest OS or the hypervisor, or while the information is in transit

Again authentication and authorization plays a large part in who is allowed to modify guest OS files and the hypervisor. You don't have to do anything special here, if they are part of the domain or treated just like a physical server the groups who are authorized to modify files should be exactly the same. God forbid you actually have some form of privileged access management tool e.g. no permanent domain admin or Root access, but actually have to checkout this access when it is needed, which is then logged and traceable - just apply the same controls. If you don't have this - well potentially the hypervisor is a big target so you may get some budget to improve overall - although I would argue it is the same risk as if you can get domain admin or Root access (you can still affect a large number of machines).

For transport - transport encryption is still the answer. Most guest OS will support ssh, kerberos etc for privileged credentials, so will most hypervisors, HTTPS, SFTP etc all still work fine, . No different to how you would protect the communications to a physical machine. In fact in a virtual world this can result in less of a performance impact because all the network communications are happening on the hypervisor backplane, so encrypting large and/or high volume bandwidth should have less of an impact and it is a lot easier to increase the CPU on guest machines to handle it even if it does.

Repudiation - attacker claims they did not perform an action

If we are talking about repudiating administrative actions I have already covered authentication, authorization and privileged access management. We can also add logging  - if you have a SIEM or a central log server simply point your new guest OS's at this, if you have Windows guest OS and/or spinning them up as needed make sure the default build and config applied contains the logging settings (e.g. from group policy or the jump start build for posix) and the agent (e.g. snare) or the external syslog settings. If you don't have this at least you will have the same local logs as you do on your physical machines. Treat the hypervisor like as Linux server and it will also have some application specific logs, so hopefully you will define some new monitoring and response actions for your security monitoring team (have one right?)

Information disclosure - information about the guest OS e.g. patch levels, the hypervisor are discovered by the attacker and/or the exception information reveals too much information. Also attacker gains access to sensitive information stored on Guest OS's or the Hypervisor.

The majority of the controls here come from the build and config both for the Guest OS's and the Hypervisor. If you are building servers or desktops you already have e.g. Windows 2008, Linux Redhat 5 you should have standard hardened builds - if you don't, use Google and checkout the vendor best practice and CIC benchmarks, luckily for you majority of OS are now becoming more secure by default - at least do the basics: turn off any services you don't need like the Apache/IIS web server, on Linux don't install any packages you don't need, change the default account passwords to strong complex ones, patch your systems regularly and do the authentication, authorization and logging steps.

For the hypervisor same steps - get the best practice config from the vendor, turn off services etc etc.

For the information stored on the guest OS and hypervisor - same controls as physical - encryption and good key management are always strong controls, you can install the same DLP controls, if you are using virtual desktops you have the option of not allowing things like USB/CD mapping removing that risk or installing the same removable media control agents

Denial of service - attackers are able to crash or reduce performance of guests OS, hypervisor, and/or virtual network.

The primary defence here is still your network controls as well as the build and config discussed above. You can get a virtual firewall with an IPS module (Cisco, Juniper all do them). Again virtual firewalls and network segmentation can work better due to virtual network operating on the backplane. Your physical network controls such as dual ISP's and ISP provided DDoS protection will still apply to the virtual servers you allow to be internet facing.

An additional concern is the degradation of all the virtual servers on running on the hypervisor by targeting one. You can best mitigate this via shares and resource limits. In fact it is easier I would suggest to configure guarantee and max CPU, memory, storage, network bandwidth on virtual guest OS than physical. With virtulization if you design your applications correctly (i.e. horizontally scalable) you also have the advantage of if you are in the middle of a DOS attack or legitimate business spike you can always easily bring extra horse power to the existing machines or simple spin up more machines to share the bandwidth.

Malware - same defences. I was reading that unless you have accounted for it in specing your hypervisor and guest OS resource shares and limits, you may have to be smart about scheduling your full AV scans (doing them on 3000 virtual desktops and servers at 3am may cause them all to come upto max utilization), same goes for the AV signature updates. So you may need to serialize these and spread them into chunks.

Elevation of privileged - attacker is able to gain admin or privileged account access to a guest OS or hypervisor.

Discussed this under tampering - two factor authentication, privileged access management, logging and monitoring, change controls - all standard stuff. Yes a lot more of the admin control is virtual (e.g. you don't have the controls of someone needing to physically go to the data center to add/remove memory to a guest OS - you can do this via the management console now so your logical controls do not get the safety net of some physical backup - as they well shouldn't.

Conclusion

Ultimately I don't believe that to virtualize or not to virtualize should be a security decision. It is a business decision (e.g. to reduce cost) and an enterprise architecture decision on roadmap and IT vision for the organization as well as best architecture to meet the business goals of performance, scalability, reliability, cost, ease of upgrade and management, capacity planning etc. We can and should be able to secure a physical or virtual IT world. If you don't have a lot of what I talked about in your current physical world here is your chance to improve the situation or just let the business get lower costs and better IT at least with no materially worse a security posture

8 comments:

  1. Like you, I see this a lot as well. Many a time I have heard that a company doesn't want to put a VMWare box in the DMZ but they are happy to put it on the internal network. I see no distinction between the two. This somehow implies that the DMZ is less secure which is just plain wrong. More than likely they do not have the proper security controls implemented in the DMZ. Like you say, if you adhere to a good security model then you can use virtualisation technology anywhere. The overall decision is not a technical one but a business one.

    ReplyDelete
  2. Agree - this is a relative mature technology now, the operational and cost benefits outweigh any potential risks introduced to the environmet.

    ReplyDelete
  3. I think you misunderstand some of the threats of the virtualized world and for that matter the cloud world. I was initially going to write up a reply to your cloud article (I will after this actually) but I'll address this first.
    I'll state up front that I agree with your statement that virtualization is a fairly mature technology at this point in time, and that the primary reasons around implementing it should be around business factors. That said security IS an important aspect. Let's take an example to make the point.
    If you have one large machine that you are running a number of VM's on top of, and let's say that you have some web server VM's, some DB vm's and some app VM's. Some of these VM's are internet facing, some of them are internal only (the app VM runs finance information), but that's all ok right because they are all in their appropriate VLAN's etc with the right network level access controls in place. You use all of the typical security controls that you outline, or that should in fact be the standard security controls. However, due to some vulnerability in your web server, someone is able to remotely drop to a shell. From that shell they then take advantage of another vuln that enables them to get control of the hypervisor. Now you can see the security problem, instead of having one individual component of your infrastructure compromised you've actually just lot the entire lot. From the hypervisor the attacker can jump onto any of the guests and from there leverage further their access into the rest of the network, not too mention have full access to anything that the guests are running.
    The above scenario is the number one concern around virtualized environments. ALL of your typical security controls rely on the attacker NOT being able to take control of the hypervisor or escape from the guest to the hypervisor. If that happens, and there are examples of vulnerabilities for all the major VM's, then all bets are off, as is your security.

    ReplyDelete
  4. strerror, actually I think you misunderstand. We understand the threats and risk, the article is about how people still refuse to avoid the risk rather than mitigate it. The example you give is full of "what ifs" and pretty ridiculous ones at that and you forgot to add that all the planets are in alignment.

    If "The above scenario is the number one concern around virtualized environments." then people should worry about

    If you have good controls such as ingress and egress filtering, strong authentication, SIEM then the bets are back on.

    ReplyDelete
  5. My example is a perfectly reasonable one that has happened many times and is routine in any hacking competition. All it takes is for a new vulnerability in a web server that grants sufficient access to the guest to initiate an attack to escape the guest. Now granted good security around these things mean that the window of opportunity would be small (you'd patch a critical vulnerability fast) but it still exists. The main point though is not that it can happen at any point in time, but the consequences if it DID happen in that kind of environment.
    I don't follow your statement "We understand the threats and risk, the article is about how people still refuse to avoid the risk rather than mitigate it.". What are you saying precisely, that people prefer to avoid doing virtualization at all because of security? That might be true in some cases, but mostly in my experience it's because people don't understand virtualization not because they are particularly worried about security. As for understanding the risks, that might be, but as Rakkhi's article missed what I outlined, I thought it worth including.

    ReplyDelete
  6. Hey Rakkhi, Charlie here. Just thought I comment on your blog... nice blog! :P

    Totally agree with you on virtualisation is a business decision although you can have too much of a good thing. Everything in one basket is never good so you just have to strike a balance between security, cost and functionality, like everything else in life.

    I'm surrounded by these VM things myself. I'm not too confident in virtual network segregation for mission critical stuff though, so I usually recommend have servers of the same risk profile residing on the same physical server(s). Of course, it all depends on $$, cost saving vs. risk taken. It's something only the business owner can decide.

    ReplyDelete
  7. Hey Ben sorry it’s taken a while for me to reply, busy with work and starting a new business and all :)

    Just wanted to rebut: “However, due to some vulnerability in your web server, someone is able to remotely drop to a shell. From that shell they then take advantage of another vuln that enables them to get control of the hypervisor."

    I agree the Hypervisor being the big target of opportunity is the main difference between physical and virtual from a security perspective.

    At a more abstract level what you are talking about is privilege escalation and I don't see how this is different for example to getting a domain admin account due to vulnerability in on an IIS server that is in a DMZ. This could is just as likely to happen and would give you full control over all the Windows servers physical or not. A Unix example is where an Apache server is compromised due to a vulnerability, the root account is obtained and due to NIS maps, rhosts, share auto mounts for SSH keys (I have seen all of these) you get access to a whole bunch more Unix servers and can leapfrog across the DMZ.

    I think people make a big deal of getting to the Hypervisor when actually the same could occur on physical servers. Also the point Mat is making is the attack tree and the probability/likelihood at each point:

    i.e.
    get access to a guest OS
    escalate privileges with vulnerability
    get access to the Hypervisor
    exploit a privilege on the Hypervisor that is sufficient to provide root level access
    do not get detected by any monitoring tools

    Also considering the impact: what are you storing on these virtual servers. The likelihood x impact may be a risk level your business is prepared to accept to gain the cost reductions and scalability benefits.

    I also disagree with: " ALL of your typical security controls rely on the attacker NOT being able to take control of the hypervisor or escape from the guest to the hypervisor."

    Some of your controls in virtualization rely on this but you can also have things such as privileged access control on the Hypervisor to let you know when anyone executes a privileged command, also monitoring controls. This means you are not entirely reliant on someone not breaking out of a guest OS because even if they do you have mitigating controls to detect them before they can do real damage

    ReplyDelete
  8. Perhaps an interesting read for the followers of these comments is:
    http://taviso.decsystem.org/virtsec.pdf

    You mention that you don't see what is different between a compromise in a physical and a virtual environment, whereas to me, that is precisely what we're talking about.
    Currently our separation extends more then just physically, ie we ensure that ONLY the type of data necessary for a particular function is present on a given box. In the virtualized world though, there are potentially a number of machines with a differing data present, and the issue I'm making is that a compromise of ONE of those virtual machines will, in the hands of a capable attacker, result in the compromise of all of that data. It's a much greater threat profile and a much greater breach IF, you were correct segregating things in the physical example.

    ReplyDelete

Author

Written by