Implementing email encryption: lessons learned

PGP Universal. 
Source Flickr. Creative Commons
I was involved in an email encryption implementation project towards the end of last year. Here are some of the lessons that I learned as well as some discussion around whether there is any point to email encryption in its current form today. Apologies in advance to my friends at PGP (now Symantec) but got to call it as I see it.

This is a companion lessons learned article to my ones on Removable media control and Data loss prevention. I have written this with quite a few technical PGP terms, I will explain some but get your background at pgp.com



Background

PGP web of trust. Source Flikr, creative commons
We chose to implement PGP - mainly because group already had a licence for it. Also our main driver was a regulatory requirement (no surprise there) and we had a short time (3 months) to get this done so no real time to do multiple proof of concepts and product evaluations.  Also to be honest there are really only two players in this space at an Enterprise level: PGP and Entrust. Although I recently found this site: https://epic.org/privacy/tools.html . Free lightweight encryption tools including for email, haven't had time to test any yet but would be interesting whether an opensource tool would offer a good alternative in an enterprise deployment.

Because we had a regulatory requirement to do both internal and external encryption we had to implement the desktop (endpoint) version of PGP and not just have a gateway. This basically means that the email is encrypted end to end from a users desktop to the recipients desktop both internal and external. The gateway model on the other hand simply encrypts the email once it leaves your network (thus no agent or software required on the endpoint).

It was a global deployment, covering three geographic regions (North America, Europe and Asia). The PGP architecture is fairly simple you have a PGP Universal server that is a software appliance - this is the heart of the operation it stores all the keys, all the policy, licence aspects and also acts as the place to submit keys and provides the webmail gateway (secure gmail type interface for external users without PGP), and connects to your SMTP gateways and LDAP.

PGP Universal can sit in-line i.e all email gets processed through it or to the side where selected email can be diverted to it for encryption (or simple use it for policy and key management and encrypt at the endpoint). All endpoint clients i.e. PGP desktop connects to the PGP Universal for licensing, policy and key management. The PGP desktop client provides desktop email encryption (i.e. end to end), whole disk (full hard disk encryption), Netshare (encrypted folders), virtual disk (encrypted volumes). There are also PGP command line, an SDK and PGP portable all which are their own software but can connect to the Universal servers.

We implemented PGP Universal 2.10 and desktop 9.10. To be fair 3.0 is a major, major update that is supposed to address some of the design and product weaknesses I talk about later but it just was not available yet. Desktop 10.0 again provides a number of bug fixes and features.

What we did right (the good)

Source Flikr. Creative Commons
Unfortunately unlike the removable media control project I was involved in, and even the DLP, I think we did more wrong that right with this, and to be fair as the main guy driving this forward I have to take a lot blame for that. I think I was the PM, the BA, the Business and Solution Architect, the Implementation Engineer, the Trainer and Lead support as well as IT security lead for this project at various times - what can I say I cared and it was a small team and hard time to get resources. We also had some fairly dodgy advice from PGP at times and first lesson learnt don't believe the solution architect  and implementation engineers believe the support guys when they say something cannot be done or is not supported.

Anyway it is a short list:
  • Have a clear idea of scope - as I said we needed to do both internal and external encryption
  • Clear business driver - regulation enough said
  • Clustered global servers - we had 3 primary servers internal and 2 gateway servers in the DMZ. This architecture provided good performance to local users and excellent resilience and redundancy
  • Use virtual servers - no reason why PGP Universal cannot be installed on VM's even in a DMZ to save money, scalability, resilience etc.
  • Intranet site for training and reference for users
  • Documentation for support - although I found out recently I may have assumed a bit too much implied knowledge in these docs
  • LDAP integration and silent install - this works quite nicely when you have a single LDAP. It basically means on first PGP Desktop install, PGP Universal authenticates the user against your LDAP and if that passes automatically generates the users key and registers them. The only user interaction required is for the user to enter their LDAP password first time.
  • FIPS compliance checks - not really sure what it does in practice but it didn't seem to break anything so why not enable it, in theory should provide better security
  • Pay for the Platinum support and some professional services hours - if you want to implement a PGP project without this good luck to you. You can always knock it down to Silver or Bronze in years 2 and beyond if it is stable

What was poor (the bad)

Source: Flikr. Creative Commons
These can be split into design (or product deficiencies) and implementation mistakes (these are the minor points really see later for the really bad ones):

Design
  • No LDAP integration  - for either the PGP Desktop user password (if you do not have SKM as a key mode: more on that later) and no SSO for admin users on PGP Universal no LDAP is just poor security practice for a critical device and makes the Joiners, Leavers, Movers (JLM) process and role based access more difficult than it needs to be
  • Key lookup and key distribution - there is no support for proxing the LDAP connection that is required for PGP to lookup keys automatically on servers that have this exposed i.e. if I am emailing job.blogs@statestreet.com and State Street have a PGP Universal server exposed allowing for query of their public keys PGP can look this up automatically. This is obviously of great benefit as it stops users having to manually exchange keys. Only problem is that for this to work PGP Universal needs a direct internet connection (or some really messy LDAP relay). This is fine for Gateway mode only where you have a PGP universal server only in the DMZ but where your endpoints are connecting to an internal PGP Universal server this does not work. I would actually recommend even if you want end to end encryption only put your PGP Universal servers in the Internet DMZ and have even your internal endpoints connecting to those
  • Blackberry PGP encryption - As soon as you start encrypting emails with PGP one of the biggest problems you will have is that your CEO cannot read that critical email on his Blackberry. PGP do have an answer to this in terms of their PGP Blackberry support package. What we found though with this: it is expensive, there is very little documentation and few people at PGP that know exactly how this works and how to implement it over the wire with BES server pushout (1- 2 BB's with Blackberry Desktop install works fine), and it is just clunky to send an an encrypted email on a BB (just too many user steps to enrol). If you have iPhones, Androids and/or Symbians good luck to you. Windows mobile should be ok because the software is more like PGP Desktop. 
  • Difficulty of external users with no PGP - they just have not cracked this nut yet. There are some good ideas on the roadmap though but sending encrypted emails to users who do not have PGP is still just not an elegant process. PDF messenger and web messenger are ok but not great, most of the time even if you send a external user an encrypted email successfully (and hell freezes over), they will simply respond back in clear text: encrypting the conversation easily and without any user interaction on both ends is what PGP have yet truly master and I bet someone like Google gets there first.

    Implementation
    • Allowing users to decide when to encrypt - I really should know better. Most people do not care at all about security. If you want security that works make it transparent or at least really, really convenient. Security works best is when you don't even know its there. We wrote the policy such that users would have press an "encrypt" button (which was a simple .Net addon for Outlook that toggled the sensitivity of the email to Confidential). The idea was that we didn't want to encrypt everything - people would just select their most sensitive information and encrypt that. The reality: hardly anyone actually did this. A much better approach is simply to write some policies like if certain words are contained, certain departments (e.g. HR, Legal, Finance, Corporate Strategy, Research and Development, Executive Management) to certain domains (e.g. all non Exec directors, all lawyers, all outsource HR companies) are automatically encrypted inside and out. Then put a process in place to manage the exceptions and the shouting :) . Further on I talk about a much more elegant approach with DLP if you have that.
    • Not having a non production environment - just due to cost and time really. But don't do it. Make sure you have a dev and test PGP Universal environment that mirrors production settings (not keys). This will mean you can easily test patches and changes prior to doing them in production. It is the basics really but how many security technology implementations lack this (because they are not production critical?)
    • Do not write your own Outlook addon - We wrote a button as I said simple .NET Microsoft Outlook addon that toggled the sensitivity setting in the email header to confidential and back to normal. We thought this would be really easy for users and provide a simple visual indicator that an email you were going to send out was encrypted or not. This was just a nightmare. It caused so many problems to various people and made debugging whether it was PGP's fault or ours very difficult. Just don't do it. With PGP Desktop 10 and above PGP have added the Outlook buttons back in, use these and then if it has a problem it is definitely PGP's fault (well they will still try to blame your users Outlook addon's and other integrating programs (e.g. CRM) but it is one less thing.

      The disastrous (the ugly)


      Design
      Source Flikr. Creative Commons
      • Server Key Mode (SKM) does not work with non gateway mode - read the background on PGP key modes here. Basically SKM is where the key is only stored on the PGP Universal server (this has the main advantages that it can be easily recovered if lost - e.g. laptop lost or stolen) and the user does not have to enter yet another password to use PGP. It is really designed for Gateway mode and we had conflicting advice on whether it would work for end to end. The solution architect said it work, the implementation engineer and support said it wouldn't. In testing we couldn't get it to work, then we found be changing a single setting (always encrypt to your key - this was redundant as PGP always does this anyway) it would work. The implementation engineer and his boss said it would then work. The short answer: it does not work. The longer answer: it does work but it gives you lots of problems like tools such as Netshare or Wholedisk do not work with it, and we had all sorts of weird bugs like blank normal emails (this may have been due to Outlook add-ons, our encrypt button or the key mode) Basically it is not worth it - suck it up with another password and choose SKCM as your key mode.
      • The joy of merging companies - of course what does PGP use as your unique identifier: your email address! How about if you get taken over and have to change your email domain. Not so much fun. In the end I think if you ever have to go through this just bite the bullet: nuke your PGP installation i.e. revoke all keys, delete all users, force them to re-install (after backing up clear text versions of any encrypted emails they wanted to keep) and start again with re-registering and generating a key with the new email address. This will be hard pill to swallow but just tie it into integration activities and mean that the encryption will just work. Mucking around with getting PGP to read the new email address from the LDAP is not worth the headache and will lead to lots of frustration where PGP cannot find the key of the user you are trying to email even though they are registered as you are using their new email address and PGP is registered with their old
      • Joy continued: multiple LDAP directories - again if you chose the silent install option and need to integrate users in a separate LDAP (even with a full two way trust relationship e.g. AD Forest) don't do it. PGP Universal just struggles with multiple LDAP's, just setup a virtual directory and point PGP at that. Setup any groups that you are using for assigning policy in this virtual directory also, it will make your life easier
      • Only ever ever do gateway mode install - do not do the PGP Desktop endpoint install unless someone like a regulator holds a gun to your head. It is just a nightmare: because it ties in so closely to the Outlook DLL you get all sorts of problems like Outlook crashes, blank emails, Outlook normal emails just not sending, if PGP does not load before Outlook those emails you thought you were sending encrypted can actually go out in clear. The endpoint software is just a nightmare. Unfortunately if you want end to end encryption, Wholedisk, Netshare or Virtual disk you have no choice. Stick to gateway mode if you can, its simple just stick a few PGP Universal servers in your DMZ, configure a few rules in Exchange and you are set: simple, transparent encryption with no user involvement and no hassle. For your Wholedisk and Netshare encryption I would recommend True crypt: it is open source thus free and brilliant. Sure you do not get the full central policy control and central key management but it is a tight trade-off in my view. If you want end to end encryption enable Outlook to Exchange TLS to cover your internal and just use Gateway.

        Is email encryption relevant today?

         Source Flickr. Creative Commons
        Slightly leading title because I would argue no. The reason is mainly the arguments I make regarding transport encryption. Basically if you are worried about interception of sensitive email don't, it is just not that big a risk. Have a look on datalossdb for incidents related to email. Even the large number records most of them are theoretical incidents:  the largest records "lost" 28,000 involves an unencrypted email being sent out but no actual costs are reported. The likelihood that it was actually intercepted is very small. If you are still worried (or need to show something to your regulator) turn on TLS between Outlook and Exchange servers (will just work with Outlook cert no need for client certs), and mandate TLS between all your email partners (or at least all the important ones). Majority over Mail Transfer Agents (MTA) now on the Internet support opportunistic TLS (even Google) which means that a lot of the email now on the Internet has transport encryption.

        Also consider the for more likely attack trees:
        • User sends right data to to the wrong person (e.g. client data A to client B)
        • User sends the wrong data to the right person (e.g. the internal HR database to the database supplier to fix a database problem)
        • User sends an email blast to everyone (e.g. reply to all mistake or disgruntled user before they leave the company) and this goes viral
        • User sends email that breaches company policy (e.g. sexist, racist, porn) internal or external users
        • User exploits open SMTP relay to spoof email (this is a great trick if your company uses email approvals - pay increase or access to the HR database anyone?)
        Email encryption doesn't really do much for you each of these, mainly because if you are not a registered user the first thing the system does is allow you to enrol (first two scenarios). Third one encryption does nothing at all, forth one nothing, fifth one signing of emails can work if people pay attention to error messages.

        Remember email encryption is not DLP software. 

        So what should you do? Implement DLP endpoint (read some lessons learned on that here) and implement a gateway email encryption server that you can direct certain traffic to for encryption / decryption. How DLP endpoint assists to reduce the risk of these scenarios:
        • User sends right data to to the wrong person (e.g. client data A to client B) - DLP policy can be setup to detect who should get what data and also to at least warn the user that this is going external and they should double check the attachment
        • User sends the wrong data to the right person (e.g. the internal HR database to the database supplier)  - certain data like credit card numbers, government identifiers, PII can just be blocked from going external or at least require the user to enter a reason that you can follow-up
        • Disgruntled user sends an email blast to everyone and this goes viral - You can alert the user that they are sending to all or replying  above a certain number of users (e.g. 10) or you can stop users who are on the "leavers" list (obtained as a real time feed from HR database) from sending external emails or emails to above 3 people
        • User sends email that breaches company policy (e.g. sexist, racist, porn) internal or external users - DLP can just block these or warn the user and require a reason
        • User exploits open SMTP relay to spoof email - just don't have open SMTP relays ;) also sign emails and block emails internally that fail the signature verification
        This is one of the main reasons that I have great hope for the Symantec purchase of PGP. PGP Universal linked to Vontu DLP could be a really valuable tool

        4 comments:

        1. It's great to see an honest and frank outlook on such a relevant technology and critical business project. We talk too much sometimes about the technical reasons why something worked or didn't work and not enough about the business reasons. Your points regarding what encryption doesn't protect against are frankly more frightening that not implementing encryption. As an industry I don't think we review completed projects enough and analyse why things happened the way they did. We just move on to the next project and learn little from our mistakes.

          ReplyDelete
        2. Your post got me thinking about implementations and why is PGP even relevant:

          http://bit.ly/9YTh6q

          To me the email portion isn't, but there are good things that come with PGP/GPG. Your post did a wonderful job of laying out the admin perspective. Going in the delicious feed for sure

          ReplyDelete
        3. I find it interesting that you say people should not be worried about someone intercepting their messages. [playing a little loose with the term intercepting here] I remember a short time ago there was a huge issue with Gmail being hacked. That is, people who had their email stored on Google's servers had that email read by people who should not have been reading it.

          I see this has the big threat (granted it's less of an issue if you host your own email server) - and the number one reason to use email encryption.

          I encrypt everywhere I can. I use TrulyMail (because it's free) but PGP and GPG work just fine for many. I think recommending people don't worry about someone else reading their messages is like recommending everyone write postcards instead of letters sealed in envelopes.

          You should do something if you want to ensure your privacy.

          ReplyDelete
        4. Very nice post about email encryption. Our biggest problem in using encrypted data exchange was the lack of security understanding with our clients. They would not install any PGP/GPG clients. We started using cloudsafe.com now which offers pub/sec key encryption as a cloud service.

          Steve

          ReplyDelete

        Author

        Written by