What is HeartBleed?
As reported in the mass media this week, a large vulnerability in OpenSSL was discovered, named the Heartbleed Bug. You probably go on secure sites (with padlocks or HTTPS) many times a day – for instance Facebook or your bank, and these all use security certificates. OpenSSL is one of the most widely used implementations of this security encryption, and the vulnerability in theory allowed malicious parties to untangle this encryption by stealing the ‘key’.
The best way I have seen it illustrated is by the excellent XKCD comic, and we have also published an article with more information on our blog.
Vidahost is a managed service provider so we had patched our systems well before the vulnerability hit the mainstream news. To show you how, let me introduce you to our systems administration team, SysOps.
How does Vidahost’s SysOps work?
Our SysOps team is made up of a dozen experienced server administrators. They are not customer facing but spend their time making sure everything runs securely and efficiently. Above you can see a picture I snapped today.
The screen at the top is a live status feed of all servers. This monitors approximately 7000 services spread over thousands of servers, and alerts the team the second anything fails. On a shared system we typically monitor HTTP (web traffic), MySQL (databases), Exim, SMTP and POP (emails), as well as ancillary services such as FTP, backups and SSH. On a dedicated machine that becomes more custom, and we may be monitoring specific websites or pages depending on customer requirements. This means that if at 3am on a Sunday morning there is a database error on a single server, we will be working to fix it within seconds.
We have been criticised before for not keeping customers up to date when we are fixing an issue; we accept that and this is why we now have status.vidahost.com. However, we are absolutely brilliant at proactive and reactive monitoring; no matter how small a service issue we immediately work to fix it without customer prompting.
How did we patch against HeartBleed?
We became aware of Heartbleed from various sources once the vulnerability was announced. The first priority was establishing which of our systems might be vulnerable. It turned out to be a minority of our cPanel servers which were running RedHat CentOS 6 – slightly over 1000 web servers. We needed to patch these systems immediately, but to individually update each server clearly would be unfeasible given the urgency of the situation.
Instead, we have a way of updating all servers simultaneously. We are enthusiastic users of Puppet, the biggest (and we think best) configuration management utility available. It allows us to configure and manage servers from a central source, deploying updates to all machines. We were able to roll out a fix for OpenSSL with just a few lines of code: updating OpenSSL, and then restarting all affected services.
It is not every day a security vulnerability makes global news but we are proud that we can react so quickly.
This of course isn’t everything that SysOps do. Next newsletter I’m going to show you how we deploy racks of servers at a time; from the cabling to the server setups.