Matt Lovell, CTO, Pulsant
Over the past few weeks another vulnerability has emerged – Shellshock – that has had an impact on many users of the affected software including cloud providers. Essentially this vulnerability enables hackers to potentially take advantage of systems by adding in a few lines of their own code. The Shellshock vulnerability, specifically, affected the LINUX program Bash GNU which had implications for cloud providers like Pulsant whose servers ultimately needed patching.
The severity and need to mitigate the vulnerability as quickly as possible meant many providers had to engage on wholesale infrastructure security updates, which can affect customers with the need to recycle or reload. However, this exposed an array of different responses from vendors in terms of completing the updates and the impact to operational service. It raised an important question for customers to consider in terms of just how does your cloud provider respond and protect infrastructure services in the event of a potential critical vulnerability.
There is no question that identified systems needed to be updated as quickly as possible. Clearly scale and security management approaches differ between cloud providers and may determine reaction times, as well as the actual processes which are leveraged to complete the updates. The nature of the update may also necessitate a reboot or recycle of the host, which requires careful scheduling of downtime to manage systems availability. While system downtime for many may be required in order maintain optimal protection of the infrastructure, the application of Shellshock has exposed cloud supplier differences in security management processes.
The team at Pulsant mobilised quickly in response to the Shellshock vulnerability without any downtime for our customers although other approaches from other suppliers have involved system downtime.
Even with advanced warning, any downtime for companies can cause disruption, cost money and ultimately affect the relationship they have with their customers and with their cloud provider. This is something we consider very carefully because the services we provide – colocation, managed hosting and cloud – aren’t always delivered straight to the end-user. We host a number of businesses within our data centres that pass those services on to their customers.
When we were made aware of Shellshock and the impact it would have on us, once the full vendor update was available, the initial step was quantifying the update process and the effect on host systems. We reacted immediately by understanding exactly how best to mitigate the vulnerability in terms of the update process whilst minimising customer impact. Once quantified, we then engaged with our customers, keeping them informed every step of the way, from initial awareness, through to testing and deploying the patch to our servers. This took place within two hours of the official release of the security update. This provides the opportunity to engage with customers, particularly where a vulnerability may necessitate a system reboot to complete the update to discuss the individual scheduling of the update whilst considering the risk of the vulnerability. In the Shellshock instance, we identified this as a critical but non-disruptive vulnerability and, as a result, we were able to patch servers immediately after the patch was released and tested. The patches were completed over a period of 36 hours with no disruptions to customer sites and no downtime.
As with dealing with any event that has the potential to impact on your and your customers’ business, the two most important factors in this instance were speed of reaction and keeping our customers informed at all times. In this way, the situation was resolved quickly and effectively, and our customers were engaged and aware throughout.