Why Security is Important for Developer Operations

After working at my current gig for about a year now (internship plus full-time), I wanted to take some time to outline my experiences transitioning to more of a developer operations role from my existing security background. The transition has been fairly straight-forward, as much of the work I do currently touches security in some aspect. However, the biggest point I've noticed is how much security considerations are involved when developing tools for a traditional developer operations requirement.

While I am a big proponent of at least some security training for everyone in a technical role, it remains to be seen just how much is "enough." Obviously, as the realm of technical fields continues to expand at its current pace, having every member of a technical team fully trained in the security of the applications they are developing is impossible. Yet it is also imperative that they at least understand the risks associated with these applications; doing so should be a requirement of a good developer.

As a "Developer Operations Engineer" (a role whose title is still rather undefined and unstandardized), I generally focus on several categories of projects: infrastructure development, monitoring and incident response, and deployments and the application lifecycle. While this is a highly compressed view of my role, each of these has a dizzying array of security concerns. While some companies offload much of these concerns to a security team, smaller companies need to remain vigilant of their impact.

In most modern startup environments, infrastructure development typically refers to everyone's favorite buzzword: the cloud. Amazon Web Services, Microsoft Azure, Google's App Engine, the list goes on. The security concerns associated with the cloud are not the focus of this post, but I do want to highlight places where security is especially important. Almost every one of these services has security enhancements that are not used as often as they should be. For example, AWS's Virtual Private Cloud is not only free, but can also greatly improve security when used properly. Yet quickly starting instances from EC2 still requires less hassle and so remains the more common choice. Another example is the use of security groups. AWS's security groups are infinitely customizable, yet simply opening a port to the world (0.0.0.0) is a tempting simpler option. While hosted infrastructure providers like Amazon and Google abstract a lot of security work away from the customer, good security practices still require active participation.

Monitoring and incident response is perhaps the area in which a lack of security can have the biggest impact. While many "DevOps" engineers view monitoring in terms of system performance, monitoring must also cover system security. Disk space, CPU utilization, and memory usage are all important indicators of a healthy system, but so too are login attempts, changes to file permissions, and unauthorized outgoing network connections. A good monitoring platform must also monitor for security events that could signal an intrusion or potential breach of security. In the same light, the response to a security event should also contain a viable plan for mitigating the same risk in the future.

Finally, the deployment of applications is a critical component of the security of an infrastructure. Because the main goal of deployments from a developer operation's standpoint is automation, any security bugs introduced once tend to be replicated. For this reason, it is imperative that the deployment process and any actors in it (Jenkins, AWS S3, etc.) are fully secured and audited often. Vulnerabilities that are present here can expand exponentially when deployments are pushed.

The role of developer operations or server engineering is rapidly changing and expanding. While it does, it is important, if not necessary, to include security in the expansion and ensure that those building a company's most critical technological parts are also trained in protecting them.

Everything is Moving to the Cloud... Even Hacking?

I promise that this won't be a buzzword-laced post about the benefits of cloud computing, the continuous move to store all our information online, or the pros and cons of relying on off-site storage. However, an interesting trend has been cropping up in terms of cloud computing and network security: the use of "the cloud" as a launching point for cyber attacks. One of the most notable incidents (and the one that inspired this post) was that of the cyber attack on Sony's Playstation Network. One of the big differences between that attack and more "conventional" hacks is that the attackers were able to harness the power of cloud computing to launch their attacks rather than relying on local servers or widespread botnets. I believe that this will quickly become a trend, if not the norm, for cyber attacks in the future. The anonymity of a cloud-based launching point, its ease-of-use, availability, power, and low costs combine to make using the cloud to launch a cyber attack not only feasible, but also tempting.

When Sony released information about the attack, one of the more noteworthy facts was that the attacked had been launched from Amazon's EC2 cloud computing infrastructure. Although hackers have previously used rented servers, this attack marks one of the most significant cases of late where a service such as Amazon's has been abused in such a manner. No longer are the masterminds behind cyber attacks required to purchase server space from shady third parties in unknown countries. Instead, they can use a legitimate service, at a fraction of the cost and with more power (Amazon's cloud is notoriously resilient).

In terms of anonymity, using cloud based services doesn't necessarily decrease your chances of detection, but it does add another layer. Now, law enforcement investigators will need to subpoena Amazon, search their records, find connecting computers, and trace from there. It's another step that only adds time and could possibly aid attackers. Renting a server and service from Amazon is as simple as signing up with a fake account and a fake credit card, something to which cyber criminals undoubtedly have easy access. In addition, using an Amazon server to launch an attack is like hiding behind a proxy without the obnoxious bandwidth reduction. Now, attacks can be launched at full speed, without being channeled through proxies. Only the commands sent to the servers need to be sent through proxies to obfuscate the identity of the attacker.

Amazon has an amazing infrastructure and you can be assured that hackers will continue to exploit it mercilessly. A question that needs to be asked, however, is how will Amazon protect against outbound attacks? They have demonstrated (for the most part) that they can secure their infrastructure from attack. But what happens when that infrastructure is doing the attacking itself rather than being attacked? Hopefully Amazon will be able to implement security that can prevent the abuse of its services.