Why Security is Important for Developer Operations

After working at my current gig for about a year now (internship plus full-time), I wanted to take some time to outline my experiences transitioning to more of a developer operations role from my existing security background. The transition has been fairly straight-forward, as much of the work I do currently touches security in some aspect. However, the biggest point I've noticed is how much security considerations are involved when developing tools for a traditional developer operations requirement.

While I am a big proponent of at least some security training for everyone in a technical role, it remains to be seen just how much is "enough." Obviously, as the realm of technical fields continues to expand at its current pace, having every member of a technical team fully trained in the security of the applications they are developing is impossible. Yet it is also imperative that they at least understand the risks associated with these applications; doing so should be a requirement of a good developer.

As a "Developer Operations Engineer" (a role whose title is still rather undefined and unstandardized), I generally focus on several categories of projects: infrastructure development, monitoring and incident response, and deployments and the application lifecycle. While this is a highly compressed view of my role, each of these has a dizzying array of security concerns. While some companies offload much of these concerns to a security team, smaller companies need to remain vigilant of their impact.

In most modern startup environments, infrastructure development typically refers to everyone's favorite buzzword: the cloud. Amazon Web Services, Microsoft Azure, Google's App Engine, the list goes on. The security concerns associated with the cloud are not the focus of this post, but I do want to highlight places where security is especially important. Almost every one of these services has security enhancements that are not used as often as they should be. For example, AWS's Virtual Private Cloud is not only free, but can also greatly improve security when used properly. Yet quickly starting instances from EC2 still requires less hassle and so remains the more common choice. Another example is the use of security groups. AWS's security groups are infinitely customizable, yet simply opening a port to the world (0.0.0.0) is a tempting simpler option. While hosted infrastructure providers like Amazon and Google abstract a lot of security work away from the customer, good security practices still require active participation.

Monitoring and incident response is perhaps the area in which a lack of security can have the biggest impact. While many "DevOps" engineers view monitoring in terms of system performance, monitoring must also cover system security. Disk space, CPU utilization, and memory usage are all important indicators of a healthy system, but so too are login attempts, changes to file permissions, and unauthorized outgoing network connections. A good monitoring platform must also monitor for security events that could signal an intrusion or potential breach of security. In the same light, the response to a security event should also contain a viable plan for mitigating the same risk in the future.

Finally, the deployment of applications is a critical component of the security of an infrastructure. Because the main goal of deployments from a developer operation's standpoint is automation, any security bugs introduced once tend to be replicated. For this reason, it is imperative that the deployment process and any actors in it (Jenkins, AWS S3, etc.) are fully secured and audited often. Vulnerabilities that are present here can expand exponentially when deployments are pushed.

The role of developer operations or server engineering is rapidly changing and expanding. While it does, it is important, if not necessary, to include security in the expansion and ensure that those building a company's most critical technological parts are also trained in protecting them.

My First Month at Mozilla

I've been working as an intern for almost a month now (3 1/2 weeks is close enough) and finally decided to get around to writing a blog post about my experiences so far. To start, Mozilla is an amazing place to work; the "we're about the open web" is not just a tag-line, it's a core principle of the entire organization.

My first week was pretty hectic. There's a phrase at Mozilla called the "Mozilla Firehose" that refers to the massive amounts of information you will take in during your first week(s) at the company. It's entirely true, although not unmanageable because there are great people to help at each step. Once I got beyond the account-setup, email-checking, bug-filing, question-asking first few days, I was able to get a good head start on what I'll be working on for the next six months.

My position at Mozilla is on the Security Assurance team as a web application security intern. Essentially, my team and I are responsible for maintaining the security of all of Mozilla's web properties as well as the investigation of security bugs and performing of security reviews for new products. It has been a very interesting position because I am exposed to new security issues each day and rarely do the same thing twice (which is great because I get bored easily). So far I have investigated XSS bugs reported by the community in a number of Mozilla's web pages, analyzed more advanced attacks such as remote code execution, observed Mozilla's web bounty program in action (they pay member's of the community for responsible disclosure of bugs), and performed a security review of an internal project known as Datazilla. I hope to continue investigating security issues as well as take on a number of additional projects.

The environment at Mozilla has been awesome. There is food around every corner (literally) and the workplace is casual and very centered around team-working. Although a number of the employees on my team work remotely, it is not difficult to use IRC or email to communicate. I have also had the opportunity to travel to Mozilla's San Francisco office which has one of the best views of any office I've ever been in. It overlooks the bay directly next to the Bay Bridge.

Although I'm only a few weeks into my internship at Mozilla, I've already been exposed to a number of great learning opportunities. I've also seen how Mozilla operates as an organization and the true commitment of the organization's members to an open web, not bound by proprietary technologies. I am looking forward to a great Summer and Fall before returning to RIT in the Winter.