Showing posts with label server. Show all posts
Showing posts with label server. Show all posts

Monday, September 1, 2014

Why Security is Important for Developer Operations

After working at my current gig for about a year now (internship plus full-time), I wanted to take some time to outline my experiences transitioning to more of a developer operations role from my existing security background. The transition has been fairly straight-forward, as much of the work I do currently touches security in some aspect. However, the biggest point I've noticed is how much security considerations are involved when developing tools for a traditional developer operations requirement.

While I am a big proponent of at least some security training for everyone in a technical role, it remains to be seen just how much is "enough." Obviously, as the realm of technical fields continues to expand at its current pace, having every member of a technical team fully trained in the security of the applications they are developing is impossible. Yet it is also imperative that they at least understand the risks associated with these applications; doing so should be a requirement of a good developer.

As a "Developer Operations Engineer" (a role whose title is still rather undefined and unstandardized), I generally focus on several categories of projects: infrastructure development, monitoring and incident response, and deployments and the application lifecycle. While this is a highly compressed view of my role, each of these has a dizzying array of security concerns. While some companies offload much of these concerns to a security team, smaller companies need to remain vigilant of their impact.

In most modern startup environments, infrastructure development typically refers to everyone's favorite buzzword: the cloud. Amazon Web Services, Microsoft Azure, Google's App Engine, the list goes on. The security concerns associated with the cloud are not the focus of this post, but I do want to highlight places where security is especially important. Almost every one of these services has security enhancements that are not used as often as they should be. For example, AWS's Virtual Private Cloud is not only free, but can also greatly improve security when used properly. Yet quickly starting instances from EC2 still requires less hassle and so remains the more common choice. Another example is the use of security groups. AWS's security groups are infinitely customizable, yet simply opening a port to the world (0.0.0.0) is a tempting simpler option. While hosted infrastructure providers like Amazon and Google abstract a lot of security work away from the customer, good security practices still require active participation.

Monitoring and incident response is perhaps the area in which a lack of security can have the biggest impact. While many "DevOps" engineers view monitoring in terms of system performance, monitoring must also cover system security. Disk space, CPU utilization, and memory usage are all important indicators of a healthy system, but so too are login attempts, changes to file permissions, and unauthorized outgoing network connections. A good monitoring platform must also monitor for security events that could signal an intrusion or potential breach of security. In the same light, the response to a security event should also contain a viable plan for mitigating the same risk in the future.

Finally, the deployment of applications is a critical component of the security of an infrastructure. Because the main goal of deployments from a developer operation's standpoint is automation, any security bugs introduced once tend to be replicated. For this reason, it is imperative that the deployment process and any actors in it (Jenkins, AWS S3, etc.) are fully secured and audited often. Vulnerabilities that are present here can expand exponentially when deployments are pushed.

The role of developer operations or server engineering is rapidly changing and expanding. While it does, it is important, if not necessary, to include security in the expansion and ensure that those building a company's most critical technological parts are also trained in protecting them.

Tuesday, June 24, 2014

Sending CollectD Metrics to Graphite

There are no shortage of tutorials on setting up collectd as an agent on a machine. However, I have found little help in the way of describing how to setup a centralized collectd collection server that aggregates statistics from multiple clients and sends them to Graphite. This post will help you do just that. It focuses on Ubuntu, but the instructions are universally applicable.

First, let's setup the central collectd server. This could be on the same machine as your Graphite server, but on large production environments, it is not recommended.

The Ubuntu collectd repositories do not contain the necessary write_graphite plugin, so you must download and install collectd manually. Download the source from the website at: http://collectd.org/download.shtml

Next, run:

tar jxf collectd-version.tar.bz2
cd collectd-version
./configure
make all install

Once collectd is installed, modify the /opt/etc/collectd.conf file to contain the following:

Hostname "hostname"
FQDNLookup true
BaseDir "/opt/collectd/var/lib/collectd"
PIDFile "/opt/collectd/var/run/collectd.pid"
PluginDir "/opt/collectd/lib/collectd"
TypesDB "/opt/collectd/share/collectd/types.db"
Interval 10


LoadPlugin network
<Plugin network>
Listen "*" "12345"
</Plugin>

LoadPlugin interface
<Plugin interface>
Interface "eth0"
</Plugin>

LoadPlugin write_graphite
<Plugin write_graphite>
<Node "graphing">
Host "localhost"
Port "2003"
Protocol "tcp"
LogSendErrors true
Prefix "collectd."
StoreRates true
AlwaysAppendDS false
EscapeCharacter "_"
</Node>
</Plugin>

Make adjustments for your network as needed.

Now, we are going install the collectd agent on the client and then tell it to send the metrics to the collectd server (not Graphite).

The clients do not need the write_graphite plugin and can use the older version of Collectd that ships with the repositories. On each client, run:

sudo apt-get install collectd collectd-utils

Then, cd into /etc/collectd. Backup the collectd.conf file as collectd.conf.bkp or similar and then create a new collectd.conf file. In it, enable the plugins you want and also add the following:

Hostname "hostname"
FQDNLookup true
BaseDir "/var/lib/collectd"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/lib/collectd"
TypesDB "/usr/share/collectd/types.db"
Interval 10
#Timeout 5
ReadThreads 5

LoadPlugin network
<Plugin network>
Server "collectd.domain.com" "12345"
</Plugin>

LoadPlugin cpu
LoadPlugin load
LoadPlugin disk
LoadPlugin memory
LoadPlugin processes

Include "/etc/collectd/filters.conf"
Include "/etc/collectd/thresholds.conf"

Be sure to configure the network plugin with your collectd server information.

Now, if you log into Graphite, you should see that your clients are sending all of their statistics to the collectd server which is then sending them to Graphite.

Friday, February 28, 2014

How to Change the Default Timezone on Amazon Ubuntu EC2 Server

After setting up a Ubuntu Server on Amazon EC2, it's annoying that the default timezone is UTC. Here is how to change it:

sudo dpkg-reconfigure tzdata

Then restart your cron service. Otherwise, the timezone should be changed. This will allow the correct times to show up in any applications you have running.

Wednesday, August 7, 2013

Running Django and Node.js on the Same Server Using Apache

I've recently started working with Node.js and have been coding a few projects that I wanted to host on my VPS. However, I already have a few sites that I've written in Django running on that server on port 80. As it turns out, it's not terribly difficult to run both Node and Django on the same server, with different domains, both utilizing port 80.

I'm going to assume that you've already installed Python, Django, and Node.js on your server and that you have working projects. There are tons of tutorials on installing these on a server, so I'm not going to use this space for that. So let's begin!

First, to create the Django site, upload your Django files to your server. I'm going to save them in the directory /user/web.

To get Django working, there is a great tutorial over at DigitalOcean here: https://www.digitalocean.com/community/articles/installing-django-on-ubuntu-12-04--4

Make sure to follow all the steps. Once you can access yourdomain.com and see your app, you are successful.

Apache is now serving your Django app via mod_wsgi. To serve the Node.js app, we're going to use a reverse proxy pointing to a running instance of the Node.js server on a different port (I'll be using 3000).

Upload all of your Node.js project files to your server. I'll be saving them in /user/web/node.

Now, enable the proxy with the following command:

sudo a2enmod proxy proxy_http

Next, create a file in /etc/apache2/sites-available for your domain such as: /etc/apache2/sites-available/sample.com

In this file, paste the following code, changing sample.com to your domain:

<VirtualHost *:80>
    ServerName sample.com
    ServerAlias www.sample.com

    ProxyRequests off

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        ProxyPass http://localhost:3000/
        ProxyPassReverse http://localhost:3000/
    </Location>
</VirtualHost>

If you need to use another port, change both locations.

Save this file, and then start your node server on the port you specified. Add the site with the following command:

sudo a2ensite sample.com

Make sure to restart your Apache service:

sudo service apache2 restart

Now, when a user visits yourdjangoapp.com, Apache will call your Django app using mod_wsgi as described in the linked tutorial above. When you visit yournodeapp.com, Apache will use a reverse proxy to call the Node.js server running on the port you specified.

You can repeat this process to add as many Node.js projects as you want, but make sure you use a different port for each.

Additionally, I recommend using a Node tool called "forever" (npm install -g forever) to keep your service running in the background and restart it if it crashes.