AWS Lambda: "Occasionally Reliable Caching"

One of the biggest misconceptions I've heard from developers working with Lambda for the first time is that every execution of a Lambda function is entirely isolated and independent. AWS may be slightly responsible for this, as their documentation states:

Each AWS Lambda function runs in its own isolated environment, with its own resources and file system view. AWS Lambda uses the same techniques as Amazon EC2 to provide security and separation at the infrastructure and execution levels.

The trouble with this statement is that, while each function may be running in an isolated environment, the executions of that function have access to the same memory and disk resources. I'll skip right to an example.

var str = 'hello';

exports.handler = function(event, context) {
console.log(str);
str = 'it\'s me';
context.succeed();
};

Create a new Lambda function using this code and then run it twice. Here's the output:

START RequestId: 031528d9-8eea-22e5-b1e5-13516c9e3426 Version: $LATEST
2015-12-07T22:37:50.125Z 031528d9-8eea-22e5-b1e5-13516c9e3426 hello
END RequestId: 031528d9-8eea-22e5-b1e5-13516c9e3426
REPORT RequestId: 031528d9-8eea-22e5-b1e5-13516c9e3426 Duration: 10.91 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 8 MB

START RequestId: 4ac77131-8aca-11e3-874c-cb361dfcaaf1 Version: $LATEST
2015-12-07T22:38:10.101Z 4ac77131-8aca-11e3-874c-cb361dfcaaf1 it's me
END RequestId: 4ac77131-8aca-11e3-874c-cb361dfcaaf1
REPORT RequestId: 4ac77131-8aca-11e3-874c-cb361dfcaaf1 Duration: 8.49 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 9 MB

Run the function again in 30 minutes. It will output "hello" again because the container had to be reinitialized.

Not so isolated, right? To be fair, AWS does claim that any code outside of the handler function is treated as global code and is only initialized once per container. However, I've seen numerous developers mistakenly add code outside of the handler that should remain private. As Lambda becomes more ubiquitous, the distinction is critical for both performance and security.

Performance

Because code outside of the handler is only initialized once, this is the perfect spot for initializing Node modules, making database connections, etc. Additionally, it's also where global caching can be added. Take the following sample code:

var CACHE = {};

exports.handler = function(event, context) {
if (CACHE[event.item]) {
return context.succeed(CACHE[event.item]);
}

// Lookup object in database
db.find(event.item, function(err, item){
if (err) return context.fail(err);

CACHE[event.item] = item;
context.succeed(item);
});
};

Adding caching outside of the handler allows cached items to be shared across multiple executions of the same function. Because AWS does not guarantee that each execution will occur on the same container (due to scaling or lack of use), I've nicknamed this form of caching "Occasionally Reliable Caching™" (ORC).

Security

As you can imagine, there are important security considerations here as well. Take the following hypothetical code:

var creditCard;

exports.handler = function(event, context) {
creditCard = {
number: event.credit_card_number,
exp: event.credit_card_expiration,
code: event.credit_card_security_code
};
// Connect to payment system to verify
payments.verify(creditCard, function(err){
if (err) {
context.fail('Invalid card');
} else {
context.succeed('Payment okay');
}
})
};

While this is honestly just as bad as storing credit card information in a global variable on a web server, outside of the request handler, the distinction isn't as clear or as well-known with Lambda. In the example above, if there are multiple executions occurring simultaneously, there is no guarantee as to which card is being verified.

This is an extreme example to indicate an obvious code error, but also note that Lambda is not yet PCI certified, so shouldn't be used for this kind of data in the first place.

As developers begin working more with Lambda, I expect this post to become unneeded; but until Lambda's execution model is more well-understood, it is important to keep in mind.

Enjoy this post? Want to learn more about Lambda? Pre-order my complete Lambda book on Amazon for only $3.99!

AWS Lambda Pricing Calculator

The AWS Lambda pricing scheme is a bit confusing to understand. To help easily calculate the cost of running Lambda functions, I've embedded my simple AWS Lambda pricing calculator below. (Click here to open in a new window).

Pre-Order the Complete Lambda Guide

Click Here to pre-order for only $3.99! The Lambda guide covers everything you need to know to get started with Lambda and deep dives into scores of topics.

More Graceful Deployments with CodeDeploy

UPDATE: As user /u/Andy_Troutman on Reddit kindly pointed out to me, the AWS team has already considered functionality very similar to this. They even wrote a similar script (long before I wrote this one). If you want the more "official" version, here it is: https://github.com/awslabs/aws-codedeploy-samples/tree/master/load-balancing/elb

If you haven't used AWS CodeDeploy before, it's a new service aimed at automating deployments across a fleet of EC2 instances. It works using an agent on the instance that polls AWS for new changes to application code. When a change is detected (you or a CI tool triggered a deployment), the instance downloads the new code and runs a series of steps you define in a YAML file. These steps can include installing dependencies, validating that the service is running, and pretty much anything you can fit in a script. CodeDeploy can be configured to deploy the code to all instances at once, one at a time, or in percentage groups.

Ideally, a new deployment revision should not cause any loss of traffic. However, once the application is installed (the code is unzipped and copied to the correct location), most services must be restarted for the changes to take effect. Personally, I use a simple Node.js process running with "forever" for 90% of my projects. When CodeDeploy finishes installing the code, I have to run "forever stop" and "forever start" for the changes to be applied. This takes about 500 to 4000 milliseconds depending on how large the application is and whether it has to make database connections or perform other startup procedures. During this time, traffic is obviously rejected and the load balancer returns a "503 - Backend Service at Capacity" error to the client.

Although ELBs do have health check options (and theoretically, you could set it to have a 1 second ping), the application restart still causes a instantaneous cut-off of all connections, followed by a failure of the health check. Until the check fails, the ELB is still sending traffic, which could amount to hundreds of connections for a highly trafficked service.

The solution to this issue is to tell the load balancer to stop sending the instance traffic and then wait for the existing connections to drain before restarting the application. Honestly, CodeDeploy could easily implement a simple option for "remove instance from the load balancer when deploying," but I've decided to recreate the effect using the start, stop, and validate scripts used by the agent.

Overview

In the next several steps, I'm going to script out the process of: 1) De-registering the instance from the load balancer 2) performing the application restart, and 3) Re-registering it once the health check passes.

Instance Preparation

Your instances must have the AWS command line tools installed. This can be done on the AMI (recommended) or during the bootstrap process. Additionally, you could even add it as a "BeforeInstall" hook (just know that it will run the install command before every deployment until it's removed).

Additionally, the instances are going to need to be able to make the necessary API calls to AWS to register and deregister themselves from the load balancer. I've allowed this using IAM roles (you are using IAM roles, right?) in CloudFormation below:

{

"Effect" : "Allow",

"Action" : [

"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",

"elasticloadbalancing:RegisterInstancesWithLoadBalancer"

],

"Resource" : [

{

"Fn::Join": [

"",

[

"arn:aws:elasticloadbalancing:",

{ "Ref" : "AWS::Region" },

":",

{ "Ref":"AWS::AccountId" },

":loadbalancer/your-elb-name"

]

]

}

]

}

You can also modify the instance's IAM role directly from the console and use the policy generator to give it the same permissions.

ELB Preparation

You should also enable connection draining on your ELB and set the time to whatever is appropriate for your application (if you're just serving webpages, 30 seconds is probably fine; if users are uploading files to your service, you may want to increase it).

CodeDeploy Files

Now that your instances have the correct permissions, you can include the code in your scripts to gracefully remove them from the ELB before running the application restart. Your scripts may differ considerably, but I have the following appspec.yml file:

version: 0.0

os: linux

files:

  - source: /

    destination: /path/to/install/location

hooks:

  AfterInstall:

    - location: deployment/stop.sh

      runas: user

  ApplicationStart:

    - location: deployment/start.sh

      runas: user

  ValidateService:

    - location: deployment/validate.sh

      runas: user

When a deployment is triggered, CodeDeploy runs the "ApplicationStop" script, downloads your artifact, runs the "BeforeInstall" script, copies the files to the correct location, runs the "AfterInstall" script, then the "ApplicationStart" script, and then finally the "ValidateService" script. As you can see, they are not all required, and I have not made use of every one.

Once the artifact is downloaded and unzipped, the "AfterInstall" script is run, which I've configured to remove the instance from the ELB, wait for the connections to drain, then stop my application:

#!/bin/bash

# Get the instance-id from AWS

INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)

# Remove the instance from the load balancer

aws elb deregister-instances-from-load-balancer --load-balancer-name elb-name --instances $INSTANCEID --region us-east-1

# Let connections drain for 30 seconds (replace with your drain time)

sleep 30

# Now stop the server

forever stop /path/to/process.js

At this point, the instance is successfully removed from the ELB, connections have been drained, and you can do whatever is needed to restart your app without worrying about loosing requests. My start.sh script restarts the server:

#!/bin/bash

forever start path/to/process.js

Finally, you should add validation to ensure your app is actually running before you re-attach the instance to the ELB. I've done this in the validate.sh script:

#!/bin/bash

# Wait for however long the service takes to be responsive

sleep 10;

res=`curl -s -I localhost/ping | grep HTTP/1.1 | awk {'print $2'}`

echo $res;

if [ $res -eq 200 ]

then

# Get the instance-id from AWS

INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)

# Add the instance back to the ELB

aws elb register-instances-with-load-balancer --load-balancer-name elb-name --instances $INSTANCEID --region us-east-1

# Wait for the instance to be detected by the ELB (set this to the health check interval)

sleep 10

exit 0;

else

exit 1;

fi

If everything is successful, CodeDeploy will complete this step and move to the next instance (assuming you're deploying one at a time). If not, the deployment will fail but the instance will remain removed from the ELB. You can either re-trigger a deployment with a fix or rollback to a previously working one.

Additional Thoughts

Depending on the size of your application, this may not fully replace proper A-B stack deployments that includes switching DNS. If you only have a few servers, taking one offline will increase the load on the others substantially. Finally, these steps will increase the time of your deployments by 30 seconds to a few minutes per server. If you have 100 servers, consider using the "percentage at a time" deployment method, but balance this will the increased load on the remaining servers.

Update Your AWS ELBs to Protect Against the FREAK Vulnerability

The recently announced "FREAK" vulnerability is yet another blight on SSL. Fortunately, AWS has been quick as always to issue another cipher update. To protect your AWS Elastic Load Balancers from FREAK, follow these steps.

1. Navigate to your ELB in the EC2 console and select the "Listeners" tab.

2. Click on "Cipher."

3. Change the dropdown to the 2015-02 cipher policy.

4. Save.

That's it! Your ELBs will now be protected.

AWS CodeDeploy: An In-Depth First Look

At Amazon's 2014 re:invent conference in Las Vegas, they announced CodeDeploy, a tool designed to simplify the process of deploying applications to groups of servers, sometimes numbering in the hundreds. The primary objective of CodeDeploy is to make deployments consistent, repeatable, and integrated with existing AWS services (you can complain about vendor lock-in now, but AWS is doing a great job of providing value for that lock-in).

I took a few hours to setup CodeDeploy and documented issues I ran into. This post is a result of a few hours of playing with the service and trying to get it running on a Ubuntu 12.04 Server (despite 14.04 being the only "officially" supported version.

First Impressions

At first glance, CodeDeploy really seems like a game changer; it's built by Amazon, integrated with their services, and a convenient way to do rolling, all-at-once, or grouped deployments. Once I started working with CodeDeploy, it felt like a solid product once I got past the first few issues. Of course, given its recent release, it also lacks a lot of support or online discussion, which left me manually digging through error logs and support forums for dependencies. While the documentation is pretty decent, there are currently only about thirty questions in the AWS forums about CodeDeploy. The biggest issue I found was that I had to manually add an alternative source for ruby2.0 on Ubuntu 12.04 and install it myself before continuing - but this was not the fault of CodeDeploy.

IAM Setup

CodeDeploy requires a moderate amount of setup to get working properly. The biggest error-prone aspect is creating the appropriate IAM roles for both the CodeDeploy service and the instances. First, I created the CodeDeploy IAM role with the following policy:

{

"PolicyName" : "AWSCodeDeployPolicy",

"PolicyDocument" : {

"Statement": [

{

"Action": [

"autoscaling:PutLifecycleHook",

"autoscaling:DeleteLifecycleHook",

"autoscaling:RecordLifecycleActionHeartbeat",

"autoscaling:CompleteLifecycleAction",

"autoscaling:DescribeAutoscalingGroups",

"autoscaling:PutInstanceInStandby",

"autoscaling:PutInstanceInService",

"ec2:Describe*"

],

"Effect": "Allow",

"Resource": "*"

}

]

}

}

This allows CodeDeploy to access the tags and autoscaling groups it needs to in order to create applications and deployment configurations.

Next, I created the instance IAM role. It is important to remember that the CodeDeploy service needs access to the autoscaling and EC2 resources listed above while the instance itself only needs access to the S3 bucket containing the CodeDeploy agent and whatever bucket you store your final compressed file in.

{

"Effect" : "Allow",

"Action" : [

"s3:Get*",

"s3:List*"

],

"Resource" : [

"arn:aws:s3:::aws-codedeploy-us-east-1/*",

"arn:aws:s3:::your-bucket/path/*"

]

}

Here's a good place to tell you what I did wrong. Being security conscious, I thought I could get away with giving the instance role GetObject permissions only. My existing deployment strategy only requires this permission to pull the file from S3. However, apparently CodeDeploy tries to list the file and its ACL before downloading, which results in an error without the additional permissions. Lesson learned.

The CodeDeploy Agent

The next step was to get the agent installed on the Ubuntu Server instance. Amazon provides its own "Amazon Linux" if you're looking for an officially AWS-supported AMI, but I'm much more familiar with Debian-based distros, so I chose to stick with that. When you launch your instance, make sure you either give it a descriptive tag or place it in an autos-scaling group.

Installing the CodeDeploy agent on Ubuntu 12.04 proved to be a bit more difficult than the documentation reveals for 14.04 (again, not the fault of CodeDeploy, just my own need to use an older version). According to AWS, all you have to do is run:

sudo apt-get update

sudo apt-get install awscli

sudo apt-get install ruby2.0

cd /home/ubuntu

sudo aws s3 cp s3://bucket-name/latest/install . --region region-name

sudo chmod +x ./install

sudo ./install auto

(http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-run-agent.html#how-to-run-agent-install-ubuntu)

However, if you try that, you'll notice it fails at the third line with:

E: Couldn't find any package by regex 'ruby2.0'

There is a yet-unanswered forum post about this here.

I decided to get Ruby installed another way. After getting it installed via rvm and rerunning the install script, it failed again, this time with:

"Dependency not satisfiable: ruby2.0"

So, I finally installed Ruby by adding an alternative source from Brightbox as documented here. In case that's ever not available, here were the steps:

sudo apt-get install software-properties-common

sudo apt-add-repository ppa:brightbox/ruby-ng

sudo apt-get update

Finally, I ran the install script yet again and it worked!

Preparing the Application

The application I wanted to deploy was a simple Node.js web app. It runs on the server using "forever," a daemon that keeps the process running in the background. To prepare it for CodeDeploy, I had to add an appspec.yml file and two scripts: a start and stop script.

The appspec.yml file looked very simple:

version: 0.0

os: linux

files:

  - source: /

    destination: /usr/local/projects/source

hooks:

  AfterInstall:

    - location: deployment/stop.sh

      runas: root

  ApplicationStart:

    - location: deployment/start.sh

      runas: root

Keep in mind that the YAML file is super-particular about spacing. There's an entire section devoted to it on the AWS docs.

Next, I added the start and stop scripts to the deployment directory of the project. Obviously they can be much more complex than this, but I'm trying to keep it relatively simple:

start.sh:

#!/bin/sh

forever start /usr/local/projects/source/server.js --flags --here;

stop.sh:

#!/bin/sh

forever stopall

Like I said, super simple, but it works. The basic premise of this is that CodeDeploy will execute each file that you provide during the correct lifecycle event, as defined by the appspec file. Besides "AfterInstall" and "ApplicationStart," there are also "ApplicationStop," "BeforeInstall," and "ValidateService." AWS provides explanations here, but keep in mind that "Install" purely means copying files to the right directories. In my example, "AfterInstall" means that CodeDeploy will wait until the files have been copied before stopping the previous running instance.

Once all of this has been done, create a compressed file of your choice (zip, tar, and tar.gz are supported on Linux, zips for Windows). Put the file in the same S3 bucket that you gave your instance permissions to earlier.

CodeDeploy Console

Within the AWS console, you can now setup your application. To do this, head to the CodeDeploy page and create a new application. Provide a name and a deployment group name. The console doesn't make this clear, but the difference is that you can have multiple deployment groups belonging to an application. For example, you could have an app called "node-app" and create a "node-app-a" deployment group and then later create a "node-app-b" group which would help with A-B style deployments.

In the tags section, enter either the autoscaling group or the tags you created earlier. If everything is successful, you should see the instance count increase.

The next section, Deployment Configuration, allows you to determine how you want your apps deployed. This is not really relevant when you only have one server, but it becomes very helpful if you have multiple servers. If you choose "one at a time," AWS will go to each server, attempt to deploy your app, and stop if any servers fail along the way. With all at once or half at a time, CodeDeploy will run in parallel accordingly. This is a much faster, but also much more dangerous option.

The service role should be the role created earlier with the necessary permissions. This role can be re-used for every application, as the permissions are the same regardless.

Finally, the application can be created. The next page is a bit confusing because it does not contain any action buttons. Instead, it says to use the command line to upload an application. Instead of doing that, head back to the main CodeDeploy page and click on "Deployments."

On this page, select your application from the list, then select the group name, paste the full S3 URL to your source into the box, and select your deployment method. Then, click deploy.

You can then see the results of the deployment.

Potential Issues

Besides the ruby dependency issue I mentioned above, I also ran into a very ambiguous error message:

UnknownError: Not Opened for Reading

This message really didn't tell me what was happening, but after logging into the instances, going to the /opt/codedeploy-agent/deployment-root directory and finding that all of the source files contained an XML error from S3 instead of the actual files, I was able to debug it. Be sure that you use all of the permissions listed above for the instance role or you might run into the same problems.

Other Options and Thoughts

Besides deploying from an S3 object, you can also tie into GitHub. While this could work, I prefer to have a 100% working source file before actually deploying to an instance. I still use Jenkins to pull my changes from GitHub, install dependencies (node modules for my apps), run tests, zip everything up, and put it on S3. Once that's done, I can launch a new deployment from the console, or even have Jenkins use the AWS CLI to launch a deployment pointed at the file it just uploaded. While I could certainly install node modules as part of the pre-install hooks on the instance itself, that is much more error prone and slower as well.

UPDATE: AWS has also informed me that there is an open-source Jenkins CodeDeploy plugin available. I've installed the plugin and it works quite nicely; you can easily specify the application name, deployment group, and deployment policy from within Jenkins. Then, it executes as a build step with the same exit codes as Jenkins. Essentially, you can push to GitHub, copy to Jenkins, run tests, then execute a CodeDeploy deployment all as a result of one push (assuming you have the appropriate webhooks).

Overall, CodeDeploy worked very well once I got it working. I was able to deploy my app multiple times in a row without issues and even tested out the "one at a time" feature with an autoscaling group. Everything worked as expected. While I don't think CodeDeploy will be a complete replacement for a tool like Jenkins or other CI suites, it does remove the last few steps and make them more tightly integrated to AWS. I highly reccommend you try CodeDeploy out, but definitely do it in a test environment first until you have the process down to a science.

Using IAM Roles and S3 to Securely Load Application Credentials

Many applications require certain information to be provided to them at run-time in order to utilize additional services. For example, applications that connect to a database require the database connection URL, a port, a username, and a password. Developers frequently utilize environment variables, which are set on the machine on which the application is running, to provide these credentials to the underlying code. Some developers, against all recommendations, will hard-code the credentials into an application, which then gets checked into git, distributed, etc.

Ideally, the credentials required by an application should not be hard-coded at all, or even accessible to processes outside of the one running the application itself. To achieve this, the application must determine what additional credentials it needs and load them prior to starting its main command.

Many applications that run on Amazon's Web Services platform have the added advantage of being hosted on EC2 instances that can assume specific IAM roles. An IAM role is essentially a definition of access rights that are provided to a particular AWS resource (an EC2 instance in this case). AWS takes care of generating temporary credentials for that instance, rotating them, and ensuring they are provided only to the assigned instance. Additionally, the AWS command line tools and various AWS language-specific SDKs will detect that they are being run on an instance using an IAM role and automatically load the necessary credentials.

As developers, we can take advantage of IAM roles to provide access to credentials that are stored in a private S3 bucket. When the application loads, it will use its IAM role to download the credentials and load them into the environment variables of the process. Then, wherever they are needed, they can simply be called by accessing the environment variable that has been defined.

As an example of this setup, here are the steps I would take to run a Node.js web server that requires some database credentials:

1. Create an S3 bucket called "organization-unique-name-credentials".

2. If you plan to have multiple applications, create a new folder for each within the bucket: "organization-unique-name-credentials/web-app," "organization-unique-name-credentials/app-two," etc. Ensure the proper access rights to each for your existing AWS users.

3. Set encryption on the bucket (you can use either AWS' key or your own).

4. Create a file called credentials.json that looks like this:

{

    "DB_URL" : "some-database-connection-string.com",

    "DB_PORT" : 3306,

    "DB_USER" : "app_user",

    "DB_PASS" : "securepass"

}

5. Upload the file to the right S3 bucket and folder (be sure to enable encryption or the upload will fail, assuming you required it for the bucket)

6. Create an IAM role for your instance. In the IAM console, click "Roles," then create a new role, enter a name, select "EC2 Service Role," and give it the following policy (add any other rights the app may need if it accesses other AWS resources):

{

  "Statement": [

    {

      "Effect": "Allow",

      "Action": [

        "s3:GetObject"

      ],

      "Resource": [

        "arn:aws:s3:::organization-unique-name-credentials/web-app/credentials.json"

      ]

    }

  ]

}

7. Launch your EC2 instance, selecting the role you just created.

8. In your code, do the following (node pseudo-code):

var AWS = require('aws-sdk);

AWS.config.region = 'us-east-1';

var s3 = new AWS.S3();

var params = {

    Bucket: 'organization-unique-name-credentials',

    Key: 'web-app/credentials.json'

}

s3.getObject(params, function(err, data) {

    if (err) {

        console.log(err);

    } else {

        data = JSON.parse(data.Body.toString());

        for (i in data) {

            console.log('Setting environment variable: ' + i);

            process.env[i] = data[i];

        }

        // Load database via db.conn({user:process.env['DB_USER'], password:process.env[

'DB_PASS']}); etc...

    }

});

9. Run the app, and you will notice that the environment variables are downloaded from S3 and are set before the database connection is attempted.

If you're using Node.js, I made a module that does exactly this: https://www.npmjs.com/package/secure-credentials

If you're not using Node.js, this technique can be applied to any language that AWS has an SDK for. While it isn't 100% hacker-proof (if someone managed to log into your instance as root, he or she could still modify the source code to display the credentials), but combined with other AWS security mechanisms such as security groups, VPCs with proper network ACLs, etc. it can certainly help. Additionally, it keeps credentials out of the source code.

One final note: if you're running this app locally to test, your machine will obviously not have an EC2 IAM role. When testing locally, it's okay to use AWS keys and secrets, but be sure to keep them in a separate file that is excluded with .gitignore.

AWS Cross-Account IAM Roles in CloudFormation

The AWS documentation is relatively sparse when it comes to creating specific IAM role types using CloudFormation. It describes the process of setting up standard roles, attaching roles to instances, etc. but doesn't mention that all of the other role types can also be created using CloudFormation.

For example, when you log into the AWS console and click on "IAM," you see a number of different roles you can create:

AWS Service Roles

Role for Cross-Account Access

Role for Identity Provider Access

However, these role types are merely just different adaptations of the same concept. In the following steps, I'll show how to create a Cross-Account Role using CloudFormation.

1. Add the following to the "Resources" section of your CloudFormation template:

"CrossAccountRole" : {

"Type" : "AWS::IAM::Role",

"Properties" : {

"AssumeRolePolicyDocument" : {

"Statement" : [

{

"Effect" : "Allow",

"Principal" : {

"AWS": "arn:aws:iam::ACCOUNT_NUMBER_HERE:root"

},

"Action" : [

"sts:AssumeRole"

]

}

]

}

}

},

2. Add another resource for the policy:

"CrossAccountPolicy" : {

"Type" : "AWS::IAM::Policy",

"Properties" : {

"PolicyName" : "IAMInstancePolicy",

"PolicyDocument" : {

"Statement" : [

{

"Effect" : "Allow",

"Action" : [

"*"

],

"Resource" : [

"*"

]

}

]

},

"Roles" : [

{ "Ref" : "CrossAccountRole" }

]

}

},

3. Adjust the account number and resources as needed:

This policy gives admin access to any account you specify. To restrict permissions, change the statement section of the policy document as desired.

How to Disable SSLv3 on AWS Elastic Load Balancers

In a blog post today, Google announced that a vulnerability in SSLv3 had been found that could allow attackers to intercept data that had previously been assumed to be secured. Luckily, a very small portion of the web (IE6 users on Windows XP) still use SSLv3, so it can safely, for the most part, be disabled to mitigate the risk from this issue.

http://googleonlinesecurity.blogspot.com/2014/10/this-poodle-bites-exploiting-ssl-30.html

UPDATE 10/15: As Andrew and Julio point out in the comments below, AWS has since updated their default cipher security policies. Replace steps 5 and 6.

To modify the ciphers on AWS ELBs, follow the following steps:

1) Log into the AWS console and click on "Load Balancers."

2) Find the load balancer that handles your site's traffic (you shouldn't need to worry about internal VPC LBs, etc.)

3) Click the "Listeners" tab

4) Find the HTTPS/443 listener and click "Edit" under the cipher column

5) Change the option to "Custom"

6) Uncheck the SSLv3 option
5) Change the policy to "ELBSecurityPolicy-2014-10" which disables SSLv3 for you.

6) Save.

This should be sufficient to mitigate this risk with the information that is currently known.

Redirect HTTP to HTTPS Behind AWS Elastic Load Balancer - Node.js and Apache

Apache

When you enable HTTPS for your website, you should enforce that HTTPS is being used by automatically redirecting users who access your site over HTTP to the HTTPS version. When using Apache, this is done via a host redirect entry with mod_rewrite:

<VirtualHost *:80>

# Beginning declarations here

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^.*$ https://%{SERVER_NAME}%{REQUEST_URI}

# Rest of entry here...
</VirtualHost>

Apache Behind an ELB

However, in the Amazon Web Services world, most applications utilize the load balancer as an SSL-termination point. This means that the traffic is decrypted at the load balancer and all traffic being sent to the final destination instances is actually sent over HTTP (from a security standpoint, this is generally an accepted practice as the load balancer and instance are in the same network, and thus secure from eavesdropping). If you used the rule above, your users would wind up in an endless redirect loop.

To fix this, the "RewriteCond" needs to be changed to:

RewriteCond %{HTTP:X-Forwarded-Proto} !https

This rule will check to see which protocol the user was using before he or she hit the load balancer - which is what HTTPS redirection should use.

Node.js

When using Node.js, the same thing can be accomplished by inserting some middleware that listens for all traffic and redirects the non-HTTPS requests to their HTTPS equivalents.

var myApp = express () ,
myServer = http.createServer(myApp);

myApp.use(function redirectHTTP(req, res, next) {
  if (!req.secure) {
    return res.redirect('https://' + req.headers.host + req.url);
  }
  next();
});

Node.js Behind an ELB

Just as in our first Apache example, this works great until it is placed behind a load balancer. If you're using a load balancer, you need to make some modifications. Instead of "req.secure", do the following:

if (req.headers['x-forwarded-proto'] && req.headers['x-forwarded-proto'].toLowerCase() === 'http') {
return res.redirect('https://' + req.headers.host + req.url);
}

Using these techniques, you should be able to enforce the new HTTPS version of your website. Remember - if you serve both an HTTPS and an HTTP version of your site, the HTTPS is rendered pointless unless you also redirect the HTTP requests.

Why Security is Important for Developer Operations

After working at my current gig for about a year now (internship plus full-time), I wanted to take some time to outline my experiences transitioning to more of a developer operations role from my existing security background. The transition has been fairly straight-forward, as much of the work I do currently touches security in some aspect. However, the biggest point I've noticed is how much security considerations are involved when developing tools for a traditional developer operations requirement.

While I am a big proponent of at least some security training for everyone in a technical role, it remains to be seen just how much is "enough." Obviously, as the realm of technical fields continues to expand at its current pace, having every member of a technical team fully trained in the security of the applications they are developing is impossible. Yet it is also imperative that they at least understand the risks associated with these applications; doing so should be a requirement of a good developer.

As a "Developer Operations Engineer" (a role whose title is still rather undefined and unstandardized), I generally focus on several categories of projects: infrastructure development, monitoring and incident response, and deployments and the application lifecycle. While this is a highly compressed view of my role, each of these has a dizzying array of security concerns. While some companies offload much of these concerns to a security team, smaller companies need to remain vigilant of their impact.

In most modern startup environments, infrastructure development typically refers to everyone's favorite buzzword: the cloud. Amazon Web Services, Microsoft Azure, Google's App Engine, the list goes on. The security concerns associated with the cloud are not the focus of this post, but I do want to highlight places where security is especially important. Almost every one of these services has security enhancements that are not used as often as they should be. For example, AWS's Virtual Private Cloud is not only free, but can also greatly improve security when used properly. Yet quickly starting instances from EC2 still requires less hassle and so remains the more common choice. Another example is the use of security groups. AWS's security groups are infinitely customizable, yet simply opening a port to the world (0.0.0.0) is a tempting simpler option. While hosted infrastructure providers like Amazon and Google abstract a lot of security work away from the customer, good security practices still require active participation.

Monitoring and incident response is perhaps the area in which a lack of security can have the biggest impact. While many "DevOps" engineers view monitoring in terms of system performance, monitoring must also cover system security. Disk space, CPU utilization, and memory usage are all important indicators of a healthy system, but so too are login attempts, changes to file permissions, and unauthorized outgoing network connections. A good monitoring platform must also monitor for security events that could signal an intrusion or potential breach of security. In the same light, the response to a security event should also contain a viable plan for mitigating the same risk in the future.

Finally, the deployment of applications is a critical component of the security of an infrastructure. Because the main goal of deployments from a developer operation's standpoint is automation, any security bugs introduced once tend to be replicated. For this reason, it is imperative that the deployment process and any actors in it (Jenkins, AWS S3, etc.) are fully secured and audited often. Vulnerabilities that are present here can expand exponentially when deployments are pushed.

The role of developer operations or server engineering is rapidly changing and expanding. While it does, it is important, if not necessary, to include security in the expansion and ensure that those building a company's most critical technological parts are also trained in protecting them.

How to Fix OpenSSL Heart Bleed Bug on Amazon ELBs

The recently discovered "Heart Bleed" bug in OpenSSL is an extremely critical security issue. Amazon has been working to get all of their environments patched to the latest version of OpenSSL that remedies the issue.

If you have Elastic Load Balancers currently using an SSL certificate that was generated via OpenSSL version 1.1.0a-f, you need to follow these streps to revoke the current certificate on your load balancer and upload a new one.

First, update OpenSSL on the machine you are going to use to generate your private key and sign your certificate. I have written another post on how to do that here: http://blog.matthewdfuller.com/2014/04/how-to-fix-openssl-heart-bleed-bug-on.html

Once you have regenerated your keys and resigned your certificate, you can upload them to your load balancers.

Within the AWS console, click "EC2" in the Services menu.

Now, click on "Load Balancers" on the left-hand side and select your load balancer instance.

Click on the "Listeners" tab and notice the existing cert:

Click "Change" and then click "Upload a new cert."

Give your cert a name, paste in the private key and cert you created earlier, and provide any chain information if needed.

Hit save and your load balancer will push the changes.

If you want to do this via the command line or API, check out the official AWS documentation: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/US_UpdatingLoadBalancerSSL.html

Find the Public Hostname of an Amazon EC2 Instance

If you need to find the public hostname of an Amazon EC2 instance, there is a simple URL which, when cURL'ed will return the hostname:

http://169.254.169.254/latest/meta-data/public-hostname

This URL only works inside the AWS infrastructure (so you must be on an instance to run it properly), but you could easily run:

curl http://169.254.169.254/latest/meta-data/public-hostname

This will return the hostname in the standard format:

ec2-54-123-256-12.compute-1.amazonaws.com

Kippo Honeypot on Amazon EC2 Instance Free Tier

A project I've had in mind for a while is to use Amazon's Cloud (specifically an EC2 instance) to setup a honeypot. Luckily, Amazon has been offering a low end, free tier of service for an EC2 instance, which is just what I need for this project. In this post I'm going to walk through exactly what I did to setup a medium-interaction honeypot known as "Kippo" on an Amazon EC2 instance completely for free (at least for one year). This isn't as straightforward as it seems because, with the free tier, you only get one IP address. This means you can't setup your honeypot for SSH on one IP and the admin/machine SSH on another. Don't worry though, we'll fix that.

Setting up an EC2 Instance

The first step to setting up the honeypot is to subscribe to Amazon's EC2 service. You'll need to go through the registration and enter a credit card, but they won't charge anything to it. You'll also need to enter a phone number to receive a code in order to verify your identity. I'm not going to walk through that process here, but you can sign up and read more here: http://aws.amazon.com/ec2/

Once your account is created, open up the AWS Management Console. It should look like this:

Click on "EC2". Now, launch an instance by clicking "Launch Instance."

Choose the "Classic Wizard" and continue. Now, select the Ubuntu Server 12.04 64-bit image (note: instances included in the free tier are marked with a star).

Choose one instance, and an instance type of "Micro" with 613MB memory (note: this is also marked with a star).

Click "continue" through instance details, nothing needs to be changed. Next, create a key pair and save the resulting .pem file. You will need this to SSH to the server. For a security group, select the quick-start group from the left.

Now, you can review your instance and create it. The instance may take some time to initialize. Once it finishes, you should see it running under "instances."

Getting an IP Address

Now, create a new elastic IP for your machine. This will allow your machine to retain a single IP through which attackers can SSH. Click on "elastic IPs" under "Network and Security." Then, allocate a new IP address. In many cases, honeypots will have two network interfaces - one for the attack surface, another for management. Each interface would have a different IP address to separate the attack surface from the management. However, Amazon's free tier allocates only a single elastic IP. Do not create a second IP or you may be charged for additional usage.

Once you have an IP, make sure it is pointing to your running instance.

SSH to the Instance

We can now connect to the instance via SSH. Make sure you are in the folder in which you saved your .pem file from Amazon. Then, ssh using the following command:

ssh -v -i <your-key>.pem ubuntu@ec2-<ip-address>.compute-1.amazonaws.com

"Ubuntu" is the default username for the instance.

Change the SSH Port

To get around the IP address restrictions, we're going to run the management SSH on a non-standard port and the honeypot on the typical port 22. This will allow us to both obscure the management connection and increase the number of attacks seen by the honeypot (almost every attacker will try port 22 for SSH first). To change ports, we need to edit the configuration file for the already-running SSH server and then restart the service. Do this carefully or you may lose access to your machine.

Begin by editing your SSH config file located here: /etc/ssh/sshd_config

At the very top of the file are the following lines:

# What ports, IPs and protocols we listen for
Port 22

Change this port number to something between 49152 and 65535. Make sure you write it down and do not forget the port number you selected.

Now, restart the SSH service by running:

/etc/init.d/ssh restart

When you run this command you will likely be disconnected from your machine. Hopefully you "restarted" and didn't "stop."

You will now need to edit the Amazon security rules within your AWS console to allow your new port on inbound connections. To do this, click "Security Groups" under "Network & Security." Then, click on the "quick-start-1" group and then the "Inbound" tab. Add your new port number and be sure to apply the changes.

You can see that my port is 50683 in this case.

Now, reconnect to the machine by running the following command. Note the added -p parameter to specify the port number.

ssh -v -i <your-key>.pem ubuntu@ec2-<ip-address>.compute-1.amazonaws.com -p <port>

*Note: you can create an SSH configuration so you don't need to specify all these options for every connection  but that is beyond the scope of this guide.

Hopefully you have reconnected to your machine. SSH is now running on a port other than 22 which will allow us to use the standard SSH port for our honeypot.

Installing Kippo

We can now install Kippo and begin configuring our honeypot. I am not going to re-write a guide for the installation process as it is well-documented and many guides already exist. This is a great guide, written for CentOS, but the process is very similar: http://www.howtoforge.com/how-to-set-up-kippo-ssh-honeypot-on-centos-5.5

*Note that you should not need to update Python. Also, when downloading the Kippo source, be sure to use the latest version as this guide is a bit old. Finally, you will need to add the IPTABLES rule to redirect port 22 traffic to port 2222.

Once everything is installed and running, you should be able to issue the command:

ssh root@<your-ip>

and be logged into your honeypot.

Viewing Logs

One of the best parts of Kippo is that it logs every interaction an attacker has with the system. These logs are saved in /home/kipuser/kippo/log/
*kipuser may be replaced with the username of the kippo user you created.

To replay the logs, copy the file "playlog.py" from kippo/utils into the kippo/log/tty folder, then issue the command:

sudo python playlog.py <log-name>.log 0

This will replay the attacker's interaction with the system.

Further Resources

http://www.howtoforge.com/how-to-set-up-kippo-ssh-honeypot-on-centos-5.5

http://code.google.com/p/kippo/wiki/KippoOnLinux

http://www.linuxlookup.com/howto/change_default_ssh_port