Showing posts with label nodejs. Show all posts
Showing posts with label nodejs. Show all posts

Monday, December 7, 2015

AWS Lambda: "Occasionally Reliable Caching"

One of the biggest misconceptions I've heard from developers working with Lambda for the first time is that every execution of a Lambda function is entirely isolated and independent. AWS may be slightly responsible for this, as their documentation states:
Each AWS Lambda function runs in its own isolated environment, with its own resources and file system view. AWS Lambda uses the same techniques as Amazon EC2 to provide security and separation at the infrastructure and execution levels.
The trouble with this statement is that, while each function may be running in an isolated environment, the executions of that function have access to the same memory and disk resources. I'll skip right to an example.

var str = 'hello';

exports.handler = function(event, context) {
console.log(str);
str = 'it\'s me';
context.succeed();
};

Create a new Lambda function using this code and then run it twice. Here's the output:

START RequestId: 031528d9-8eea-22e5-b1e5-13516c9e3426 Version: $LATEST
2015-12-07T22:37:50.125Z 031528d9-8eea-22e5-b1e5-13516c9e3426 hello
END RequestId: 031528d9-8eea-22e5-b1e5-13516c9e3426
REPORT RequestId: 031528d9-8eea-22e5-b1e5-13516c9e3426 Duration: 10.91 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 8 MB

START RequestId: 4ac77131-8aca-11e3-874c-cb361dfcaaf1 Version: $LATEST
2015-12-07T22:38:10.101Z 4ac77131-8aca-11e3-874c-cb361dfcaaf1 it's me
END RequestId: 4ac77131-8aca-11e3-874c-cb361dfcaaf1
REPORT RequestId: 4ac77131-8aca-11e3-874c-cb361dfcaaf1 Duration: 8.49 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 9 MB

Run the function again in 30 minutes. It will output "hello" again because the container had to be reinitialized.

Not so isolated, right? To be fair, AWS does claim that any code outside of the handler function is treated as global code and is only initialized once per container. However, I've seen numerous developers mistakenly add code outside of the handler that should remain private. As Lambda becomes more ubiquitous, the distinction is critical for both performance and security.

Performance

Because code outside of the handler is only initialized once, this is the perfect spot for initializing Node modules, making database connections, etc. Additionally, it's also where global caching can be added. Take the following sample code:

var CACHE = {};

exports.handler = function(event, context) {
if (CACHE[event.item]) {
return context.succeed(CACHE[event.item]);
}

// Lookup object in database
db.find(event.item, function(err, item){
if (err) return context.fail(err);

CACHE[event.item] = item;
context.succeed(item);
});
};

Adding caching outside of the handler allows cached items to be shared across multiple executions of the same function. Because AWS does not guarantee that each execution will occur on the same container (due to scaling or lack of use), I've nicknamed this form of caching "Occasionally Reliable Caching" (ORC).

Security

As you can imagine, there are important security considerations here as well. Take the following hypothetical code:

var creditCard;

exports.handler = function(event, context) {
creditCard = {
number: event.credit_card_number,
exp: event.credit_card_expiration,
code: event.credit_card_security_code
};
// Connect to payment system to verify
payments.verify(creditCard, function(err){
if (err) {
context.fail('Invalid card');
} else {
context.succeed('Payment okay');
}
})
};

While this is honestly just as bad as storing credit card information in a global variable on a web server, outside of the request handler, the distinction isn't as clear or as well-known with Lambda. In the example above, if there are multiple executions occurring simultaneously, there is no guarantee as to which card is being verified.

This is an extreme example to indicate an obvious code error, but also note that Lambda is not yet PCI certified, so shouldn't be used for this kind of data in the first place.

As developers begin working more with Lambda, I expect this post to become unneeded; but until Lambda's execution model is more well-understood, it is important to keep in mind.

Enjoy this post? Want to learn more about Lambda? Pre-order my complete Lambda book on Amazon for only $3.99!

Thursday, January 15, 2015

Using IAM Roles and S3 to Securely Load Application Credentials

Many applications require certain information to be provided to them at run-time in order to utilize additional services. For example, applications that connect to a database require the database connection URL, a port, a username, and a password. Developers frequently utilize environment variables, which are set on the machine on which the application is running, to provide these credentials to the underlying code. Some developers, against all recommendations, will hard-code the credentials into an application, which then gets checked into git, distributed, etc.

Ideally, the credentials required by an application should not be hard-coded at all, or even accessible to processes outside of the one running the application itself. To achieve this, the application must determine what additional credentials it needs and load them prior to starting its main command.

Many applications that run on Amazon's Web Services platform have the added advantage of being hosted on EC2 instances that can assume specific IAM roles. An IAM role is essentially a definition of access rights that are provided to a particular AWS resource (an EC2 instance in this case). AWS takes care of generating temporary credentials for that instance, rotating them, and ensuring they are provided only to the assigned instance. Additionally, the AWS command line tools and various AWS language-specific SDKs will detect that they are being run on an instance using an IAM role and automatically load the necessary credentials.

As developers, we can take advantage of IAM roles to provide access to credentials that are stored in a private S3 bucket. When the application loads, it will use its IAM role to download the credentials and load them into the environment variables of the process. Then, wherever they are needed, they can simply be called by accessing the environment variable that has been defined.

As an example of this setup, here are the steps I would take to run a Node.js web server that requires some database credentials:

1. Create an S3 bucket called "organization-unique-name-credentials".

2. If you plan to have multiple applications, create a new folder for each within the bucket: "organization-unique-name-credentials/web-app," "organization-unique-name-credentials/app-two," etc. Ensure the proper access rights to each for your existing AWS users.

3. Set encryption on the bucket (you can use either AWS' key or your own).

4. Create a file called credentials.json that looks like this:

{
    "DB_URL" : "some-database-connection-string.com",
    "DB_PORT" : 3306,
    "DB_USER" : "app_user",
    "DB_PASS" : "securepass"
}

5. Upload the file to the right S3 bucket and folder (be sure to enable encryption or the upload will fail, assuming you required it for the bucket)

6. Create an IAM role for your instance. In the IAM console, click "Roles," then create a new role, enter a name, select "EC2 Service Role," and give it the following policy (add any other rights the app may need if it accesses other AWS resources):

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::organization-unique-name-credentials/web-app/credentials.json"
      ]
    }
  ]
}

7. Launch your EC2 instance, selecting the role you just created.

8. In your code, do the following (node pseudo-code):

var AWS = require('aws-sdk);
AWS.config.region = 'us-east-1';
var s3 = new AWS.S3();

var params = {
    Bucket: 'organization-unique-name-credentials',
    Key: 'web-app/credentials.json'
}

s3.getObject(params, function(err, data) {
    if (err) {
        console.log(err);
    } else {
        data = JSON.parse(data.Body.toString());
        for (i in data) {
            console.log('Setting environment variable: ' + i);
            process.env[i] = data[i];
        }

        // Load database via db.conn({user:process.env['DB_USER'], password:process.env[
'DB_PASS']}); etc...
    }
});

9. Run the app, and you will notice that the environment variables are downloaded from S3 and are set before the database connection is attempted.

If you're using Node.js, I made a module that does exactly this: https://www.npmjs.com/package/secure-credentials

If you're not using Node.js, this technique can be applied to any language that AWS has an SDK for. While it isn't 100% hacker-proof (if someone managed to log into your instance as root, he or she could still modify the source code to display the credentials), but combined with other AWS security mechanisms such as security groups, VPCs with proper network ACLs, etc. it can certainly help. Additionally, it keeps credentials out of the source code.

One final note: if you're running this app locally to test, your machine will obviously not have an EC2 IAM role. When testing locally, it's okay to use AWS keys and secrets, but be sure to keep them in a separate file that is excluded with .gitignore.

Thursday, October 30, 2014

Node.js "Error: too many parameters at queryparse"

If you've been using Express' body-parser and attempting to process large data submissions (aka forms with more than 1000 elements or 1000 sub-elements, you may have run into the following error:

Error: too many parameters
    at queryparse (/project/node_modules/body-parser/lib/types/urlencoded.js:120:17)
    at parse (/project/node_modules/body-parser/lib/types/urlencoded.js:64:9)

This is due to the fact the urlencode defaults to 1000 parameters by default. If you have a large form or just an abnormally large JSON submission, you'll need to increase this limit by doing the following:

var bodyParser = require('body-parser');

app.use(bodyParser.urlencoded({
        extended: false,
    parameterLimit: 10000,
    limit: 1024 * 1024 * 10
}));
app.use(bodyParser.json({
        extended: false,
    parameterLimit: 10000,
    limit: 1024 * 1024 * 10
}));


This will allow you to provide up to 10,000 parameters (increase as needed) and 10 MB of data (also adjustable).

Wednesday, July 2, 2014

Use google-passport Authentication in Node.js Projects without Google+

NPM passport-google and 400 Error "OpenID auth request contains an unregistered domain"

If you've been using the Node.js module "passport-google" for authentication in your projects you will now notice that new projects are receiving an error stating:

"400 That's an error. OpenID auth request contains an unregistered domain."

This issue is due to the fact that Google has deprecated their OpenID API for new domains beginning in May 2014. Old projects which had previously been used should not have an issue (although you should upgrade at some point), but new projects will not be allowed to authenticate.

There are two fixes for this:

1) Convert your application to using the new, Google+ sign-in. This will require users to have a Google+ profile and approve your application to access it. The passport-google-plus module located on GitHub can do just that: https://github.com/sqrrrl/passport-google-plus

2) Convert your application to use OAuth2.0 signin. Your users will not need to have a Google+ profile and this new method is the closest match to the old. The passport-google-oauth module can help with this: https://github.com/jaredhanson/passport-google-oauth

If you choose the second option, be sure to only provide the userinfo scope (and not the google.plus scope):

passport.authenticate('google', { scope: ['https://www.googleapis.com/auth/userinfo.email'] })

One additional note is that you will now be required to register your application at https://console.developers.google.com and create a client ID and secret (which are used in the passport module).

Wednesday, August 7, 2013

Running Django and Node.js on the Same Server Using Apache

I've recently started working with Node.js and have been coding a few projects that I wanted to host on my VPS. However, I already have a few sites that I've written in Django running on that server on port 80. As it turns out, it's not terribly difficult to run both Node and Django on the same server, with different domains, both utilizing port 80.

I'm going to assume that you've already installed Python, Django, and Node.js on your server and that you have working projects. There are tons of tutorials on installing these on a server, so I'm not going to use this space for that. So let's begin!

First, to create the Django site, upload your Django files to your server. I'm going to save them in the directory /user/web.

To get Django working, there is a great tutorial over at DigitalOcean here: https://www.digitalocean.com/community/articles/installing-django-on-ubuntu-12-04--4

Make sure to follow all the steps. Once you can access yourdomain.com and see your app, you are successful.

Apache is now serving your Django app via mod_wsgi. To serve the Node.js app, we're going to use a reverse proxy pointing to a running instance of the Node.js server on a different port (I'll be using 3000).

Upload all of your Node.js project files to your server. I'll be saving them in /user/web/node.

Now, enable the proxy with the following command:

sudo a2enmod proxy proxy_http

Next, create a file in /etc/apache2/sites-available for your domain such as: /etc/apache2/sites-available/sample.com

In this file, paste the following code, changing sample.com to your domain:

<VirtualHost *:80>
    ServerName sample.com
    ServerAlias www.sample.com

    ProxyRequests off

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        ProxyPass http://localhost:3000/
        ProxyPassReverse http://localhost:3000/
    </Location>
</VirtualHost>

If you need to use another port, change both locations.

Save this file, and then start your node server on the port you specified. Add the site with the following command:

sudo a2ensite sample.com

Make sure to restart your Apache service:

sudo service apache2 restart

Now, when a user visits yourdjangoapp.com, Apache will call your Django app using mod_wsgi as described in the linked tutorial above. When you visit yournodeapp.com, Apache will use a reverse proxy to call the Node.js server running on the port you specified.

You can repeat this process to add as many Node.js projects as you want, but make sure you use a different port for each.

Additionally, I recommend using a Node tool called "forever" (npm install -g forever) to keep your service running in the background and restart it if it crashes.