Creating Multiple Opsview Pager Duty Services

Pager Duty, the popular alerting system for sysadmins, provides easy integration with a wide variety of tools. Recently, I have been using Opsview, a fork of Nagios, for system monitoring. Integrating Opsview with Pager Duty is fairly simple when creating a "catch all" user that sends every alert to Pager Duty. However, creating fine-tuned alerts associated with different Pager Duty services is a bit more complex.

The following steps should be completed for every service that requires a Pager Duty integration:

1. Within Pager Duty, click the "Services" tab and then click "Add New Service."
   
   
   
   
2. Provide a name for the service, select an escalation policy, and then choose "Opsview" in the "Integration" menu.
   
   
   
3. On the next page, copy your Service API Key
4. At this point, you need to prepare Opsview for Pager Duty integration if you haven't done so already. I am not going to repeat these steps because Pager Duty has an excellent guide here. Follow the steps under "On Your Server" and "In Opsview" up to step 13.
5. When you create a new user in Opsview, give it a unique name for the service. In this case, I am using "Pager Duty - Flickr".
6. The role should be Administrator and the password should be something long, complex, and not easily guessable. You will never need it again, so feel free to make it 100 characters.
7. Click Next
8. In the "Pagerduty Service Key" box, enter the key you copied in step 3
9. Submit the changes and you should now see your user listed
10. Now, click on "No Notifications Configured" next to your user to be taken to the notifications page
11. On this page, enter a name, check the box to notify by Pagerduty, and select all of the host groups and services related to that service only
12. Edit the other defaults as needed, then submit the changes
13. Be sure to reload Opsview so your changes take effect
14. Now, you can repeat this process of creating a new user for each service you want to monitor separately

CURL POST JSON Data from a File

Sometimes I need to POST JSON data to an API endpoint for testing that is too large to fit easily on the command line in the typical "{"id":1,"string":"something"}" format. Instead, it is much easier to just save the entire JSON structure to a file and then reference that file in the CURL.

For example, with the JSON data saved to temp.json:

curl -X POST -H 'content-type: application/json' -d @temp.json http://dev.com/api/

It's as simple as that!

How to Fix OpenSSL Heart Bleed Bug on Amazon ELBs

The recently discovered "Heart Bleed" bug in OpenSSL is an extremely critical security issue. Amazon has been working to get all of their environments patched to the latest version of OpenSSL that remedies the issue.

If you have Elastic Load Balancers currently using an SSL certificate that was generated via OpenSSL version 1.1.0a-f, you need to follow these streps to revoke the current certificate on your load balancer and upload a new one.

First, update OpenSSL on the machine you are going to use to generate your private key and sign your certificate. I have written another post on how to do that here: http://blog.matthewdfuller.com/2014/04/how-to-fix-openssl-heart-bleed-bug-on.html

Once you have regenerated your keys and resigned your certificate, you can upload them to your load balancers.

Within the AWS console, click "EC2" in the Services menu.

Now, click on "Load Balancers" on the left-hand side and select your load balancer instance.

Click on the "Listeners" tab and notice the existing cert:

Click "Change" and then click "Upload a new cert."

Give your cert a name, paste in the private key and cert you created earlier, and provide any chain information if needed.

Hit save and your load balancer will push the changes.

If you want to do this via the command line or API, check out the official AWS documentation: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/US_UpdatingLoadBalancerSSL.html

How to Fix OpenSSL Heart Bleed Bug on Ubuntu

If you're looking for how to update your Amazon Elastic Load Balancer, click here instead.

The recently discovered "Heart Bleed" bug in OpenSSL is an extremely critical security issue. Fixing it is relatively simple now that Ubuntu has pushed out changes to their repositories containing a fixed version of OpenSSL.

The following steps need to be run on each server that you generated a certificate or private key on. If you are using one certificate on multiple servers, then the cert needs to be revoked and regenerated on one of them and then pushed to each of the other servers.

UPDATE: Thanks to anonymous commenter for pointing out that relying solely on the build information is not completely accurate. Versions earlier than 1.0.1 are not vulnerable (although you should upgrade now that a fix is live for the latest version).

First, to make sure you (for some reason) don't have the latest version, run the following commands:

openssl version -b

openssl version -a

The response will look like:

OpenSSL 1.0.1 14 Mar 2012

built on: Wed Jan  8 20:45:51 UTC 2014

If the date is not more recent than older than "Mon Apr  7 20:33:29 UTC 2014" and the version is 1.0.1, then you are vulnerable to the Heart Bleed bug.

UPDATE: Reworded the above to make it clearer that the vulnerable versions were built before April 7th.

UPDATE: As James points out in the comments, different versions may have been built at different times, thus you should rely only on the date, not the time. Anything before Apr 7 is considered vulnerable.

Next, update your repositories:

sudo apt-get update

Once this finishes, upgrade openssl:

sudo apt-get upgrade openssl

sudo apt-get install openssl libssl1.0.0

UPDATE: use the install command to upgrade only openssl and libssl rather than upgrading everything on the server.

Once the upgrade finishes, check the version again. It should now read "Apr 7" or later.

Now, you need to regenerate your certificate using a new private key. This process is the same as it as always been, but I am including the link here for posterity's sake:

https://www.digitalocean.com/community/articles/how-to-create-a-ssl-certificate-on-apache-for-ubuntu-12-04

(Use step 3 and replace the key and cert names with your existing ones to overwrite them).

Once finished, you need to restart your Apache server and any services using SSL.

Update: Now with video:

Find the Public Hostname of an Amazon EC2 Instance

If you need to find the public hostname of an Amazon EC2 instance, there is a simple URL which, when cURL'ed will return the hostname:

http://169.254.169.254/latest/meta-data/public-hostname

This URL only works inside the AWS infrastructure (so you must be on an instance to run it properly), but you could easily run:

curl http://169.254.169.254/latest/meta-data/public-hostname

This will return the hostname in the standard format:

ec2-54-123-256-12.compute-1.amazonaws.com

Custom Opsview NRPE Client Agent Service Checks

To install custom checks on Opsview Agents, there are a number of steps that need to be taken. I will assume that the OpsView daemon has already been installed and is running (if not, you can download it here: http://www.opsview.com/technology/downloads/extras/opsview-agents).

First, log into the client on which you want to install a custom check. Then, navigate to the /usr/local/nagios/libexec folder. This is where we will place the actual custom check script you have written. In this case, I will call it "check_test" (it must begin with "check_" to follow the naming conventions. Additionally, the check must return an exit code indicating its status ("0" for "Success", "1" for "Down", etc.).

Now, that you have saved the check_test file in this directory, make it executable and change the owner and group to "nagios".

sudo chmod +x check_test
sudo chown nagios check_test
sudo chgrp nagios check_test

Now, your check is ready to execute, but it must be made available to the NRPE configuration. To do this, navigate to /usr/local/nagios/etc and create a folder called "nrpe_local" if it does not exist already. Then navigate into the folder.

sudo mkdir nrpe_local
cd nrpe_local

Here, create a file called "override.cfg" and change the owner and group to "nagios" just as you did with the script before.

sudo touch override.cfg
sudo chown nagios override.cfg
sudo chgrp nagios override.cfg

Now, open the file in your favorite text editor and add the following line:

command[check_test]=/usr/local/nagios/libexec/check_test

Where "check_test" is the name of your check that you saved earlier.

Save the file, exit, and restart the opsview agent.

sudo service opsview-agent restart

Now, head over to the Opsview web interface on the Opsview server side. Go to Settings > Service Checks and click the + to add a new check. Give it a name "Test Check" and a description of what it does. Then put it into a Service Group or create a new one. It shouldn't need any dependencies, and you can add it to a Host Template as needed. The check type should be "Active Check" and the period, interval, max check attempts, and retry interval can be whatever tou like.

Under "plugin" select "check_nrpe" (not the name of your plugin, which likely won't be in that list).

Under "Arguments" enter the following:

-H $HOSTADDRESS$ -c check_test

Where "check_test is the name of your check. Submit the changes and you should be complete. Reload the configuration and you will see the check show up and it can be added to your host.

How to Change the Default Timezone on Amazon Ubuntu EC2 Server

After setting up a Ubuntu Server on Amazon EC2, it's annoying that the default timezone is UTC. Here is how to change it:

sudo dpkg-reconfigure tzdata

Then restart your cron service. Otherwise, the timezone should be changed. This will allow the correct times to show up in any applications you have running.