Sending CollectD Metrics to Graphite

There are no shortage of tutorials on setting up collectd as an agent on a machine. However, I have found little help in the way of describing how to setup a centralized collectd collection server that aggregates statistics from multiple clients and sends them to Graphite. This post will help you do just that. It focuses on Ubuntu, but the instructions are universally applicable.

First, let's setup the central collectd server. This could be on the same machine as your Graphite server, but on large production environments, it is not recommended.

The Ubuntu collectd repositories do not contain the necessary write_graphite plugin, so you must download and install collectd manually. Download the source from the website at: http://collectd.org/download.shtml

Next, run:

tar jxf collectd-version.tar.bz2
cd collectd-version
./configure
make all install

Once collectd is installed, modify the /opt/etc/collectd.conf file to contain the following:

Hostname "hostname"
FQDNLookup true
BaseDir "/opt/collectd/var/lib/collectd"
PIDFile "/opt/collectd/var/run/collectd.pid"
PluginDir "/opt/collectd/lib/collectd"
TypesDB "/opt/collectd/share/collectd/types.db"
Interval 10


LoadPlugin network
<Plugin network>
Listen "*" "12345"
</Plugin>

LoadPlugin interface
<Plugin interface>
Interface "eth0"
</Plugin>

LoadPlugin write_graphite
<Plugin write_graphite>
<Node "graphing">
Host "localhost"
Port "2003"
Protocol "tcp"
LogSendErrors true
Prefix "collectd."
StoreRates true
AlwaysAppendDS false
EscapeCharacter "_"
</Node>
</Plugin>

Make adjustments for your network as needed.

Now, we are going install the collectd agent on the client and then tell it to send the metrics to the collectd server (not Graphite).

The clients do not need the write_graphite plugin and can use the older version of Collectd that ships with the repositories. On each client, run:

sudo apt-get install collectd collectd-utils

Then, cd into /etc/collectd. Backup the collectd.conf file as collectd.conf.bkp or similar and then create a new collectd.conf file. In it, enable the plugins you want and also add the following:

Hostname "hostname"
FQDNLookup true
BaseDir "/var/lib/collectd"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/lib/collectd"
TypesDB "/usr/share/collectd/types.db"
Interval 10
#Timeout 5
ReadThreads 5

LoadPlugin network
<Plugin network>
Server "collectd.domain.com" "12345"
</Plugin>

LoadPlugin cpu
LoadPlugin load
LoadPlugin disk
LoadPlugin memory
LoadPlugin processes

Include "/etc/collectd/filters.conf"
Include "/etc/collectd/thresholds.conf"

Be sure to configure the network plugin with your collectd server information.

Now, if you log into Graphite, you should see that your clients are sending all of their statistics to the collectd server which is then sending them to Graphite.

Creating Multiple Opsview Pager Duty Services

Pager Duty, the popular alerting system for sysadmins, provides easy integration with a wide variety of tools. Recently, I have been using Opsview, a fork of Nagios, for system monitoring. Integrating Opsview with Pager Duty is fairly simple when creating a "catch all" user that sends every alert to Pager Duty. However, creating fine-tuned alerts associated with different Pager Duty services is a bit more complex.

The following steps should be completed for every service that requires a Pager Duty integration:

1. Within Pager Duty, click the "Services" tab and then click "Add New Service."
   
   
   
   
2. Provide a name for the service, select an escalation policy, and then choose "Opsview" in the "Integration" menu.
   
   
   
3. On the next page, copy your Service API Key
4. At this point, you need to prepare Opsview for Pager Duty integration if you haven't done so already. I am not going to repeat these steps because Pager Duty has an excellent guide here. Follow the steps under "On Your Server" and "In Opsview" up to step 13.
5. When you create a new user in Opsview, give it a unique name for the service. In this case, I am using "Pager Duty - Flickr".
6. The role should be Administrator and the password should be something long, complex, and not easily guessable. You will never need it again, so feel free to make it 100 characters.
7. Click Next
8. In the "Pagerduty Service Key" box, enter the key you copied in step 3
9. Submit the changes and you should now see your user listed
10. Now, click on "No Notifications Configured" next to your user to be taken to the notifications page
11. On this page, enter a name, check the box to notify by Pagerduty, and select all of the host groups and services related to that service only
12. Edit the other defaults as needed, then submit the changes
13. Be sure to reload Opsview so your changes take effect
14. Now, you can repeat this process of creating a new user for each service you want to monitor separately

Custom Opsview NRPE Client Agent Service Checks

To install custom checks on Opsview Agents, there are a number of steps that need to be taken. I will assume that the OpsView daemon has already been installed and is running (if not, you can download it here: http://www.opsview.com/technology/downloads/extras/opsview-agents).

First, log into the client on which you want to install a custom check. Then, navigate to the /usr/local/nagios/libexec folder. This is where we will place the actual custom check script you have written. In this case, I will call it "check_test" (it must begin with "check_" to follow the naming conventions. Additionally, the check must return an exit code indicating its status ("0" for "Success", "1" for "Down", etc.).

Now, that you have saved the check_test file in this directory, make it executable and change the owner and group to "nagios".

sudo chmod +x check_test
sudo chown nagios check_test
sudo chgrp nagios check_test

Now, your check is ready to execute, but it must be made available to the NRPE configuration. To do this, navigate to /usr/local/nagios/etc and create a folder called "nrpe_local" if it does not exist already. Then navigate into the folder.

sudo mkdir nrpe_local
cd nrpe_local

Here, create a file called "override.cfg" and change the owner and group to "nagios" just as you did with the script before.

sudo touch override.cfg
sudo chown nagios override.cfg
sudo chgrp nagios override.cfg

Now, open the file in your favorite text editor and add the following line:

command[check_test]=/usr/local/nagios/libexec/check_test

Where "check_test" is the name of your check that you saved earlier.

Save the file, exit, and restart the opsview agent.

sudo service opsview-agent restart

Now, head over to the Opsview web interface on the Opsview server side. Go to Settings > Service Checks and click the + to add a new check. Give it a name "Test Check" and a description of what it does. Then put it into a Service Group or create a new one. It shouldn't need any dependencies, and you can add it to a Host Template as needed. The check type should be "Active Check" and the period, interval, max check attempts, and retry interval can be whatever tou like.

Under "plugin" select "check_nrpe" (not the name of your plugin, which likely won't be in that list).

Under "Arguments" enter the following:

-H $HOSTADDRESS$ -c check_test

Where "check_test is the name of your check. Submit the changes and you should be complete. Reload the configuration and you will see the check show up and it can be added to your host.