Modern Monitoring with Sensu

Configuring a Sensu Server

To get a Sensu server up and working we’ll need to do the following:

  1. Install / Enable the EPEL repository.
  2. Install erlang.
  3. Install and configure the rabbitmq service.
  4. Install and configure the redis service.
  5. Install the sensu repository.
  6. Install and configure the sensu and uchiwa services.
  7. Install and configure the sensu agent.
  8. Configure system / service checks.
  9. Configure email alerts.

This can all be done on one box, but at scale you can have several sensu servers work in union… but more on that later.

You’ll need to install and enable the epel repository:

yum install -y epel-\*
# or something like
rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

With the epel repo installed we can install erlang.

yum install erlang -y

Next we need rabbitmq. RabbitMQ acts as the messaging / queuing mechanism between the agents and the servers.

rpm -Uvh http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.1/rabbitmq-server-3.2.1-1.noarch.rpm

Start the service, and enable it at boot time.

service rabbitmq-server start
chkconfig rabbitmq-server on

In order for us to make use of RabbitMQ we’ll need to do a bit of configuration. This is a working configuration using self-signed SSL certificates:

/etc/rabbitmq/rabbitmq.config:

\[
    {rabbit, \[
    {ssl\_listeners, \[5671\]},
    {ssl\_options, \[{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                   {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                   {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                   {verify,verify\_none},
                   {fail\_if\_no\_peer\_cert,false}\]}
  \]}
\].

Do not use these setting in production, as the last two options disable certificate validation and the requirement to have the certificate on the client. This is intended for demonstration purposes only.

You will also need to install certificates to /etc/rabbitmq/ssl/ for the listener to start.

Once the configurations are in place, restart the service with the new settings:

service rabbitmq-server restart

Now we’re going to add a data center (sensu), a user, and grant the user rights to that data center.

rabbitmqctl add\_vhost /sensu
rabbitmqctl add\_user sensu mypass
rabbitmqctl set\_permissions -p /sensu sensu '.\*' '.\*' '.\*'

In line #1 there, the /sensu was the data center path that we were creating. In your environment you could create it as /sensu-us-west-2 or anything else that you wanted. Line #2 creates a RabbitMQ username and password for accessing the service. Line #3 grants the a user permission in our data center path, with permissions for configuring, writing, and reading.

Next we need redis installed and running. Redis maintains sensu’s state.

yum install -y redis
chkconfig redis on
service redis start

The default settings for redis are pretty sane for just local testing, although you would need to tune them in a Production environment. When possible I recommend using a service like ElastiCache.

Next we need to add the sensu repository:

echo '\[sensu\]
name=sensu
baseurl=http://repos.sensuapp.org/yum/el/$basearch/
gpgcheck=0
enabled=1' | sudo tee /etc/yum.repos.d/sensu.repo

Now we can install sensu and sensu’s dashboard, uchiwa:

yum install -y sensu uchiwa

Next we need to configure Sensu. We’re going to split out our configuration into several different files under the /etc/sensu/conf.d/ directory.

/etc/sensu/conf.d/api.json:

{
  "api": {
    "host": "localhost",
    "port": 4567,
    "user": "admin",
    "password": "mypass"
  }
}

/etc/sensu/conf.d/rabbitmq.json:

{
  "rabbitmq": {
    "ssl": {
      "cert\_chain\_file": "/etc/sensu/ssl/cert.pem",
      "private\_key\_file": "/etc/sensu/ssl/key.pem"
    },
    "host": "localhost",
    "port": 5671,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "mypass"
  }
}

/etc/sensu/conf.d/redis.json:

{
  "redis": {
    "host": "localhost",
    "port": 6379
  }
}

That’s all the configuring that we need to get our Sensu server working, although in production you would want to configure everything with certificates.

To get the Uchiwa web interface working is much simpler:

/etc/sensu/uchiwa.json:

{
  "sensu": \[
    {
      "name": "Sensu",
      "host": "127.0.0.1",
      "ssl": false,
      "port": 4567,
      "user": "admin",
      "pass": "mypass",
      "path": "",
      "timeout": 5000
    }
  \],
  "uchiwa": {
    "user": "",
    "pass": "",
    "port": 3000,
    "stats": 10,
    "refresh": 10000
  }
}

If you want a user and password to protect your uchiwa dashboard add them in. And again, in production you would want to configure everything with certificates.

Once you’ve configured sensu and uchiwa start the services with:

chkconfig sensu-server on
chkconfig sensu-client on
chkconfig uchiwa on
service sensu-server start
service sensu-api start
service sensu-client start
service uchiwa start

And with everything started you can view the dashboard at: http://[ip]:3000

Configuring a Sensu Agent

Configuring a sensu agent is relatively easy. Again make sure that you’ve installed the sensu repo and sensu package. Then the only configuration that the client needs is RabbitMQ and a client configuration file.

/etc/sensu/conf.d/client.json

{
  "client": {
    "name": "clientname",
    "address": "ipaddr",
    "subscriptions": \[ "all" \]
  }
}

Replace “clientname” with the name of the client as you want it to appear. Also replace “ipaddr” with the ip address or resolvable DNS hostname.

When configuring a client, the concept of subscriptions is important. You’ll notice above that currently this client is subscribed to a subscription called “all”. Subscriptions are similar to Nagios’s host and service groups.

Next we need to configure our RabbitMQ configuration file as well:

/etc/sensu/conf.d/rabbitmq.json

{
  "rabbitmq": {
    "ssl": {
      "cert\_chain\_file": "/etc/sensu/ssl/cert.pem",
      "private\_key\_file": "/etc/sensu/ssl/key.pem"
    },
    "host": "192.168.250.100",
    "port": 5671,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "mypass"
  }
}

RabbitMQ is how the Sensu server will receive data from the agent.

With the configuration done we can enable and start the agent:

chkconfig sensu-client on
service sensu-client start

Configuring Sensu Checks

At this point we should have a working sensu server with at least one client connected. We should have access to a dashboard that allows us to monitor the system status.

With Sensu, you write check definition files and place them in /etc/sensu/conf.d/ on the server. We’re going to make a check that confirms that a cron process is running.

/server/sensu/conf.d/check_cron.json:

{
  "checks": {
    "cron\_check": {
      "handlers": \["default"\],
      "command": "/etc/sensu/plugins/check-procs.rb -p crond -C 1 ",
      "interval": 60,
      "subscribers": \[ "all" \]
    }
  }
}

So what we’ve done is created a check called “cron_check” that runs an arbitrary command (in this case a ruby script I’ve copied to the system) at a specified interval on all the systems that are part of the “all” subscription and sends the results to the default handler (which does nothing at the moment).

After we restart the sensu-server we can check the dashboard to see the status of this check.

What’s important to note is that you can run any command as a check - that is I could run pretty much anything I want as a check. This does include things like rm -rf / and shutdown now.

Configuring Sensu Handlers

Right now we have a check running on a client, that reports as up or down on the uchiwa dashboard. This is fine for some things, but typically we’ll want an emailed alert. Checks pass data (such as success or failure) to a handler. The handler then does “something” with that data. Like send an email.

Here’s a basic handler definition:

/etc/sensu/conf.d/handler_mailer.json

{
  "handlers": {
    "mailer": {
      "type": "pipe",
      "command": "/etc/sensu/handlers/mailer.rb"
    }
  }
}

With some additional configuring for the script:

/etc/sensu/conf.d/mailer.json:

{
  "mailer": {
    "admin\_gui": "http://sensu:3000/",
    "mail\_from": "mailfrom",
    "mail\_to": "mailto",
    "smtp\_address": "smtpaddr",
    "smtp\_port": "smtpport",
    "smtp\_domain": "smtpdomain"
  }
}

You will also need this, the actual handler script.

There are several things you can do with handlers, like sending data to graphite, etc. You can also apply filtering so that staging produces no alert but production does.

$ vagrant up

You can bootstrap this example with the pre-configured vagrant environment available in this repo.