Site Monitoring and Operations – Part 2

Building Nagios

We earmarked a single server to act as our host for all virtual servers. Named “Wintermute” (some cyberpunk fans will get a laugh out of this), it was a Windows 2003 RC2 server that had previously been our demo/Roadshow server for the NYC show in May.

Downloading and installing VMWare is pretty simple – just follow the default prompts. You can download the Nagios virtual appliance we used here - http://www.vmware.com/vmtn/appliances/directory/372. This particular virtual appliance is helpful because it includes two additional tools to help configure Nagios. Monarch and Monarch-EZ are open-source applications that provide a web-based configuration tool and this makes the job of configuring Nagios easier. Monarch as a web-based tool is fairly intuitive and we added our production servers in about 15-20 minutes.

But what should we check? Our IIS servers are checked for HTTP and Ping availability; This just means that Nagios verifies that the box can be pinged and is responsive on Port 80. Our mail servers are checked for Ping, HTTP, SMTP and POP services since our users access their mail from the web, different mail applications and mobile devices.

What turned into a very-large problem was getting Nagios to email us when it noted a problem on a server. The virtual appliance uses Sendmail and Exim4 to act as a mail transfer agent – I’m not a Unix guy by nature – how was I going to get this to work? After a few hours looking at various pages, I found this page which turned out to be one I wished I had found in the beginning - http://pkg-exim4.alioth.debian.org/README/README.Debian.html. We still needed to modify some configuration and alias files using vi, but the majority of configuration was handled via the dpkg-reconfigure exim4-config command. By performing this reconfiguration – we were able to immediately get Exim4 to send mail to our mail server and configured a distribution list to forward all Nagios traffic to several people. Although it wasn’t an issue, we also checked to allow SMTP traffic from the virtual appliance to our Exchange server.

Now we were ready for testing. Since no one was using POP mail services on our second mail server, I killed our POP virtual server in Exchange 2003 and waited to see how long Nagios would take to notice it. Not long, as it turned out. In about 2 minutes, I had the “Warning” message appear in our web tool and 2 minutes after that, several messages showed up in our mailboxes to let us know that POP wasn’t responding on our mail server. I restarted the POP virtual server and got several “Recovery” messages after that.

As a nice-to-have, we configured a manual DNS entry to allow the virtual appliance to be accessed from http://nagios/nagios. Manual DNS entries for different web servers is a lot easier than IP addresses for your team to remember.

In our last entry, we will talk about the Burn-in process and Lessons Learned. There are a few things you want to watch out for.

- “Site Monitoring and Operations” continues in Part 3…