On Jun 26, 2008, at 5:22 AM, Rev. Jeffrey Paul wrote:

Hi. I've a (theoretically) simple problem and I'm wondering how others solve it.

I've recently deployed ~40 Linux instances on ~20 different Dell blades and PowerEdges (we're big on virtualization), a few 7204s and 3560s, and assorted switchable PDUs and whatnot.

We need to monitor standard things like cpu, memory, disk usage on all OSes. This is straightforward with net-snmp. It would also be cool if I could monitor more esoteric things, like ntp synchronization status, i/o statistics, etc.

Other stuff we really need to keep an eye on is hardware - redundant PSU status in our 7204s and Dells, temperatures and voltages (one of our colos in New York peaked at over 40C a few weeks ago, for instance), and disk array status (I'd like to know of a failed disk in a hardware RAID5 before I get calls about performance issues). Our blade chassis have DRACs in them and I think they export this data via SNMP (I'm trying to avoid the use of SNMP traps), but not all of our other PowerEdges have the DRACs in them so some of this information may need to be pulled via IPMI from within the host OS. Presumably the Cisco gear makes the temperature available via SNMP.

Finally, service checks - standard stuff (dns, http, https, ssh, smtp).

Now, to the questions.

1) Is SNMP the best way to do this? Obviously some of the data (service checks) will need to be collected other ways.

2) Is there any good solution that does both logging/trending of this data and also notification/monitoring/alerting? I've used both Nagios and Cacti in the past, and, due to the number of individual things being monitored (3-5 items per OS instance, 5-10 items per physical server, 10-50 things per network device), setting them both up independently seems like a huge pain. Also, I've never really liked Nagios that much.

I recently entertained the idea of writing a CGI that output all of this information in a standard format (csv?), distributing and installing it, then collecting it periodically at a central location and doing all the rrd/notification myself, but then realized that this problem must've been solved a million times already.

There's got to be a better way. What do you guys use?

(I'm not opposed to non-free solutions, provided they work better.)

You may want to have a look at Zenoss, http://www.zenoss.com/ Cheers, Andrew

Rev. Jeffrey Paul (sneak) writes:

1) Is SNMP the best way to do this? Obviously some of the data (service checks) will need to be collected other ways.

SNMP, the vendor MIBs + SNMP extensions for monitoring hardware specifics (PSU, etc...), and something like Nagios to do the TCP/network checks.

2) Is there any good solution that does both logging/trending of this data and also notification/monitoring/alerting? I've used both Nagios and Cacti in the past, and, due to the number of individual things being monitored (3-5 items per OS instance, 5-10 items per physical server, 10-50 things per network device), setting them both up independently seems like a huge pain. Also, I've never really liked Nagios that much.

Well, you could look at Zabbix, Hyperic, ZenOSS, OpenNMS and see if they cut it better for you, but the trick with Nagios is to use a DB and generate the include files automatically, then have some other more user friendly tools to populate the DB. Or use templates extensively. Then make sure your plugins output performance data for perf.data monitoring, and use something like NagiosGraph http://nagiosgraph.wiki.sourceforge.net/ or PNP4Nagios: http://www.pnp4nagios.org/pnp/about#system_requirements http://nagiosplug.sourceforge.net/developer-guidelines.html#AEN203 http://www.pnp4nagios.org/pnp/screenshots

I recently entertained the idea of writing a CGI that output all of this information in a standard format (csv?), distributing and installing it, then collecting it periodically at a central location and doing all the rrd/notification myself, but then realized that this problem must've been solved a million times already.

Yes :) But check out the above links, and with a bit of planning and a small amount of coding/adapting existing components, it will work out.

There's got to be a better way. What do you guys use?

We rewrote our own NMS from scratch :)

(I'm not opposed to non-free solutions, provided they work better.)

We sell our solution, so I'm biased, but do check out the Nagios route, it works well enough for small to medium, and larger installations with careful planning (problem with Nagios is how to make it perform with thousands of hosts). Hth, Phil

Rev. Jeffrey Paul wrote:

Hi. I've a (theoretically) simple problem and I'm wondering how others solve it.

Taken one at a time, mos of them are simple. Most of life is like that.

1) Is SNMP the best way to do this? Obviously some of the data (service checks) will need to be collected other ways.

I've actually been out of the admin biz for some time but back in the day I was very found of SNMP tools for all sorts of reporting. For output I liked MRTG for most things, WhatsUpGold had some nice features if you would rather pay money. For alarms, I used some unix hack or another (home-made). I also used home-made hacks to gather data about things that did not have a suitable SNMP interface.

2) Is there any good solution that does both logging/trending of this data and also notification/monitoring/alerting? I've used both Nagios and Cacti in the past, and, due to the number of individual things being monitored (3-5 items per OS instance, 5-10 items per physical server, 10-50 things per network device), setting them both up independently seems like a huge pain. Also, I've never really liked Nagios that much.

See MRTG, RRD, et al.

I recently entertained the idea of writing a CGI that output all of this information in a standard format (csv?), distributing and installing it, then collecting it periodically at a central location and doing all the rrd/notification myself, but then realized that this problem must've been solved a million times already.

There's got to be a better way. What do you guys use?

I had the luxury of management that thought managing was a good idea, so I had a machine pretty much dedicated to systems management and all the machines (including routers, bridges, hubs, and such) reported to it. We had a web interface to the MRTG and MRTG-like presentations.

(I'm not opposed to non-free solutions, provided they work better.)

Just before the fired me for being too old, they bought all the HP and cisco stuff in the world. I do not recommend any of it. -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs

Rev. Jeffrey Paul wrote:

Hi. I've a (theoretically) simple problem and I'm wondering how others solve it.

I've recently deployed ~40 Linux instances on ~20 different Dell blades and PowerEdges (we're big on virtualization), a few 7204s and 3560s, and assorted switchable PDUs and whatnot.

We need to monitor standard things like cpu, memory, disk usage on all OSes. This is straightforward with net-snmp. It would also be cool if I could monitor more esoteric things, like ntp synchronization status, i/o statistics, etc.

Other stuff we really need to keep an eye on is hardware - redundant PSU status in our 7204s and Dells, temperatures and voltages (one of our colos in New York peaked at over 40C a few weeks ago, for instance), and disk array status (I'd like to know of a failed disk in a hardware RAID5 before I get calls about performance issues). Our blade chassis have DRACs in them and I think they export this data via SNMP (I'm trying to avoid the use of SNMP traps), but not all of our other PowerEdges have the DRACs in them so some of this information may need to be pulled via IPMI from within the host OS. Presumably the Cisco gear makes the temperature available via SNMP.

Finally, service checks - standard stuff (dns, http, https, ssh, smtp).

Now, to the questions.

1) Is SNMP the best way to do this? Obviously some of the data (service checks) will need to be collected other ways.

2) Is there any good solution that does both logging/trending of this data and also notification/monitoring/alerting? I've used both Nagios and Cacti in the past, and, due to the number of individual things being monitored (3-5 items per OS instance, 5-10 items per physical server, 10-50 things per network device), setting them both up independently seems like a huge pain. Also, I've never really liked Nagios that much.

I recently entertained the idea of writing a CGI that output all of this information in a standard format (csv?), distributing and installing it, then collecting it periodically at a central location and doing all the rrd/notification myself, but then realized that this problem must've been solved a million times already.

There's got to be a better way. What do you guys use?

I wrote an NMS to do something along these lines. It's focussed more towards graphing than alerting. It knows where to find Dell/Cisco temperature monitors via SNMP and will keep track of hardware and OS types/versions. It's probably still not really ready for general consumption, but if you think it would be useful to you, give me a shout and I'll see if I can help you make it work properly for you. http://www.project-observer.org I wrote it mostly due to my own absolute hatred of Nagios and disappointment at the other NMSes around (where are the asthetics?)! :) Thanks, adam.