What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features. Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there? Mike
It's a bit old but still works well. Russel Fulton and I worked on this when I was down in NZ. You still need to run syslog-ng but this allows you to ignore, warn, alert on logs via regex. http://www.ip-solutions.net/syslog-ng/ Cheers, Harry On 09/30/2011 09:50 AM, harbor235 wrote:
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
A sub question to this would be - is anyone using an app or client that will forward windows OS events to said collector? I've seen Loglogic and others. Was just curious if you've used a small scale version to collect security events - log on, log off, etc...? -----Original Message----- From: Harry Hoffman [mailto:hhoffman@ip-solutions.net] Sent: Friday, September 30, 2011 6:56 AM To: nanog@nanog.org Subject: Re: events It's a bit old but still works well. Russel Fulton and I worked on this when I was down in NZ. You still need to run syslog-ng but this allows you to ignore, warn, alert on logs via regex. http://www.ip-solutions.net/syslog-ng/ Cheers, Harry On 09/30/2011 09:50 AM, harbor235 wrote:
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
http://code.google.com/p/eventlog-to-syslog/ On Oct 4, 2011, at 11:47 AM, Jones, Barry wrote:
A sub question to this would be - is anyone using an app or client that will forward windows OS events to said collector? I've seen Loglogic and others. Was just curious if you've used a small scale version to collect security events - log on, log off, etc...?
-----Original Message----- From: Harry Hoffman [mailto:hhoffman@ip-solutions.net] Sent: Friday, September 30, 2011 6:56 AM To: nanog@nanog.org Subject: Re: events
It's a bit old but still works well. Russel Fulton and I worked on this when I was down in NZ.
You still need to run syslog-ng but this allows you to ignore, warn, alert on logs via regex.
http://www.ip-solutions.net/syslog-ng/
Cheers, Harry
On 09/30/2011 09:50 AM, harbor235 wrote:
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
I've been testing ManageEngines Syslog application. It works pretty good so far, I haven't really hammered it with a lot of devices. Splunk is suppose to be king of the hill I hear, but so is their pricing.....
Date: Fri, 30 Sep 2011 09:50:29 -0400 Subject: events From: harbor235@gmail.com To: nanog@nanog.org
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
We use splunk works ok except with the amount of text data you can process with it (depends on license). -B On Fri, Sep 30, 2011 at 7:50 AM, harbor235 <harbor235@gmail.com> wrote:
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
-- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments Disclaimer: http://goldmark.org/jeff/stupid-disclaimers/
Is it really that expensive, and WORTH the expense?
Date: Fri, 30 Sep 2011 10:37:22 -0600 Subject: Re: events From: pfunix@gmail.com To: harbor235@gmail.com CC: nanog@nanog.org
We use splunk works ok except with the amount of text data you can process with it (depends on license).
-B
On Fri, Sep 30, 2011 at 7:50 AM, harbor235 <harbor235@gmail.com> wrote:
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
-- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments
Disclaimer: http://goldmark.org/jeff/stupid-disclaimers/
On Fri, Sep 30, 2011 at 11:21 AM, Brandon Kim <brandon.kim@brandontek.com> wrote:
Is it really that expensive, and WORTH the expense?
IMO, from price quotes I've gotten in the past, it's astronomically expensive. As for worth it...depends. If you're dealing with events for say payment processing systems, it might be. But as a general use tool, it's way outside of being worth it. You license based on the incoming bytes of logging data. But you still have to buy the hardware to process it. They also expect you to pay for that license time and time again.
Thank you! That's a bummer about the way they license their product. All it takes is another "splunk" company to come out with something just as competitive.... I've been happy with my basic ManageEngine's syslog, but I may be looking at Solarwinds too...
Date: Fri, 30 Sep 2011 11:36:58 -0600 Subject: Re: events From: mloftis@wgops.com To: brandon.kim@brandontek.com CC: pfunix@gmail.com; harbor235@gmail.com; nanog@nanog.org
On Fri, Sep 30, 2011 at 11:21 AM, Brandon Kim <brandon.kim@brandontek.com> wrote:
Is it really that expensive, and WORTH the expense?
IMO, from price quotes I've gotten in the past, it's astronomically expensive. As for worth it...depends. If you're dealing with events for say payment processing systems, it might be. But as a general use tool, it's way outside of being worth it. You license based on the incoming bytes of logging data. But you still have to buy the hardware to process it. They also expect you to pay for that license time and time again.
On 2011-09-30, at 2:13 PM, Brandon Kim wrote:
I've been happy with my basic ManageEngine's syslog, but I may be looking at Solarwinds too...
I've just installed the Splunk eval myself, but I'm curious about your ManageEngine experiences. I don't have any interest in using ManageEngine as an NMS; I have a couple of tools that I use for that already. Can you use ManageEngine's syslog without having to set it up to monitor all of your devices first? Have you looked at the TRAP support in ManageEngine?
Have you tried qradar? It's rather good On 30 Sep 2011, at 19:21, Jason Lixfeld <jason@lixfeld.ca> wrote:
On 2011-09-30, at 2:13 PM, Brandon Kim wrote:
I've been happy with my basic ManageEngine's syslog, but I may be looking at Solarwinds too...
I've just installed the Splunk eval myself, but I'm curious about your ManageEngine experiences. I don't have any interest in using ManageEngine as an NMS; I have a couple of tools that I use for that already. Can you use ManageEngine's syslog without having to set it up to monitor all of your devices first? Have you looked at the TRAP support in ManageEngine?
On Fri, Sep 30, 2011 at 2:44 PM, Ukpong Ukpong <ukpong.ukpong@gmail.com> wrote:
Have you tried qradar? It's rather good
I've used Splunk and QRadar; both are available as free VMware appliances with limitations on log volume, sufficient for testing. Or if you're mostly looking at webserver/proxy/firewall logs, Sawmill is worth checking out. I've also been looking into using Lancope's replicator to take in syslog UDP and send copies to multiple loggers, since some appliances only support a single syslog destination. Kevin
Good question, we do not use manageengine for NMS and I have no desire to use them either. I tried their NMS platform last year and it was "ok", the interface just seemed a little clunky.... Setting up ManageEngine syslog was a breeze and now we get alerts based on what kind of messages we want, it's pretty hands off, I'm sure you could fine tune it further... But I hear that solarwinds NPM has syslog built into it, so I'm thinking of going with one product that covers it all....
Subject: Re: events From: jason@lixfeld.ca Date: Fri, 30 Sep 2011 14:21:38 -0400 To: nanog@nanog.org
On 2011-09-30, at 2:13 PM, Brandon Kim wrote:
I've been happy with my basic ManageEngine's syslog, but I may be looking at Solarwinds too...
I've just installed the Splunk eval myself, but I'm curious about your ManageEngine experiences. I don't have any interest in using ManageEngine as an NMS; I have a couple of tools that I use for that already. Can you use ManageEngine's syslog without having to set it up to monitor all of your devices first? Have you looked at the TRAP support in ManageEngine?
I'm obviously biased as I'm the Head Geek here at SolarWinds but if you need any help or guidance with our products feel free to ping me off list. Josh -----Original Message----- From: Brandon Kim [mailto:brandon.kim@brandontek.com] Sent: Friday, September 30, 2011 1:14 PM To: mloftis@wgops.com Cc: nanog group Subject: RE: events Thank you! That's a bummer about the way they license their product. All it takes is another "splunk" company to come out with something just as competitive.... I've been happy with my basic ManageEngine's syslog, but I may be looking at Solarwinds too...
Date: Fri, 30 Sep 2011 11:36:58 -0600 Subject: Re: events From: mloftis@wgops.com To: brandon.kim@brandontek.com CC: pfunix@gmail.com; harbor235@gmail.com; nanog@nanog.org
On Fri, Sep 30, 2011 at 11:21 AM, Brandon Kim <brandon.kim@brandontek.com> wrote:
Is it really that expensive, and WORTH the expense?
IMO, from price quotes I've gotten in the past, it's astronomically expensive. As for worth it...depends. If you're dealing with events for say payment processing systems, it might be. But as a general use tool, it's way outside of being worth it. You license based on the incoming bytes of logging data. But you still have to buy the hardware to process it. They also expect you to pay for that license time and time again.
Use Splunk here. Cheers, RR On Fri, Sep 30, 2011 at 9:50 AM, harbor235 <harbor235@gmail.com> wrote:
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/30/2011 09:50 AM, harbor235 wrote:
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
We've made some great strides in OpenNMS in the area of syslog event processing. The upcoming 1.10 release will be much easier to get going, particularly since we now have pluggable message parsers -- you no longer need Wireshark and a black belt in regular expressions to start receiving events from syslog sources. We've also made it possible to split the syslog rules across multiple files, which makes maintaining your own rules much easier compared to the old monolithic style. It's still not going to be Splunk-easy to configure, but it's now darned close to Netcool OMNIbus syslogd probe-easy. Plus you get pretty JasperReports reports based on your events like this one (or roll your own): http://opennms.org/~jeffg/event-analysis-sample.pdf Also flexible event notifications, event de-duplication, and SNMP trap handling as well as service-assurance polling, performance data collection via SNMP, HTTP, WMI, SQL/JDBC, and other protocols. Oh yeah, it's 100% free / libre / open source software. And you can get support for it from my employer. PR hat off, - -jeff -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6GEB0ACgkQB3953+hexDrEPACfRzSKZxijkirgVgTA0OTRrGjX 27IAoJ7Ef0Cv33zRsYVN50YNbL3tVvLq =5v3H -----END PGP SIGNATURE-----
Jeff, When is 1.10 going to be released? thx, /bs On Fri, Sep 30, 2011 at 11:53 AM, Jeff Gehlbach <jeffg@opennms.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 09/30/2011 09:50 AM, harbor235 wrote:
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
We've made some great strides in OpenNMS in the area of syslog event processing. The upcoming 1.10 release will be much easier to get going, particularly since we now have pluggable message parsers -- you no longer need Wireshark and a black belt in regular expressions to start receiving events from syslog sources. We've also made it possible to split the syslog rules across multiple files, which makes maintaining your own rules much easier compared to the old monolithic style.
It's still not going to be Splunk-easy to configure, but it's now darned close to Netcool OMNIbus syslogd probe-easy. Plus you get pretty JasperReports reports based on your events like this one (or roll your own):
http://opennms.org/~jeffg/event-analysis-sample.pdf
Also flexible event notifications, event de-duplication, and SNMP trap handling as well as service-assurance polling, performance data collection via SNMP, HTTP, WMI, SQL/JDBC, and other protocols.
Oh yeah, it's 100% free / libre / open source software. And you can get support for it from my employer.
PR hat off, - -jeff -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk6GEB0ACgkQB3953+hexDrEPACfRzSKZxijkirgVgTA0OTRrGjX 27IAoJ7Ef0Cv33zRsYVN50YNbL3tVvLq =5v3H -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/04/2011 01:33 AM, Brian Spade wrote:
When is [OpenNMS] 1.10 going to be released?
When it's done :) Most likely this month. The unit tests are failing right now: http://bamboo.internal.opennms.com:8085/ But that means that we know where the bugs are :) The 1.9.91 (aka 1.10.0rc2) release is quite solid, and we hope that Tuesday's 1.9.92 (RC3) will be the final release candidate. If you give it a try and run into trouble, be sure to hit the project mailing lists and IRC channel. - -jeff -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6McyQACgkQB3953+hexDrOyQCgqu/MGMXAhfREgwytLkSpq9yQ SLYAn3RWWmvGMi06Hbl1062zoqXTinM8 =13RE -----END PGP SIGNATURE-----
Hi Mike, We have used octopussy ( http://www.8pussy.org/dokuwiki/doku.php?id=home yes it is work safe :-) ) with ok results. Have used sec ( simple event correlator http://simple-evcorr.sourceforge.net/ ) to some success in simple cases. Currently having another look at this myself and the following look interesting, but have not deployed them yet http://logstash.net/ http://graylog2.org/about Ben On 30 Sep 2011, at 14:50, harbor235 wrote:
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
8pussy.org ? -- Leigh Porter On 4 Oct 2011, at 10:59, "Ben Roeder" <ben.roeder@sohonet.co.uk> wrote:
Hi Mike, We have used octopussy ( http://www.8pussy.org/dokuwiki/doku.php?id=home yes it is work safe :-) ) with ok results. Have used sec ( simple event correlator http://simple-evcorr.sourceforge.net/ ) to some success in simple cases.
Currently having another look at this myself and the following look interesting, but have not deployed them yet http://logstash.net/ http://graylog2.org/about
Ben On 30 Sep 2011, at 14:50, harbor235 wrote:
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
+1 for SEC, minimal hit on the cpu like most parsing tools, the regexp can be painful but it is fairly extensible. Once you get used to it you'll love it. On 10/04/2011 05:58 AM, Ben Roeder wrote:
Hi Mike, We have used octopussy ( http://www.8pussy.org/dokuwiki/doku.php?id=home yes it is work safe :-) ) with ok results. Have used sec ( simple event correlator http://simple-evcorr.sourceforge.net/ ) to some success in simple cases.
Currently having another look at this myself and the following look interesting, but have not deployed them yet http://logstash.net/ http://graylog2.org/about
Ben On 30 Sep 2011, at 14:50, harbor235 wrote:
What is everyone using to collect, alert, and analyze syslog data? I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
I've tried quite a few solutions. And the solution that works for engineers who know linux and text parsing, is often ill-suited to many operations folks. I have to admit, Splunk is nice and I prefer it, but the price it outrageous. If I'm logging from 500 routers/switches, I can likely get away with a reasonable 5gb/day license. However, any firewall logging per-connection statistics towards anything reasonably busy will quickly chew through the 5gb in no time with a single device, and I don't like paying more in software licensing to log than I did for the firewall itself. This, combined with the removal of e-mail alerts in the 4.0 version when upgrading from 3.0 resulting in breakage without warning and no downgrade path, irked me. So that solution is out. I've also heard of a coworker liking a solution called PHP-SYSLOG-NG. It's claim to fame was putting the events in a database so they are easily and quickly searchable. I didn't explore it further when I looked about a year ago, as it was clear further development had ceased as the author had turned it into a commercial solution called logzilla. I haven't explored pricing. I now use SEC/simple event coorelator linked by someone below. It works adequately well if you can write a REGEX which matches what you're watching for and an output action. Performance is acceptable, but there is some hit. However, it can keep the logs available in text file format which is nice for data parsing with command line tools for certain cases, where many of the database alternatives don't. The one thing SEC is missing that I would enjoy, is a community based rules database for common alerts in network products. I believe there are adequate open source solutions, but the best seem to be the commercial products, IMHO. On Tue, Oct 4, 2011 at 8:27 AM, Jason LeBlanc <jml@packetpimp.org> wrote:
+1 for SEC, minimal hit on the cpu like most parsing tools, the regexp can be painful but it is fairly extensible. Once you get used to it you'll love it.
On 10/04/2011 05:58 AM, Ben Roeder wrote:
Hi Mike, We have used octopussy ( http://www.8pussy.org/** dokuwiki/doku.php?id=home<http://www.8pussy.org/dokuwiki/doku.php?id=home> yes it is work safe :-) ) with ok results. Have used sec ( simple event correlator http://simple-evcorr.** sourceforge.net/ <http://simple-evcorr.sourceforge.net/> ) to some success in simple cases.
Currently having another look at this myself and the following look interesting, but have not deployed them yet http://logstash.net/ http://graylog2.org/about
Ben On 30 Sep 2011, at 14:50, harbor235 wrote:
What is everyone using to collect, alert, and analyze syslog data?
I am looking for something that can generate reports as well as support multiple vendors. We have done some home grown stuff in the past but would be interested in something that incorprates all the best features.
Soalrwinds, splunk, fwanalog, and others come to mind, any other good ones out there?
Mike
participants (18)
-
Beavis -
Ben Roeder -
Brandon Kim -
Brian Spade -
harbor235 -
Harry Hoffman -
Jason LeBlanc -
Jason Lixfeld -
Jeff Gehlbach -
jeff murphy -
Jones, Barry -
Kevin Kadow -
Leigh Porter -
Michael Loftis -
PC -
Rafael Rodriguez -
Stephens, Josh -
Ukpong Ukpong