improving signal to noise ratio from centralized network syslogs
Hey All, Centralized logging is a good thing. However, what happens is that every repetitive, annoying but not (usually) important thing fills up the log with reams of what you are not looking for. Networks are a noisy place and silencing every logged condition is impractical and sometimes undesirable. What I am interested in is an automated zoom-in zoom-out tool to mask the repetition of "normal" events and allow the unusual to stand out. Add to that an ability to identify gaps in the background noise. (The dog that didnt bark) What I am not interested in are solutions based upon preconfigured filters and definitions and built in analysis for supported (prepopulated definitions) platforms, this is all about pattern mining/masking and should be self discoverable. Ideally a command tool to generate static versions of the analysis coupled with a web platform (with zoom +- buttons) for realtime. I made a crude run of it with SLCT, using its generated patterns to grep -v, and that in and of itself was useful, but needs a bit of work. Also, its not quite real time. Any ideas would be greatly appreciated. Joe
On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon@jmaimon.com> wrote:
Hey All,
Centralized logging is a good thing. However, what happens is that every repetitive, annoying but not (usually) important thing fills up the log with reams of what you are not looking for.
Networks are a noisy place and silencing every logged condition is impractical and sometimes undesirable.
What I am interested in is an automated zoom-in zoom-out tool to mask the repetition of "normal" events and allow the unusual to stand out.
Add to that an ability to identify gaps in the background noise. (The dog that didnt bark)
What I am not interested in are solutions based upon preconfigured filters and definitions and built in analysis for supported (prepopulated definitions) platforms, this is all about pattern mining/masking and should be self discoverable. Ideally a command tool to generate static versions of the analysis coupled with a web platform (with zoom +- buttons) for realtime.
I made a crude run of it with SLCT, using its generated patterns to grep -v, and that in and of itself was useful, but needs a bit of work. Also, its not quite real time.
Any ideas would be greatly appreciated.
Not cheap, but Splunk comes to mind.
Joe
-- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
Splunk is the obvious solution that most organizations with a mature security group will likely already have in their portfolio. Going a step further, and with an abundance of skill, ability, and forethought: either ELK (or any derivative there of such as: Elasticache, Fluentd, Kibana), or rsyslog|syslog-ng + database + loganalzyer. Grep-fu will pay dividends in any of the three options (do nothing, go proprietary, go open). ~Steven On Fri, Jan 26, 2018 at 1:01 AM, Michael Loftis <mloftis@wgops.com> wrote:
On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon@jmaimon.com> wrote:
Hey All,
Centralized logging is a good thing. However, what happens is that every repetitive, annoying but not (usually) important thing fills up the log with reams of what you are not looking for.
Networks are a noisy place and silencing every logged condition is impractical and sometimes undesirable.
What I am interested in is an automated zoom-in zoom-out tool to mask the repetition of "normal" events and allow the unusual to stand out.
Add to that an ability to identify gaps in the background noise. (The dog that didnt bark)
What I am not interested in are solutions based upon preconfigured filters and definitions and built in analysis for supported (prepopulated definitions) platforms, this is all about pattern mining/masking and should be self discoverable. Ideally a command tool to generate static versions of the analysis coupled with a web platform (with zoom +- buttons) for realtime.
I made a crude run of it with SLCT, using its generated patterns to grep -v, and that in and of itself was useful, but needs a bit of work. Also, its not quite real time.
Any ideas would be greatly appreciated.
Not cheap, but Splunk comes to mind.
Joe
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
-- Steven M. Miano http://stevenmiano.com
ELK stack. Java RAM devoring monster but Kibana makes indexing easy. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 01/26/18 01:01, Michael Loftis wrote:
On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon@jmaimon.com> wrote:
Hey All,
Centralized logging is a good thing. However, what happens is that every repetitive, annoying but not (usually) important thing fills up the log with reams of what you are not looking for.
Networks are a noisy place and silencing every logged condition is impractical and sometimes undesirable.
What I am interested in is an automated zoom-in zoom-out tool to mask the repetition of "normal" events and allow the unusual to stand out.
Add to that an ability to identify gaps in the background noise. (The dog that didnt bark)
What I am not interested in are solutions based upon preconfigured filters and definitions and built in analysis for supported (prepopulated definitions) platforms, this is all about pattern mining/masking and should be self discoverable. Ideally a command tool to generate static versions of the analysis coupled with a web platform (with zoom +- buttons) for realtime.
I made a crude run of it with SLCT, using its generated patterns to grep -v, and that in and of itself was useful, but needs a bit of work. Also, its not quite real time.
Any ideas would be greatly appreciated.
Not cheap, but Splunk comes to mind.
Joe
+1 for Graylog, you can pour ALL your syslog data into it, and then configure what are called streams. Streams are a way to whittle down the incoming log flows and see something LESS than everything. You can create a stream that only shows these 6 devices, or one that only shows log info from the RPD daemon on your Juniper routers. In your case, you could use the stream rules to create a stream that filters out the background noise with regex expressions. You're not losing anything, you still have the full log data captured, and you can see it in the portal, but if you click on one of your streams, you see filtered data based, on your rulesets. We've been using it for about 2 years now I think. It's open source, easy to set up, supports LDAP, multiple input types (beyond just udp syslog), and the community is pretty solid. Sincerely, Casey Russell Network Engineer [image: KanREN] <http://www.kanren.net> [image: phone]785-856-9809 2029 Becker Drive, Suite 282 Lawrence, Kansas 66047 [image: linkedin] <https://www.linkedin.com/company/92399?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A92399%2Cidx%3A1-1-1%2CtarId%3A1440002635645%2Ctas%3AKanREN> [image: twitter] <https://twitter.com/TheKanREN> [image: twitter] <http://www.kanren.net/feed/> need support? <support@kanren.net> On Fri, Jan 26, 2018 at 10:41 AM, Alain Hebert <ahebert@pubnix.net> wrote:
ELK stack.
Java RAM devoring monster but Kibana makes indexing easy.
----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles <https://maps.google.com/?q=50+boul.+St-Charles&entry=gmail&source=g> P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
On 01/26/18 01:01, Michael Loftis wrote:
On Thu, Jan 25, 2018 at 8:11 PM Joe Maimon <jmaimon@jmaimon.com> wrote:
Hey All,
Centralized logging is a good thing. However, what happens is that every repetitive, annoying but not (usually) important thing fills up the log with reams of what you are not looking for.
Networks are a noisy place and silencing every logged condition is impractical and sometimes undesirable.
What I am interested in is an automated zoom-in zoom-out tool to mask the repetition of "normal" events and allow the unusual to stand out.
Add to that an ability to identify gaps in the background noise. (The dog that didnt bark)
What I am not interested in are solutions based upon preconfigured filters and definitions and built in analysis for supported (prepopulated definitions) platforms, this is all about pattern mining/masking and should be self discoverable. Ideally a command tool to generate static versions of the analysis coupled with a web platform (with zoom +- buttons) for realtime.
I made a crude run of it with SLCT, using its generated patterns to grep -v, and that in and of itself was useful, but needs a bit of work. Also, its not quite real time.
Any ideas would be greatly appreciated.
Not cheap, but Splunk comes to mind.
Joe
On Thu, Jan 25, 2018 at 11:10:02PM -0500, Joe Maimon wrote:
What I am interested in is an automated zoom-in zoom-out tool to mask the repetition of "normal" events and allow the unusual to stand out.
This is an approach outlined by Marcus Ranum years ago; he called it "artificial stupidity", and it works. (Of course, an inverse check that makes sure routine boring things are still happening is also a good idea.) You could use any number of elaborate (and sometimes expensive) tools to do this, but I recommend rolling your own with Perl or similar. This is goodness for two reasons: first, it forces you to look at your own data, which is really helpful. You'll be surprised at what you find if you've never done it before. Second, it lets you customize for your environment at every step. I have written dozens of these, some as trivial as a few lines of code, some quite extensive. None of them "solve" the problem per se, they just all take bites out of it. But this admittedly-simplistic (and deliberately so) approach has flagged a lot of issues, and because it's simple, it's easy to connect to other monitoring/alerting plumbing. ---rsk
From the systems side we got HoneycombIO which shifts a bit to calling itself events rather than logs management. I don't know anyone else who's tried using it for networks per se but that's on my "interesting tech tools explorations" medium length list. -george Sent from my iPhone
On Jan 31, 2018, at 7:17 AM, Rich Kulawiec <rsk@gsp.org> wrote:
On Thu, Jan 25, 2018 at 11:10:02PM -0500, Joe Maimon wrote: What I am interested in is an automated zoom-in zoom-out tool to mask the repetition of "normal" events and allow the unusual to stand out.
This is an approach outlined by Marcus Ranum years ago; he called it "artificial stupidity", and it works. (Of course, an inverse check that makes sure routine boring things are still happening is also a good idea.)
You could use any number of elaborate (and sometimes expensive) tools to do this, but I recommend rolling your own with Perl or similar. This is goodness for two reasons: first, it forces you to look at your own data, which is really helpful. You'll be surprised at what you find if you've never done it before. Second, it lets you customize for your environment at every step.
I have written dozens of these, some as trivial as a few lines of code, some quite extensive. None of them "solve" the problem per se, they just all take bites out of it. But this admittedly-simplistic (and deliberately so) approach has flagged a lot of issues, and because it's simple, it's easy to connect to other monitoring/alerting plumbing.
---rsk
participants (8)
-
Alain Hebert -
Casey Russell -
Edwin Pers -
George William Herbert -
Joe Maimon -
Michael Loftis -
Rich Kulawiec -
Steven Miano