Re: improving signal to noise ratio from centralized network syslogs
--- jmaimon@jmaimon.com wrote: Centralized logging is a good thing. However, what happens is that every repetitive, annoying but not (usually) important thing fills up the log with reams of what you are not looking for. --------------------------------------- Apologies, I'm late to the party. But I just want to add one thing for the archives. It's along with what Rich Kulawiec said, "it forces you to look at your own data, which is really helpful. You'll be surprised at what you find if you've never done it before." This is accurate. It's fun to see what your network is putting out. This is all from memory (I've done it so many times it's in there permanently... :-) as I don't have a unix server or a router in front of me to use, so don't hold me to exact details... And it's mainly for the newbies. Have all the routers send to one syslog file, switches to another and other devices to a third on a *nix box: For example, send the router messages to /var/log/router.log and the switch messages to /var/log/switch.log This is done with the 'logging facility' command on the devices: After defining your syslog server's IP address and the level of messaging you want (I set it to debug because I want to see everything): on the routers: logging facility local0 on the switches: logging facility local1 on the logging server in: /etc/rsyslog.conf local0.* /var/log/router.log local1.* /var/log/switch.log Use logrotate to manage the log files as they can get quite large. Then, you can watch your network in real time like so (below is all one line): tail -f /var/log/router.log /var/log/switch.log | egrep -vi 'term1|term2|termN' 'egrep -v' takes out all the lines you don't want to see while the syslog messages scroll across the screen. Say there is a battery condition on router1 and a duplex mismatch on a switch I don't want to see: tail -f /var/log/router.log /var/log/switch.log | egrep -vi 'router1.*battery|switch1.*duplex.*mismatch' For me, N can get to 40-50 sometimes, so I put it into a shell script like so: vi log.sh --------------------------- #! /bin/sh tail -f /var/log/router.log /var/log/switch.log | egrep -v 'term1|term2|termN' --------------------------- then, run it like so: ./log.sh It's all netgeek fun-n-games from there on. :) scott
hey,
This is done with the 'logging facility' command on the devices:
After defining your syslog server's IP address and the level of messaging you want (I set it to debug because I want to see everything):
on the routers: logging facility local0 on the switches: logging facility local1
Alternative, and more universal, way to do it is to use multiple IPs for syslog server. Then configure correct syslog server IP on the device. syslog-ng and others can all do filtering to different destinations based on the IP where message was received. -- tarko
I really recommend setting up fluentd, and then routing logging from there - it makes it very easy to keep auditor-appeasing logs, while also having important stuff sending pages. Log aggregation, organization, and search is a hard problem, other people have already done it and provided it as a service, and chances are its NOT a core competency or secret sauce at your organization. Once you get your logs in one routing system, you can do a lot with them, but stop rolling your own. This is a prime area for most companies to buy something that works better, for less than the cost of developing in house. And if you run your own aggregation layer - then you can easily try out a bunch of different systems and add/remove them easily. :) Also, you may want to see one level of logs, but your auditors might wanna see another, and your engineers/sec team might wanna do some analytics on them. Being able to provide a solution for everyone who needs network logs at whatever detail level they ask for will make you popular at your organization. On Sun, Feb 4, 2018 at 12:21 AM, Tarko Tikan <tarko@lanparty.ee> wrote:
hey,
This is done with the 'logging facility'
command on the devices:
After defining your syslog server's IP address and the level of messaging you want (I set it to debug because I want to see everything):
on the routers: logging facility local0 on the switches: logging facility local1
Alternative, and more universal, way to do it is to use multiple IPs for syslog server. Then configure correct syslog server IP on the device.
syslog-ng and others can all do filtering to different destinations based on the IP where message was received.
-- tarko
In addition to that, you can use some fancy awk colour coding, so you can make it highlight certain lines based on content.. I use this for my e-mail logs, but I’m sure it could be adapted: tail -n 1000 -f /var/log/mail-submission.log | grep smtp.*relay | awk ' /sent/ {print "\033[32m" $0 "\033[39m"} /bounced/ {print "\033[31m" $0 "\033[39m"} /deferred/ {print "\033[33m" $0 "\033[39m"} '
On 4 Feb 2018, at 5:49 am, Scott Weeks <surfer@mauigateway.com> wrote:
--- jmaimon@jmaimon.com wrote: Centralized logging is a good thing. However, what happens is that every repetitive, annoying but not (usually) important thing fills up the log with reams of what you are not looking for. ---------------------------------------
Apologies, I'm late to the party. But I just want to add one thing for the archives. It's along with what Rich Kulawiec said, "it forces you to look at your own data, which is really helpful. You'll be surprised at what you find if you've never done it before." This is accurate. It's fun to see what your network is putting out.
This is all from memory (I've done it so many times it's in there permanently... :-) as I don't have a unix server or a router in front of me to use, so don't hold me to exact details...
And it's mainly for the newbies.
Have all the routers send to one syslog file, switches to another and other devices to a third on a *nix box: For example, send the router messages to /var/log/router.log and the switch messages to /var/log/switch.log This is done with the 'logging facility' command on the devices:
After defining your syslog server's IP address and the level of messaging you want (I set it to debug because I want to see everything):
on the routers: logging facility local0 on the switches: logging facility local1
on the logging server in: /etc/rsyslog.conf local0.* /var/log/router.log local1.* /var/log/switch.log
Use logrotate to manage the log files as they can get quite large.
Then, you can watch your network in real time like so (below is all one line):
tail -f /var/log/router.log /var/log/switch.log | egrep -vi 'term1|term2|termN'
'egrep -v' takes out all the lines you don't want to see while the syslog messages scroll across the screen.
Say there is a battery condition on router1 and a duplex mismatch on a switch I don't want to see:
tail -f /var/log/router.log /var/log/switch.log | egrep -vi 'router1.*battery|switch1.*duplex.*mismatch'
For me, N can get to 40-50 sometimes, so I put it into a shell script like so:
vi log.sh
--------------------------- #! /bin/sh
tail -f /var/log/router.log /var/log/switch.log | egrep -v 'term1|term2|termN' ---------------------------
then, run it like so: ./log.sh
It's all netgeek fun-n-games from there on. :)
scott
On 2018-02-03 15:49, Scott Weeks wrote:
Then, you can watch your network in real time like so (below is all one line):
tail -f /var/log/router.log /var/log/switch.log | egrep -vi 'term1|term2|termN'
'egrep -v' takes out all the lines you don't want to see while the syslog messages scroll across the screen.
Syslog-ng can do regex filtering on messages also. So instead of doing an 'egrep -v' on a huge file after it has been logged, you can put your filter right into the syslog-ng configuration, and have those filtered messages output to a file (or any other output that syslog-ng supports). The result is a smaller file to search and work with. We implemented a simple email alerter using this functionality. In syslog-ng, we set up two filters. One filter does the 'egrep -v': filter f_email_msg { not message("%PKT_INFRA-LINEPROTO-.*[0-9/]+\\.") # filter out subinterface up/downs and not message("%PKT_INFRA-LINEPROTO-.*Multilink") and not message("%PKT_INFRA-LINEPROTO-.*Serial") and not message("%PKT_INFRA-LINEPROTO-.*Tunnel") # etc }; Another filter applied to the messages filters messages to just our core devices: filter f_email_sources { host("192.0.2.1") or host("192.0.2.2") or host("192.0.2.3") or host("192.0.2.4") or host("192.0.2.5") or host("192.0.2.6") }; Then those are tied together in a syslog-ng rule that outputs to a file: destination d_email_log { file("/var/log/syslog-ng/alert/alerts.log" template("$HOST:$MSG\n") create_dirs(yes) ); }; log { source(s_devices); filter(f_email_sources); filter(f_email_msg); destination(d_email_log); }; A lightweight Python script that runs as a daemon checks that file once every 10 seconds, and if the file length is non-zero, it sends the contents of the file in an email to the admins. A shell script run as a cron job would work equally as well. (Also, for emailed syslogs, there is more incentive for the admin to keep her or his message filter up to date, as opposed to a file the administrator must manually examine. Otherwise the admin has a full inbox :) ) It's very simple and stable, and has worked better than the commercial product we used to use for this purpose. -Brian
participants (5)
-
Brian Knight -
Jippen -
Scott Weeks -
Shane Short -
Tarko Tikan