Hey, I got a weird problem with my syslog-ng setup, im logging from alot of cisco machines and that works great. The problem is that when i "pass" this further to a shell program, some lines disapere. My destination looks like this destination hosts { file("/var/log/ciscorouters/$HOST.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); program("/scripts/irc/syslog_wrapper_new.sh" template(t_irctempl)); }; The "/var/log/ciscorouters/$HOST.log" writes correct, but the data thats putted trough to "/scripts/irc/syslog_wrapper_new.sh" only get the first line, if it gets flooded (like 5 rows per second). Do anyone of you have any idea of what might be the problem? Regards, Peter
It's hard to say without seeing the actual script. Is your script running as a daemon or are you counting on syslog-ng to start your program per message. If the latter, that's probably not the best strategy. On Sat, May 10, 2014 at 2:24 AM, Peter Persson <webbax@webbax.se> wrote:
Hey,
I got a weird problem with my syslog-ng setup, im logging from alot of cisco machines and that works great. The problem is that when i "pass" this further to a shell program, some lines disapere.
My destination looks like this destination hosts { file("/var/log/ciscorouters/$HOST.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); program("/scripts/irc/syslog_wrapper_new.sh" template(t_irctempl)); }; The "/var/log/ciscorouters/$HOST.log" writes correct, but the data thats putted trough to "/scripts/irc/syslog_wrapper_new.sh" only get the first line, if it gets flooded (like 5 rows per second).
Do anyone of you have any idea of what might be the problem?
Regards, Peter
Off topic. The issue is with the daemon, not your devices. https://lists.balabit.hu/mailman/listinfo/syslog-ng On Sat, May 10, 2014 at 4:24 AM, Peter Persson <webbax@webbax.se> wrote:
Hey,
I got a weird problem with my syslog-ng setup, im logging from alot of cisco machines and that works great. The problem is that when i "pass" this further to a shell program, some lines disapere.
My destination looks like this destination hosts { file("/var/log/ciscorouters/$HOST.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); program("/scripts/irc/syslog_wrapper_new.sh" template(t_irctempl)); }; The "/var/log/ciscorouters/$HOST.log" writes correct, but the data thats putted trough to "/scripts/irc/syslog_wrapper_new.sh" only get the first line, if it gets flooded (like 5 rows per second).
Do anyone of you have any idea of what might be the problem?
Regards, Peter
-- jamie rishaw // .com.arpa@j <- reverse it. ish. "...let's consider this world like a family and care about each other..." -Malala Yousafzai
Another off topic (question) - what kind of fronted UI you use with syslog-ng? I see log analyser based on PHP is common. In my tests it worked fine but it’s major issue I saw was that I couldn’t sort all logs based on individual hosts/devices. What kind of open source web UI everyone is using, just wondering? Thanks. On 11-May-2014, at 12:19 am, jamie rishaw <j@arpa.com> wrote:
Off topic. The issue is with the daemon, not your devices.
https://lists.balabit.hu/mailman/listinfo/syslog-ng
On Sat, May 10, 2014 at 4:24 AM, Peter Persson <webbax@webbax.se> wrote:
Hey,
I got a weird problem with my syslog-ng setup, im logging from alot of cisco machines and that works great. The problem is that when i "pass" this further to a shell program, some lines disapere.
My destination looks like this destination hosts { file("/var/log/ciscorouters/$HOST.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); program("/scripts/irc/syslog_wrapper_new.sh" template(t_irctempl)); }; The "/var/log/ciscorouters/$HOST.log" writes correct, but the data thats putted trough to "/scripts/irc/syslog_wrapper_new.sh" only get the first line, if it gets flooded (like 5 rows per second).
Do anyone of you have any idea of what might be the problem?
Regards, Peter
-- jamie rishaw // .com.arpa@j <- reverse it. ish.
"...let's consider this world like a family and care about each other..." -Malala Yousafzai
-- Anurag Bhatia anuragbhatia.com
I use kibana / elasticsearch -Blake On Sat, May 10, 2014 at 2:15 PM, Anurag Bhatia <me@anuragbhatia.com> wrote:
Another off topic (question) - what kind of fronted UI you use with syslog-ng? I see log analyser based on PHP is common. In my tests it worked fine but it’s major issue I saw was that I couldn’t sort all logs based on individual hosts/devices.
What kind of open source web UI everyone is using, just wondering?
Thanks.
On 11-May-2014, at 12:19 am, jamie rishaw <j@arpa.com> wrote:
Off topic. The issue is with the daemon, not your devices.
https://lists.balabit.hu/mailman/listinfo/syslog-ng
On Sat, May 10, 2014 at 4:24 AM, Peter Persson <webbax@webbax.se> wrote:
Hey,
I got a weird problem with my syslog-ng setup, im logging from alot of cisco machines and that works great. The problem is that when i "pass" this further to a shell program, some lines disapere.
My destination looks like this destination hosts { file("/var/log/ciscorouters/$HOST.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); program("/scripts/irc/syslog_wrapper_new.sh" template(t_irctempl)); }; The "/var/log/ciscorouters/$HOST.log" writes correct, but the data thats putted trough to "/scripts/irc/syslog_wrapper_new.sh" only get the first line, if it gets flooded (like 5 rows per second).
Do anyone of you have any idea of what might be the problem?
Regards, Peter
-- jamie rishaw // .com.arpa@j <- reverse it. ish.
"...let's consider this world like a family and care about each other..." -Malala Yousafzai
-- Anurag Bhatia anuragbhatia.com
Peter, it's a bit difficult to tell what's going on without seeing the rest of the syslog-ng configuration and your script's source code. However, a couple possibilities come to mind: - Your script is only reading one line at a time. syslog-ng starts a program() output persistently and expects that it can send multiple messages into its pipe to your script's stdin. - Messages are being buffered inside of syslog-ng. Check out the flush_lines() and flush_timeout() flags to syslog-ng's program() output. Find the right page for your version, but here's v3.3.: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide... - Messages are being buffered in your shell or script. Maybe try some non-blocking IO with a smallish buffer to see data as it comes in before a whole line or block fills and flushes in. To Anurag's question about open source log management with a WebUI, I agree with Blake: logstash ingesting syslog and inputting it into elasticsearch makes for a great backend for Kibana. The logstash grok filter is great for pulling apart and indexing weird vendor-specific logging formats: http://logstash.net/docs/1.4.1/filters/grok Cheers, jof On Sat, May 10, 2014 at 2:24 AM, Peter Persson <webbax@webbax.se> wrote:
Hey,
I got a weird problem with my syslog-ng setup, im logging from alot of cisco machines and that works great. The problem is that when i "pass" this further to a shell program, some lines disapere.
My destination looks like this destination hosts { file("/var/log/ciscorouters/$HOST.log" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); program("/scripts/irc/syslog_wrapper_new.sh" template(t_irctempl)); }; The "/var/log/ciscorouters/$HOST.log" writes correct, but the data thats putted trough to "/scripts/irc/syslog_wrapper_new.sh" only get the first line, if it gets flooded (like 5 rows per second).
Do anyone of you have any idea of what might be the problem?
Regards, Peter
participants (6)
-
Anurag Bhatia -
Blake Dunlap -
Gary Josack -
jamie rishaw -
Jonathan Lassoff -
Peter Persson