Spiffy Netflow tools?
Howdy! Checking out various Netflow tools and wanted to see what others are using? Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others. Also curious about on-prem solutions as well. Thanks! Mike
Hey Mike. Kentik does on-prem, too. Full disclosure: I work for Kentik and I’m glad you think we’re cool :-) Dan On Mon, Mar 12, 2018 at 4:26 PM <mike.lyon@gmail.com> wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
-- Thanks, Dan
I'm very fond of nfsen/nfdump for on-prem. Setup is not complicated at all and plugins are widely available. Also inbefore Solarwinds... -Matt On Mar 12, 2018 18:25, <mike.lyon@gmail.com> wrote: Howdy! Checking out various Netflow tools and wanted to see what others are using? Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others. Also curious about on-prem solutions as well. Thanks! Mike
On 2018-03-13 00:24, mike.lyon@gmail.com wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
Kentik is probably top of the foodchain right now. But they are certainly not alone in the biz. Ontop of my head... * Flowmon * Talaia * Arbor Peakflow * Deepfield * Pmacct + supporting toolkit * NFsen/Nfdump/AS-stats * Put kibana/ES infront of any collector * Solarwinds something something * Different vendor toolkits -- hugge
Plixer is also interesting. nfdump works great with NetFlow but support for IPFIX is somehow limited to basics. -- Babak On 13 Mar 2018, at 3:20, Fredrik Korsbäck wrote:
On 2018-03-13 00:24, mike.lyon@gmail.com wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
Kentik is probably top of the foodchain right now.
But they are certainly not alone in the biz. Ontop of my head...
* Flowmon * Talaia * Arbor Peakflow * Deepfield * Pmacct + supporting toolkit * NFsen/Nfdump/AS-stats * Put kibana/ES infront of any collector * Solarwinds something something * Different vendor toolkits
-- hugge
Disclaimer: Am Plixer engineer. If you want to take it for a spin, you can download a fully functional OVA/QCOW2 30 day eval from the plixer website. I can also get you access to an AWS AMI as well. I don’t want to turn this into an Ad. So DM if you need any info/access. Mike Krygeris On Tue, Mar 13, 2018 at 11:52 AM Babak Farrokhi <babak@farrokhi.net> wrote:
Plixer is also interesting.
nfdump works great with NetFlow but support for IPFIX is somehow limited to basics.
-- Babak
On 13 Mar 2018, at 3:20, Fredrik Korsbäck wrote:
On 2018-03-13 00:24, mike.lyon@gmail.com wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
Kentik is probably top of the foodchain right now.
But they are certainly not alone in the biz. Ontop of my head...
* Flowmon * Talaia * Arbor Peakflow * Deepfield * Pmacct + supporting toolkit * NFsen/Nfdump/AS-stats * Put kibana/ES infront of any collector * Solarwinds something something * Different vendor toolkits
-- hugge
+1 for Plixer Scrutinizer 2018-03-17 19:42 GMT-03:00 Michael Krygeris <me@krygerism.com>:
Disclaimer: Am Plixer engineer. If you want to take it for a spin, you can download a fully functional OVA/QCOW2 30 day eval from the plixer website. I can also get you access to an AWS AMI as well. I don’t want to turn this into an Ad. So DM if you need any info/access.
Mike Krygeris
On Tue, Mar 13, 2018 at 11:52 AM Babak Farrokhi <babak@farrokhi.net> wrote:
Plixer is also interesting.
nfdump works great with NetFlow but support for IPFIX is somehow limited to basics.
-- Babak
On 13 Mar 2018, at 3:20, Fredrik Korsbäck wrote:
On 2018-03-13 00:24, mike.lyon@gmail.com wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
Kentik is probably top of the foodchain right now.
But they are certainly not alone in the biz. Ontop of my head...
* Flowmon * Talaia * Arbor Peakflow * Deepfield * Pmacct + supporting toolkit * NFsen/Nfdump/AS-stats * Put kibana/ES infront of any collector * Solarwinds something something * Different vendor toolkits
-- hugge
Also +1 for plixer scrutinizer. On 3/19/2018 10:16 AM, Gustavo Santos wrote:
+1 for Plixer Scrutinizer
2018-03-17 19:42 GMT-03:00 Michael Krygeris <me@krygerism.com>:
Disclaimer: Am Plixer engineer. If you want to take it for a spin, you can download a fully functional OVA/QCOW2 30 day eval from the plixer website. I can also get you access to an AWS AMI as well. I don’t want to turn this into an Ad. So DM if you need any info/access.
Mike Krygeris
On Tue, Mar 13, 2018 at 11:52 AM Babak Farrokhi <babak@farrokhi.net> wrote:
Plixer is also interesting.
nfdump works great with NetFlow but support for IPFIX is somehow limited to basics.
-- Babak
On 13 Mar 2018, at 3:20, Fredrik Korsbäck wrote:
On 2018-03-13 00:24, mike.lyon@gmail.com wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
Kentik is probably top of the foodchain right now.
But they are certainly not alone in the biz. Ontop of my head...
* Flowmon * Talaia * Arbor Peakflow * Deepfield * Pmacct + supporting toolkit * NFsen/Nfdump/AS-stats * Put kibana/ES infront of any collector * Solarwinds something something * Different vendor toolkits
-- hugge
-- Rick Coloccia, Jr. Network Manager State University of NY College at Geneseo 1 College Circle, 119 South Hall Geneseo, NY 14454 V: 585-245-5577 F: 585-245-5579
On Tue 2018-Mar-13 00:50:26 +0100, Fredrik Korsbäck <hugge@nordu.net> wrote:
Kentik is probably top of the foodchain right now.
But they are certainly not alone in the biz. Ontop of my head...
* Flowmon * Talaia * Arbor Peakflow * Deepfield * Pmacct + supporting toolkit * NFsen/Nfdump/AS-stats * Put kibana/ES infront of any collector
Logstash has a netflow plugin as of 5.x or something (https://www.elastic.co/guide/en/logstash/current/netflow-module.html) to act as a collector. A walkthrough: http://www.routereflector.com/2017/07/elk-as-a-free-netflow-ipfix-collector-... Using the logstash module setup thing adds a whole bunch of pretty netflow graphs and visualizations and such into Kibana for you. Caveat: Supports netflow v5 and v9, but does not indicate support for IPFIX explicitly. It definitely does not support sFlow, though if you really want you can stick sflowtool in front of it to translate sFlow->netflow, e.g. http://blog.sflow.com/2011/12/sflowtool.html.
* Solarwinds something something * Different vendor toolkits
-- hugge
-- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
There is also https://github.com/robcowart/elastiflow which uses the ELK stack. Luke Guillory Vice President – Technology and Innovation Tel: 985.536.1212 Fax: 985.536.0300 Email: lguillory@reservetele.com Reserve Telecommunications 100 RTC Dr Reserve, LA 70084 _________________________________________________________________________________________________ Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. . -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Hugo Slabbert Sent: Tuesday, March 13, 2018 10:44 AM To: Fredrik Korsbäck Cc: nanog@nanog.org Subject: Re: Spiffy Netflow tools? On Tue 2018-Mar-13 00:50:26 +0100, Fredrik Korsbäck <hugge@nordu.net> wrote:
Kentik is probably top of the foodchain right now.
But they are certainly not alone in the biz. Ontop of my head...
* Flowmon * Talaia * Arbor Peakflow * Deepfield * Pmacct + supporting toolkit * NFsen/Nfdump/AS-stats * Put kibana/ES infront of any collector
Logstash has a netflow plugin as of 5.x or something (https://www.elastic.co/guide/en/logstash/current/netflow-module.html) to act as a collector. A walkthrough: http://www.routereflector.com/2017/07/elk-as-a-free-netflow-ipfix-collector-... Using the logstash module setup thing adds a whole bunch of pretty netflow graphs and visualizations and such into Kibana for you. Caveat: Supports netflow v5 and v9, but does not indicate support for IPFIX explicitly. It definitely does not support sFlow, though if you really want you can stick sflowtool in front of it to translate sFlow->netflow, e.g. http://blog.sflow.com/2011/12/sflowtool.html.
* Solarwinds something something * Different vendor toolkits
-- hugge
-- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
+1 for ElastiFlow. Couldn't be easier to set up and run. Logstash has native support for netflow and sflow now via codecs. Kibana is an easy-to-use dashboard. I trimmed out a bunch of stuff in the ElastiFlow config that assumed a unidirectional network (like a corporate site). On Tue, Mar 13, 2018 at 8:48 AM, Luke Guillory <lguillory@reservetele.com> wrote:
There is also https://github.com/robcowart/elastiflow which uses the ELK stack.
Luke Guillory Vice President – Technology and Innovation
Tel: 985.536.1212 Fax: 985.536.0300 Email: lguillory@reservetele.com
Reserve Telecommunications 100 RTC Dr Reserve, LA 70084
____________________________________________________________ _____________________________________
Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. .
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Hugo Slabbert Sent: Tuesday, March 13, 2018 10:44 AM To: Fredrik Korsbäck Cc: nanog@nanog.org Subject: Re: Spiffy Netflow tools?
On Tue 2018-Mar-13 00:50:26 +0100, Fredrik Korsbäck <hugge@nordu.net> wrote:
Kentik is probably top of the foodchain right now.
But they are certainly not alone in the biz. Ontop of my head...
* Flowmon * Talaia * Arbor Peakflow * Deepfield * Pmacct + supporting toolkit * NFsen/Nfdump/AS-stats * Put kibana/ES infront of any collector
Logstash has a netflow plugin as of 5.x or something (https://www.elastic.co/guide/en/logstash/current/netflow-module.html) to act as a collector.
A walkthrough: http://www.routereflector.com/2017/07/elk-as-a-free-netflow- ipfix-collector-and-visualizer/
Using the logstash module setup thing adds a whole bunch of pretty netflow graphs and visualizations and such into Kibana for you.
Caveat: Supports netflow v5 and v9, but does not indicate support for IPFIX explicitly. It definitely does not support sFlow, though if you really want you can stick sflowtool in front of it to translate sFlow->netflow, e.g. http://blog.sflow.com/2011/12/sflowtool.html.
* Solarwinds something something * Different vendor toolkits
-- hugge
-- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
How scalable is ElastiFlow ? Let say I will dump 90kflow/s, how big elasticsearch farm do I need to comfortably store and work with at least couple weeks of data ? right now in NFSEN it takes about 3T in disk space and minutes for simple reports if it spans few time default time intervals. Thank you. On Tue, Mar 13, 2018 at 6:18 PM, Chase Christian <madsushi@gmail.com> wrote:
+1 for ElastiFlow. Couldn't be easier to set up and run. Logstash has native support for netflow and sflow now via codecs. Kibana is an easy-to-use dashboard. I trimmed out a bunch of stuff in the ElastiFlow config that assumed a unidirectional network (like a corporate site).
On Tue, Mar 13, 2018 at 8:48 AM, Luke Guillory <lguillory@reservetele.com> wrote:
There is also https://github.com/robcowart/elastiflow which uses the ELK stack.
Luke Guillory Vice President – Technology and Innovation
Tel: 985.536.1212 Fax: 985.536.0300 Email: lguillory@reservetele.com
Reserve Telecommunications 100 RTC Dr Reserve, LA 70084
____________________________________________________________ _____________________________________
Disclaimer: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material which should not disseminate, distribute or be copied. Please notify Luke Guillory immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. .
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Hugo Slabbert Sent: Tuesday, March 13, 2018 10:44 AM To: Fredrik Korsbäck Cc: nanog@nanog.org Subject: Re: Spiffy Netflow tools?
On Tue 2018-Mar-13 00:50:26 +0100, Fredrik Korsbäck <hugge@nordu.net> wrote:
Kentik is probably top of the foodchain right now.
But they are certainly not alone in the biz. Ontop of my head...
* Flowmon * Talaia * Arbor Peakflow * Deepfield * Pmacct + supporting toolkit * NFsen/Nfdump/AS-stats * Put kibana/ES infront of any collector
Logstash has a netflow plugin as of 5.x or something (https://www.elastic.co/guide/en/logstash/current/netflow-module.html) to act as a collector.
A walkthrough: http://www.routereflector.com/2017/07/elk-as-a-free-netflow- ipfix-collector-and-visualizer/
Using the logstash module setup thing adds a whole bunch of pretty netflow graphs and visualizations and such into Kibana for you.
Caveat: Supports netflow v5 and v9, but does not indicate support for IPFIX explicitly. It definitely does not support sFlow, though if you really want you can stick sflowtool in front of it to translate sFlow->netflow, e.g. http://blog.sflow.com/2011/12/sflowtool.html.
* Solarwinds something something * Different vendor toolkits
-- hugge
-- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal
-- -- Vitaly Nikolaev
FlowViewer is a robust user interface complement to Carnegie Mellon's SiLK netflow capture and analysis tool suite. FlowViewer provides the user with text/graphical analysis tools, multiple dashboards, long-term tracking of filtered sets, automatic storage management, raw netflow packet analysis, etc.. All open-source. Easy install. Runs on Linux. FlowViewer: https://sourceforge.net/projects/flowviewer/ SiLK: https://tools.netsa.cert.org/silk/ Joe Loiacono -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of mike.lyon@gmail.com Sent: Monday, March 12, 2018 7:25 PM To: NANOG list <nanog@nanog.org> Subject: Spiffy Netflow tools? Howdy! Checking out various Netflow tools and wanted to see what others are using? Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others. Also curious about on-prem solutions as well. Thanks! Mike DXC Technology Company -- This message is transmitted to you by or on behalf of DXC Technology Company or one of its affiliates. It is intended exclusively for the addressee. The substance of this message, along with any attachments, may contain proprietary, confidential or privileged information or information that is otherwise legally exempt from disclosure. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate any part of this message. If you have received this message in error, please destroy and delete all copies and notify the sender by return e-mail. Regardless of content, this e-mail shall not operate to bind DXC Technology Company or any of its affiliates to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of e-mail for such purpose.
Not necessarily (only) for *flow, but very nice combo: Luca Deri's ntopng+nprobe (https://www.ntop.org/products/traffic-analysis/ntop/) ***Stefan On Mon, Mar 12, 2018, 6:26 PM <mike.lyon@gmail.com> wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
Mike, All of the architecture's listed are pretty good. Nfsen is great if you have multiple routers exporting various netflow versions with a single daemon, but its a bit older and not as pretty/quick as something using elastic. Team Cymru has a netflow analyzer that matches your netflow data to known 'bad IPs'. http://www.team-cymru.org/Flow-Sonar.html Thanks, Scott Thanks, Scott On 3/12/18 7:24 PM, mike.lyon@gmail.com wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
IPFIXcol+fbitdump is what we use for our IPFIX measurements: https://github.com/CESNET/ipfixcol/ Can do NetFlow v5/v9 and sFlow as well. luuk On Mon 12 Mar 2018, 16:24, mike.lyon@gmail.com wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
+1 ElastiFlow, the templates are great, a great quickstart to using netflow on elk stack. -Vinny Stipo On Wed, Mar 14, 2018 at 2:57 AM, Luuk Hendriks <luuk.hendriks@utwente.nl> wrote:
IPFIXcol+fbitdump is what we use for our IPFIX measurements: https://github.com/CESNET/ipfixcol/
Can do NetFlow v5/v9 and sFlow as well.
luuk
On Mon 12 Mar 2018, 16:24, mike.lyon@gmail.com wrote:
Howdy!
Checking out various Netflow tools and wanted to see what others are using?
Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others.
Also curious about on-prem solutions as well.
Thanks! Mike
Stipo wrote:
+1 ElastiFlow, the templates are great, a great quickstart to using netflow on elk stack.
out of curiosity, I set up a test ElastiFlow installation on a small site recently. It's completely gorgeous from an eye candy point of view and it's pretty easy to see how you could tap into the ELK APIs to do interesting data mangling. On the down-side, it used ~40x the amount of disk space that nfsen used for the same accounting period, and even though it was only handling less than 1G traffic at a NF sample rate of 1:10, logstash and elastisearch managed to peg between 4-6 cores on the server which was handling it. Granted, these were only E5606 (2011-era Westmere Xeon) cpus, but even still there was an alarming mismatch between the amount of compute power required compared to the amount of netflow traffic being handled. It would be interesting to hear the sort of cpu requirements needed for larger installations. Obviously you can scale elkstack sideways, so it wouldn't be difficult to build out something which performed well. The issue is that burning cpu time can become an expensive proposition. Nick
Netflow Auditor In-house solution. The interface takes some getting used to, but you can pull a-n-y-t-h-i-n-g from it. Easy setup, great support, highly scalable, priced well. Best regards, -Alex -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of mike.lyon@gmail.com Sent: Monday, March 12, 2018 7:25 PM To: NANOG list Subject: Spiffy Netflow tools? Howdy! Checking out various Netflow tools and wanted to see what others are using? Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others. Also curious about on-prem solutions as well. Thanks! Mike This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you.
(To the thread in general) Those of us using RouterOS have to suffer a bit longer to get ASN-usefulness out of these tools. Well, natively. I'm just about done with using pmacct to inject the ASN into into a local Flow Analyzer. Maybe I can figure out at some point how to get pmacct to spit out a new netflow with the ASN information so these other tools can work out of the box. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "mike lyon" <mike.lyon@gmail.com> To: "NANOG list" <nanog@nanog.org> Sent: Monday, March 12, 2018 6:24:51 PM Subject: Spiffy Netflow tools? Howdy! Checking out various Netflow tools and wanted to see what others are using? Kentik is cool. Are they the only SaaS based flow digester? I don’t seem to see any others. Also curious about on-prem solutions as well. Thanks! Mike
participants (20)
-
Alex Lembesis -
Babak Farrokhi -
Chase Christian -
Daniel Rohan -
Fredrik Korsbäck -
Gustavo Santos -
Hugo Slabbert -
Loiacono, Joe -
Luke Guillory -
Luuk Hendriks -
Matt Erculiani -
Michael Krygeris -
Mike Hammett -
mike.lyon＠gmail.com -
Nick Hilliard -
Rick Coloccia -
Scott Fisher -
Stefan -
Stipo -
Vitaly Nikolaev