I’m still using nfsen/nfdump Been looking at manageengine netflow analyzer lately and liking it, we might be buying some time on Calix flowanalyze which might be an improved version of xangati Aaron

On Dec 30, 2018, at 10:44 PM, Michael Gehrmann <mgehrmann@atlassian.com> wrote:

Add Flowtraq to your list.

Cheers Mike

...

On Mon, 31 Dec 2018 at 14:30, Erik Sundberg <ESundberg@nitelusa.com> wrote: Hi Nanog….

We are looking at replacing our Netflow collector. I am wonder what other service providers are using to collect netflow data off their Core and Edge Routers. Pros/Cons… What to watch out for any info would help.

We are mainly looking to analyze the netflow data. Bonus if it does ddos detection and mitigation.

We are looking at

ManageEngine Netflow Analyzer

PRTG

Plixer – Scrutinizer

PeakFlow

Kentik

Solarwinds NTA

Thanks in advance…

Erik

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

I just recently rolled out Elastiflow. Lots of great information. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Michel 'ic' Luczak" <lists@benappy.com> To: "Erik Sundberg" <ESundberg@nitelusa.com> Cc: nanog@nanog.org Sent: Monday, December 31, 2018 3:40:40 AM Subject: Re: Service Provider NetFlow Collectors Don’t underestimate good old ELK https://www.elastic.co/guide/en/logstash/current/netflow-module.html + https://github.com/robcowart/elastiflow BR, ic On 31 Dec 2018, at 04:29, Erik Sundberg < ESundberg@nitelusa.com > wrote: Hi Nanog…. We are looking at replacing our Netflow collector. I am wonder what other service providers are using to collect netflow data off their Core and Edge Routers. Pros/Cons… What to watch out for any info would help. We are mainly looking to analyze the netflow data. Bonus if it does ddos detection and mitigation. We are looking at ManageEngine Netflow Analyzer PRTG Plixer – Scrutinizer PeakFlow Kentik Solarwinds NTA Thanks in advance… Erik CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

We were on Arbor for quite some time, but are now moving to Kentik. Mark. On 3/Jan/19 05:37, Nick Peelman wrote:

We rolled a large(ish) ElasticSearch cluster last year out of SuperMicro Microclouds (3U, 8 nodes per chassis, Xeon-D based processors), mostly 32GB of RAM per node, and M.2 PCIe SSDs as well as HDD storage. ES is a finicky beast to maintain. It can handle a node completely dying or disappearing from the network, but not when one runs out of space (at least not gracefully). Maintaining retention and rotation is tedious at best (yay curator). We’re dumping a boatload of log data there, as well as Flow data using Elastiflow, which provides the necessary collector bits as well as all the pretty Kibana graphs and stuff. Probably overbuilt, but I can pretty much keep whatever logs we want in perpetuity, we have plenty of headroom, and searching is incredibly fast.

ELK is an awesome set of tools, but be warned, there be dragons. Admin’ing even a small cluster can be time consuming and frustrating, and requires a pretty stout linux and server background, or at least some really good troubleshooting skills and an ability to turn to the code when the docs fall short. Doing a larger cluster could easily be a full time job. Still, all in all, I’m happy with the cost of ours, including my time building it and continued time maintaining it, compared to what the yearly outlay was going to be for Kentik.

-nick

On 31 Dec 2018, at 11:40, Mike Hammett <nanog@ics-il.net<mailto:nanog@ics-il.net>> wrote:

I just recently rolled out Elastiflow. Lots of great information.

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com<http://www.ics-il.com/>

Midwest-IX http://www.midwest-ix.com<http://www.midwest-ix.com/>

________________________________ From: "Michel 'ic' Luczak" <lists@benappy.com<mailto:lists@benappy.com>> To: "Erik Sundberg" <ESundberg@nitelusa.com<mailto:ESundberg@nitelusa.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Monday, December 31, 2018 3:40:40 AM Subject: Re: Service Provider NetFlow Collectors

Don’t underestimate good old ELK https://www.elastic.co/guide/en/logstash/current/netflow-module.html + https://github.com/robcowart/elastiflow

BR, ic

On 31 Dec 2018, at 04:29, Erik Sundberg <ESundberg@nitelusa.com<mailto:ESundberg@nitelusa.com>> wrote:

Hi Nanog….

We are looking at replacing our Netflow collector. I am wonder what other service providers are using to collect netflow data off their Core and Edge Routers. Pros/Cons… What to watch out for any info would help.

We are mainly looking to analyze the netflow data. Bonus if it does ddos detection and mitigation.

We are looking at ManageEngine Netflow Analyzer PRTG Plixer – Scrutinizer PeakFlow Kentik Solarwinds NTA

Thanks in advance…

Erik

________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

+1 Kentik as well, DDoS, RTBH, Netflow. Cloud based so I don't have to worry about it. On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" <nanog-bounces@nanog.org on behalf of bryan@shout.net> wrote: +1 Kentik ... We've been using their DDoS/RTBH mitigation with good success. On 12/31/18 3:52 AM, Eric Lindsjö wrote: > Hi, > > We use kentik and we're very happy. Works great, tons of new features > coming along all the time. Going to start looking into ddos detection > and mitigation soon. > > Would recommend. > > Kind regards, > Eric Lindsjö > > > On 12/31/2018 04:29 AM, Erik Sundberg wrote: >> >> Hi Nanog…. >> >> We are looking at replacing our Netflow collector. I am wonder what >> other service providers are using to collect netflow data off their >> Core and Edge Routers. Pros/Cons… What to watch out for any info would >> help. >> >> We are mainly looking to analyze the netflow data. Bonus if it does >> ddos detection and mitigation. >> >> We are looking at >> >> ManageEngine Netflow Analyzer >> >> PRTG >> >> Plixer – Scrutinizer >> >> PeakFlow >> >> Kentik >> >> Solarwinds NTA >> >> Thanks in advance… >> >> Erik >> >> >> ------------------------------------------------------------------------ >> >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, >> files or previous e-mail messages attached to it may contain >> confidential information that is legally privileged. If you are not >> the intended recipient, or a person responsible for delivering it to >> the intended recipient, you are hereby notified that any disclosure, >> copying, distribution or use of any of the information contained in or >> attached to this transmission is STRICTLY PROHIBITED. If you have >> received this transmission in error please notify the sender >> immediately by replying to this e-mail. You must destroy the original >> transmission and its attachments without reading or saving in any >> manner. Thank you. >

We do have a minimum for commercial service that's more like $1500/mo but we are coming out with a free tier in Q1 with lower retention (among other deltas, but including fully slice and dice flow analytics +BGP that it sounded like Erik might be looking for). Feel free to ping me if anyone would like to help us test the free tier in January. Thanks, Avi Freedman CEO, Kentik

Doesn't Kentik cost like $2000 a month minimum?

On Mon, Dec 31, 2018 at 11:57 AM Matthew Crocker <matthew@corp.crocker.com> wrote:

...

+1 Kentik as well, DDoS, RTBH, Netflow. Cloud based so I don't have to worry about it.

On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" < nanog-bounces@nanog.org on behalf of bryan@shout.net> wrote:

+1 Kentik ...

We've been using their DDoS/RTBH mitigation with good success.

On 12/31/18 3:52 AM, Eric Lindsjö wrote: > Hi, > > We use kentik and we're very happy. Works great, tons of new features > coming along all the time. Going to start looking into ddos detection > and mitigation soon. > > Would recommend. > > Kind regards, > Eric Lindsjö > > > On 12/31/2018 04:29 AM, Erik Sundberg wrote: >> >> Hi Nanog…. >> >> We are looking at replacing our Netflow collector. I am wonder what >> other service providers are using to collect netflow data off their >> Core and Edge Routers. Pros/Cons… What to watch out for any info would >> help. >> >> We are mainly looking to analyze the netflow data. Bonus if it does >> ddos detection and mitigation. >> >> We are looking at >> >> ManageEngine Netflow Analyzer >> >> PRTG >> >> Plixer – Scrutinizer >> >> PeakFlow >> >> Kentik >> >> Solarwinds NTA >> >> Thanks in advance… >> >> Erik >> >> >> ------------------------------------------------------------------------ >> >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, >> files or previous e-mail messages attached to it may contain >> confidential information that is legally privileged. If you are not >> the intended recipient, or a person responsible for delivering it to >> the intended recipient, you are hereby notified that any disclosure, >> copying, distribution or use of any of the information contained in or >> attached to this transmission is STRICTLY PROHIBITED. If you have >> received this transmission in error please notify the sender >> immediately by replying to this e-mail. You must destroy the original >> transmission and its attachments without reading or saving in any >> manner. Thank you. >

I would advise against InfluxDB in this case - flow data has a very high (and open) tag cardinality which is not suited to Influx (although their recently new index format has improved this). I’m currently pushing sFlow through Pmacct —> Kafka —> Clickhouse (columnar store) with a summing merge tree database engine. Clickhouse is very fast for queries across columns as well as aggregating down them (e.g. summing number of bytes). For example this is the results of a query of nearly a year’s worth of MAC-to-MAC flows (7-tuple) queried for the last 7 days between two given sets of MACs: 2016 rows in set. Elapsed: 0.208 sec. Processed 17.56 million rows, 1.03 GB (84.51 million rows/s., 4.97 GB/s.) There is also a Grafana datasource plugin for Clickhouse :) - Tim

On 2 Jan 2019, at 7:50 pm, H I Baysal <hibaysal@gmail.com> wrote:

PMACCT (Works Awesome) push to influxdb ( Works awesome)

With some custom scripts to add/match interface descriptions. And you can query whatever you want in grafana :D And grafana has a nice API for rendering a dashboardgraph to a PNG and you can send this png to whatever chat/bot or mail you want.

And all for free with 99% of accuracy.

(Mucho gracias to Paulo :D )

On 01-01-19 05:56, Avi Freedman wrote:

...

We do have a minimum for commercial service that's more like $1500/mo but we are coming out with a free tier in Q1 with lower retention (among other deltas, but including fully slice and dice flow analytics +BGP that it sounded like Erik might be looking for).

Feel free to ping me if anyone would like to help us test the free tier in January.

Thanks,

Avi Freedman CEO, Kentik

...

Doesn't Kentik cost like $2000 a month minimum?

On Mon, Dec 31, 2018 at 11:57 AM Matthew Crocker <matthew@corp.crocker.com> wrote:

...

+1 Kentik as well, DDoS, RTBH, Netflow. Cloud based so I don't have to worry about it.

On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" < nanog-bounces@nanog.org on behalf of bryan@shout.net> wrote:

+1 Kentik ...

We've been using their DDoS/RTBH mitigation with good success.

On 12/31/18 3:52 AM, Eric Lindsjö wrote: > Hi, > > We use kentik and we're very happy. Works great, tons of new features > coming along all the time. Going to start looking into ddos detection > and mitigation soon. > > Would recommend. > > Kind regards, > Eric Lindsjö > > > On 12/31/2018 04:29 AM, Erik Sundberg wrote: >> >> Hi Nanog…. >> >> We are looking at replacing our Netflow collector. I am wonder what >> other service providers are using to collect netflow data off their >> Core and Edge Routers. Pros/Cons… What to watch out for any info would >> help. >> >> We are mainly looking to analyze the netflow data. Bonus if it does >> ddos detection and mitigation. >> >> We are looking at >> >> ManageEngine Netflow Analyzer >> >> PRTG >> >> Plixer – Scrutinizer >> >> PeakFlow >> >> Kentik >> >> Solarwinds NTA >> >> Thanks in advance… >> >> Erik >> >> >> ------------------------------------------------------------------------ >> >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, >> files or previous e-mail messages attached to it may contain >> confidential information that is legally privileged. If you are not >> the intended recipient, or a person responsible for delivering it to >> the intended recipient, you are hereby notified that any disclosure, >> copying, distribution or use of any of the information contained in or >> attached to this transmission is STRICTLY PROHIBITED. If you have >> received this transmission in error please notify the sender >> immediately by replying to this e-mail. You must destroy the original >> transmission and its attachments without reading or saving in any >> manner. Thank you. >

This is correct, With a flow database you want to be able to say: “show me all HTTP traffic from subnet a.b.c.0/24” which requires you to either keep individual IPs or aggregate subnets. Combined with port and protocol data for both source and destination, the series count shoots way above 10M. - Tim

On 2 Jan 2019, at 20:20, Saku Ytti <saku@ytti.fi> wrote:

Hey Tim,

...

I would advise against InfluxDB in this case - flow data has a very high (and open) tag cardinality which is not suited to Influx (although their recently new index format has improved this).

I'm not entirely sure I understand. Does this mean the permutations of tags are high, i.e. series count is high? If so, isn't this general problem and advice against all TSDBs? If so, I fully agree, you couldn't/shouldn't make for example IP addresses your tags, potentially creating 2**32*2 series without any other tags, it's rather non-sensical proposal in TSDB.

Influx themselves comment that >10M series is likely infeasible. So you need unique tag combinations to be low millions at most. -- ++ytti

Hey, On Wed, 2 Jan 2019 at 14:40, H I Baysal <hibaysal@gmail.com> wrote:

That absolutely depends on the amount of TAGs you use, and how you aggregate, etc. I am collecting DSTAS, SRCAS, en DST AS per IP. And influx is not even sweating a single drop....

We have a 4 Tbps of traffic during peak, and as well as pmacct and influxdb or running very very smooth.

How many series do you have in the DB? Your explanation makes it unclear to me what labels you have 'per IP' is ambiguous to me. If only DST_IP is tag and you have low amount of IPs in or behind your network, it seems very feasible indeed. -- ++ytti

That’s a much better cardinality (AS based) but it’s not the general case. Even if you want per-prefix information I’d argue that Influx would still not handle the load (~700k ^ 2 cardinality). For limited tag-sets it would do the trick. I never did attempt to push it to Influx with some foresight that it’d be suboptimal for my ultimate use cases. I wanted a solution that could handle a wide range of use cases without having to worry about limits on tag-sets. I found Clickhouse able to do what I wanted in a performant way. - Tim

On 2 Jan 2019, at 20:37, H I Baysal <hibaysal@gmail.com> wrote:

Hi Tim,

That absolutely depends on the amount of TAGs you use, and how you aggregate, etc. I am collecting DSTAS, SRCAS, en DST AS per IP. And influx is not even sweating a single drop....

We have a 4 Tbps of traffic during peak, and as well as pmacct and influxdb or running very very smooth.

(With the mentioned aggregations I can see what a single customer costs with Transit, Peering and IX (per IP even if needed) ) And dst AS per port/description/ethernet name

From your mail i derive that you just pushed everything to influx from flows, you have to be a bit smarter with the layout, aggregations and continuous queries. (collect what you need)

...

On 02-01-19 13:08, Tim Raphael wrote: I would advise against InfluxDB in this case - flow data has a very high (and open) tag cardinality which is not suited to Influx (although their recently new index format has improved this).

I’m currently pushing sFlow through Pmacct —> Kafka —> Clickhouse (columnar store) with a summing merge tree database engine. Clickhouse is very fast for queries across columns as well as aggregating down them (e.g. summing number of bytes).

For example this is the results of a query of nearly a year’s worth of MAC-to-MAC flows (7-tuple) queried for the last 7 days between two given sets of MACs:

2016 rows in set. Elapsed: 0.208 sec. Processed 17.56 million rows, 1.03 GB (84.51 million rows/s., 4.97 GB/s.)

There is also a Grafana datasource plugin for Clickhouse :)

- Tim

...

On 2 Jan 2019, at 7:50 pm, H I Baysal <hibaysal@gmail.com> wrote:

PMACCT (Works Awesome) push to influxdb ( Works awesome)

With some custom scripts to add/match interface descriptions. And you can query whatever you want in grafana :D And grafana has a nice API for rendering a dashboardgraph to a PNG and you can send this png to whatever chat/bot or mail you want.

And all for free with 99% of accuracy.

(Mucho gracias to Paulo :D )

...

On 01-01-19 05:56, Avi Freedman wrote: We do have a minimum for commercial service that's more like $1500/mo but we are coming out with a free tier in Q1 with lower retention (among other deltas, but including fully slice and dice flow analytics +BGP that it sounded like Erik might be looking for).

Feel free to ping me if anyone would like to help us test the free tier in January.

Thanks,

Avi Freedman CEO, Kentik

...

Doesn't Kentik cost like $2000 a month minimum?

On Mon, Dec 31, 2018 at 11:57 AM Matthew Crocker <matthew@corp.crocker.com> wrote:

...

+1 Kentik as well, DDoS, RTBH, Netflow. Cloud based so I don't have to worry about it.

On 12/31/18, 11:37 AM, "NANOG on behalf of Bryan Holloway" < nanog-bounces@nanog.org on behalf of bryan@shout.net> wrote:

+1 Kentik ...

We've been using their DDoS/RTBH mitigation with good success.

On 12/31/18 3:52 AM, Eric Lindsjö wrote: > Hi, > > We use kentik and we're very happy. Works great, tons of new features > coming along all the time. Going to start looking into ddos detection > and mitigation soon. > > Would recommend. > > Kind regards, > Eric Lindsjö > > > On 12/31/2018 04:29 AM, Erik Sundberg wrote: >> >> Hi Nanog…. >> >> We are looking at replacing our Netflow collector. I am wonder what >> other service providers are using to collect netflow data off their >> Core and Edge Routers. Pros/Cons… What to watch out for any info would >> help. >> >> We are mainly looking to analyze the netflow data. Bonus if it does >> ddos detection and mitigation. >> >> We are looking at >> >> ManageEngine Netflow Analyzer >> >> PRTG >> >> Plixer – Scrutinizer >> >> PeakFlow >> >> Kentik >> >> Solarwinds NTA >> >> Thanks in advance… >> >> Erik >> >> >> ------------------------------------------------------------------------ >> >> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, >> files or previous e-mail messages attached to it may contain >> confidential information that is legally privileged. If you are not >> the intended recipient, or a person responsible for delivering it to >> the intended recipient, you are hereby notified that any disclosure, >> copying, distribution or use of any of the information contained in or >> attached to this transmission is STRICTLY PROHIBITED. If you have >> received this transmission in error please notify the sender >> immediately by replying to this e-mail. You must destroy the original >> transmission and its attachments without reading or saving in any >> manner. Thank you. >

Hey Phil, What use cases are you trying to work on? I work for Kentik on the product side of things and this kind of info is very interesting for me to hear. Happy to take your reply in a DM or here. Dan On Wed, Jan 2, 2019 at 6:01 AM Phil Lavin <phil.lavin@cloudcall.com> wrote:

...

Doesn't Kentik cost like $2000 a month minimum?

We recently got a quote from Kentik and I fell off my chair. The annual cost was slightly more than the total upfront purchase cost of the hardware they were collecting Flow from and was significantly more than the total cost each year of running the network (when factoring in power, cross-connects, transit, etc.). I'm told that collecting Flow and BGP from two routers which both do < 10Gbit is "~$32K USD per year for Kentik Pro" - even that feels disproportionally expensive. It does look like an excellent product though I'm struggling to find a business case that justifies the cost on relatively small (sub 2 Tbit/s) networks.

-- Thanks, Dan