Re: DDOS, IDS, RTBH, and Rate limiting
Cisco ASRs and MXs with inline jflow can do hundreds of K flows/second without affecting packet forwarding.
Yes, i agree,those are good for netflow, but when they already exist in network.
Does it worth to buy ASR, if L3 switch already doing the job (BGP/ACL/rate-limit/routing)?
Not suggesting that anyone should change out their gear though per my other message, I've seen SPAN make things go wonky on almost every vendor that ISPs use for switching.
Well, if it is available, except hardware limitations, there is second obstacle, software licensing cost. On latest JunOS, for example on EX2200, you need to purchase license (EFL), and if am not wrong it is $3000 for 48port units.
So if only sFlow feature is on stake, it worth to think, to purchase license, or to purchase server. Prices for JFlow license on MX, just for 5/10G is way above cost of very decent server.
I believe that smaller MXs can run it for free. Larger providers we've worked with often have magic cookies they can call in to get it enabled, but I understand you're talking about the smaller-provider (or at least ~ 10gig per POP across multiple POPs) case. We see a lot of Brocade for switching in hosting providers, which makes sFlow easy, of course.
And with the right setup you can run FastNetMon or other tools in addition to generating flow that can be of use for other purposes as well...
Technically there is ipt_NETFLOW, that can generate netflow on same box, for statistical/telemetry purposes. But i am not sure it is possible to run them together.
At frac 10gig you can just open pcap on a 10gig interface on a Linux box getting a tap, of course. What we did was use myricom cards and the myri_snf drivers and take from the single-consumer ring buffers into large in-RAM ring buffers, and make those ring buffers available via LD_PRELOAD or cli tools to allow flow, snort, p0f, tcpdump, etc to all be run at the same time at 10gig. The key for that is not going through the kernel IP stack, though.
But taps can be difficult or at least time consuming for people to put in at scale. Even, we've seen, for folks with 10G networks. Often because they can get 90% of what they need for 4 different business purposes from just flow :)
About scaling, i guess it depends on proper deployment strategy and sysadmins/developers capabilities. For example to deploy new ruleset for my pcap-based "homemade" analyser to 150 probes across the country - is just one click.
Sounds cool. You should write up that use case. Hopefully you've secured the metadata/command push channel well enough :)
Best regards, Denys
Avi Freedman | Your flow has something to show you; can you see it? | CEO, CloudHelix | (avi at cloudhelix dot com) | my name one word on skype |
On 2014-11-22 18:00, freedman@freedman.net wrote:
Cisco ASRs and MXs with inline jflow can do hundreds of K flows/second without affecting packet forwarding.
Yes, i agree,those are good for netflow, but when they already exist in network.
Does it worth to buy ASR, if L3 switch already doing the job (BGP/ACL/rate-limit/routing)?
Not suggesting that anyone should change out their gear though per my other message, I've seen SPAN make things go wonky on almost every vendor that ISPs use for switching. Well, i always try to stay on safe side. Additionally, sure, i do mirror for RX only, RX+TX often can exceed interface rate too :)
Well, if it is available, except hardware limitations, there is second obstacle, software licensing cost. On latest JunOS, for example on EX2200, you need to purchase license (EFL), and if am not wrong it is $3000 for 48port units.
So if only sFlow feature is on stake, it worth to think, to purchase license, or to purchase server. Prices for JFlow license on MX, just for 5/10G is way above cost of very decent server.
I believe that smaller MXs can run it for free. Larger providers we've worked with often have magic cookies they can call in to get it enabled, but I understand you're talking about the smaller-provider (or at least ~ 10gig per POP across multiple POPs) case.
We see a lot of Brocade for switching in hosting providers, which makes sFlow easy, of course.
Oh, Brocade, recent experience with ServerIron taught me new lesson, that i can't do bonding on ports as i want, it has limitations about even/odd port numbers and etc. Most amazing part i just forgot, that i have this ServerIron, and it is a place where i run DDoS protection (but it works perfectly over "tap" way). Thanks for reminding about this vendor :)
And with the right setup you can run FastNetMon or other tools in addition to generating flow that can be of use for other purposes as well...
Technically there is ipt_NETFLOW, that can generate netflow on same box, for statistical/telemetry purposes. But i am not sure it is possible to run them together.
At frac 10gig you can just open pcap on a 10gig interface on a Linux box getting a tap, of course.
What we did was use myricom cards and the myri_snf drivers and take from the single-consumer ring buffers into large in-RAM ring buffers, and make those ring buffers available via LD_PRELOAD or cli tools to allow flow, snort, p0f, tcpdump, etc to all be run at the same time at 10gig.
The key for that is not going through the kernel IP stack, though.
Ntop's pf_ring, which is basically same idea, but can run on Intel cards. Just maybe because never had myricom in hands, and it is difficult to obtain them here.
But taps can be difficult or at least time consuming for people to put in at scale. Even, we've seen, for folks with 10G networks. Often because they can get 90% of what they need for 4 different business purposes from just flow :)
About scaling, i guess it depends on proper deployment strategy and sysadmins/developers capabilities. For example to deploy new ruleset for my pcap-based "homemade" analyser to 150 probes across the country - is just one click.
Sounds cool. You should write up that use case. Hopefully you've secured the metadata/command push channel well enough :)
For servers it is ssh with key authentication, and push system doesn't contain private key, it is forwarded over ssh agent from developer pc. Sure, it is better also to sign by assymmetric crypto update also, keep keys on smartcard, but in this case it is not necessary. --- Best regards, Denys
On 11/22/2014 11:18 AM, Denys Fedoryshchenko wrote:
We see a lot of Brocade for switching in hosting providers, which makes sFlow easy, of course. Oh, Brocade, recent experience with ServerIron taught me new lesson,
On 2014-11-22 18:00, freedman@freedman.net wrote: that i can't do bonding on ports as i want, it has limitations about even/odd port numbers and etc. Most amazing part i just forgot, that i have this ServerIron, and it is a place where i run DDoS protection (but it works perfectly over "tap" way). Thanks for reminding about this vendor :)
I just hope you're not talking FCX's.... if you upgrade those to 8.x firmware, you'll lose sflow on the 10gb ports. Once you upgrade, they send a corrupted sflow packet, and at *far* less then the rate that you configure. Even if you adjust your parser to compensate for the corrupt packet, they're still dropping the large majority of samples, making sflow pretty much useless. It's been several months since we reported this, and we're still waiting on a fix.
Hello, folks! Thank you for a very useful feedback! I'm so sorry for my negative vision of netflow :( It's nice protocol but I haven't equpment with ability to generate netflow on wire speed and I use mirror/SPAN instead. I competely redesigned attack-analyzer subsystem and can process sampled data now. I just added sFLOW v5 suport to FastNetMon and you can try it now. In near future I will add netflow v5 support. With sFLOW support my tool can detect attack on 40-100GE links and more! Thanks for sFLOW architecture! :) Thank you! On Sun, Nov 23, 2014 at 2:53 AM, Brian Rak <brak@gameservers.com> wrote:
On 11/22/2014 11:18 AM, Denys Fedoryshchenko wrote:
On 2014-11-22 18:00, freedman@freedman.net wrote:
We see a lot of Brocade for switching in hosting providers, which makes sFlow easy, of course.
Oh, Brocade, recent experience with ServerIron taught me new lesson, that i can't do bonding on ports as i want, it has limitations about even/odd port numbers and etc. Most amazing part i just forgot, that i have this ServerIron, and it is a place where i run DDoS protection (but it works perfectly over "tap" way). Thanks for reminding about this vendor :)
I just hope you're not talking FCX's.... if you upgrade those to 8.x firmware, you'll lose sflow on the 10gb ports. Once you upgrade, they send a corrupted sflow packet, and at *far* less then the rate that you configure. Even if you adjust your parser to compensate for the corrupt packet, they're still dropping the large majority of samples, making sflow pretty much useless.
It's been several months since we reported this, and we're still waiting on a fix.
-- Sincerely yours, Pavel Odintsov
Hello, folks! Thank you for a very useful feedback! I'm so sorry for my negative vision of netflow :( It's nice protocol but I haven't equpment with ability to generate netflow on wire speed and I use mirror/SPAN instead. I competely redesigned attack-analyzer subsystem and can process sampled data now. I just added sFLOW v5 suport to FastNetMon and you can try it now. In near future I will add netflow v5 support. With sFLOW support my tool can detect attack on 40-100GE links and more! Thanks for sFLOW architecture! :) You can check new version here: https://github.com/FastVPSEestiOu/fastnetmon Thank you! On Sun, Nov 23, 2014 at 2:53 AM, Brian Rak <brak@gameservers.com> wrote:
On 11/22/2014 11:18 AM, Denys Fedoryshchenko wrote:
On 2014-11-22 18:00, freedman@freedman.net wrote:
We see a lot of Brocade for switching in hosting providers, which makes sFlow easy, of course.
Oh, Brocade, recent experience with ServerIron taught me new lesson, that i can't do bonding on ports as i want, it has limitations about even/odd port numbers and etc. Most amazing part i just forgot, that i have this ServerIron, and it is a place where i run DDoS protection (but it works perfectly over "tap" way). Thanks for reminding about this vendor :)
I just hope you're not talking FCX's.... if you upgrade those to 8.x firmware, you'll lose sflow on the 10gb ports. Once you upgrade, they send a corrupted sflow packet, and at *far* less then the rate that you configure. Even if you adjust your parser to compensate for the corrupt packet, they're still dropping the large majority of samples, making sflow pretty much useless.
It's been several months since we reported this, and we're still waiting on a fix.
-- Sincerely yours, Pavel Odintsov
On 2 Dec 2014, at 17:18, Pavel Odintsov wrote:
In near future I will add netflow v5 support.
Good job - you should really go for NetFlow v9 when you can, as it supports IPv6 and MPLS labels. Next would be IPFIX. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Hello, folks! NetFlow v5 and v9 support have just added to FastNetMon: https://github.com/FastVPSEestiOu/fastnetmon Now you can catch DDoS attacks and collect data from sFLOW v5, NetFlow v5/v9 and even from mirror port with PF_RING in one tool simultaneously! Will be very glad for feedback and testing! On Wed, Dec 3, 2014 at 7:57 AM, Roland Dobbins <rdobbins@arbor.net> wrote:
On 2 Dec 2014, at 17:18, Pavel Odintsov wrote:
In near future I will add netflow v5 support.
Good job - you should really go for NetFlow v9 when you can, as it supports IPv6 and MPLS labels.
Next would be IPFIX.
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
-- Sincerely yours, Pavel Odintsov
participants (5)
-
Brian Rak -
Denys Fedoryshchenko -
freedman＠freedman.net -
Pavel Odintsov -
Roland Dobbins