-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Dear Ryan,

I am going to assume you want it to spit out 10G clean, what size dirty traffic are you expecting it to handle?

Great question! Let's say between 6Gbps and 8Gbps dirty. Thank you! Rob.

On Nov 17 2019, at 2:18 pm, Rabbi Rob Thomas <robt@cymru.com> wrote:

Hello, NANOG!

I'm in the midst of rebuilding/upgrading our backbone and peering - sessions cheerfully accepted :) - and am curious what folks recommend in the DDoS mitigation appliance realm? Ideally it would be capable of 10Gbps and circa 14Mpps rate of mitigation. If you have a recommendation, I'd love to hear it and the reasons for it. If you have an alternative to an appliance that has worked well for you (we're a mix of Cisco and Juniper), I'm all ears.

Private responses are fine, and I'm happy to summarize back to the list if there is interest.

Thank you! Rob.

- -- Rabbi Rob Thomas Team Cymru "It is easy to believe in freedom of speech for those with whom we agree." - Leo McKern -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEDcVjavXj08cL/QwdQ+hhYvqF8o0FAl3RzkAACgkQQ+hhYvqF 8o1J6Q//ZUgytaLqJoKV6i39pXmVH7Yxau5jThSfHEdLk9n1dQzrLCfM28vyUTQr 93TeikXMvEXi8mG5vFXjQAkaNLbKPLJnpydIwRe3vbDxl6pkzrWF3XF5dy9dZ0rl IWcpe1ngVmT/FGTFm5T26woEAmvg4CLjP9Fm8nMHLKp29xRgd8SKs7jDxtZZx68g BkdJiFGXdVP/oKUslYzDTIUdhUwckAeJKxFfsvdgN6Ybz70yckLeyfwZwo9pNcjj W8yYWchGEtPMKidtupAATYKkKcZQp0gvObRXwDeGR4y+4YoJlTU5L+bNAr+xmsgi hIy9YKs3/0uhOFPBbcN+sconQqTCyWA2eyXlCGlT1dnMvM7SbXDeD8R4IxqQeQ9i JSZJiUhtfQFVqNnufqbeI0im/onSbyqv+IUPFKug5wU2hXY04YnoRcFMGwufIugj pUSUqlkh4pmTe8so+JMOYHzH186fuVRKtNnScqkGPeKxEM+vp2Ou4hCaHyWPfTb1 aLKBY6LeJK6oWWOPArk8m8nVjvTKdYZh6XvlCeiA/lOy8a6rGVKLN8uX2QRVGZFE 5TE0XpoH+0MAqhO57ZiT8Uvs7D0Gpdc0ZJ3HQUj005SwJ1l4vGeq/jPhTnBEtcEO fIu9tyqlWDuIeZfuMGG1lXrL+OUtfA8TJomizvyPBwzMfvTX4bU= =vQSp -----END PGP SIGNATURE-----

I would say you are making some assumptions that are not fact based. The OP is very knowledgeable and would not mince words or waste bandwidth. Let us see what he has to say in regards to your remarks. He will be able to make this more clear once he has read what people have stated in other responses. Respectfully, of course, Richard Golodner On 11/17/19 8:12 PM, Töma Gavrichenkov wrote:

Peace,

On Mon, Nov 18, 2019, 1:49 AM Rabbi Rob Thomas <robt@cymru.com <mailto:robt@cymru.com>> wrote:

> I am going to assume you want it to spit out 10G clean, what size > dirty traffic are you expecting it to handle?

Great question!  Let's say between 6Gbps and 8Gbps dirty.

As someone making a living as a DDoS mitigation engineer for the last 10 years (minus 1 month) I should say your threat model is sort of unusual.  Potential miscreants today should be assumed to have much more to show you even on a daily basis.

Is it like you also have something filtering upstream for you, e.g. flowspec-enabled peers?

-- Töma

On 2019-11-18 04:23, Richard wrote:

I would say you are making some assumptions that are not fact based. The OP is very knowledgeable and would not mince words or waste bandwidth. Let us see what he has to say in regards to your remarks. He will be able to make this more clear once he has read what people have stated in other responses.

Respectfully, of course, Richard Golodner On 11/17/19 8:12 PM, Töma Gavrichenkov wrote:

...

Peace,

On Mon, Nov 18, 2019, 1:49 AM Rabbi Rob Thomas <robt@cymru.com> wrote:

...

...

I am going to assume you want it to spit out 10G clean, what size dirty traffic are you expecting it to handle?

Great question! Let's say between 6Gbps and 8Gbps dirty.

As someone making a living as a DDoS mitigation engineer for the last 10 years (minus 1 month) I should say your threat model is sort of unusual. Potential miscreants today should be assumed to have much more to show you even on a daily basis.

Is it like you also have something filtering upstream for you, e.g. flowspec-enabled peers?

-- Töma

...

AFAIK new threats (SYN+ACK amplification) can't be mitigated over flowspec and they can reach 40+Gbps easily.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Dear Töma,

Potential miscreants today should be assumed to have much more to show you even on a daily basis.

Oh, indeed! :)

Is it like you also have something filtering upstream for you, e.g. flowspec-enabled peers?

That is correct. Be well, Rob. - -- Rabbi Rob Thomas Team Cymru "It is easy to believe in freedom of speech for those with whom we agree." - Leo McKern -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEDcVjavXj08cL/QwdQ+hhYvqF8o0FAl3SBH4ACgkQQ+hhYvqF 8o3k7Q//YSgohuoaE6bB7uopBUTbdg6uiHc1TNFGrYA+vf9idXpYuc7V0092hA3k xxF1iSsChQgqFB03syWgI3j5+NIHGS2LLYHVmsgQWOBLbsr2dvp09MoVTWDh7ODR zex5g/ZyOn6Tdj3oJkeKzcwi49kgYEAxXqaU1nt7jFS0pFBKeXuaLomLIm5hPaNO qhHkNg7BnRlm3Vr+rdcXFYFnlZIBkCVXi6I4E5xBWzu4r3TPJU/LYGPstZFt/coF vc86Ry90rsdm9xuo7se2LTpXimL8Qzqcj5MP948JCc3TyS+ZGecGq9QovEDsX4SC I9bOX8jmYThO1HgOFXKt9y6dl8J1Mi98KpZL82Gc1gspeFzdJ4FdyUddpA9+glda IvDKI2o+pP9dqignczvlEiExiSCHDe/5DnVYcAvwUwI59gyKBdGkgGGVu8cPTIAt 8qV8SHrYoVD/3tM3h89ZvReJwZk8wo7JG0cxwqRJqlBbCKPSMSYm35ps/L+7rYNL ApxiFwu3pWfx5HBBpbQ/KugVyBA2KOg2qMVs07FM3D0CIy6soUqjWsn/hMMbRZaE zpyhTjBKpyvOXFLQWThDoahDUzTS/KFGvMS/JMeAz35gg1p5zHtbXGY8WCb63B1l oefiZEIVJImNfEvw+SViK6RSK/OMfhg0yN0NAhVEyifxPxhrdIM= =enDs -----END PGP SIGNATURE-----

On 18/11/2019 13:50, Mike Hammett wrote:

I would like the list to know that not all targets attract such large attacks. I know many eyeball ISPs that encounter less than 10 gig attacks, which can be reasonably absorbed\mitigated. Online gamers looking to boot someone else from the game aren't generally committing

...

100 gigs of resources to an attack.

There are two very good reasons to use 'surgical' amounts of traffic in attacks: 1. Concealing the size of your botnet 2. Reducing the damage to the end user's ISP, and thus reducing the likelihood that they escalate the attack to the authorities (because who's got the time to do that for an individual subscriber?) The shift to "just enough to knock the customer off without killing the whole network" happened around ~2015 in my capacity, at least. -- Tom

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello, NANOG! My thanks again to all who responded with suggestions, tips, and further considerations. I appreciate it very much! As promised, here is my pithy summary of your detailed suggestions. I've included URLs for those who may wish to conduct further research. We've not made our selection yet, and likely won't until early 2020. At present I'm busy building out our new backbone, and thus can't yet offer up my own recommendation. Who needs sleep? :D Several folks shared their architecture and deployment recommendations, which were quite insightful. Placement of these devices, and in particular a centralized monitoring solution for distributed deployments, were keys to success. There were no support concerns for any of these suggestions. Folks have used open source and freeware, but generally recommended commercial offerings. These required less manual intervention. It was aces to see so many folks employing techniques such as flowspec and RTBH. DDoS appliance recommendations: . Anycast and fat pipes - Multiple votes . Massive peering - Multiple votes - Be ready for peering requests from me :) . Arbor Netscout - Multiple votes - Consistently labeled as "expensive" - https://www.netscout.com/arbor-ddos . RioRey - Multiple votes - http://www.riorey.com/ . Juniper routers MX240 or MX480 - https://www.juniper.net/us/en/products-services/routing/mx-series/mx240/ - https://www.juniper.net/us/en/products-services/routing/mx-series/mx480/ . NFOCUS ADS - ADS 8000 is the scrubbing box - ADS-m is the monitoring box - NTS is the box which uses Netflow to find unwanted traffic - https://nsfocusglobal.com/anti-ddos-system-ads/ . Wanguard+Wanfilter - https://www.andrisoft.com/software/wanguard - https://www.andrisoft.com/software/wanguard/ddos-mitigation-protecti on . A10 Thunder ADC - https://a10networks.optrics.com/products/application-delivery.aspx . FastNetMon - Free or inexpensive - https://fastnetmon.com/ Thank you! Rob, the routing rabbi. - -- Rabbi Rob Thomas Team Cymru "It is easy to believe in freedom of speech for those with whom we agree." - Leo McKern -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEDcVjavXj08cL/QwdQ+hhYvqF8o0FAl3n97AACgkQQ+hhYvqF 8o1zdA//aSCm5pVs2O6g88cqTMkOP9RMHndPv0HMSSbaGTKvLEgfO+Vb3uC//GrU GqOVPdq2DqMk0iYnplRFqXIGD1wPT6q6m141FCm0srh6Wza4Q4+9uRoOMoNFDGu4 +PWjKTlThUyu2GzpTEDehMU1ruN0cXtKSNa3Pz9CXTNLcDDf5d1L+Jdfci6I7kKp 6flJG6IIuxDXKMhByywmYW2pEGfMqqgKK6maqyICwtvA4rL/rB54cwvNjE8fnhuY qboqkYXQDFO0+8+lVeWQXVCh5NGD8HfD+pZ7h4sLEp6/6WMivQ7WBZdno7wMW73U vexICCPq5zSfcir7ME4BIBfSRpDZZODBAe6T2EQ9X/ehy+iJEnnQV7NZ96nHLOZc dCTY29XC4Un1kAWN0HfNP7be8SuXmFt4VcuuOVzlUuwoBIDzUX9+eDgoZN2uRYvd ev27CL3dr1RAuWLRzauOz6nJGiKqZ2Hh1JhEaqAxC4V+zJfeGMuNiqazJ1SjDVkG lAufVLdjsIy7AoCjkJI7diVQ6QuBR70w0p9l8rFaJ5rc/Ef9OzLR8Po4QlJHstLD IaD9IKCoqnlucxFQmHA45Zp+h+EZvo32lg4Cy3rDv4NweoFhzgxpq6ER1IvS3k4T zhiAsZxKPwitwxNdRUg0Qb1wFq3gwa9nDUv3Z0cy6+CE/zSg0KU= =hYKB -----END PGP SIGNATURE-----

FastNetMon is awesome, but its a detection tool with no mitigation capacity whatsoever.

Does is not, though, provide the ability to hook into RTBH or Flowspec setups? -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal On Thu 2019-Dec-05 10:31:30 +0100, Alexander Lyamin <la@qrator.net> wrote:

FastNetMon is awesome, but its a detection tool with no mitigation capacity whatsoever.

On 12/5/19 1:43 PM, Hugo Slabbert wrote:

...

FastNetMon is awesome, but its a detection tool with no mitigation capacity whatsoever.

Does is not, though, provide the ability to hook into RTBH or Flowspec setups?

Yes it does provide RTBH hook. I evaluated fastnetmon using exactly the 'quick setup' and found it to have some serious problems with false alarms and statistical anomalies, at least when using pure netflow data (did not try sampled mode).  Hosts that were not in fact receiving >100mbps traffic (a traffic level I predetermined as 'attack' for a given network segment), would occasionally get flagged as such (and rtbh activated), while 2 real attacks that came during the testing period (60 days for me) went completely unnoticed. Support seemed to concede that sampled mode is really the only accurate method, and which by this time I'd expended all my interest. Great concept, cool integration, just not ready for prime time. MIke-

I had intended to use the paid version once the 'free trial' proved to work, but for the previously mentioned reasons it did not and I gave up. Would still love to have this style of solution in my network and still open to other solutions, just haven't really found anything else. On 1/28/20 2:46 PM, Colton Conor wrote:

Mike,

What did you end up going with if not fastnetmon? Were you using their paid or free version?

On Thu, Dec 5, 2019 at 4:45 PM Mike <mike-nanog@tiedyenetworks.com <mailto:mike-nanog@tiedyenetworks.com>> wrote:

On 12/5/19 1:43 PM, Hugo Slabbert wrote: >> FastNetMon is awesome, but its a detection tool with no mitigation >> capacity whatsoever. > > Does is not, though, provide the ability to hook into RTBH or Flowspec > setups? >

Yes it does provide RTBH hook.

I evaluated fastnetmon using exactly the 'quick setup' and found it to have some serious problems with false alarms and statistical anomalies, at least when using pure netflow data (did not try sampled mode).  Hosts that were not in fact receiving >100mbps traffic (a traffic level I predetermined as 'attack' for a given network segment), would occasionally get flagged as such (and rtbh activated), while 2 real attacks that came during the testing period (60 days for me) went completely unnoticed. Support seemed to concede that sampled mode is really the only accurate method, and which by this time I'd expended all my interest. Great concept, cool integration, just not ready for prime time.

MIke-

Check out Wanguard -- Dmitry Sherman From: NANOG <nanog-bounces@nanog.org> on behalf of Colton Conor <colton.conor@gmail.com> Date: Wednesday, 29 January 2020 at 0:47 To: Mike <mike-nanog@tiedyenetworks.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Recommended DDoS mitigation appliance? Mike, What did you end up going with if not fastnetmon? Were you using their paid or free version? On Thu, Dec 5, 2019 at 4:45 PM Mike <mike-nanog@tiedyenetworks.com<mailto:mike-nanog@tiedyenetworks.com>> wrote: On 12/5/19 1:43 PM, Hugo Slabbert wrote:

...

FastNetMon is awesome, but its a detection tool with no mitigation capacity whatsoever.

Does is not, though, provide the ability to hook into RTBH or Flowspec setups?

Yes it does provide RTBH hook. I evaluated fastnetmon using exactly the 'quick setup' and found it to have some serious problems with false alarms and statistical anomalies, at least when using pure netflow data (did not try sampled mode). Hosts that were not in fact receiving >100mbps traffic (a traffic level I predetermined as 'attack' for a given network segment), would occasionally get flagged as such (and rtbh activated), while 2 real attacks that came during the testing period (60 days for me) went completely unnoticed. Support seemed to concede that sampled mode is really the only accurate method, and which by this time I'd expended all my interest. Great concept, cool integration, just not ready for prime time. MIke-

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello, NANOG! Thank you to all who have generously given your time to respond publicly and privately. I have a long list of things to research while configuring our shiny new Juniper routers. :) I'll summarize to the list shortly. Be well! Rob, the routing rabbi. - -- Rabbi Rob Thomas Team Cymru "It is easy to believe in freedom of speech for those with whom we agree." - Leo McKern -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEDcVjavXj08cL/QwdQ+hhYvqF8o0FAl3TQCsACgkQQ+hhYvqF 8o2Iqg/+IyWtI8vPfEX6tcnoaFTz+fU86wAHzlr9ZQ4OuuGIutNYMOgl7BoJ709E qDb8iZTZuMtj3zfPXd/YsYbC2gFrqkybYzcPKK2Pd/m+iYUEgy6ckthzzdKENWFF CBaPfSMJIBUH2lcIl97JYxRGvzE0ffLYScCkI0UWWDr8ZHJb5oqA+u8zs4FGgDDI 7bY2eQcw+ZoCcah1uMeVCFC2WPIa4OxmJeIECK29ROOlutJ6bqNMEElD9148Q1W7 MUeoyfOaJxY1U+NjkyTDSF8MwMXsOxwPWE4z0GlHPdGaBB/ksuFohiM/eIIjf0+O XAj27WRkpbuBT8jYd1IT/ljVjJzruI11x97Vln3S0Zi9mP62n1VED1377jMezzzQ YBXLsEllXu5TcTUbFdt9n+0F8CIn1eo8klsGi8UtjsGV7tDGx9leZ91tCChpdvIa KQALYSkMu4AeODpcceBNfQ/GBimUpuKWEDPWg6FPDyZgkdYvBOJCAm71yFUzXYO/ vaQ7ZqVTyRosvM+hO7xVotDXSZgT2PtBLJNfdWk7NMJvBlS5xNl/6Gb6UZd261a8 0LEu5Yta1iFk+zWN8lb7yA0nATFhQBjz1ClqFscnzSirM5BLNIMUgRNyDAgg9UcY +ytzSKl/XaTLBXeKfxXr+Ju0HjYsjxBlGWr605VRpu5a/QzyJ5o= =/Ew7 -----END PGP SIGNATURE-----

So is Imperva similar to how Kentik operates? What was it priced liked?

It is a nice model as you don't need additional hardware or virtual appliances on-prem, which cuts down on the CAPEX cost. Like everyone else, they price the scrubbing based on your clean traffic levels. Price I have is circa $73,000 a year for 250mbit clean traffic and circa $94,000 a year for 500mbit clean traffic. Reasonably good value if you get attacked a lot - a very expensive insurance policy if not. Yearly pricing is broadly on par with Radware, Arbor and A10 (Verisign).

Phil, This sounds like a different model to me. Kentik I think averages out around $500 per 10G per month. Kentik doesn't do any scrubbing however. Does anyone have guide to DDoS services? Seems like there is a wide array of pricing and technology options. On Tue, Feb 4, 2020 at 7:50 AM Phil Lavin <phil.lavin@cloudcall.com> wrote:

...

So is Imperva similar to how Kentik operates? What was it priced liked?

It is a nice model as you don't need additional hardware or virtual appliances on-prem, which cuts down on the CAPEX cost. Like everyone else, they price the scrubbing based on your clean traffic levels. Price I have is circa $73,000 a year for 250mbit clean traffic and circa $94,000 a year for 500mbit clean traffic. Reasonably good value if you get attacked a lot - a very expensive insurance policy if not. Yearly pricing is broadly on par with Radware, Arbor and A10 (Verisign).

Hopefully you would be sending those flows out a different circuit than the one that’s going to get swamped with a DDoS otherwise... it might just take a while to mitigate that ;-) depending on the type obviously. -- J. Hellenthal The fact that there's a highway to Hell but only a stairway to Heaven says a lot about anticipated traffic volume.

On Feb 3, 2020, at 11:01, Javier Juan <javier.juan@gmail.com> wrote:

 Hi !

I was looking around (a couple years ago) for mitigation appliances (Riorey, Arbor, F5 and so on).... but the best and almost affordable solution I found was Incapsula/Imperva. https://docs.imperva.com/bundle/cloud-application-security/page/introducing/...

Basically, You send your flows to Imperva on cloud for analysis. As soon as they find DDoS attack , they activate mitigation. It´s some kind of elegant-hybrid solution without on-premise appliances . Just check it out :)

Regards,

JJ

...

On Sun, Nov 17, 2019 at 11:20 PM Rabbi Rob Thomas <robt@cymru.com> wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

Hello, NANOG!

I'm in the midst of rebuilding/upgrading our backbone and peering - sessions cheerfully accepted :) - and am curious what folks recommend in the DDoS mitigation appliance realm? Ideally it would be capable of 10Gbps and circa 14Mpps rate of mitigation. If you have a recommendation, I'd love to hear it and the reasons for it. If you have an alternative to an appliance that has worked well for you (we're a mix of Cisco and Juniper), I'm all ears.

Private responses are fine, and I'm happy to summarize back to the list if there is interest.

Thank you! Rob. - -- Rabbi Rob Thomas Team Cymru "It is easy to believe in freedom of speech for those with whom we agree." - Leo McKern -----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEDcVjavXj08cL/QwdQ+hhYvqF8o0FAl3Rx08ACgkQQ+hhYvqF 8o0snw/8CxTOujcodNh/huMXZaUNlMNoNRz3IoPqBiAP9BZomMz9xqlpDW/qvWBF xhoJ07C0O0mo5ilNjnPR308uifIBu6ylw02PshOCU06dV0afgtndxGg5AoG9npUV 7uCi2afWaf22dq5TwKLut8QPNNQJTRzndX88xJw9MzzoBTemxRtM7ft4H3UhJ0hv oKo83FCNZQt36I+GZA9GBJeXM+o0f5h0w6fhRqARzttf6brJZdXgROyIQ7jptGuZ N3Yrjk/8RM4XKMnYbtIwl8NS3c0nEGN3ndn+Bz7p2FE7QJrZKonk/o03dvr2kU0Y 7gUQliOOzV9EsptVGyLCVyDJSElvXTBaps0giEVZhdmEIDJPWvBc+93j1g7xbmti 27lT6+5qBmEN0oKJWxXgtw9/n1yX9vsc7tXlgYDoXGhIlszdB3baRao1tYEp8BBQ hTGAULRfHe94tRzvOOQUQIuhzNcK1Q4E2jU6kzBB1wJsBD4zuHk+QIJLSHBmmnka VNKlQ+5zP8dmSMBp6k4feqAtt3hy0Bj+34FbdQZYPutIe3VXHEjpWI3jI9vKjhtC g7U/9CQIjVUl2APn1IllArpUpETBlNq7dSeJNUN/4Xh+eHglUnEn/m2kFG5mizmP d0YvLEVe0/+WzDUz+y3KxDVP5tdJT1VM46FHIgeiB4KrWNGRPUo= =uuel -----END PGP SIGNATURE-----