We've had an increasing rate of DoS attacks that spew tens-of-thousands of small UDP packets to a destination on our network. We are getting roughly 2x our entire normal pps across all providers through one interface, or about 4x normal through the individual interface. The Cisco 7206VXR/NPE-G1 CPU melts (>95% load vs 15% average, 20% normal peak) when this hits. I'm using CEF and ip-route-cache flow on the outside interface. Unicast RPF is also enabled on the interface. Unicast RPF in conjunction with a BGP black-hole generator handles TCP attacks fairly well. Two questions: - Are there any knobs I should be turning in the Cisco config to help with mitigate this? - Are there any platforms that deal with high PPS/small packet more gracefully? We are looking at a network refresh and aren't locked into Cisco as a vendor (although our current IP network consists entirely of Cisco gear). Our current aggregate (all providers, in- plus out-bound) bandwidth is ~500Mbs, but projected growth is 1Gbs within the year. Thanks, Rick
On Dec 13, 2008, at 2:15 AM, Rick Ernst wrote:
- Are there any platforms that deal with high PPS/small packet more gracefully?
S/RTBH can deal with any type of packet-flooding DDoS at layer-3, up to the capacity of the platform in question. It sounds as if a) you should investigate getting DDoS mitigation assistance from your upstreams and/or b) moving from your currently software-based platform to a hardware-based platform at your edge to provide increased performance (this holds true irrespective of which vendor you select for your edge platform). If you move to a hardware-based edge platform, be sure to first investigate all the particulars of its uRPF implementation so as to ensure that you can use it for S/RTBH, and if at all possible, test it before buying. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // +852.9133.2844 mobile History is a great teacher, but it also lies with impunity. -- John Robb
Couple of things come to mind: 1. Take a packet capture to see some UDP traffic characteristics, based on which traffic rate-limiting may be configured by your upstream providers, so that this traffic doesn't saturate your pipes, and maybe the ISP can even drop it. That is if they're willing to help you. 2. As far as hardware is concerned, we're in the same boat as far as various UDP/ICMP floods, and our Juniper M10i's handle it with no issues (running multiple BGP sessions, OSPF, firewall sets/access lists). Sincerely, David Kotlerewsky, Sr. Network Engineer ------------------------------------------------- OVERSEE.NET 515 S. Flower Street, Suite 4400 Los Angeles, CA 90071 ph 213.408.0080 x1458 cell 310.350.0399 www.oversee.net dkotlerewsky@oversee.net Confidentiality Warning: this email contains information intended for the use of the individual or entity named above. If the reader of this e-mail is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, any dissemination, publication or copying of this e-mail is prohibited. The sender does not accept any responsibility for any loss, disruption or damage to your data or computer system that may occur while using data contained in it, or transmitted with this e-mail. If you have received this e-mail in error, please immediately notify us by return e-mail. Thank you. -----Original Message----- From: Rick Ernst [mailto:ernst@easystreet.com] Sent: Friday, December 12, 2008 10:15 AM To: nanog@nanog.org Subject: UDP DoS mitigation? We've had an increasing rate of DoS attacks that spew tens-of-thousands of small UDP packets to a destination on our network. We are getting roughly 2x our entire normal pps across all providers through one interface, or about 4x normal through the individual interface. The Cisco 7206VXR/NPE-G1 CPU melts (>95% load vs 15% average, 20% normal peak) when this hits. I'm using CEF and ip-route-cache flow on the outside interface. Unicast RPF is also enabled on the interface. Unicast RPF in conjunction with a BGP black-hole generator handles TCP attacks fairly well. Two questions: - Are there any knobs I should be turning in the Cisco config to help with mitigate this? - Are there any platforms that deal with high PPS/small packet more gracefully? We are looking at a network refresh and aren't locked into Cisco as a vendor (although our current IP network consists entirely of Cisco gear). Our current aggregate (all providers, in- plus out-bound) bandwidth is ~500Mbs, but projected growth is 1Gbs within the year. Thanks, Rick
On Dec 13, 2008, at 2:27 AM, David Kotlerewsky wrote:
2. As far as hardware is concerned, we're in the same boat as far as various UDP/ICMP floods, and our Juniper M10i's handle it with no issues (running multiple BGP sessions, OSPF, firewall sets/access lists).
Right - a hardware-based platform is required to deal with high pps rates (the Cisco equivalent is the ASR1000; I'm not familiar with boxes from other vendors, but I'm pretty sure there are others in this same class). ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // +852.9133.2844 mobile History is a great teacher, but it also lies with impunity. -- John Robb
Although the problem we had wasn't DoS, but rather high packet rates for market data, we saw a huge improvement by moving from a 7204VRX to a 7600 platform. Going from a software switched environment to a hardware one help deal with large number of packet drops during peaks of burst activity. We looked at the ASR1000, but found the price too high. Although cisco doesn't promote it, the 7604 with the Sup32 engine (WS-SUP32-GE-3B) with 8 x GE interfaces is a very cost effective hardware router. -----Original Message----- From: Rick Ernst [mailto:ernst@easystreet.com] Sent: Friday, December 12, 2008 1:15 PM To: nanog@nanog.org Subject: UDP DoS mitigation? We've had an increasing rate of DoS attacks that spew tens-of-thousands of small UDP packets to a destination on our network. We are getting roughly 2x our entire normal pps across all providers through one interface, or about 4x normal through the individual interface. The Cisco 7206VXR/NPE-G1 CPU melts (>95% load vs 15% average, 20% normal peak) when this hits. I'm using CEF and ip-route-cache flow on the outside interface. Unicast RPF is also enabled on the interface. Unicast RPF in conjunction with a BGP black-hole generator handles TCP attacks fairly well. Two questions: - Are there any knobs I should be turning in the Cisco config to help with mitigate this? - Are there any platforms that deal with high PPS/small packet more gracefully? We are looking at a network refresh and aren't locked into Cisco as a vendor (although our current IP network consists entirely of Cisco gear). Our current aggregate (all providers, in- plus out-bound) bandwidth is ~500Mbs, but projected growth is 1Gbs within the year. Thanks, Rick
Replying to my own since there are currently about a dozen responses. - Hardware/ASIC routers are a consistent response. We are currently evaluating Juniper for other reasons, but I'll add DoS mitigation to mix. - Upstream involvement: We get transit from 701, 1239, etc. I've had mixed results getting timely responses from our upstreams. It's useful for long-term issues, but I need as much local and timely control as I can get. - I'm not having a problem with pipe bandwidth, but high pps. - uRPF and RTBH helped internally, but anything passing through that upstream connection was impacted. - This instance was a DoS, not DDoS. Single source and destination, but the source (assuming no spoofing) was in Italy. Turning off netflow seemed to help, but the attack itself stopped at about the same time. Also, thanks for the offers of individual help in mitigation, although I'd be concerned that "Hey, can somebody block traffic {from} or {to}?" would be an interesting experiment in a socially-engineered DoS. Finally, there were some suggestions "S/RTBH". RTBH I get, but my Google-fu is weak on S/RTBH. Details? Thanks, Rick On Fri, December 12, 2008 10:15, Rick Ernst wrote:
We've had an increasing rate of DoS attacks that spew tens-of-thousands of small UDP packets to a destination on our network. We are getting roughly 2x our entire normal pps across all providers through one interface, or about 4x normal through the individual interface. The Cisco 7206VXR/NPE-G1 CPU melts (>95% load vs 15% average, 20% normal peak) when this hits.
I'm using CEF and ip-route-cache flow on the outside interface. Unicast RPF is also enabled on the interface. Unicast RPF in conjunction with a BGP black-hole generator handles TCP attacks fairly well.
Two questions: - Are there any knobs I should be turning in the Cisco config to help with mitigate this? - Are there any platforms that deal with high PPS/small packet more gracefully?
We are looking at a network refresh and aren't locked into Cisco as a vendor (although our current IP network consists entirely of Cisco gear). Our current aggregate (all providers, in- plus out-bound) bandwidth is ~500Mbs, but projected growth is 1Gbs within the year.
Thanks, Rick
Rick Ernst wrote on 2008-12-13:
- This instance was a DoS, not DDoS. Single source and destination, but the source (assuming no spoofing) was in Italy. Turning off netflow seemed to help, but the attack itself stopped at about the same time.
Before moving to hardware based platforms, we used a lot of G1s on sticks. One of the advantages of this is the ability to filter DOS traffic on the switch in front of the router - anything 2950 or higher (with L3 snooping capabilities) can do this with an access list. Router1 Gi0/1 ----- Gi0/1 Switch1 Gi0/2 ----- Upstream On Switch1 configure something like: access-list 100 deny ip host x.x.x.x access-list 100 permit ip any any interface GigabitEthernet0/2 ip access-group 100 in So if your topology allows for it, this is a great short term fix. Note that this means you lose high speed convergence due to immediate link state notifications, and should use aggressive timers to compensate. -- Ian Henderson, CCIE #14721 Senior Network Engineer, iiNet Limited
* Rick Ernst:
We've had an increasing rate of DoS attacks that spew tens-of-thousands of small UDP packets to a destination on our network. We are getting roughly 2x our entire normal pps across all providers through one interface, or about 4x normal through the individual interface. The Cisco 7206VXR/NPE-G1 CPU melts (>95% load vs 15% average, 20% normal peak) when this hits.
I'm using CEF and ip-route-cache flow on the outside interface.
Is the UDP stream a single flow, or does it consist of lots of different flows?
participants (6)
-
David Kotlerewsky -
Florian Weimer -
Ian Henderson -
Matthew Huff -
Rick Ernst -
Roland Dobbins