Blackholes and IXs and Completing the Attack.
Hi, I have been working away on remote trigger blackholing and community based client initiated blackholing into transit ASes. It got me thinking that while this works great with a handful of upstream transit peers it does not really scale very well at an Internet Exchange with a high overhead configuring things for many peers. Plus if your IX connection is saturated that means legitimate traffic must be getting degraded - even if your router is coping and blackholing the interconnect is still flat lined. The only ways into an AS are via transit, public IX or private interconnects. If we want to extend the blackholing to secure IXs peers as well as into transits. So my idea.... Is to have an IX route reflector configured with ACLs locking it down to exclusively BGP with the IX peer IP of the member. The IX route reflector would be configured to have per peer prefix filters per peer auto generated from registered AS macro for each peer from the RIPE,ARIN,APNIC etc databases. This should mean the router will not accept announcements for any /32 that is not part of the routes announced by that AS (it would be even better to tie it down to a match on origin AS as well). Plus the router will only talk to IX peers - no global transit. This hopefully will ensure a relatively protected router that is only accessible from the edge routers we want and also secured to only accept filtered announcements for black holing and in consequence enable the system to be trusted similar to Team Cymaru. Then all a member AS of the exchange does is announce any /32 from their IP block that they would like other members to Null route in their AS to this reflector. There are people way smarter than me on this list and the above is not implemented at any of the IXs I am connected to, so why is the above a dumb idea / what have I missed that makes the above unworkable because it does seem kind of obvious now I have done some work with this. Kind Regards Ben Butler ++++++++++++++++++++++++++++++++++++++++++ C2 Internet Ltd Globe House, The Gullet, Nantwich, Cheshire, CW5 5RL E mailto:ben.butler@c2internet.net W http://www.c2internet.net/ B1 http://c2internet.blogspot.com/ B2 http://c2noc.blogspot.com/ T +44-(0)845-658-0020 F +44-(0)845-658-0070 All quotes & services from C2 are bound by our standard terms and conditions which are available on our website at: http://www.c2internet.net/legal/main.htm#tandc C2 Internet Limited is a company registered in England and Wales with company number 03910154 Our VAT Registration number is GB 752 7650 17
ben.butler@c2internet.net ("Ben Butler") writes:
... This hopefully will ensure a relatively protected router that is only accessible from the edge routers we want and also secured to only accept filtered announcements for black holing and in consequence enable the system to be trusted similar to Team Cymaru. ...
This sounds like another attempt to separate the Internet's control plane from its data plane, and most such attempts do succeed and are helpful (like NSP OOB, or like enterprise-level anycast of DNS). However, I'm not sure that remote triggered blackholes are a good direction, worthy of the protection you're proposing, for three reasons. First, because large NSP's simply cannot afford the risk associated with letting a third party, automatically and without controls or audits, decide in real time what sources or destinations shall become unreachable. With all respect (which is a lot) for spamhaus and cymru and even MAPS (which I had a hand in, back in the day), feeding BGP null-routes to a multinational backbone is a privilege that ISO9000 and SarBox and liability insurance providers don't usually want to extend. Second, because many backbone routers in use today can't do policy routing routing (which is in this case dropping packets because their source address, not their destination address, has a particular community associated with it) at line speed. Note, this is many-not-all -- I'm perfectly aware that lots of backbone routers can do this but not everybody has them or can afford them and those who have them tend to be the multinational NSPs discussed earlier. To prevent our DDoS protection reflexes from lowering an attacker's cost (by automatically blackholing victims to protect the nonvictims), we have to be able to blackhole the abusive traffic by source, not by destination. Third, because many OPNs (other people's networks) still don't filter on source address on their customer-facing edge, and thus allow spoofed-source traffic to exit toward "the core" or toward a victim's NSP who cannot filter by source due to path ambiguities inherent in "the core", any wide scale implementation of this, even if we could get trusted automation of it at scale and even if everybody had policy-routing-at-like-speed, would just push the attackers toward spoofed-source. That means a huge amount of work and money for the world, without changing the endgame for attackers and victims at all. (See BCP38 and SAC004 for prior rants on this controversial topic.)
Hi, I was not proposing he Null routing of the attack source in the other ISPs network but the destination in my network being Null routed as a destination from your network out. This has no danger to the other network as it is my network that is going to be my IP space that is blackholed in your network, and the space blackholed is going to be an address that is being knocked of the air anyway under DoS and we are trying to minimise collateral damage. I cant see where the risk to the large NSP is - given that the route reflector will only reflect /32s that legitimately originate (as a destination not a source) in the AS announcing them as please blackhole. For complete clarity: AS13005 announces 213.170.128.0/19 and has its route object in RIPE as being announced by AS13005. My router at IX - BENIX say - announces 213.170.128.1/32 to the router reflector their, the announcement from my IX assigned address 212.121.34.30 is known to be my router on the exchange, and I am announcing a /32 from my AS for a route object registered as being announced by my AS - so the reflector accepts my announcement and reflects it to any other members that choose to peer with this reflector - effectively this is a please blackhole this destination in AS13005 - the other members that receive this announcement can then deal with it in anyway they see fit from ignoring it to setting next-hop 192.0.2.1 -> Null0. The effect of this would be that any BotNet controlled hosts in the other member network would now be able to drop any attack traffic in their network on destination at their customer aggregation routers. I think you might have thought I was suggesting we blackhole sources in other peoples networks - this is definatly not what I was saying. So, given we all now understand each other - why is no one doing the above? At the end of the day if an IX member doesn't want the announcements don't peer with the blackhole reflector, simple, and it will get Null routed as soon as it hits my edge router at the IX - it would just seem more sensible to enable people to block the traffic before it traverses the IX and further back in their own networks. So????? Ben -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Paul Vixie Sent: 02 February 2008 17:32 To: nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack. ben.butler@c2internet.net ("Ben Butler") writes:
... This hopefully will ensure a relatively protected router that is only accessible from the edge routers we want and also secured to only accept filtered announcements for black holing and in consequence enable the system to be trusted similar to Team Cymaru. ...
This sounds like another attempt to separate the Internet's control plane from its data plane, and most such attempts do succeed and are helpful (like NSP OOB, or like enterprise-level anycast of DNS). However, I'm not sure that remote triggered blackholes are a good direction, worthy of the protection you're proposing, for three reasons. First, because large NSP's simply cannot afford the risk associated with letting a third party, automatically and without controls or audits, decide in real time what sources or destinations shall become unreachable. With all respect (which is a lot) for spamhaus and cymru and even MAPS (which I had a hand in, back in the day), feeding BGP null-routes to a multinational backbone is a privilege that ISO9000 and SarBox and liability insurance providers don't usually want to extend. Second, because many backbone routers in use today can't do policy routing routing (which is in this case dropping packets because their source address, not their destination address, has a particular community associated with it) at line speed. Note, this is many-not-all -- I'm perfectly aware that lots of backbone routers can do this but not everybody has them or can afford them and those who have them tend to be the multinational NSPs discussed earlier. To prevent our DDoS protection reflexes from lowering an attacker's cost (by automatically blackholing victims to protect the nonvictims), we have to be able to blackhole the abusive traffic by source, not by destination. Third, because many OPNs (other people's networks) still don't filter on source address on their customer-facing edge, and thus allow spoofed-source traffic to exit toward "the core" or toward a victim's NSP who cannot filter by source due to path ambiguities inherent in "the core", any wide scale implementation of this, even if we could get trusted automation of it at scale and even if everybody had policy-routing-at-like-speed, would just push the attackers toward spoofed-source. That means a huge amount of work and money for the world, without changing the endgame for attackers and victims at all. (See BCP38 and SAC004 for prior rants on this controversial topic.)
You could achieve the exact same result simply by not advertising the network to your peers, or by advertising a bogus route (prefixing a known bogon AS for the addresses you want null-routed). I realize you would have to subnet/deaggregate your netblocks, and therefore could wind up with a prefix-length issue for peers who won't accept routes longer than a certain mask, in some cases, but if you are already being DDOSed, this should represent SOME improvement. If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong. The bigger issue with all these approaches is that they run afoul of a patent applied for by AT&T: http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u =%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=200 60031575&OS=20060031575&RS=20060031575 USPTO App Number 20060031575
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Ben Butler Sent: Saturday, February 02, 2008 12:17 PM To: Paul Vixie; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
Hi,
I was not proposing he Null routing of the attack source in the other ISPs network but the destination in my network being Null routed as a destination from your network out.
This has no danger to the other network as it is my network that is going to be my IP space that is blackholed in your network, and the space blackholed is going to be an address that is being knocked of the air anyway under DoS and we are trying to minimise collateral damage. I cant see where the risk to the large NSP is - given that the route reflector will only reflect /32s that legitimately originate (as a destination not a source) in the AS announcing them as please blackhole.
For complete clarity: AS13005 announces 213.170.128.0/19 and has its route object in RIPE as being announced by AS13005. My router at IX - BENIX say - announces 213.170.128.1/32 to the router reflector their, the announcement from my IX assigned address 212.121.34.30 is known to be my router on the exchange, and I am announcing a /32 from my AS for a route object registered as being announced by my AS - so the reflector accepts my announcement and reflects it to any other members that choose to peer with this reflector - effectively this is a please blackhole this destination in AS13005 - the other members that receive this announcement can then deal with it in anyway they see fit from ignoring it to setting next-hop 192.0.2.1 -> Null0.
The effect of this would be that any BotNet controlled hosts in the other member network would now be able to drop any attack traffic in their network on destination at their customer aggregation routers.
I think you might have thought I was suggesting we blackhole sources in other peoples networks - this is definatly not what I was saying.
So, given we all now understand each other - why is no one doing the above?
At the end of the day if an IX member doesn't want the announcements don't peer with the blackhole reflector, simple, and it will get Null routed as soon as it hits my edge router at the IX - it would just seem more sensible to enable people to block the traffic before it traverses the IX and further back in their own networks.
So?????
Ben
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Paul Vixie Sent: 02 February 2008 17:32 To: nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack.
ben.butler@c2internet.net ("Ben Butler") writes:
... This hopefully will ensure a relatively protected router that is only accessible from the edge routers we want and also secured to only accept filtered announcements for black holing and in consequence enable the system to be trusted similar to Team Cymaru. ...
This sounds like another attempt to separate the Internet's control plane from its data plane, and most such attempts do succeed and are helpful (like NSP OOB, or like enterprise-level anycast of DNS). However, I'm not sure that remote triggered blackholes are a good direction, worthy of the protection you're proposing, for three reasons.
First, because large NSP's simply cannot afford the risk associated with letting a third party, automatically and without controls or audits, decide in real time what sources or destinations shall become unreachable. With all respect (which is a lot) for spamhaus and cymru and even MAPS (which I had a hand in, back in the day), feeding BGP null-routes to a multinational backbone is a privilege that ISO9000 and SarBox and liability insurance providers don't usually want to extend.
Second, because many backbone routers in use today can't do policy routing routing (which is in this case dropping packets because their source address, not their destination address, has a particular community associated with it) at line speed. Note, this is many-not-all -- I'm perfectly aware that lots of backbone routers can do this but not everybody has them or can afford them and those who have them tend to be the multinational NSPs discussed earlier. To prevent our DDoS protection reflexes from lowering an attacker's cost (by automatically blackholing victims to protect the nonvictims), we have to be able to blackhole the abusive traffic by source, not by destination.
Third, because many OPNs (other people's networks) still don't filter on source address on their customer-facing edge, and thus allow spoofed-source traffic to exit toward "the core" or toward a victim's NSP who cannot filter by source due to path ambiguities inherent in "the core", any wide scale implementation of this, even if we could get trusted automation of it at scale and even if everybody had policy-routing-at-like-speed, would just push the attackers toward spoofed-source. That means a huge amount of work and money for the world, without changing the endgame for attackers and victims at all. (See BCP38 and SAC004 for prior rants on this controversial topic.)
On Feb 2, 2008 3:39 PM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
The bigger issue with all these approaches is that they run afoul of a patent applied for by AT&T:
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u =%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=200 60031575&OS=20060031575&RS=20060031575
USPTO App Number 20060031575
Somene from ATT may want to consider pulling this patent application since it seems to fail on prior art... http://www.nanog.org/mtg-0410/soricelli.html presented by a juniper employee (Joe Soricelli ) and Wayne Gustavus from Verizon. IANAL though...
ATT has no reason to pull their application, what needs to happen is that the publisher of the prior art contact the USPTO. If ATT willingly failed to note the prior art in their app, that may be a problem, but it isn't their duty to report ALL prior art, just the stuff they know about. IANAL, but I have filed some patents, and reviewed a bunch more.
-----Original Message----- From: christopher.morrow@gmail.com [mailto:christopher.morrow@gmail.com] On Behalf Of Christopher Morrow Sent: Saturday, February 02, 2008 12:58 PM To: Tomas L. Byrnes Cc: Ben Butler; Paul Vixie; nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack.
On Feb 2, 2008 3:39 PM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
The bigger issue with all these approaches is that they run afoul of a patent applied for by AT&T:
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1
&u
=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=2
00 60031575&OS=20060031575&RS=20060031575
USPTO App Number 20060031575
Somene from ATT may want to consider pulling this patent application since it seems to fail on prior art...
http://www.nanog.org/mtg-0410/soricelli.html
presented by a juniper employee (Joe Soricelli ) and Wayne Gustavus from Verizon. IANAL though...
On Feb 2, 2008 11:40 PM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
ATT has no reason to pull their application, what needs to happen is that the publisher of the prior art contact the USPTO.
If ATT willingly failed to note the prior art in their app, that may be a problem, but it isn't their duty to report ALL prior art, just the stuff they know about.
sweetness, hopefully Wayne or Verizon (they have lots of lawyers) or Juniper will ping USPTO... or not, I suppose I don't care directly anymore :)
IANAL, but I have filed some patents, and reviewed a bunch more.
-----Original Message----- From: christopher.morrow@gmail.com [mailto:christopher.morrow@gmail.com] On Behalf Of Christopher Morrow Sent: Saturday, February 02, 2008 12:58 PM To: Tomas L. Byrnes Cc: Ben Butler; Paul Vixie; nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack.
On Feb 2, 2008 3:39 PM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
The bigger issue with all these approaches is that they run afoul of a patent applied for by AT&T:
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1
&u
=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=2
00 60031575&OS=20060031575&RS=20060031575
USPTO App Number 20060031575
Somene from ATT may want to consider pulling this patent application since it seems to fail on prior art...
http://www.nanog.org/mtg-0410/soricelli.html
presented by a juniper employee (Joe Soricelli ) and Wayne Gustavus from Verizon. IANAL though...
"If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong." Well then they wouldn't be peering with this route reflector in the first place. -----Original Message----- From: Tomas L. Byrnes [mailto:tomb@byrneit.net] Sent: 02 February 2008 20:39 To: Ben Butler; Paul Vixie; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack. You could achieve the exact same result simply by not advertising the network to your peers, or by advertising a bogus route (prefixing a known bogon AS for the addresses you want null-routed). I realize you would have to subnet/deaggregate your netblocks, and therefore could wind up with a prefix-length issue for peers who won't accept routes longer than a certain mask, in some cases, but if you are already being DDOSed, this should represent SOME improvement. If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong. The bigger issue with all these approaches is that they run afoul of a patent applied for by AT&T: http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u =%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=200 60031575&OS=20060031575&RS=20060031575 USPTO App Number 20060031575
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Ben Butler Sent: Saturday, February 02, 2008 12:17 PM To: Paul Vixie; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
Hi,
I was not proposing he Null routing of the attack source in the other ISPs network but the destination in my network being Null routed as a destination from your network out.
This has no danger to the other network as it is my network that is going to be my IP space that is blackholed in your network, and the space blackholed is going to be an address that is being knocked of the air anyway under DoS and we are trying to minimise collateral damage. I cant see where the risk to the large NSP is - given that the route reflector will only reflect /32s that legitimately originate
(as a destination not a source) in the AS announcing them as please blackhole.
For complete clarity: AS13005 announces 213.170.128.0/19 and has its route object in RIPE as being announced by AS13005. My router at IX - BENIX say - announces 213.170.128.1/32 to the router
reflector their, the announcement from my IX assigned address 212.121.34.30 is known to be my router on the exchange, and I am announcing a /32 from my AS for a route object registered as being announced by my AS - so the reflector accepts my announcement and reflects it to any other members that choose to peer with this reflector - effectively this is a please blackhole this destination in AS13005 - the other members that receive this announcement can then deal with it in anyway they see fit from ignoring it to setting next-hop 192.0.2.1 -> Null0.
The effect of this would be that any BotNet controlled hosts in the other member network would now be able to drop any attack traffic in their network on destination at their customer aggregation routers.
I think you might have thought I was suggesting we blackhole sources in other peoples networks - this is definatly not what I was saying.
So, given we all now understand each other - why is no one doing the above?
At the end of the day if an IX member doesn't want the announcements don't peer with the blackhole reflector, simple, and it will get Null routed as soon as it hits my edge router at the IX - it would just seem more sensible to enable people to block the traffic before it traverses the IX and further back in their own networks.
So?????
Ben
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Paul Vixie Sent: 02 February 2008 17:32 To: nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack.
ben.butler@c2internet.net ("Ben Butler") writes:
... This hopefully will ensure a relatively protected router that is only accessible from the edge routers we want and also secured to only accept filtered announcements for black holing and in consequence enable the system to be trusted similar to Team Cymaru. ...
This sounds like another attempt to separate the Internet's control plane from its data plane, and most such attempts do succeed and are helpful (like NSP OOB, or like enterprise-level anycast of DNS). However, I'm not sure that remote triggered blackholes are a good direction, worthy of the protection you're proposing, for three reasons.
First, because large NSP's simply cannot afford the risk associated with letting a third party, automatically and without controls or audits, decide in real time what sources or destinations shall become unreachable. With all respect (which is a lot) for spamhaus and cymru
and even MAPS (which I had a hand in, back in the day), feeding BGP null-routes to a multinational backbone is a privilege that ISO9000 and SarBox and liability insurance providers don't usually want to extend.
Second, because many backbone routers in use today can't do policy routing routing (which is in this case dropping packets because their source address, not their destination address, has a particular community associated with it) at line speed. Note, this is many-not-all -- I'm perfectly aware that lots of backbone routers can do this but not everybody has them or can afford them and those who have them tend
to be the multinational NSPs discussed earlier. To prevent our DDoS protection reflexes from lowering an attacker's cost (by automatically blackholing victims to protect the nonvictims),
we have to be able to blackhole the abusive traffic by source, not by destination.
Third, because many OPNs (other people's networks) still don't filter on source address on their customer-facing edge, and thus allow spoofed-source traffic to exit toward "the core" or toward a victim's NSP who cannot filter by source due to path ambiguities inherent in "the core", any wide scale implementation of this, even if we could get trusted automation of it at scale and even if everybody had policy-routing-at-like-speed, would just push the attackers toward spoofed-source. That means a huge amount of work and money for the world, without changing the endgame for attackers and victims at all. (See BCP38 and SAC004 for prior rants on this controversial topic.)
While I am not sure I fully understand your suggestion, I don't think it would be that hard to set up manually. Sure it would require asking the individual peers for their black hole communities, but of they don't have one they are unlikely to honor the infrastructure you describe anyway. Assume your network is set up to discard packets marked with community 13005:666 Get a list of your peers blackhole communities, when you announce the route from a location on your network, tag it with community 13005:666 but also 1111:777, 2222:888 etc. for the individual peers from the source. This prevents you from having to update multiple policies in multiple locations for each attack. As long as they accept the /32 announced to them with their black hole community, they should discard the traffic without sending it to you. Not all peers will have a blackhole community, but you need some way to know when the attack is over to know when to withdraw the route, and they are useful for this. If you are real lazy, on the router you announce the black hole from, add an export policy that says from community 13005:666, then community add 1111:777, 2222:888 etc. This way you only need to: 1. Update one policy in one place when peers change 2. Announce the route from one location adding one community to it.
"Well then they wouldn't be peering with this route reflector " Well then, the utility is probably close to 0, isn't it? I doubt most of the sources of DDOS traffic, especially those without ingress source filtering, are going to peer with your route reflector. What's their economic incentive to expend the engineering time to do so? I sincerely doubt that any backbone provider will filter at a /32. That means they have to check EVERY PACKET AT FULL IP DEST against your AS advertised routes. Since most backbone routers build circuits at the /18 and above mask on MPLS, just to keep up with traffic, I sincerely doubt they are going to expend the CPU, and potentially RAM, never mind prefix table entries (you know, those things we're running out of) to have a full table of every host that every hoster says is being DDOSed. In this case, there's a clear economic cost, for no economic benefit (they do actually make money delivering that DDOS traffic). A better approach would be to move your DDOS target and all the rest of its co-subnet hosts into a different /24, update the DNS RRs, and cease advertising that /24. If you really want to be nice, they don't need to renumber, you just need to stop advertising the target subnet, change the DNS RR's and NAT at your borders, if you control DNS and IP. The added benefit of this is that you can swap them back when the DDOs is over, and they get to stay up while it's happening. All you need to do this is some spare, never to be allocated, IP space. Works with the existing infrastructure, doesn't require an "Internet God", and in general, is likely to be more effective in the real world. That whole "rough consensus and running code" ethos of the IETF thing, as opposed to the "Cathedral" mentality of the ITU (and ICANNt), which your approach would imply. You control which routes you advertise, after all. Plus, AT&T's legal beagles don't get to send you a dunning notice when their patent gets granted.
-----Original Message----- From: Ben Butler [mailto:ben.butler@c2internet.net] Sent: Saturday, February 02, 2008 2:42 PM To: Tomas L. Byrnes; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
"If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong."
Well then they wouldn't be peering with this route reflector in the first place.
-----Original Message----- From: Tomas L. Byrnes [mailto:tomb@byrneit.net] Sent: 02 February 2008 20:39 To: Ben Butler; Paul Vixie; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
You could achieve the exact same result simply by not advertising the network to your peers, or by advertising a bogus route (prefixing a known bogon AS for the addresses you want null-routed). I realize you would have to subnet/deaggregate your netblocks, and therefore could wind up with a prefix-length issue for peers who won't accept routes longer than a certain mask, in some cases, but if you are already being DDOSed, this should represent SOME improvement.
If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong.
The bigger issue with all these approaches is that they run afoul of a patent applied for by AT&T:
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HI TOFF&p=1&u =%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=P G01&s1=200 60031575&OS=20060031575&RS=20060031575
USPTO App Number 20060031575
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Ben Butler Sent: Saturday, February 02, 2008 12:17 PM To: Paul Vixie; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
Hi,
I was not proposing he Null routing of the attack source in the other ISPs network but the destination in my network being Null routed as a destination from your network out.
This has no danger to the other network as it is my network that is going to be my IP space that is blackholed in your network, and the space blackholed is going to be an address that is being knocked of the air anyway under DoS and we are trying to minimise collateral damage. I cant see where the risk to the large NSP is - given that the route reflector will only reflect /32s that legitimately originate
(as a destination not a source) in the AS announcing them as please blackhole.
For complete clarity: AS13005 announces 213.170.128.0/19 and has its route object in RIPE as being announced by AS13005. My router at IX - BENIX say - announces 213.170.128.1/32 to the router
reflector their, the announcement from my IX assigned address 212.121.34.30 is known to be my router on the exchange, and I am announcing a /32 from my AS for a route object registered as being announced by my AS - so the reflector accepts my announcement and reflects it to any other members that choose to peer with this reflector - effectively this is a please blackhole this destination in AS13005 - the other members that receive this announcement can then deal with it in anyway they see fit from ignoring it to setting next-hop 192.0.2.1 -> Null0.
The effect of this would be that any BotNet controlled hosts in the other member network would now be able to drop any attack traffic in their network on destination at their customer aggregation routers.
I think you might have thought I was suggesting we blackhole sources in other peoples networks - this is definatly not what I was saying.
So, given we all now understand each other - why is no one doing the above?
At the end of the day if an IX member doesn't want the announcements don't peer with the blackhole reflector, simple, and it will get Null routed as soon as it hits my edge router at the IX - it would just seem more sensible to enable people to block the traffic before it traverses the IX and further back in their own networks.
So?????
Ben
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Paul Vixie Sent: 02 February 2008 17:32 To: nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack.
ben.butler@c2internet.net ("Ben Butler") writes:
... This hopefully will ensure a relatively protected router that is only accessible from the edge routers we want and also secured to only accept filtered announcements for black holing and in consequence enable the system to be trusted similar to Team Cymaru. ...
This sounds like another attempt to separate the Internet's control plane from its data plane, and most such attempts do succeed and are helpful (like NSP OOB, or like enterprise-level anycast of DNS). However, I'm not sure that remote triggered blackholes are a good direction, worthy of the protection you're proposing, for three reasons.
First, because large NSP's simply cannot afford the risk associated with letting a third party, automatically and without controls or audits, decide in real time what sources or destinations shall become unreachable. With all respect (which is a lot) for spamhaus and cymru
and even MAPS (which I had a hand in, back in the day), feeding BGP null-routes to a multinational backbone is a privilege that ISO9000 and SarBox and liability insurance providers don't usually want to extend.
Second, because many backbone routers in use today can't do policy routing routing (which is in this case dropping packets because their source address, not their destination address, has a particular community associated with it) at line speed. Note, this is many-not-all -- I'm perfectly aware that lots of backbone routers can do this but not everybody has them or can afford them and those who have them tend
to be the multinational NSPs discussed earlier. To prevent our DDoS protection reflexes from lowering an attacker's cost (by automatically blackholing victims to protect the nonvictims),
we have to be able to blackhole the abusive traffic by source, not by destination.
Third, because many OPNs (other people's networks) still don't filter on source address on their customer-facing edge, and thus allow spoofed-source traffic to exit toward "the core" or toward a victim's NSP who cannot filter by source due to path ambiguities inherent in "the core", any wide scale implementation of this, even if we could get trusted automation of it at scale and even if everybody had policy-routing-at-like-speed, would just push the attackers toward spoofed-source. That means a huge amount of work and money for the world, without changing the endgame for attackers and victims at all. (See BCP38 and SAC004 for prior rants on this controversial topic.)
I sincerely doubt that any backbone provider will filter at a /32. That means they have to check EVERY PACKET AT FULL IP DEST against your AS advertised routes. Since most backbone routers build circuits at the /18 and above mask on MPLS, just to keep up with traffic, I sincerely doubt they are going to expend the CPU, and potentially RAM, never mind prefix table entries (you know, those things we're running out of) to have a full table of every host that every hoster says is being DDOSed. In this case, there's a clear economic cost, for no economic benefit (they do actually make money delivering that DDOS traffic). "most backbone routers build circuits at the /18 and above mask on MPLS" -
On Sat, 2 Feb 2008, Tomas L. Byrnes wrote: that part is seriously funny. However: a) Yes, if such proposal was to be widely accepted, it would generate more entries in RIB/FIB. b) However, if this service was actually operated by IX's, the limits to prevent "too much" growth could be applied centrally (max-prefixes per ASN, automatic removal of those routes after X days, unless manually requested by host, etc). c) Since only your peers will have those :666 entries, it is less "route growth" than than the alternative of announcing the affected block as /24 (which you seem to suggest).
A better approach would be to move your DDOS target and all the rest of its co-subnet hosts into a different /24, update the DNS RRs, and cease advertising that /24. That...is...perverted. Not to mention, you can't "cease advertising /24". what you would need to do is to deaggregate your (say) /20 into /21, /22, /23 and /24. That's 3 extra entries in FIB for everyone in the world to carry.
If you really want to be nice, they don't need to renumber, you just need to stop advertising the target subnet, change the DNS RR's and NAT at your borders, if you control DNS and IP. The added benefit of this is that you can swap them back when the DDOs is over, and they get to stay up while it's happening. All you need to do this is some spare, never to be allocated, IP space. That...is...perverted.
-alex [not speaking as mlc anything]
Hi, I am no lawyer, but I question whether ATT can patent anything that uses the existing technology and commands implemented in BGP. The idea that you can patent a persons intent in advertising a prefix in BGP seems somewhat bizarre. Surely a patent relates to the use of a new bit of technology that they have developed. Btw, I think I might be a good idea to do all sort of things, that doesn't mean I can file legally enforceable patents in case someone in the future shares my point of view. With regards to: "A better approach would be to move your DDOS target and all the rest of its co-subnet hosts into a different /24, update the DNS RRs, and cease advertising that /24." I just think you don't get it. Apart from the impracticality of renumbering all the other non-effected hosts in the same /24 and all the associated DNS records and the amount of time that is going to take. Plus then announce another /24 for the renumbered hosts. You are are saying that I should de-aggregate the /19 announcement into components so that I can stop advertising the original /24 - because your worried about route table prefix pollution!!!! If I blackhole a /32 to my transits, it does not appear in your routing table. If I blackhole a /32 to one of my peers and you are not even at the same IX, then there is no change of it appearing in your route table either. Conversely you seem to be suggesting I announce a bunch of prefixes to everyone? Kind Regards Ben -----Original Message----- From: Tomas L. Byrnes [mailto:tomb@byrneit.net] Sent: 03 February 2008 07:54 To: Ben Butler; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack. "Well then they wouldn't be peering with this route reflector " Well then, the utility is probably close to 0, isn't it? I doubt most of the sources of DDOS traffic, especially those without ingress source filtering, are going to peer with your route reflector. What's their economic incentive to expend the engineering time to do so? I sincerely doubt that any backbone provider will filter at a /32. That means they have to check EVERY PACKET AT FULL IP DEST against your AS advertised routes. Since most backbone routers build circuits at the /18 and above mask on MPLS, just to keep up with traffic, I sincerely doubt they are going to expend the CPU, and potentially RAM, never mind prefix table entries (you know, those things we're running out of) to have a full table of every host that every hoster says is being DDOSed. In this case, there's a clear economic cost, for no economic benefit (they do actually make money delivering that DDOS traffic). A better approach would be to move your DDOS target and all the rest of its co-subnet hosts into a different /24, update the DNS RRs, and cease advertising that /24. If you really want to be nice, they don't need to renumber, you just need to stop advertising the target subnet, change the DNS RR's and NAT at your borders, if you control DNS and IP. The added benefit of this is that you can swap them back when the DDOs is over, and they get to stay up while it's happening. All you need to do this is some spare, never to be allocated, IP space. Works with the existing infrastructure, doesn't require an "Internet God", and in general, is likely to be more effective in the real world. That whole "rough consensus and running code" ethos of the IETF thing, as opposed to the "Cathedral" mentality of the ITU (and ICANNt), which your approach would imply. You control which routes you advertise, after all. Plus, AT&T's legal beagles don't get to send you a dunning notice when their patent gets granted.
-----Original Message----- From: Ben Butler [mailto:ben.butler@c2internet.net] Sent: Saturday, February 02, 2008 2:42 PM To: Tomas L. Byrnes; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
"If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong."
Well then they wouldn't be peering with this route reflector in the first place.
-----Original Message----- From: Tomas L. Byrnes [mailto:tomb@byrneit.net] Sent: 02 February 2008 20:39 To: Ben Butler; Paul Vixie; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
You could achieve the exact same result simply by not advertising the network to your peers, or by advertising a bogus route (prefixing a known bogon AS for the addresses you want null-routed). I realize you would have to subnet/deaggregate your netblocks, and therefore could wind up with a prefix-length issue for peers who won't accept routes longer than a certain mask, in some cases, but if you are already being DDOSed, this should represent SOME improvement.
If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong.
The bigger issue with all these approaches is that they run afoul of a
patent applied for by AT&T:
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HI TOFF&p=1&u =%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=P G01&s1=200 60031575&OS=20060031575&RS=20060031575
USPTO App Number 20060031575
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Ben Butler Sent: Saturday, February 02, 2008 12:17 PM To: Paul Vixie; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
Hi,
I was not proposing he Null routing of the attack source in the other ISPs network but the destination in my network being Null routed as a destination from your network out.
This has no danger to the other network as it is my network that is going to be my IP space that is blackholed in your network, and the space blackholed is going to be an address that is being knocked of the air anyway under DoS and we are trying to minimise collateral damage. I cant see where the risk to the large NSP is - given that the route reflector will only reflect /32s that legitimately originate
(as a destination not a source) in the AS announcing them as please blackhole.
For complete clarity: AS13005 announces 213.170.128.0/19 and has its route object in RIPE as being announced by AS13005. My router at IX - BENIX say - announces 213.170.128.1/32 to the router
reflector their, the announcement from my IX assigned address 212.121.34.30 is known to be my router on the exchange, and I am announcing a /32 from my AS for a route object registered as being announced by my AS - so the reflector accepts my announcement and reflects it to any other members that choose to peer with this reflector - effectively this is a please blackhole this destination in AS13005 - the other members that receive this announcement can then deal with it in anyway they see fit from ignoring it to setting next-hop 192.0.2.1 -> Null0.
The effect of this would be that any BotNet controlled hosts in the other member network would now be able to drop any attack traffic in their network on destination at their customer aggregation routers.
I think you might have thought I was suggesting we blackhole sources in other peoples networks - this is definatly not what I was saying.
So, given we all now understand each other - why is no one doing the above?
At the end of the day if an IX member doesn't want the announcements don't peer with the blackhole reflector, simple, and it will get Null routed as soon as it hits my edge router at the IX - it would just seem more sensible to enable people to block the traffic before it traverses the IX and further back in their own networks.
So?????
Ben
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Paul Vixie Sent: 02 February 2008 17:32 To: nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack.
ben.butler@c2internet.net ("Ben Butler") writes:
... This hopefully will ensure a relatively protected router that is only accessible from the edge routers we want and also secured to only accept filtered announcements for black holing and in consequence enable the system to be trusted similar to Team Cymaru. ...
This sounds like another attempt to separate the Internet's control plane from its data plane, and most such attempts do succeed and are helpful (like NSP OOB, or like enterprise-level anycast of DNS). However, I'm not sure that remote triggered blackholes are a good direction, worthy of the protection you're proposing, for three reasons.
First, because large NSP's simply cannot afford the risk associated with letting a third party, automatically and without controls or audits, decide in real time what sources or destinations shall become unreachable. With all respect (which is a lot) for spamhaus and cymru
and even MAPS (which I had a hand in, back in the day), feeding BGP null-routes to a multinational backbone is a privilege that ISO9000 and SarBox and liability insurance providers don't usually want to extend.
Second, because many backbone routers in use today can't do policy routing routing (which is in this case dropping packets because their source address, not their destination address, has a particular community associated with it) at line speed. Note, this is many-not-all -- I'm perfectly aware that lots of backbone routers can do this but not everybody has them or can afford them and those who have them tend
to be the multinational NSPs discussed earlier. To prevent our DDoS protection reflexes from lowering an attacker's cost (by automatically blackholing victims to protect the nonvictims),
we have to be able to blackhole the abusive traffic by source, not by destination.
Third, because many OPNs (other people's networks) still don't filter on source address on their customer-facing edge, and thus allow spoofed-source traffic to exit toward "the core" or toward a victim's NSP who cannot filter by source due to path ambiguities inherent in "the core", any wide scale implementation of this, even if we could get trusted automation of it at scale and even if everybody had policy-routing-at-like-speed, would just push the attackers toward spoofed-source. That means a huge amount of work and money for the world, without changing the endgame for attackers and victims at all. (See BCP38 and SAC004 for prior rants on this controversial topic.)
IANAL, but my understanding of patents is that anyone can patent any innovative combination of existing things, if it produces a new effect or is used in a new way. According to the patent lawyers I've worked with, that is, in fact, the most common type of patent. As an example, Watt's steam engine combined a piston, a balance beam, a wheel, and a crank, all of which existed before, but his invention was patented. The ATT patent is on black-holing using BGP null routing to mitigate DDOS. As far as your proposal goes, perhaps I had misunderstood. My understanding was that you wanted everyone in the Internet to accept routes from an AS that was intended to be black-holed. Those routes would be hosts (/32) that were, in fact, placed in the route server by some trusted community (presumably hosters). The benefit of this would be to eliminate the collateral damage to other customers of the hosters caused by traffic targeting the DDOS target also denying service to them. If I have misunderstood, then I'm sorry, but my responses were based on the following assumptions: 1: The routes to be black-holed would all be /32s 2: The routes to be black-holed would have to be, in order to be effective, accepted by routers that carried the vast majority of the Internet's traffic, since you want to drop DDOS traffic as close to its source as possible. In my mind that would, at the very least include all the routers at major aggregation and peering points. 3: Backbone routers can't reasonably filter on a bunch of /32s and also forward traffic at wire speed. 4: It would be much harder to get all the ingress networks, which include all sorts of small local and regional ISPs, to join such a scheme than it would be to get larger ISPS to do so, assuming item 3 above is not true. 5: When one /32 is under DDOS, the rest of the hosts served by the same links are also effectively DOSed, ergo renumbering them out of the DOSed space, while painful, might be less painful than continuing to deal with the DOS. 6: You control what routes you advertise, and therefore can, effectively, make any prefix as short as or shorter than the shortest prefix accepted by your peers go dark, simply by stopping advertising it. 7: The increase in route prefixes caused by disaggregating would, assuming that there were multiple DDOS targets at any one time, not be materially larger than that caused by accepting a bunch of /32s, and may well be less. 8: Disaggregation can be done now, with the tools currently available, and requires no additional hardware, software, or legal agreements. It seems that you are saying that you're just asking your immediate peers to accept this private AS from you, to black-hole traffic to that AS locally, and not propagate the routes to their peers. That's completely different than what I thought you were talking about, which was some sort of Internet-wide black-hole AS that everyone was supposed to peer with. What you and your immediate peers do is between you and them, and I could see this as working quite well on that basis. All it's doing, however, is absorbing the DDOS one step upstream, which is probably a place that can do so more easily, and clearly is doing so any time the DDOS is getting through. In any event, as I pointed out, null routing via BGP to defend against DDOS is Patent Pending ATT, so unless you can get the patent not to issue, building a network based on it runs the risk of the patent troll. I've said my piece on this. Where I've made errors or misspoken, I'm happy to stand corrected. If I've offended anyone, I'm sorry. Best wishes to all. May your packets flow, your sessions connect, and your pagers remain silent.
-----Original Message----- From: Ben Butler [mailto:ben.butler@c2internet.net] Sent: Sunday, February 03, 2008 5:31 AM To: Tomas L. Byrnes; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
Hi,
I am no lawyer, but I question whether ATT can patent anything that uses the existing technology and commands implemented in BGP. The idea that you can patent a persons intent in advertising a prefix in BGP seems somewhat bizarre. Surely a patent relates to the use of a new bit of technology that they have developed.
Btw, I think I might be a good idea to do all sort of things, that doesn't mean I can file legally enforceable patents in case someone in the future shares my point of view.
With regards to:
"A better approach would be to move your DDOS target and all the rest of its co-subnet hosts into a different /24, update the DNS RRs, and cease advertising that /24."
I just think you don't get it. Apart from the impracticality of renumbering all the other non-effected hosts in the same /24 and all the associated DNS records and the amount of time that is going to take. Plus then announce another /24 for the renumbered hosts. You are are saying that I should de-aggregate the /19 announcement into components so that I can stop advertising the original /24 - because your worried about route table prefix pollution!!!!
If I blackhole a /32 to my transits, it does not appear in your routing table. If I blackhole a /32 to one of my peers and you are not even at the same IX, then there is no change of it appearing in your route table either. Conversely you seem to be suggesting I announce a bunch of prefixes to everyone?
Kind Regards
Ben
-----Original Message----- From: Tomas L. Byrnes [mailto:tomb@byrneit.net] Sent: 03 February 2008 07:54 To: Ben Butler; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
"Well then they wouldn't be peering with this route reflector "
Well then, the utility is probably close to 0, isn't it?
I doubt most of the sources of DDOS traffic, especially those without ingress source filtering, are going to peer with your route reflector. What's their economic incentive to expend the engineering time to do so?
I sincerely doubt that any backbone provider will filter at a /32. That means they have to check EVERY PACKET AT FULL IP DEST against your AS advertised routes. Since most backbone routers build circuits at the /18 and above mask on MPLS, just to keep up with traffic, I sincerely doubt they are going to expend the CPU, and potentially RAM, never mind prefix table entries (you know, those things we're running out of) to have a full table of every host that every hoster says is being DDOSed. In this case, there's a clear economic cost, for no economic benefit (they do actually make money delivering that DDOS traffic).
A better approach would be to move your DDOS target and all the rest of its co-subnet hosts into a different /24, update the DNS RRs, and cease advertising that /24.
If you really want to be nice, they don't need to renumber, you just need to stop advertising the target subnet, change the DNS RR's and NAT at your borders, if you control DNS and IP. The added benefit of this is that you can swap them back when the DDOs is over, and they get to stay up while it's happening. All you need to do this is some spare, never to be allocated, IP space.
Works with the existing infrastructure, doesn't require an "Internet God", and in general, is likely to be more effective in the real world.
That whole "rough consensus and running code" ethos of the IETF thing, as opposed to the "Cathedral" mentality of the ITU (and ICANNt), which your approach would imply.
You control which routes you advertise, after all.
Plus, AT&T's legal beagles don't get to send you a dunning notice when their patent gets granted.
-----Original Message----- From: Ben Butler [mailto:ben.butler@c2internet.net] Sent: Saturday, February 02, 2008 2:42 PM To: Tomas L. Byrnes; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
"If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong."
Well then they wouldn't be peering with this route reflector in the first place.
-----Original Message----- From: Tomas L. Byrnes [mailto:tomb@byrneit.net] Sent: 02 February 2008 20:39 To: Ben Butler; Paul Vixie; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
You could achieve the exact same result simply by not advertising the network to your peers, or by advertising a bogus route (prefixing a known bogon AS for the addresses you want null-routed). I realize you would have to subnet/deaggregate your netblocks, and therefore could wind up with a prefix-length issue for peers who won't accept routes longer than a certain mask, in some cases, but if you are already being DDOSed, this should represent SOME improvement.
If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong.
The bigger issue with all these approaches is that they run afoul of a
patent applied for by AT&T:
http://appft1.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HI TOFF&p=1&u =%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=P G01&s1=200 60031575&OS=20060031575&RS=20060031575
USPTO App Number 20060031575
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Ben Butler Sent: Saturday, February 02, 2008 12:17 PM To: Paul Vixie; nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack.
Hi,
I was not proposing he Null routing of the attack source in the other ISPs network but the destination in my network being Null routed as a destination from your network out.
This has no danger to the other network as it is my network that is going to be my IP space that is blackholed in your network, and the space blackholed is going to be an address that is being knocked of the air anyway under DoS and we are trying to minimise collateral damage. I cant see where the risk to the large NSP is - given that the route reflector will only reflect /32s that legitimately originate
(as a destination not a source) in the AS announcing them as please blackhole.
For complete clarity: AS13005 announces 213.170.128.0/19 and has its route object in RIPE as being announced by AS13005. My router at IX - BENIX say - announces 213.170.128.1/32 to the router
reflector their, the announcement from my IX assigned address 212.121.34.30 is known to be my router on the exchange, and I am announcing a /32 from my AS for a route object registered as being announced by my AS - so the reflector accepts my announcement and reflects it to any other members that choose to peer with this reflector - effectively this is a please blackhole this destination in AS13005 - the other members that receive this announcement can then deal with it in anyway they see fit from ignoring it to setting next-hop 192.0.2.1 -> Null0.
The effect of this would be that any BotNet controlled hosts in the other member network would now be able to drop any attack traffic in their network on destination at their customer aggregation routers.
I think you might have thought I was suggesting we blackhole sources in other peoples networks - this is definatly not what I was saying.
So, given we all now understand each other - why is no one doing the above?
At the end of the day if an IX member doesn't want the announcements don't peer with the blackhole reflector, simple, and it will get Null routed as soon as it hits my edge router at the IX - it would just seem more sensible to enable people to block the traffic before it traverses the IX and further back in their own networks.
So?????
Ben
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Paul Vixie Sent: 02 February 2008 17:32 To: nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack.
ben.butler@c2internet.net ("Ben Butler") writes:
... This hopefully will ensure a relatively protected router that is only accessible from the edge routers we want and also secured to only accept filtered announcements for black holing and in consequence enable the system to be trusted similar to Team Cymaru. ...
This sounds like another attempt to separate the Internet's control plane from its data plane, and most such attempts do succeed and are helpful (like NSP OOB, or like enterprise-level anycast of DNS). However, I'm not sure that remote triggered blackholes are a good direction, worthy of the protection you're proposing, for three reasons.
First, because large NSP's simply cannot afford the risk associated with letting a third party, automatically and without controls or audits, decide in real time what sources or destinations shall become unreachable. With all respect (which is a lot) for spamhaus and cymru
and even MAPS (which I had a hand in, back in the day), feeding BGP null-routes to a multinational backbone is a privilege that ISO9000 and SarBox and liability insurance providers don't usually want to extend.
Second, because many backbone routers in use today can't do policy routing routing (which is in this case dropping packets because their source address, not their destination address, has a particular community associated with it) at line speed. Note, this is many-not-all -- I'm perfectly aware that lots of backbone routers can do this but not everybody has them or can afford them and those who have them tend
to be the multinational NSPs discussed earlier. To prevent our DDoS protection reflexes from lowering an attacker's cost (by automatically blackholing victims to protect the nonvictims),
we have to be able to blackhole the abusive traffic by source, not by destination.
Third, because many OPNs (other people's networks) still don't filter on source address on their customer-facing edge, and thus allow spoofed-source traffic to exit toward "the core" or toward a victim's NSP who cannot filter by source due to path ambiguities inherent in "the core", any wide scale implementation of this, even if we could get trusted automation of it at scale and even if everybody had policy-routing-at-like-speed, would just push the attackers toward spoofed-source. That means a huge amount of work and money for the world, without changing the endgame for attackers and victims at all. (See BCP38 and SAC004 for prior rants on this controversial topic.)
On Feb 3, 2008 2:53 PM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
3: Backbone routers can't reasonably filter on a bunch of /32s and also forward traffic at wire speed.
yes they can. the size of the individual route doesn't matter to the devices in question, the NUMBER of routes does... (as does the associated change-rate of that number, but that's a story for another day)
4: It would be much harder to get all the ingress networks, which include all sorts of small local and regional ISPs, to join such a scheme than it would be to get larger ISPS to do so, assuming item 3 above is not true.
some already do this though... not in quite the manner Ben's aiming to do, but there are folks that accept BGP feeds in order to drop traffic inside there network(s).
5: When one /32 is under DDOS, the rest of the hosts served by the same links are also effectively DOSed, ergo renumbering them out of the DOSed space, while painful, might be less painful than continuing to deal with the DOS.
you have not had to deal with renumbering I presume? not a raft of end-users (consumers nevermind businesses). Why is the assumption that the surrounding space is a /24 relevant exactly? The aggregation scheme used inside any particular network isn't necessarily '/24 per pop/link/service-area'... renumbering for DDoS isn't really a workable solution, save the distinct case when you own the IP in question and services it provides (and other ancillary bits/bytes related to said ip/device/thingy).
8: Disaggregation can be done now, with the tools currently available, and requires no additional hardware, software, or legal agreements.
your point here is that perhaps instead of this scheme one would just advertise the max-prefix-length (/24 currently) from a 'better' place on your network and suck all the 'bad' traffic (all traffic in point of fact) for the attacked destination via a transit/peer/place which can deal with it properly? This isn't a bad solution, and it gives you some control on the traffic stream, it does have the penalty to everyone else of 'one more route in the RIB/FIB'... which I think was Ben's vote against this method. (also not a bad vote...) anyway, the idea behind multi-as blackholing has been (and apparently contunues to get) rehashed a few times over the last 5-8 years... good luck! -Chris
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
anyway, the idea behind multi-as blackholing has been (and apparently continues to get) rehashed a few times over the last 5-8 years... good luck!
It seems that way. People seem to forget about the conversations and work around 2000 - 2002. We not only had RTBH (static), multi AS RTBH, Source based RTBH (why uRPF Loose check was created), BGP Community based packet filtering (QPPB - source or destination), Backscatter Traceback (Chris and Brian's cool technique), Customer triggered RTBH (another Chris and Brian trick), BGP Shunts (originally created for the Great Firewall), MAPS's grow (which had multi-AS eBGP multihops BGP RTBHs back in 1997 for anti-SPAM filtering), and then all the BGP Flow-Spec work. We even have a RFC - 3882 Configuring BGP to Block Denial-of-Service Attacks. by D. Turk. published in September 2004. This is a good conversation for NANOG, but we really need to build up some FAQ so we don't keep going over the same things every year. Barry -----BEGIN PGP SIGNATURE----- Version: PGP 8.1 iQA/AwUBR6Ys/7/UEA/xivvmEQK3pwCg/a7329AxsnBgmPT9kmHoSWXhd1AAnA8d COSRO/CaIVnFOu0BIjbh5snD =HANY -----END PGP SIGNATURE-----
Hi Barry, Thank you for some really useful pointers, I am off to do some more reading. Kind Regards Ben -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Barry Greene (bgreene) Sent: 03 February 2008 21:07 To: Christopher Morrow; Tomas L. Byrnes Cc: nanog@merit.edu Subject: RE: Blackholes and IXs and Completing the Attack. -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
anyway, the idea behind multi-as blackholing has been (and apparently continues to get) rehashed a few times over the last 5-8 years... good
luck!
It seems that way. People seem to forget about the conversations and work around 2000 - 2002. We not only had RTBH (static), multi AS RTBH, Source based RTBH (why uRPF Loose check was created), BGP Community based packet filtering (QPPB - source or destination), Backscatter Traceback (Chris and Brian's cool technique), Customer triggered RTBH (another Chris and Brian trick), BGP Shunts (originally created for the Great Firewall), MAPS's grow (which had multi-AS eBGP multihops BGP RTBHs back in 1997 for anti-SPAM filtering), and then all the BGP Flow-Spec work. We even have a RFC - 3882 Configuring BGP to Block Denial-of-Service Attacks. by D. Turk. published in September 2004. This is a good conversation for NANOG, but we really need to build up some FAQ so we don't keep going over the same things every year. Barry -----BEGIN PGP SIGNATURE----- Version: PGP 8.1 iQA/AwUBR6Ys/7/UEA/xivvmEQK3pwCg/a7329AxsnBgmPT9kmHoSWXhd1AAnA8d COSRO/CaIVnFOu0BIjbh5snD =HANY -----END PGP SIGNATURE-----
Hi, <snip> "your point here is that perhaps instead of this scheme one would just advertise the max-prefix-length (/24 currently) from a 'better' place on your network and suck all the 'bad' traffic (all traffic in point of fact) for the attacked destination via a transit/peer/place which can deal with it properly? This isn't a bad solution, and it gives you some control on the traffic stream, it does have the penalty to everyone else of 'one more route in the RIB/FIB'... which I think was Ben's vote against this method. (also not a bad vote...)" </snip> Personally, I would achieve this using multiple sinkholes at the edge in IGP rather than advertising an extra /24 in BGP to suck it to one router. PA Deagregation as evidenced in some of my other posts is a pet hate of mine. PI space (and for that matter v6 PI please) is not - especially if we clean up the act on the PA front. Because, my research into anti DoS is still work in progress, I was going to sort out traffic mitigation through completing the attack first, then move on to investigating classification using sinks, and then worry about scrubbing / source filtering last. I just haven't got around at looking at sinks and scrubbing yet. I fully accept there is no single silver bullet for all situations and circumstances, but equally a tactic should be as effective as possible when it is selected and deployed - which started this thread. And I am trying to advocate being able to extend completing the attack beyond just transit feeds that is all. I don't know about other people our multiple Internet Exchange peak interconnect capacity versus our transit peak capacity is a significant %. While effectively securing my AS as a whole against the sources that reach me via transit, currently I cant do the same trick with XPs. Now the number of end host systems that I reach via peers is obviously a lot less than transit but the potential is still there as an unsecured ingress which could cause problems either through router/wan overload or interconnect congestion causing packet loss for other traffic. Either isn't good. In the absence of an alternative, it appears that in the scenario that I am under DoS, have blackholed a /32 to transit but my interconnect with an IX is saturated to the detriment of customer traffic - that the only thing I can sensibly do to resolve the situation is to temporally admin down / remove my prefix announcement from the IX peerings to shift the load to transit. This also doesn't seem very sensible. Kind Regards Ben -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Christopher Morrow Sent: 03 February 2008 20:56 To: Tomas L. Byrnes Cc: nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack. On Feb 3, 2008 2:53 PM, Tomas L. Byrnes <tomb@byrneit.net> wrote:
3: Backbone routers can't reasonably filter on a bunch of /32s and also forward traffic at wire speed.
yes they can. the size of the individual route doesn't matter to the devices in question, the NUMBER of routes does... (as does the associated change-rate of that number, but that's a story for another day)
4: It would be much harder to get all the ingress networks, which include all sorts of small local and regional ISPs, to join such a scheme than it would be to get larger ISPS to do so, assuming item 3 above is not true.
some already do this though... not in quite the manner Ben's aiming to do, but there are folks that accept BGP feeds in order to drop traffic inside there network(s).
5: When one /32 is under DDOS, the rest of the hosts served by the same links are also effectively DOSed, ergo renumbering them out of the DOSed space, while painful, might be less painful than continuing to deal with the DOS.
you have not had to deal with renumbering I presume? not a raft of end-users (consumers nevermind businesses). Why is the assumption that the surrounding space is a /24 relevant exactly? The aggregation scheme used inside any particular network isn't necessarily '/24 per pop/link/service-area'... renumbering for DDoS isn't really a workable solution, save the distinct case when you own the IP in question and services it provides (and other ancillary bits/bytes related to said ip/device/thingy).
8: Disaggregation can be done now, with the tools currently available,
and requires no additional hardware, software, or legal agreements.
your point here is that perhaps instead of this scheme one would just advertise the max-prefix-length (/24 currently) from a 'better' place on your network and suck all the 'bad' traffic (all traffic in point of fact) for the attacked destination via a transit/peer/place which can deal with it properly? This isn't a bad solution, and it gives you some control on the traffic stream, it does have the penalty to everyone else of 'one more route in the RIB/FIB'... which I think was Ben's vote against this method. (also not a bad vote...) anyway, the idea behind multi-as blackholing has been (and apparently contunues to get) rehashed a few times over the last 5-8 years... good luck! -Chris
On Feb 3, 2008 5:18 PM, Ben Butler <ben.butler@c2internet.net> wrote:
Hi,
<snip> "your point here is that perhaps instead of this scheme one would just advertise the max-prefix-length (/24 currently) from a 'better' place on your network and suck all the 'bad' traffic (all traffic in point of fact) for the attacked destination via a transit/peer/place which can deal with it properly?
This isn't a bad solution, and it gives you some control on the traffic stream, it does have the penalty to everyone else of 'one more route in the RIB/FIB'... which I think was Ben's vote against this method. (also not a bad vote...)" </snip>
Personally, I would achieve this using multiple sinkholes at the edge in IGP rather than advertising an extra /24 in BGP to suck it to one router.
Oops, I think I wasn't clear, my point was you could force traffic off of most peers and transits and onto a single transit by advertising the most specific global route possible (/24 today) through a single transit. This way you can force all of the world to find your attacked host through a place you choose, rather than 'everywhere'. Shifting around things in your IGP isn't going to help the rest-of-the-worlds view of your problem... My proposal would allow you to normalize traffic on your peer/transit links save one (or a smaller selection of them)... You could extend this to pulling the /24 down some sacrificial link (t1 sort of thing) as well, of course. You could also reverse the logic and either drop the route toward peers or extend the path via as-prepend...
I fully accept there is no single silver bullet for all situations and circumstances, but equally a tactic should be as effective as possible when it is selected and deployed - which started this thread. And I am trying to advocate being able to extend completing the attack beyond just transit feeds that is all.
Sure, and as I and Barry said, there have been several iterations of this discussion, not that that's a bad thing just a note that this is ground covered at least a few times.
I don't know about other people our multiple Internet Exchange peak interconnect capacity versus our transit peak capacity is a significant %. While effectively securing my AS as a whole against the sources that reach me via transit, currently I cant do the same trick with XPs. Now
sure you can, just don't have the traffic arrive there, draw it elsewhere, somewhere you are better prepared to deal with the problem... Something about fighting battles on your terms not theirs?
traffic - that the only thing I can sensibly do to resolve the situation is to temporally admin down / remove my prefix announcement from the IX peerings to shift the load to transit. This also doesn't seem very sensible.
I'd couch this in the following terms: "Don't be where the flood is, or deal with it where you are best equipped to..." There are many option, getting 'peer' folks to do BHR things for you isn't simple (most times they don't want you traffic engineering inside their network...), getting a transit to is another story, most times they have this facility it's just a matter of finding someone inside their support crew to get you the right bits/setup. Extra BGP sessions and unbounded /32 growth doesn't bode well for this plan either... anyway, it'll be interesting to watch the discussion progress. -Chris
If you're trying to do it on a /32 basis, I doubt you'd find too many border router operators interested in accepting a route that small, but I may be wrong.
I can think of at least one Global Tier One that offers a service that allows one to do exactly this (ie. advertise a /30 or /32 to them with a specific community) and blackhole it. -- Matthew Moyle-Croft - Internode/Agile - Networks Level 5, 150 Grenfell Street, Adelaide, SA 5000 Australia Email: mmc@internode.com.au Web: http://www.on.net Direct: +61-8-8228-2909 Mobile: +61-419-900-366 Reception: +61-8-8228-2999 Fax: +61-8-8235-6909 "The difficulty lies, not in the new ideas, but in escaping from the old ones" - John Maynard Keynes
On Feb 2, 2008, at 1:16 PM, Ben Butler wrote:
So, given we all now understand each other - why is no one doing the above?
Some folks are doing this, just not via some third-party route servers. For example, either via customer peering sessions, or other BGP interconnections between peers. E.g.: http://www.nanog.org/mtg-0402/morrow.html I'm not sure it's ideal to employ third-party route servers for this purpose, as it only increases the attacks/error surface. I suppose if folks rely on it for native peering then it might be reasonable.
At the end of the day if an IX member doesn't want the announcements don't peer with the blackhole reflector, simple, and it will get Null routed as soon as it hits my edge router at the IX - it would just seem more sensible to enable people to block the traffic before it traverses the IX and further back in their own networks.
Yes, helping to ease effects of collateral damage from large-scale attacks by conveying drop policies to upstream ASes for prefixes which you originate. And perhaps as significant, being able to allow the target AS to remove those policies dynamically, rather than having to contact each upstream AS that appears to have imposed them manually out-of-band. I think Paul's comments were more regarding the fact that destination-based blackhole routing for mitigation *effectively completes the attack*, which is often times undesirable. Inter-domain source-based blackhole routing is pretty much a non-option. One other offshoot is that advertised more-specifics are going to further contribute to routing AND forwarding table bloat, and a single new prefixes might result in 10+ new paths in the iBGP RIBs. If you do implement something like this you may wish to scope advertisement only to adjacent ASes via NO_EXPORT or the like, to scope both more-specific propagation, and to ensure that some lack of consistent drop community semantic interpretation doesn't hose something. Also, if you impose this as a standard attack response mechanism recall that you lose visibility of attack scales, and knowing just when to resume normal forwarding policies is a bit more complex. As such, your policy sets may want to provide hooks that enable selective prefix advertise/withdraw drop policies so that it can be applied or removed incrementally. -danny
"destination-based blackhole routing for mitigation *effectively completes the attack*, which is often times undesirable. Inter-domain source-based blackhole routing is pretty much a non-option." That is why I put Completing the Attack in my subject line - and didnt attempt to sujest this as an approach for source based filtering. "If you do implement something like this you may wish to scope advertisement only to adjacent ASes via NO_EXPORT or the like, to scope both more-specific propagation, and to ensure that some lack of consistent drop community semantic interpretation doesn't hose something." It is upto the receiving AS what they do with the announcement and whether they advertise it to their BGP customers or not - they shouldnt be announcing to anyone else anyway. I would sugest they dont, but I wouldnt presume. Advertisments to transits do not get propigated outside their AS. "I suppose if folks rely on it for native peering then it might be reasonable." This is envisage to be used between members of the same Internet Exchange only. Where there are so many peers the overhead of trying to do this on a peer peer basis / agreeing comunities is going to be a pain in the arse. And that this gives an easier way to get to the goal - we do after all trust the IX operator to run the critical bit of infrastrucutre which is the exchange. Kind Regards Ben -----Original Message----- From: Danny McPherson [mailto:danny@tcb.net] Sent: 02 February 2008 20:49 To: Ben Butler Cc: NANOG NANOG Subject: Re: Blackholes and IXs and Completing the Attack. On Feb 2, 2008, at 1:16 PM, Ben Butler wrote:
So, given we all now understand each other - why is no one doing the above?
Some folks are doing this, just not via some third-party route servers. For example, either via customer peering sessions, or other BGP interconnections between peers. E.g.: http://www.nanog.org/mtg-0402/morrow.html I'm not sure it's ideal to employ third-party route servers for this purpose, as it only increases the attacks/error surface. I suppose if folks rely on it for native peering then it might be reasonable.
At the end of the day if an IX member doesn't want the announcements don't peer with the blackhole reflector, simple, and it will get Null routed as soon as it hits my edge router at the IX - it would just seem more sensible to enable people to block the traffic before it traverses the IX and further back in their own networks.
Yes, helping to ease effects of collateral damage from large-scale attacks by conveying drop policies to upstream ASes for prefixes which you originate. And perhaps as significant, being able to allow the target AS to remove those policies dynamically, rather than having to contact each upstream AS that appears to have imposed them manually out-of-band. I think Paul's comments were more regarding the fact that destination-based blackhole routing for mitigation *effectively completes the attack*, which is often times undesirable. Inter-domain source-based blackhole routing is pretty much a non-option. One other offshoot is that advertised more-specifics are going to further contribute to routing AND forwarding table bloat, and a single new prefixes might result in 10+ new paths in the iBGP RIBs. If you do implement something like this you may wish to scope advertisement only to adjacent ASes via NO_EXPORT or the like, to scope both more-specific propagation, and to ensure that some lack of consistent drop community semantic interpretation doesn't hose something. Also, if you impose this as a standard attack response mechanism recall that you lose visibility of attack scales, and knowing just when to resume normal forwarding policies is a bit more complex. As such, your policy sets may want to provide hooks that enable selective prefix advertise/withdraw drop policies so that it can be applied or removed incrementally. -danny
I was not proposing he Null routing of the attack source in the other ISPs network but the destination in my network being Null routed as a destination from your network out.
i explained why this is bad -- it lowers the attacker's costs in what amounts to an economics war. they can get a web site taken down by its own provider just by attacking it. they need fewer resources for their attack once they know the provider's going to blackhole the victim.
This has no danger to the other network as it is my network that is going to be my IP space that is blackholed in your network, and the space blackholed is going to be an address that is being knocked of the air anyway under DoS and we are trying to minimise collateral damage.
your collateral damage is of precious little interest to someone else's backbone staff, unless they can route-filter the potential announcements so that you are unable to also remotely blackhole addresses you don't advertise. i explained this as an insurance/ISO9000 problem.
I think you might have thought I was suggesting we blackhole sources in other peoples networks - this is definatly not what I was saying.
i explained why this would be a more sensible approach, but STILL unworkable.
So, given we all now understand each other - why is no one doing the above?
now that we've rehashed what we both said, i think we're done here.
Hi, "i explained why this is bad -- it lowers the attacker's costs in what amounts to an economics war. they can get a web site taken down by its own provider just by attacking it. they need fewer resources for their attack once they know the provider's going to blackhole the victim." I thought the cold war nuclear arms race had shown up to be truly MAD. Who is paying for this ever escalating capacity of infrastructure as a way to survive large DoS attacks. Smaller attacks can be absorbed, but I really cant see a strategy of endlessly upgrading network router and WAN infrastructure to ensure enough head room ideal capacity is a particularly economically sensible approach to the problem. Ben -----Original Message----- From: vixie@vix.com [mailto:vixie@vix.com] On Behalf Of Paul Vixie Sent: 02 February 2008 21:37 To: Ben Butler Cc: nanog@merit.edu Subject: Re: Blackholes and IXs and Completing the Attack.
I was not proposing he Null routing of the attack source in the other ISPs network but the destination in my network being Null routed as a destination from your network out.
i explained why this is bad -- it lowers the attacker's costs in what amounts to an economics war. they can get a web site taken down by its own provider just by attacking it. they need fewer resources for their attack once they know the provider's going to blackhole the victim.
This has no danger to the other network as it is my network that is going to be my IP space that is blackholed in your network, and the space blackholed is going to be an address that is being knocked of the air anyway under DoS and we are trying to minimise collateral damage.
your collateral damage is of precious little interest to someone else's backbone staff, unless they can route-filter the potential announcements so that you are unable to also remotely blackhole addresses you don't advertise. i explained this as an insurance/ISO9000 problem.
I think you might have thought I was suggesting we blackhole sources in other peoples networks - this is definatly not what I was saying.
i explained why this would be a more sensible approach, but STILL unworkable.
So, given we all now understand each other - why is no one doing the above?
now that we've rehashed what we both said, i think we're done here.
participants (10)
-
Alex Pilosov -
Barry Greene (bgreene) -
Ben Butler -
Christopher Morrow -
Danny McPherson -
Matthew Moyle-Croft -
Paul Vixie -
Paul Vixie -
Rick Astley -
Tomas L. Byrnes