Unwanted Traffic Removal Service (UTRS)
Friends and colleagues, Yesterday I briefly discussed a new project we've recently launched and for which invited participation from the NANOG 62 attendees. This is a not so subtle wider request for consideration. UTRS is essentially a community RTBH that people have suggested to us would be a good service to provide, so we're giving it a go. Yesterday's lightning talk page has the slide deck I presented: <https://www.nanog.org/meetings/abstract?id=2436> I've also put some additional technical detail here: <http://www.cymru.com/jtk/misc/utrs.html> If you have an assigned ASN and this sounds like something you or your network would like to take advantage of, drop me a line, I'd sure like to hear about it. If you think this is a terrible idea and want to express all that is wrong with it, tell me that too, I can take it. Even if you're unsure if you would ultimately deploy it in production, but are interested in testing it, do contact me. We won't expend energy on this project and will drop it if we there isn't sufficient interest from the community, but enough trial users would be encouraging. Kindly, John
Dear John, On Wed, Oct 08, 2014 at 08:59:00AM -0500, John Kristoff wrote:
UTRS is essentially a community RTBH that people have suggested to us would be a good service to provide, so we're giving it a go.
FYI, there are various projects which are similar to this concept: http://www.de-cix.net/products-services/de-cix-frankfurt/blackholing/ https://ripe68.ripe.net/presentations/369-bgp_bh_ripe.pdf https://wiki.rtbh.me/
If you think this is a terrible idea and want to express all that is wrong with it, tell me that too, I can take it.
Just like chicory, personally I don't like it. Yes, Cymru has build a reputation as clearing house for redistribution of security related information. But... (aside from any local safety net filter), it's quite a leap to allow a single entity to inject blackholes for any prefix. There are various flavors at the moment in terms of validation (please correct me if I am wrong): The Polish blackholing project only allows blackholes which fall within the set of prefixes which an ASN originates, the DE-CIX BS service accepts anything that is a subset of your AS-SET. Both approaches have their downsides: you can make any AS or AS-SET a member of your AS-SET and thereby gain a degree of control on the RTBH server, and for $500/year you can register any route-object you want in RADB. RIPE is the only RIR which has the IRR service as a truely integral part of the database, allowing advanced automatable authentication schemes for purposes such as these. However, they only administrate for a subset of the Internet, making this direction inpractical for a universal solution. Might I suggest an alternative approach, without central validation or need for a clearing house: IXPs could offer BGP or API triggered ACLs which are inserted into the peering fabric and only affect the participant's peering port(s). This way, any blackholing (either correctly applied or malicious) only affects the initator of that blackhole and nobody else. Advantages are that aclserver does not require peers to cooperate with each other and no validation is required. Kind regards, Job
On Wed, 8 Oct 2014 16:42:38 +0200 Job Snijders <job@instituut.net> wrote:
Just like chicory, personally I don't like it. Yes, Cymru has build a reputation as clearing house for redistribution of security related information. But... (aside from any local safety net filter), it's quite a leap to allow a single entity to inject blackholes for any prefix.
Hi Job, Thanks for your comments. I'm aware of some other projects, including another one, much more elaborate, talked about in another session at NANOG this week. Do note, UTRS does not allow a single entity to inject black holes for any prefix, only a limited number of /32's for their own prefixes. The presentation and the information page I linked to have some additional details.
IXPs could offer BGP or API triggered ACLs which are inserted into the peering fabric and only affect the participant's peering port(s). This way, any blackholing (either correctly applied or malicious) only affects the initator of that blackhole and nobody else. Advantages are that aclserver does not require peers to cooperate with each other and no validation is required.
I've heard of some IXPs recently offering this service, sounds great. It has also been suggested we might talk to ISPs how to RTBH to their customers and see if there was a way for those routes to be passed further along, perhaps to something like UTRS for further dissemination. I'm not sure that would work, but it was an interesting idea too. Thanks for your comments, John
On Wed, Oct 8, 2014 at 10:42 AM, Job Snijders <job@instituut.net> wrote:
Just like chicory, personally I don't like it. Yes, Cymru has build a reputation as clearing house for redistribution of security related information. But... (aside from any local safety net filter), it's quite a leap to allow a single entity to inject blackholes for any prefix.
Hi Job, We already do that on the /24 level... it's called "BGP" and we all have fun when a service provider in one of the 'Stans injects a route overriding Youtube. How could John's system do worse? Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/> May I solve your unusual networking challenges?
information. But... (aside from any local safety net filter), it's quite a leap to allow a single entity to inject blackholes for any prefix.
Spamhaus has been distributing their DROP list by BGP for years. The world hasn't ended, and I can't recall the last time it had a plausible false positive. http://www.spamhaus.org/drop/ R's, John
On Wed, Oct 08, 2014 at 04:02:21PM -0000, John Levine wrote:
information. But... (aside from any local safety net filter), it's quite a leap to allow a single entity to inject blackholes for any prefix.
Spamhaus has been distributing their DROP list by BGP for years. The world hasn't ended, and I can't recall the last time it had a plausible false positive.
The Spamhaus context is different: I know people take that DROP BGP feed and converting it into ACLs applicable locally on their mailservers, I don't know of people injecting these prefixes straight into the core. Kind regards, Job
On Wed, Oct 08, 2014 at 04:42:38PM +0200, Job Snijders wrote:
There are various flavors at the moment in terms of validation (please correct me if I am wrong): The Polish blackholing project only allows blackholes which fall within the set of prefixes which an ASN originates, the DE-CIX BS service accepts anything that is a subset of your AS-SET.
There is also "dynamic validation" approach: blackhole route is considered valid for injection if and only if there is a covering less-specific route with the best-path pointing to the same exit point as blackhole route. (definition of "exit point" can vary from "next ASn is the same we received blackhole from" to "both as-path and next-hops must be the same and aggregate route must be marked as customer's one"). This approach has its downside too: it requires you to run task-specific bgp speaker. Worse yet, usually you have to write that speaker :) -- In theory, there is no difference between theory and practice. But, in practice, there is.
Hello, Am 08.10.2014 um 16:42 schrieb Job Snijders:
If you think this is a terrible idea and want to express all that is wrong with it, tell me that too, I can take it.
I support the RTBH idea. I also set up https://wiki.rtbh.me/ for this some time ago and there is also a discuss mailing list where already 65 people are subscribed. Currently there is not much discussion on the list, but you all can change this ;-) Also there is no other content besides the mailing list in the wiki right now. I have removed it because somebody thought that it may be confusing for some people on the list as he wanted to use it for a general RTBH discussion instead of my project. I will try to restore it in the next days...
Just like chicory, personally I don't like it. Yes, Cymru has build a reputation as clearing house for redistribution of security related information. But... (aside from any local safety net filter), it's quite a leap to allow a single entity to inject blackholes for any prefix.
What I do not like at this UTRS idea is that I cannot announce a prefix via BGP. Somebody has to inject it for me. I would like to announce it in real time and not with delay because of manual approval. One problem that I also see here is that this single entity could be forced by someone (eg. government) to blackhole some prefix. If this ever happens such a project will have to be terminated.
There are various flavors at the moment in terms of validation (please correct me if I am wrong): The Polish blackholing project only allows blackholes which fall within the set of prefixes which an ASN originates, the DE-CIX BS service accepts anything that is a subset of your AS-SET.
Both approaches have their downsides: you can make any AS or AS-SET a member of your AS-SET and thereby gain a degree of control on the RTBH server, and for $500/year you can register any route-object you want in RADB.
Allowing ASN to blackhole a prefix based on AS sets is dangerous from my point of view. In the RIPE database you can add any AS to your AS set without verification. Ok, it doesn't make much difference because most IP transit providers also filter on the AS set, but a worldwide announced /24 prefix is much more visible than a /32 blackhole route that is only announced to the participants. My idea was that an ASN can only announce more specifics prefixes (not just /32) from officially registered routes. Downstream ASN should also be able to define their upstream ASN who may blackhole their prefixes if they do not set up their own session to the RTBH route servers. Most IP transit providers have an RTBH service for their customers that these downstream ASN could use. Usually they do not offer such a service for peers. This is the useful part here. We also had some DDoS attacks via IPv6. I think it's important to also have such a service for IPv6. Starting with IPv4 is ok and better than nothing, but IPv6 should not be on the roadmap for 2018 ;-)
Might I suggest an alternative approach, without central validation or need for a clearing house:
IXPs could offer BGP or API triggered ACLs which are inserted into the peering fabric and only affect the participant's peering port(s). This way, any blackholing (either correctly applied or malicious) only affects the initator of that blackhole and nobody else. Advantages are that aclserver does not require peers to cooperate with each other and no validation is required.
Why is there no validation required when this is done by an IXP? "All peers are my customers and I do trust them"? What about private peerings via PNIs? Regards, Chris -- Individual Network Berlin e.V. : support@in-berlin.de : vorstand@in-berlin.de Tel +49-30-45494343 ::: Fax +49-30-45494344 ::: Web https://www.in-berlin.de/ IN-Berlin e.V. : Christian Seitz (1. Vors.) : Lehrter Str. 53 :: 10557 Berlin Amtsgericht Charlottenburg 95 - VR 15669 Nz ::::::: USt.Ident-Nr. DE188894648
Hi Christian, On Thu, Oct 09, 2014 at 10:58:05PM +0200, Christian Seitz wrote:
<snip>
Why is there no validation required when this is done by an IXP? "All peers are my customers and I do trust them"? What about private peerings via PNIs?
Validation is not required because the requester can only affect his or her own peering ports. Conceptually the IXP participant sets an ACL, just not on their own ingress routerport but on the egress port just across the crossconnect, this prevents congestion of said crossconnect. Modern switching/routing equipment used in IXPs can do far more then mere L2 switching. Lots of chipsets these days allow you to apply combined layer-3 + layer-2 ACLs, an example would be "Discard traffic if (destination IP is A && destination MAC is B)". The blackhole route is announced to a special network component which I dub the "ACL Server". The ACL Server is operated by the IXP (exabgp + customizations?). The ACL Server takes the prefix and associated origin (origin is a static lookup table based on source IP or ASN, IXPs know who their members are) and then inserts a layer3+layer2 ACL into their switching fabric. If the IXP, on every ingress port, have a ACL that says "drop traffic to this IP if the dest MAC address corresponds with the originator of the ACL request", the ddos traffic doesn't even hit the IXPs backbone, and only affects the peering participant who requested the ACL to be inserted. So the blackhole route only lives inside the IXP's switching fabric so to speak, as an ACL only applicable to the participant itself. Regarding implementation: some IXPs might have to screenscrape/expect to apply ACLs on their switches with clogin, others will just convert the announcement and insert flowspec or sdn rules in the fabric. It is the IXP's job to sanitize the ACL trigger in such a way that it only applies to the peering ports of the participant requesting it. I hope this clarifies a bit. Kind regards, Job
On Thu, 09 Oct 2014 22:58:05 +0200 Christian Seitz <chris@in-berlin.de> wrote:
What I do not like at this UTRS idea is that I cannot announce a prefix via BGP. Somebody has to inject it for me. I would like to announce it in real time and not with delay because of manual approval.
While true today, it might not be true for long. It requires code to be written in order to perform the desired verification we want before blindly passing along an announcement. Code we're not motivated to write if there is insufficient interest in UTRS. Interest is looking good, so the code may soon follow. In other words, this a valid complaint, but it may have a limited life span.
One problem that I also see here is that this single entity could be forced by someone (eg. government) to blackhole some prefix. If this ever happens such a project will have to be terminated.
I've heard this once before too. I admit we probably can't provide a satisfactory answer to some who will be so distrustful of government or influence peddling to win them over, but I'll try to offer a response that I hope is fairly reasonable and satisfies the majority, and presumably any of the actual participants. There are legal questions, maneuvers and responses that might be interesting to speculate on, but I'll say simply this. Team Cymru, while established and operated within the U.S., is a global organization with team members outside of the U.S. and we rely heavily on the cooperation of global partners to do what we do. If we could be compelled to announce a black hole by someone, government or otherwise, the cooperation and inherent trust we might have with the Internet community is probably gone and we are likely finished as an organization. It would be counter to our very existence and so on that basis I hope most would agree is extremely unlikely to occur. Now if someone came up to me with a gun to my head and said type the equivalent of "ip route foonet mask 192.0.2.1" or die, I might just type it out of self preservation.
We also had some DDoS attacks via IPv6. I think it's important to also have such a service for IPv6. Starting with IPv4 is ok and better than nothing, but IPv6 should not be on the roadmap for 2018 ;-)
You are only the second person I've heard from to explicitly state as such. This is actually not terribly hard to do and I'm pretty certain could be done way before 2018. Simple to start, careful and necessary improvements as we go. Thanks for your comments Chris, John
I understand the concerns but it seems to me that there are already plenty of ways for any large government to black hole whatever they want and they do not need UTRS to do so. The only thing stopping (most) governments from doing this regularly are fears of turning the Internet into another arms race. It's a stigma thing like the different between launching the first nuke vs. being the responder. We all know they do a lot of cyber stuff out there but it is mostly behind a veil of deniability. First, if they have access to a tier 1 carrier (or at least enough carriers to make an impact) in their jurisdiction they could just order that carrier to do it with whatever court system (or not) is required. Most large governments also have enough connectivity to bury a route by brute force. The only thing stopping (most) governments from doing this regularly are fears of turning the Internet into another arms race and possibly losing easy access to that resource. We all know they do a lot of cyber crime stuff out there but it is mostly behind a veil of deniability. There has actually been more black hole events that occur by accident or as part of denial of service attacks than government launched. The global routing structure of the Internet has always been highly cooperative and vulnerable to a bad actor at a lot of points. My only real concern with UTRS is designing a system that cannot be gamed or exploited to turn it into a very effective DoS weapon system. I admit that I don't know enough about how it works to make that decision yet. Steven Naslund Chicago IL
Subject: Re: Unwanted Traffic Removal Service (UTRS)
On Thu, 09 Oct 2014 22:58:05 +0200 Christian Seitz <chris@in-berlin.de> wrote:
What I do not like at this UTRS idea is that I cannot announce a prefix via BGP. Somebody has to inject it for me. I would like to announce it in real time and not with delay because of manual approval.
While true today, it might not be true for long. It requires code to be written in order to perform the desired verification we want before blindly passing along an announcement. Code we're not motivated to write if there is >insufficient interest in UTRS. Interest is looking good, so the code may soon follow. In other words, this a valid complaint, but it may have a limited life span.
One problem that I also see here is that this single entity could be forced by someone (eg. government) to blackhole some prefix. If this ever happens such a project will have to be terminated.
I've heard this once before too. I admit we probably can't provide a satisfactory answer to some who will be so distrustful of government or influence peddling to win them over, but I'll try to offer a response that I hope is >fairly reasonable and satisfies the majority, and presumably any of the actual participants.
There are legal questions, maneuvers and responses that might be interesting to speculate on, but I'll say simply this. Team Cymru, while established and operated within the U.S., is a global organization with team members outside >of the U.S. and we rely heavily on the cooperation of global partners to do what we do. If we could be compelled to announce a black hole by someone, government or otherwise, the cooperation and inherent trust we might have with >the Internet community is probably gone and we are likely finished as an organization. It would be counter to our very existence and so on that basis I hope most would agree is extremely unlikely to occur. Now if someone came up to >me with a gun to my head and said type the equivalent of "ip route foonet mask 192.0.2.1" or die, I might just type it out of self preservation.
We also had some DDoS attacks via IPv6. I think it's important to also have such a service for IPv6. Starting with IPv4 is ok and better than nothing, but IPv6 should not be on the roadmap for 2018 ;-)
You are only the second person I've heard from to explicitly state as such. This is actually not terribly hard to do and I'm pretty certain could be done way before 2018. Simple to start, careful and necessary improvements as we >go.
Thanks for your comments Chris,
John
At 22:58 09/10/2014 +0200, Christian Seitz wrote:
Allowing ASN to blackhole a prefix based on AS sets is dangerous from my point of view. In the RIPE database you can add any AS to your AS set without verification. Ok, it doesn't make much difference because most IP transit providers also filter on the AS set, but a worldwide announced /24 prefix is much more visible than a /32 blackhole route that is only announced to the participants.
See: http://www.ripe.net/ripe/mail/archives/routing-wg/2014-June/002696.html -Hank
On Wed, Oct 8, 2014 at 9:59 AM, John Kristoff <jtk@cymru.com> wrote:
If you think this is a terrible idea and want to express all that is wrong with it, tell me that too, I can take it.
Hi John, It's a good idea, I think, but it has a problem: it's an escalation in an arms race that doesn't end well for the blue team. If we ever get good at keeping traffic to a single IP far enough away to not cripple us, the attacker need only spray the /24. Or spray our entire address space, easily identifiable from our BGP announcement. All this effort on our end and it took the attacker 15 minutes to modify his code. Two general types of DDOS traffic: botnets and forged source addresses. For the botnets, lots of real machines, each with a legitimate source IP address, we need to get to a router interface as close to each source address as we can get. Then temporarily shut down traffic from that source address crossing that link until the data flow suggests the problem traffic has ceased. Even if we have to pay the ISP who owns that link to do it for us. Quickly find it with automation. Quickly authenticate the attack flow. Quickly pay for remediation. For the address forgers, we need some kind of public detection system where ISPs who care provide the trace tools that let us figure out where the rogue attacking our network is _actually_ coming from. After which we can pay the ISP to interdict any traffic destined for anywhere in our network which enters from that link. Quickly with automation We can't win the arms race based on the destination; we'll only win it if we find a way to zero in on and interdict the source. Regards, Bill Herrin P.S. Also worth noting that paying a DDOS mitigation service can already accomplish the best-case result from something like UTRS. The mitigator announces the affected /24, sinks the attacked IP address and tunnels the rest of the packets back to us. Expensive but easy peasy. -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/> May I solve your unusual networking challenges?
participants (8)
-
Alexandre Snarskii
-
Christian Seitz
-
Hank Nussbacher
-
Job Snijders
-
John Kristoff
-
John Levine
-
Naslund, Steve
-
William Herrin