Re: BCP38 making it work, solving problems
It's better than a sharp stick in the eye, I'll tell ya, lad. Listen to me: It's called a "best current practice" for a reason -- people should do it. Not sit and around and endlessly discuss it (we've already done that a thousand times). I wrote it, I stand beside it. I'm sick of hearing why people haven't implemented it yet -- it's almost five years later and there's simply no excuse. It's sickening. - fergie Ref: ftp://ftp.rfc-editor.org/in-notes/bcp/bcp38.txt -- Sean Donelan <sean@donelan.com> wrote: But BCP38 doesn't immediately help the ISP. -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg@netzero.net or fergdawg@sbcglobal.net
the problem is that isp security folk doing actual measurement see very little spoofing. it's easy for the bad folk to get real bots. and tcp bad things are more popular and desirable, e.g. spam, ... and tcp does not work from spoofed addresses. isp security folk have limited resources. so why should they spend them where there is little perceived benefit when there are gaping holes elsewhere that need those resources? yes, it's a nice thing to do. but, in today's disasterous economy, so are 42 other things that won't get done. so it's growing slowly. when it solves critical problems, it'll grow more quickly. randy
RB> Date: Sun, 10 Oct 2004 20:14:01 -0700 RB> From: Randy Bush RB> when it solves critical problems, it'll grow more quickly. Maybe. * Use 25/TCP for SMTP and 587/TCP for submission * Block outbound SMTP by default, but allow for the clueful * Run SMTP authentication * Let each authenticated user have whitelisted sender addresses that they can use * Limit whitelist size * Add a delay and/or rate limit to whitelist additions. Not perfect, and certainly subject to social engineering and possible automated attack on the whitelist mechanism, but it should decrease the number of cable/DSL pipes filled with forged mail transmissions. This isn't the first time I've suggested it, and I'm sure others have, too. Not once have I received a response to the extent of "I'd love to implement this if it existed". People are worried about OPNs, not their own networks. IMNSHO, worrying about N-1 ASNs scales far more poorly than worrying about one ASN. Of course, note the parallel to BCP38; I'm sure someone will be quick to point out that, unless adopted universally, forged mail will still exist. Enter SPF as a bandaid on the receiving side, and rehash that discussion. Combine with BMF, DNSBLs, and one is well on the way to much cleaner mail. Has anyone on NANOG ever solved a jigsaw puzzle with 500+ pieces? Or are "age 2 to 4" puzzles too difficult, as even they tend to have around ten pieces per puzzle? Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _________________________________________________________________ DO NOT send mail to the following addresses: davidc@brics.com -*- jfconmaapaq@intc.net -*- sam@everquick.net Sending mail to spambait addresses is a great way to get blocked.
On Mon, 11 Oct 2004, Fergie (Paul Ferguson) wrote:
I wrote it, I stand beside it. I'm sick of hearing why people haven't implemented it yet -- it's almost five years later and there's simply no excuse. It's sickening.
it's cheaper to ignore bcp38 than to implement it. operators are reactive to abuse, not proactive. though this is slowly changing as abuse becomes a significant % of network traffic. -Dan
On Mon, Oct 11, 2004 at 02:58:59AM +0000, Fergie (Paul Ferguson) wrote:
It's better than a sharp stick in the eye, I'll tell ya, lad.
Listen to me: It's called a "best current practice" for a reason -- people should do it. Not sit and around and endlessly discuss it (we've already done that a thousand times).
I wrote it, I stand beside it. I'm sick of hearing why people haven't implemented it yet -- it's almost five years later and there's simply no excuse. It's sickening.
Tell it to the vendors of the layer 3 customer attach devices. I was just speaking about this with a major "up and coming layer 2/3 switch vendor" the other day, who had no implementation or real plans to implement uRPF in the immediate future. It apears that there are no enterprise customers asking for this feature (not a shock), despite the fact that they are probably a leading generator of hacked machines throwing bad packets down reasonably big pipes. uRPF support is hardly universal outside of Cisco and Juniper, and even Juniper didn't add uRPF until semi-recently. I know Foundry still doesn't have support for it (not really anyways, they added something they call uRPF and have you activate through "ip verify unicast reverse-path" but it isn't really uRPF, just a way to prevent outside networks from spoofing local addresses, and you have to manually tag every interface as internal or external or the addresses aren't protected), and I'm pretty sure Extreme doesn't have it (or at least I've never seen anyone use it, but I don't follow 0 day Extreme code). Short of the Cisco L3 switch solutions, I'm not aware of any other vendor with a L3 switch who has uRPF functionality. I can't think of a single web hoster (or at least someone who wasn't a real network operator who got into hosting somehow) who even knows what uRPF is, let alone implements it for their customers. We're counting on their IP providers to filter the bad packets from the hundreds of FE connected servers on burstable GigE pipes out there, but some of them just aren't. I can name several major well-known "cheap" networks who don't do any uRPF filtering of their customers, and a couple more who don't do any real prefix-lists on their BGP speaking customers, only prefix-limits. There are even fewer vendors who are implementing automation for filtering basic BGP speaking customers, such as Juniper's ability (sort-of) to re-use a route filtering prefix-list for source address filtering in a firewall. It's not quite perfect, since you can't do length modifiers (upto /24, etc) in a prefix-list, and since you have to create a seperate policy-statement (with all routes listed in route-filter statements), prefix-list, AND firewall filter for each and every customer, but at least it is a start. If you don't have this, or uRPF at all, you are left trying to script ACLs based on interface configurations, static routes, and/or registered prefixes, and the bottom line is that most networks aren't going to do it if it takes that much work. If and when all the vendors who are making boxes that are supposed to be used for customer aggregation start implementing uRPF, especially considering all the enterprises, universities, and international network using these L3 switches as their sole routing equipment (think price), maybe we'll start seeing some more progress. And of course, those remaining service providers with the gear to implement it who havn't done so need to be yelled at more. Unfortunately, no customer ever complains (or takes their business elsewhere) when their provider doesn't do source address filtering, only when they are getting a DoS attack and their provider can't do anything about it. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
At 05:41 PM 10/11/2004, Richard A Steenbergen wrote:
On Mon, Oct 11, 2004 at 02:58:59AM +0000, Fergie (Paul Ferguson) wrote:
It's better than a sharp stick in the eye, I'll tell ya, lad.
Listen to me: It's called a "best current practice" for a reason -- people should do it. Not sit and around and endlessly discuss it (we've already done that a thousand times).
I wrote it, I stand beside it. I'm sick of hearing why people haven't implemented it yet -- it's almost five years later and there's simply no excuse. It's sickening.
Tell it to the vendors of the layer 3 customer attach devices. I was just speaking about this with a major "up and coming layer 2/3 switch vendor" the other day, who had no implementation or real plans to implement uRPF in the immediate future. It apears that there are no enterprise customers asking for this feature (not a shock), despite the fact that they are probably a leading generator of hacked machines throwing bad packets down reasonably big pipes.
I've removed the rest of your message, talking about which vendors do or don't have what capabilities. While I agree it'd be nice if more vendors offered automated tools for implementing ingress filtering, such tools are unnecessary in most corporate network cases, thus the lack of corporate customers asking for the feature. In reality every device offering access control lists capable of filtering on source IP address can and does have sufficient capability to implement BCP38. While I appreciate the desire to have a single switch solution, like was possible with BCP34, it's a bit more complex to do in this case. It is, however, disingenuous to say that devices don't support BCP38 because they don't have an automated widget to implement it. Keep in mind that uRPF is an implementation of BCP38 capability, and other implementations are entirely possible. This was probably obvious to you, but others reading might find the clarification useful.
On Mon, Oct 11, 2004 at 06:03:08PM -0400, Daniel Senie wrote:
I've removed the rest of your message, talking about which vendors do or don't have what capabilities. While I agree it'd be nice if more vendors offered automated tools for implementing ingress filtering, such tools are unnecessary in most corporate network cases, thus the lack of corporate customers asking for the feature. In reality every device offering access control lists capable of filtering on source IP address can and does have sufficient capability to implement BCP38.
While I appreciate the desire to have a single switch solution, like was possible with BCP34, it's a bit more complex to do in this case. It is, however, disingenuous to say that devices don't support BCP38 because they don't have an automated widget to implement it. Keep in mind that uRPF is an implementation of BCP38 capability, and other implementations are entirely possible.
This was probably obvious to you, but others reading might find the clarification useful.
Yes if a box has source address packet filtering capabilities you can filter packets by source address ("Duh"). This doesn't mean that it is going to be sane or easy to implement the filtering by manually maintaining an acl of every prefix/host on every interface where you could have a customer or corporate box injecting spoofed packets into the network. I believe there are plenty of corporate networks out there that are far too complex to maintain with manual ACLs, I believe the reason that no one cares is simply because... no one cares. :) If you expect people to be able to maintain these filters on any scale, they need tools. Certainly uRPF is a good tool to do this, and certainly someone could implement some others that are different, but the complete lack of any tool, especially on the boxes where you should be doing this filtering, counts as a failure in my book. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
At 07:51 PM 10/11/2004, Richard A Steenbergen wrote:
On Mon, Oct 11, 2004 at 06:03:08PM -0400, Daniel Senie wrote:
I've removed the rest of your message, talking about which vendors do or don't have what capabilities. While I agree it'd be nice if more vendors offered automated tools for implementing ingress filtering, such tools are unnecessary in most corporate network cases, thus the lack of corporate customers asking for the feature. In reality every device offering access control lists capable of filtering on source IP address can and does have sufficient capability to implement BCP38.
While I appreciate the desire to have a single switch solution, like was possible with BCP34, it's a bit more complex to do in this case. It is, however, disingenuous to say that devices don't support BCP38 because they don't have an automated widget to implement it. Keep in mind that uRPF is an implementation of BCP38 capability, and other implementations are entirely possible.
This was probably obvious to you, but others reading might find the clarification useful.
Yes if a box has source address packet filtering capabilities you can filter packets by source address ("Duh"). This doesn't mean that it is going to be sane or easy to implement the filtering by manually maintaining an acl of every prefix/host on every interface where you could have a customer or corporate box injecting spoofed packets into the network. I believe there are plenty of corporate networks out there that are far too complex to maintain with manual ACLs, I believe the reason that no one cares is simply because... no one cares. :)
One of your arguments presented was that corporate customers weren't asking for unicast RPF, and I responded that corporate customers are not in need of automated mechanisms to implement BCP38, since in most cases their networks are EDGE networks, and it's quite simple to filter your egress points to ensure you don't send out any spoofed packets. You laid out a complaint against the equipment makers claiming they weren't implementing automated mechanisms BECAUSE the corporate customers were not asking for such tools, and I simply pointed out they shouldn't be expected to do so. If network operators need features, they need to ask for them when talking with potential vendors. Network operators need to ensure downstreams don't advertise AS's they're not supposed to. Last I looked, that required some custom work (whether done by scripting or whatever, it's done off the router and pushed in). At the same time you're building those lists, you could be building ACLs. Some border routers will do this just fine, others won't. Next time you're qualifying routers for possible use, maybe the ability to handle wire speed acls might be worth testing?
If you expect people to be able to maintain these filters on any scale, they need tools.
Why do those tools need to be built into the router? Are your tools for maintaining AS path filteirng built into the routers? Are the tools to archive and compare router configurations most of you use built into the routers?
Certainly uRPF is a good tool to do this, and certainly someone could implement some others that are different, but the complete lack of any tool, especially on the boxes where you should be doing this filtering, counts as a failure in my book.
I disagree that the tools must be an integral part of the router software. Perhaps it's time to think outside the (router) box?
Daniel Senie wrote:
One of your arguments presented was that corporate customers weren't asking for unicast RPF, and I responded that corporate customers are not in need of automated mechanisms to implement BCP38, since in most cases their networks are EDGE networks, and it's quite simple to filter your egress points to ensure you don't send out any spoofed packets.
There is, of course, the issue of multihomed networks, or networks that have satellite connectivity etc emitting spoofed source packets. Yes I know that multihoming customers must make sure packets going out to the internet over a link match the route advertised out that link .. but stupid multihoming implementations do tend to ensure that lots of people will yell loudly, and yell loudly enough for several tickets to be escalated well beyond tier 1 NOC support desks, for ISPs to kind of think twice before they put uRPF filters in .. srs
On Tue, 12 Oct 2004, Suresh Ramasubramanian wrote:
Daniel Senie wrote:
One of your arguments presented was that corporate customers weren't asking for unicast RPF, and I responded that corporate customers are not in need of automated mechanisms to implement BCP38, since in most cases their networks are EDGE networks, and it's quite simple to filter your egress points to ensure you don't send out any spoofed packets.
There is, of course, the issue of multihomed networks, or networks that have satellite connectivity etc emitting spoofed source packets.
a common occurance we've seen is a customer of a customer NOT announcing , nor planning on announcing, their routes to their upstream#1 which they use ONLY for outbound traffic (cheap transit for instance, and perhaps only for some portions of their total sources) though they announce to upstreams#2-N the proper sources to gather the return traffic. These things make uRPF 'difficult'. As to the entire conversation I think more folks will implement BCP38, or parts atleast, as they replace gear which is not capable of dealing with parts of the implementation of BCP38. Hopefully new gear is being tested/certified/bought that can do wirespeed filtering on all interfaces. -Chris
* christopher.morrow@mci.com (Christopher L. Morrow) [Tue 12 Oct 2004, 05:18 CEST]:
a common occurance we've seen is a customer of a customer NOT announcing , nor planning on announcing, their routes to their upstream#1 which they use ONLY for outbound traffic (cheap transit for instance, and perhaps only for some portions of their total sources) though they announce to upstreams#2-N the proper sources to gather the return traffic. These things make uRPF 'difficult'.
You could use uRPF-loose there, or the customer could do: ! route-map outbound-only permit 10 match prefix-list myprefixes set community no-export ! And bash the people who, in this age, don't have "neighbor x.y.z.a send-community" on all their BGP sessions. -- Niels (who recently had a CCIE claim that he was "not aware of a single ISP accepting communities from its peers" - well, my experience begs to differ, with his employer a rare and lonely exception to the rule)
On Tue, 12 Oct 2004, Niels Bakker wrote:
* christopher.morrow@mci.com (Christopher L. Morrow) [Tue 12 Oct 2004, 05:18 CEST]:
a common occurance we've seen is a customer of a customer NOT announcing , nor planning on announcing, their routes to their upstream#1 which they use ONLY for outbound traffic (cheap transit for instance, and perhaps only for some portions of their total sources) though they announce to upstreams#2-N the proper sources to gather the return traffic. These things make uRPF 'difficult'.
You could use uRPF-loose there, or the customer could do:
! route-map outbound-only permit 10 match prefix-list myprefixes set community no-export !
this does not address the problem, the customer's customer isn't announcing routes for this traffic so there is nothing to no-export :( Example: the 'chris.net' network is a customer of MCI, his customer "bakker.net". 'bakker.net' decides 'chris.net' has priced transit cheaply this year/month/day and choses not to accept traffic from 'chris.net' but send all outbound traffic through 'chris.net'. 'chris.net' never seens routes for the sources sending this traffic, yet passes it along to the upstream, which also has no routes for 'bakker.net' via 'chris.net'. Regardless, the point here is: "Things seem like they may be getting better, as 'security' requirements are now firmly being included into new equipment purchases."
... Example:
the 'chris.net' network is a customer of MCI, his customer "bakker.net". 'bakker.net' decides 'chris.net' has priced transit cheaply this year/month/day and choses not to accept traffic from 'chris.net' but send all outbound traffic through 'chris.net'. 'chris.net' never seens routes for the sources sending this traffic, yet passes it along to the upstream, which also has no routes for 'bakker.net' via 'chris.net'.
uPRF is only one of several ways to implement BCP38. you could do it with contracts and reverse-SLA's and thus no technology (on your side) at all: demand that a customer pay 10X his bill, or $1.00 per packet, whichever is lower, if they emit packets with source addresses no explicitly named in the contract. why pay for expensive upgrades on your end of the link, when all you really care about is that BCP38's rules are observed?
Regardless, the point here is: "Things seem like they may be getting better, as 'security' requirements are now firmly being included into new equipment purchases."
that is of course good news. but it demonstrates a pitfall in CFO-think, which is the belief that participating in assymetric cost:benefit efforts (where uunet bears the cost of an upgrade in order for all the non-uunet parts of the internet to get the benefit of less spoofed traffic, and the abuse incident costs don't drop nearly enough to pay for the upgrades) is essentially a selfless act. we all want cleaner ddos flows. when we get ddos'd, we want to be able to look at the source addresses, look 'em up in whois, and call the launch-isp, and get things stopped. we want to be able to turn on flow shaping and know that an attacker can't cause us to use an arbitrarily large number of buckets. we *all* want these things. even the bad guys, who are often the victim of ddos attacks by other bad guys, want these things. how are we going to get there? the first thing is, some nets who want the internet to work this way have to implement BCP38 in their own corner of the internet. then they have to start de-peering with nets who don't do it, and offer a better rate to customers who do it than to those who don't. then they have to de-peer with anyone who doesn't require their peers and customers to do it. then they have to refuse as customers anyone who won't do it. it's all very simple, and it's inevitable. you and your CFO's have a couple of choices to make. first thing is, do you want the insurance companies, government regulatory agencies, and ISO9000 people to be making these rules or do you want to make them at the technical and business level? (this is called "industry self regulation" and it's an either-or proposition.) second choice to make is, do you want to do this early enough that you can fold it into your normal purchase/retire cycle, or do you want it rammed down your throat later in an irritating, short-timeframe, expensive, embarrassing way? fergie complained about the lack of chutzpah among a community who used to be distinguishable from other technical communities by that very chutzpah. i agreed but pointed out that the same engineers are here, but with vastly fewer of their buddies to help out, and vastly more supervision from the CFO than used to be the case. HOWEVER, there is still an opportunity to show some leadership, and GET THIS DONE without waiting for fireman's fund and a bunch of ISO9000 wonks to have to recognize it in a corporate risk profile. -- Paul Vixie
uPRF is only one of several ways to implement BCP38. you could do it with contracts and reverse-SLA's and thus no technology (on your side) at all: demand that a customer pay 10X his bill, or $1.00 per packet, whichever is lower, if they emit packets with source addresses no explicitly named in the contract. why pay for expensive upgrades on your end of the link, when all you really care about is that BCP38's rules are observed?
No reasonably sized provider is going to do that. There is too much competition, most of which is based on price. Until the companies creating the price pressure die (as in die completely, not re-emerge under a new, slightly different name), there is going to be no financial insentive for anyone to spend money improving their network. Let me underline, I am not talking about smaller ISPs, smaller networks or smaller service providers.
that is of course good news. but it demonstrates a pitfall in CFO-think, which is the belief that participating in assymetric cost:benefit efforts (where uunet bears the cost of an upgrade in order for all the non-uunet parts of the internet to get the benefit of less spoofed traffic, and the abuse incident costs don't drop nearly enough to pay for the upgrades) is essentially a selfless act.
Rubbish. CFO speak is what keeps the companies alive. Engineer-speak typically lands the company in chapter 11. Companies in Chapter 11 have too many operational decisions dictated by the courts, and those that think CFO speak would greally hate to hear courts on the topic.
we all want cleaner ddos flows. when we get ddos'd, we want to be able to look at the source addresses, look 'em up in whois, and call the launch-isp, and get things stopped. we want to be able to turn on flow shaping and know that an attacker can't cause us to use an arbitrarily large number of buckets. we *all* want these things. even the bad guys, who are often the victim of ddos attacks by other bad guys, want these things.
It is possible that _nanog_ subscribers want this. I am not quite clear how one can make that generalization about those behind kor.net, those in .ru, .ua etc. Finally,
how are we going to get there? the first thing is, some nets who want the internet to work this way have to implement BCP38 in their own corner of the internet. then they have to start de-peering with nets who don't do it, and offer a better rate to customers who do it than to those who don't. then they have to de-peer with anyone who doesn't require their peers and customers to do it. then they have to refuse as customers anyone who won't do it.
Last time I checked it was 2004, not 1998. The companies are financed by revenues that they generate, not IPOs or VCs based on a promise of enormous payoff sometime down the road. Cash is the king.
it's all very simple, and it's inevitable. you and your CFO's have a couple of choices to make. first thing is, do you want the insurance companies, government regulatory agencies, and ISO9000 people to be making these rules or do you want to make them at the technical and business level?
Yes, I do. This will level playing field and hopefully force a few of the big networks out of business completely, decreasing price pressure on this service. A drop in the price pressure will create an opportunity for those companies to spend the money (should they want to or be forced to) to be better internet citizens. This is just the cold blooded economic reality. The same reality which dictates that only smaller companies can enfore strict anti-spam policies, and prevent their customers from behaving badly. Alex
on Tue, Oct 12, 2004 at 12:49:54PM -0400, alex@yuriev.com wrote:
This is just the cold blooded economic reality. The same reality which dictates that only smaller companies can enfore strict anti-spam policies, and prevent their customers from behaving badly.
You want cold-blooded economic reality? I'm nominally a consultant, programmer, writer, and sysadmin - in that order. Over the past two years, I've spent over half of my time dealing with spam that I would never see if the rest of you had your acts together. You're costing me money and detracting from my ability to earn my keep. And yet, over those past two years, I've cut my spam load inbound by 98-99% with very few FPs (on the order of one report per week, which is far less time than I spend proactively blocking unwanted mail from badly policed networks). I taught myself the dark art of writing sendmail rulesets, built a database of ~7K rDNS naming conventions, ~18K poorly configured mail sources that inundate my servers with bounces from mail I didn't send, ~1.7K legitimate mail hosts that have relayed spam, 419 advance fee fraud scams, phishing frauds, or otherwise so utterly failed to police your own network egress points that we've blocked mail from you altogether. Where are the rest of you and why aren't you doing this? If I, and my little 7-man company, can afford to have me solve the problem on our end, why the heck can't you do the same? If I can do it, as the lone admin in a 7-man company, I should think that anyone on this list could do it. But I don't see it. All I see is arguing over whether or not to implement this or that BCP, or this or that antispam measure, or this or that anti-backscatter measure, or this or that antivirus measure. Get on with it. Email is dying, if it isn't already dead. The killer app isn't the Web, it's email. If email dies, who will want Internet services? You'll all be out pounding pavement and wishing you'd had the guts to do what you know is right and necessary. It's not possible for anyone here to say they didn't see it coming. The Cassandras have been screaming loudly, and often clearly, for years. Profoundly disappointed in the utter moral bankruptcy and cowardice of the NANOG constituency, Steve -- join us! http://hesketh.com/about/careers/web_designer.html join us! hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com join us! http://hesketh.com/about/careers/account_manager.html join us!
If I, and my little 7-man company, can afford to have me solve the problem on our end, why the heck can't you do the same?
You can do it because you are a 7-man company. So can I. However, companies the size of Sprint cannot do it. Alex
alex@yuriev.com [12/10/04 13:16 -0400]:
If I, and my little 7-man company, can afford to have me solve the problem on our end, why the heck can't you do the same?
You can do it because you are a 7-man company. So can I. However, companies the size of Sprint cannot do it.
Most filtering that I've seen (email, router, whatever) that just works great for a 7 man company will not work when you serve several million users, that's a fact. One false positive report per week from 7 users. How many per week - or per day - when you have 40 million users, is a question that gets answered real fast. A lot of the bad filtering (or lack of filtering, for that matter) decisions I've seen at large network providers and ISPs is generally where they are also unresponsive to their users and to the internet community that reports stuff to them (quite a few places I could name where most role accounts seem to funnel straight to /dev/null) srs
on Wed, Oct 13, 2004 at 07:09:10AM +0530, Suresh Ramasubramanian wrote:
alex@yuriev.com [12/10/04 13:16 -0400]:
If I, and my little 7-man company, can afford to have me solve the problem on our end, why the heck can't you do the same?
You can do it because you are a 7-man company. So can I. However, companies the size of Sprint cannot do it.
Most filtering that I've seen (email, router, whatever) that just works great for a 7 man company will not work when you serve several million users, that's a fact.
A 7 man Web app development company with ~100 or so hosted POP mail accounts, FYI. Not that it matters. If I can write rules that block spam with forged headers, so can any damn body else. And I'm tired of getting the bounces from those who feel it's not possible. Some examples of headers from mail that other ISPs have felt compelled by their size to accept and then bounce back to me: Received: from dslam129-213-59-62.adsl.zonnet.nl (62.59.213.129) by 0 with SMTP; 27 Aug 2004 21:20:16 -0000 Received: from thewordfactory.com (mail.thewordfactory.com [216.27.21.211]) by dslam129-213-59-62.adsl.zonnet.nl (Postfix) with ESMTP id 0453B70AB1 for <hermes510@philonline.com>; Fri, 27 Aug 2004 15:29:58 -0500 The second Received header is forged. The first is from a dynamic DSL line. The message was accepted by mail.philonline.com ([203.177.71.7]) and bounced back to me in a message that didn't even have a Message-ID header, letting me know they are so dumb they accept forged mail from dynamic IPs and only then analyze it to see if the user exists or if the forged sender should be notified. Received: from ip84-RND_DIGIT[2-3]-RND_DIGIT[2-3]-RND_DIGIT[2-3].4tvMsWC8okzw.customer.aviamail.zzn.com (50-RND_DIGIT[2-3]-RND_DIGIT[2-3]-RND_DIGIT[2-3].customer.aviamail.zzn.com [24.135.29.42]) by ezEG4GA1.aviamail.zzn.com (RND_DIGIT[1-3].RND_DIGIT[1-3].RND_DIGIT[1-3]/RND_DIGIT[1-3].RND_DIGIT[1-3].RND_DIGIT[1-3]) with SMTP id B3tc6UKcaTVY This was in a bounce from mail.cygentech.com (geoanalysis36.cust.viawest.net [216.150.205.36]). We've been seeing these headers for more than a year now, and blocking them almost as long. But these idiots can't see that backscattering them at me is rather stupid. Just a few of the others who've done this (accepted mail with completely bogus RNDizer headers from broken spamware): plesk4.merkury.com (216-86-139-44.tns.net [216.86.139.44] (may be forged)) smtp03.adnc.com (smtp03.adnc.com [206.251.248.23]) cobble.phpwebhosting.com (cobble.phpwebhosting.com [64.65.61.211]) date.marketorder.com ([64.65.150.229]) (by way of postini) exchgkom.TRI.NET (mail.tecolote.com [65.170.33.20]) DNS2.TCBINC.NET (DNS2.TCBINC.NET [64.124.116.30]) mail-in-01.netsonic.net (mail-in-01.netsonic.net [66.180.172.253]) ms1.tcnoc.com (ms1.tcnoc.com [63.150.10.30]) exchange3.corp.bcs-tx.com (ip204.bcs-tx.com [216.136.93.204] (may be forged)) [...etc...] My full list of backscatter hosts is ~17K entries long. And those are just the ones who've hit my servers. Never mind the charter/rr/att/hotmail/etc. hosts that are also listed - I'm not just talking about random Exchange boxes here - I'm talking about every major ISP with which I am familiar. Want to know if you're responsible for backscatter abuse? Just ask. As you know, Suresh, Outblaze does the same thing. Listed here as hosts we no longer accept null sender mail from: mta1-1.us4.outblaze.com BOUNCER spf0.us4.outblaze.com BOUNCER spf10.us4.outblaze.com BOUNCER spf1.us4.outblaze.com BOUNCER spf2.us4.outblaze.com BOUNCER spf4-1.us4.outblaze.com BOUNCER spf5-1.us4.outblaze.com BOUNCER spf7-17.us4.outblaze.com BOUNCER At least you've said you're moving towards fixing it. Thanks.
One false positive report per week from 7 users. How many per week - or per day - when you have 40 million users, is a question that gets answered real fast.
I don't even want to go into the backscatter messages that show that: - the sender forged the IP/domain of the recipient into HELO/EHLO - the recipient accepts mail for unknown accounts - the relaying host forwarded Webmail from Nigeria without proper Received headers added for blocking purposes - you name the obvious spamsign Come on, there is so much obvious crud that should be checked that isn't being checked - we could reduce backscatter by a third or more simply by refusing during SMTP handshake messages from hosts that forge the receipient IP/hostname/domain in HELO, or to users that don't exist, or if virus filters were clueful enough to drop, rather than emit DSNs, for known virus-originated messages that always forge the sender.
A lot of the bad filtering (or lack of filtering, for that matter) decisions I've seen at large network providers and ISPs is generally where they are also unresponsive to their users and to the internet community that reports stuff to them (quite a few places I could name where most role accounts seem to funnel straight to /dev/null)
I wish I could say that was where most of them were confined. But what I see is a widespread lack of clue and the will to do something to fix email. You see Dave Crocker on Farber's list, saying that most spam is perfectly well-formed email, and I wonder what planet he's smoking. At /least/ 30% of the mail we reject is simply riddled with spamsigns and violations of basic RFCs. And I'd say that the same is true of the mail we get as backscatter from clueless morons pretending to be large ISP mail admins. -- join us! http://hesketh.com/about/careers/web_designer.html join us! hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com join us! http://hesketh.com/about/careers/account_manager.html join us!
Date: 12 Oct 2004 16:22:17 +0000 From: Paul Vixie <vixie@vix.com> To: nanog@merit.edu Subject: Re: BCP38 making it work, solving problems
[ ... ] how are we going to get there? the first thing is, some nets who want the internet to work this way have to implement BCP38 in their own corner of the internet. then they have to start de-peering with nets who don't do it, and offer a better rate to customers who do it than to those who don't. then they have to de-peer with anyone who doesn't require their peers and customers to do it. then they have to refuse as customers anyone who won't do it. [ ... ]
As it was "in the old days": first clean up your own act and then start pointing at others that they're doing "it" wrong. It's a mentality I see very rarely amongst the larger ISP's who've been part of that 'I' in their name since the very early beginning. Regards, JP Velders
At 01:11 PM 10/19/04 +0200, JP Velders wrote:
As it was "in the old days": first clean up your own act and then start pointing at others that they're doing "it" wrong.
hear hear... But Paul knows and in fact does that. He is pointing out the difficulty of getting people to do basic things that are for their own benefit. For example, how many ISPs use TCP MD5 to limit the possibility of a BGP/TCP connection getting hijacked or disrupted by a ddos attack? But this has been in the code since ~1990, and was put there because of a fairly serious and specific attack that was made on Internet routing, and benefits primarily the ISP that enables the procedure in that it knows that its routes are coming to it from systems it has chosen to trust. Ingress filters help the ISP that installs them, in that a certain class of attacks are prevented among customers of the ISP. Would it be better if all ISPs and all edge networks put appropriate filters in place? Absolutely. But even if they do not, the ISP saves itself that much trouble. Where ingress filters don't help, of course, is when the attacks come from an apparently-legitimate address. Then we are off to other tools.
For example, how many ISPs use TCP MD5 to limit the possibility of a BGP/TCP connection getting hijacked or disrupted by a ddos attack?
i hope none use it for the latter, as it will not help. more and more use it for the former. why? becuase they perceived the need to solve an immediate problem, a weakness in a vendor's code.
Where ingress filters don't help, of course, is when the attacks come from an apparently-legitimate address.
many folk see that this is the vast majority of the cases. hence, one reason for the lack of adoption of rfc 2827 randy
Date: Tue, 19 Oct 2004 09:21:46 -0700 From: Randy Bush <randy@psg.com> Subject: Re: BCP38 making it work, solving problems
For example, how many ISPs use TCP MD5 to limit the possibility of a BGP/TCP connection getting hijacked or disrupted by a ddos attack?
i hope none use it for the latter, as it will not help. more and more use it for the former. why? becuase they perceived the need to solve an immediate problem, a weakness in a vendor's code.
Uhm, you might need to run that by me again... Hijacking the connection is in a completely different class as someone bombarding you with a bunch of forged BGP packets to close down a session. Without that MD5 checksum you are quite vulnerable to that. I haven't seen a vendor come up with a solution to that, because the problem is on a much more vendor-neutral level... Regards, JP Velders PS: ofcourse that MD5 option also causes problems for peerings to come back "up" again if you have to reboot/reload *without* properly closing them... :( Hey, pro's and con's are part of the job ;)
On Tue, Oct 19, 2004 at 07:14:32PM +0200, JP Velders scribed:
Date: Tue, 19 Oct 2004 09:21:46 -0700 From: Randy Bush <randy@psg.com> Subject: Re: BCP38 making it work, solving problems
For example, how many ISPs use TCP MD5 to limit the possibility of a BGP/TCP connection getting hijacked or disrupted by a ddos attack?
i hope none use it for the latter, as it will not help. more and more use it for the former. why? becuase they perceived the need to solve an immediate problem, a weakness in a vendor's code.
Uhm, you might need to run that by me again...
Hijacking the connection is in a completely different class as someone bombarding you with a bunch of forged BGP packets to close down a session. Without that MD5 checksum you are quite vulnerable to that. I haven't seen a vendor come up with a solution to that, because the problem is on a much more vendor-neutral level...
Unless you're worried about an adversary who taps into your fiber, how is MD5 checksums any better than anti spoofing filters that protect your BGP peering sessions? The only benefit I see is that you can actually verify that your peer is using md5 checksums, instead of having to take them on faith that they won't permit someone to spoof their router's address. -Dave -- work: dga@lcs.mit.edu me: dga@pobox.com MIT Laboratory for Computer Science http://www.angio.net/
Date: Tue, 19 Oct 2004 13:20:08 -0400 From: David G. Andersen <dga@lcs.mit.edu> Subject: Re: BCP38 making it work, solving problems
[ ... ] Unless you're worried about an adversary who taps into your fiber, how is MD5 checksums any better than anti spoofing filters that protect your BGP peering sessions? The only benefit I see is that you can actually verify that your peer is using md5 checksums, instead of having to take them on faith that they won't permit someone to spoof their router's address.
How much control do 'they' have over the ways 'someone' can spoof ? With large providers who don't see any harm in allowing possibly spoofed traffic through, you cannot exclude the possibility that an ISP connected to an IX "leaks" those spoofed packets onto the IX. (or leaks RFC1918 space... I know of a few examples / mails ;D) In the current world - where you cannot exclude either one - you're much better off 'safe' then 'sorry'... Implementing BCP38 (to come back on-topic) is just plain good neighbourhood policy. I don't go building 2.5 meter tall fences around my house because I don't want my neighbour's plants in my garden. No, we come to an understanding that whenever his plants get out of control in my garden I can cut them back, but that he will also trim them more often. In most cases it will go like that, the minority of when it doesn't go like that, you start filtering / whatever, just like we do now. Regards, JP Velders
On Oct 19, 2004, at 1:14 PM, JP Velders wrote:
jacking the connection is in a completely different class as someone bombarding you with a bunch of forged BGP packets to close down a session. Without that MD5 checksum you are quite vulnerable to that. I haven't seen a vendor come up with a solution to that, because the problem is on a much more vendor-neutral level...
We haven't talked about this in a few months, so what the hell.... Have you actually done the work to see how many packets it takes to shut down a session with and without MD5 enabled? (The question is rhetorical, since your post shows that you have not.) Back on topic, the MD5 debate is not an exact apples-to-apples comparison of BCP38. Stopping people from shutting down your BGP sessions is not the same as letting your customer hurt me while claiming to be a third party. Put another way, MD5 on BGP sessions is a personal choice per network. I made my decision. You are welcome and encouraged to make your own. Neither will effect the other, except where our two networks meet. (And then I am positive we can come to some mutual understanding.) BCP38 is not a personal decision. Not implementing it hurts the whole Internet, not just your little corner. -- TTFN, patrick
On Wed, 20 Oct 2004, Patrick W Gilmore wrote:
Have you actually done the work to see how many packets it takes to shut down a session with and without MD5 enabled? (The question is rhetorical, since your post shows that you have not.)
Just a bit more sauce for the goose...enabling MD5 on BGP peers under certain latest in their train IOS versions will immediately crash IOS. Guess how I know that? ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On 20 Oct 2004, at 21:49, Jon Lewis wrote:
On Wed, 20 Oct 2004, Patrick W Gilmore wrote:
Have you actually done the work to see how many packets it takes to shut down a session with and without MD5 enabled? (The question is rhetorical, since your post shows that you have not.)
Just a bit more sauce for the goose...enabling MD5 on BGP peers under certain latest in their train IOS versions will immediately crash IOS.
Guess how I know that?
In a similar vein, upgrading from certain flavours of 12.0 to certain flavours of 12.2 seems to cause password hashes to become disfunctional, leaving BGP sessions down after the first reboot until the passwords are re-entered in plain text from the command line. (Guess how I know that, too). This is not a statement in support of not using MD5 auth passwords. Just a reminder to include MD5 passwords in your lab testing before deployment, if you use them. Joe
There is, of course, the issue of multihomed networks, or networks that have satellite connectivity etc emitting spoofed source packets.
y'know, SAC004 (http://www.icann.org/committees/security/sac004.txt) is only four pages long, and one of those is references. you should read it before you call multihomed networks an "issue" wrt BCP38 deployment. in fact, you should read it, and BCP38, and BCP84, before participating in this discussion at all, either here, or at the bar-bofs next week. -- Paul Vixie
Paul Vixie wrote:
There is, of course, the issue of multihomed networks, or networks that have satellite connectivity etc emitting spoofed source packets.
y'know, SAC004 (http://www.icann.org/committees/security/sac004.txt) is only four pages long, and one of those is references. you should read it before you call multihomed networks an "issue" wrt BCP38 deployment. in fact, you should read it, and BCP38, and BCP84, before participating in this discussion at all, either here, or at the bar-bofs next week.
Multihomed networks are not an issue. Stupid implementations of these? Yes sure. srs
On 10/12/04 9:06 AM, "Paul Vixie" <vixie@vix.com> wrote:
There is, of course, the issue of multihomed networks, or networks that have satellite connectivity etc emitting spoofed source packets.
y'know, SAC004 (http://www.icann.org/committees/security/sac004.txt) is only four pages long, and one of those is references. you should read it before you call multihomed networks an "issue" wrt BCP38 deployment. in fact, you should read it, and BCP38, and BCP84, before participating in this discussion at all, either here, or at the bar-bofs next week.
Excerpt from the text quoted above: 2.3. For a DDoS attack to succeed more than once, the launch points must remain anonymous. Therefore, forged IP source addresses are used. From the victim's point of view, a DDoS attack seems to come from everywhere at once, even from many IP addresses that are unallocated or otherwise invalid. How many people have seen "forged" spoofed IP addresses being used for DOS attacks lately? Bora
On Oct 12, 2004, at 12:50 PM, Bora Akyol wrote:
2.3. For a DDoS attack to succeed more than once, the launch points must remain anonymous. Therefore, forged IP source addresses are used. From the victim's point of view, a DDoS attack seems to come from everywhere at once, even from many IP addresses that are unallocated or otherwise invalid.
How many people have seen "forged" spoofed IP addresses being used for DOS attacks lately?
<raises hand> Not saying that I have not see non-forged DoS attacks too, or even which is more common, just saying they exist, are happening today, and cause non-trivial problems for some providers. From my _personal_ experience (not my company, not a scientific sampling), it appears non-spoofed sources are a bigger problem. But ignoring spoofed sources would be a mistake, IMHO. -- TTFN, patrick
On Tue, 12 Oct 2004, Bora Akyol wrote:
Excerpt from the text quoted above:
2.3. For a DDoS attack to succeed more than once, the launch points must remain anonymous. Therefore, forged IP source addresses are used. From the victim's point of view, a DDoS attack seems to come from everywhere at once, even from many IP addresses that are unallocated or otherwise invalid.
How many people have seen "forged" spoofed IP addresses being used for DOS attacks lately?
it does still happen... I've not run the numbers for our reactions to say '50% spoofed/50% non-spoofed' but it certainly seems like 'more' are non-spoofed lately. This could be a simple swing of the pendulum, or other 'better' things like more people egress filtering. -Chris
At 09:50 AM 12-10-04 -0700, Bora Akyol wrote:
How many people have seen "forged" spoofed IP addresses being used for DOS attacks lately?
For the week starting Sept 12, our dark space telescope "saw" 1675 spoofed DDOS attacks. No idea how many non-spoofed attacks took place during the same time period. -Hank
Bora
At 04:59 AM 13-10-04 -0700, Randy Bush wrote:
For the week starting Sept 12, our dark space telescope "saw" 1675 spoofed DDOS attacks.
any idea why someone(s) is ddosing dark space? seems a bit silly.
No one is DDOSing dark space. The dark space telescope picks up the "richochets" caused by DDOS. CAIDA created a nice 90 second video clip explaining this at: http://www.caida.org/outreach/resources/animations/passive_monitoring/backsc... -Hank
randy
any idea why someone(s) is ddosing dark space? seems a bit silly. No one is DDOSing dark space. The dark space telescope picks up the "richochets" caused by DDOS.
sorry. i guess i should give up getting more sleep and go make coffee. randy
Randy Bush wrote:
For the week starting Sept 12, our dark space telescope "saw" 1675 spoofed DDOS attacks.
any idea why someone(s) is ddosing dark space? seems a bit silly.
Something like this I rather fancy ... http://lists.planet-lab.org/pipermail/announce/2004-April/000012.html
[Planetlab-announce] network telescope Larry Peterson llp at CS.Princeton.EDU Thu Apr 15 16:31:24 EDT 2004
* Previous message: [Planetlab-announce] Cambridge meeting * Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
I have a request to make of PlanetLab sites... There is an effort underway to collect information about worms, ddos backscatter, and other suspect activity on the Internet. Our ability to do this sort of analysis would be greatly enhanced by having access to IP address blocks spread across the Internet, for example, at as many PlanetLab sites as possible. My specific request is for sites to contribute blocks of, say, 16 otherwise unused addresses to PlanetLab. I have attached a note from Vern Paxson outlining the idea, a so called "network telescope". Please let me know if your site would be willing to contribute (assign) some number of addresses to PlanetLab for this purpose.
Larry
------------------------------
A basic challenge for analyzing Internet-scale malicious phenomena such as worms and automated scanning is acquiring sufficiently broad visibility into their workings. Monitoring at a single location may for example miss the early stages of a worm's spread or, more generally, lack the diverse perspectives necessary for capturing large-scale behavior.
A powerful tool for acquiring such broader visibility is a "network telescope". Network telescopes monitor traffic sent to communication dead-ends such as unallocated portions of the IP address space or ports on endhosts for which no server is listening. Since there is no legitimate reason for a host to send packets to those destinations, such traffic provides strong evidence of malicious activity - including DDoS backscatter, port scanning, and probe activity from worms.
Our goal is to build a large-scale telescope with significantly more sampling breadth and diversity than current telescopes. This telescope will be structured as two layers. Its front-end sensors will be spread across a large number of address blocks and monitoring points to achieve sampling diversity. We'll use both unallocated address blocks (which attackers can learn about fairly easily) but also unused subblocks within allocated blocks. This latter "dark address space" is much more difficult for an attacker to learn about and also enables highly diverse distribution of the sensors.
It's this latter that we're hoping can be done in conjunction with PL nodes. In particular, the way we picture it working is that the PL nodes will have multiple addresses assigned to them. A monitor running on the host then tunnels traffic it receives on the extra addresses over to the analysis point. It could also tunnel traffic sent to its "normal" address but for which there's no listener.
One crucial issue with building a large telescope is *filtering*. For a very large telescope, the volume of data collected can be enormous. However, for many forms of analysis we can often filter out a great deal of the traffic. For example, for worm detection we can drop traffic seen by the sensor rather than forwarding it if the traffic does not correspond to worm activity of possible interest (e.g., it's instead DoS backscatter, or activity from known worms). Because PL nodes can do computation before they forward traffic over the tunnel (unlike, for example, telescope sensors based on using routers), they make ideal platforms for developing such filtering.
How many people have seen "forged" spoofed IP addresses being used for DOS attacks lately?
syn-flood protection, and random TCP ISS, are now common enough that spoofed-source isn't effective for TCP flows. if you want to bring down somebody's web server then blackhats really do have to use real addresses. however, if you just want to make their web server unreachable, then you can either overload their DNS infrastructure or just congest their upstreams, and you don't need to use real addresses for that. i've never seen a dns attack that didn't have 50% or more packets coming from spoofed sources, though due to loose-mode uRPF, most spoofed sources in the last year or so have been from addresses for which a route exists. -- Paul Vixie
On 13 Oct 2004, Paul Vixie wrote:
How many people have seen "forged" spoofed IP addresses being used for DOS attacks lately?
syn-flood protection, and random TCP ISS, are now common enough that spoofed-source isn't effective for TCP flows. if you want to bring down somebody's web server then blackhats really do have to use real addresses.
of course the docs were written a couple years ago, and things have changed a lot in that time. the proliference of and ease of establishing bot networks is such that their controllers dont care if you track them and shut them down as they are easily replaced Steve
i've never seen a dns attack that didn't have 50% or more packets coming from spoofed sources, though due to loose-mode uRPF, most spoofed sources in the last year or so have been from addresses for which a route exists. -- Paul Vixie
reiterating a sometimes heretical idea... are you refering to things like 172.17.0.0/16 where only a couple hundred of those numbers have real services, e.g. all the services are in 172.17.22.0/24 and the spoofed addresses are in 172.17.128.0/17 space? or... why do people insist on injecting routes to non-existent things? a route table entry is a route table entry, regardless of the scope. --bill
i've never seen a dns attack that didn't have 50% or more packets coming from spoofed sources, though due to loose-mode uRPF, most spoofed sources in the last year or so have been from addresses for which a route exists. -- Paul Vixie
reiterating a sometimes heretical idea...
are you refering to things like 172.17.0.0/16 where only a couple hundred of those numbers have real services, e.g. all the services are in 172.17.22.0/24 and the spoofed addresses are in 172.17.128.0/17 space?
In that example, if the receiver uses loose-mode uRPF to filter ingress and carries within their AS only the minimal set of RFC1918 routes corresponding to their actual use of it, then spoofed-source packets "from" 172.17.22.0/24 would be allowed in by loose-mode uRPF because a route table entry exists. Packets "from" the rest of 172.17.0.0/16 would be filtered. Loose-mode uRPF does not protect you from spoofed-source packets where the source address falls within a prefix that you use internally, whether that space is RFC1918 space or not. Likewise, loose-mode uRPF does not (completely) protect you from spoofed-source packets where the source address falls within a prefix that is valid externally, not necessarily via the reverse path on which you're receiving the packet. The first can be addressed with additional filtering, as has been mentioned. The second is a harder problem, because of the business decisions of some providers to source packets from prefixes that they do not announce.
or... why do people insist on injecting routes to non-existent things? a route table entry is a route table entry, regardless of the scope.
Is this where you advocate that providers only announce the parts of their assigned blocks that are in use? Stephen
or... why do people insist on injecting routes to non-existent things? a route table entry is a route table entry, regardless of the scope.
Is this where you advocate that providers only announce the parts of their assigned blocks that are in use?
seems like a good lead in, so yes - i advocate folks only announce what they use. may play old-hob on the ISP that likes to use some other metric for accepting announcements, (e.g. RIR or other routing registry DB) and will no doubt increase the tension on justification of proxy announcements, but overall, this seems to be a good goal. thanks for letting me rant. :) --bill
On (13/10/04 18:43), bmanning@vacation.karoshi.com wrote:
Is this where you advocate that providers only announce the parts of their assigned blocks that are in use?
seems like a good lead in, so yes - i advocate folks only announce what they use. may play old-hob on the ISP that likes to use some other metric for accepting announcements, (e.g. RIR or other routing registry DB) and will no doubt increase the tension on justification of proxy announcements, but overall, this seems to be a good goal.
i thought that a common goal was to announce only your aggregates (which would inherently include unused space) rather than announce all your deag'ed space (which should be more well used) two of my acl's have dropped ~9million packets destined for bogon networks in the past week (only about 20 Mbps on this router)...bogon sources are probably equivalent (although i havent been logging it, but in light of recent discussions, i probably will start again)
thanks for letting me rant. :)
yep - me too /joshua -- **** END NET CENSORSHIP - FREE RAS ****
Date: Wed, 13 Oct 2004 18:43:45 +0000 From: bmanning@vacation.karoshi.com Sender: owner-nanog@merit.edu
or... why do people insist on injecting routes to non-existent things? a route table entry is a route table entry, regardless of the scope.
Is this where you advocate that providers only announce the parts of their assigned blocks that are in use?
seems like a good lead in, so yes - i advocate folks only announce what they use. may play old-hob on the ISP that likes to use some other metric for accepting announcements, (e.g. RIR or other routing registry DB) and will no doubt increase the tension on justification of proxy announcements, but overall, this seems to be a good goal.
First, we do accept prefixes from most ASes based on RIR. Second, we don't simply assign address space sequentially from our assigned spaces. We have an addressing plan that leaves the assignments deliberately sparse to allow for better management and the ability to keep our PA assignments to a site contiguous. To only announce the active space would increase the number of routes we announce by about 80%. If everyone did this, the routing table would increase massively. So would the time to compute the routes which might lead to some really bad instability for some routers.
thanks for letting me rant. :)
Any time, Bill. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634
On Wed, Oct 13, 2004 at 12:54:44PM -0700, Kevin Oberman wrote:
Date: Wed, 13 Oct 2004 18:43:45 +0000 From: bmanning@vacation.karoshi.com Sender: owner-nanog@merit.edu
or... why do people insist on injecting routes to non-existent things? a route table entry is a route table entry, regardless of the scope.
Is this where you advocate that providers only announce the parts of their assigned blocks that are in use?
seems like a good lead in, so yes - i advocate folks only announce what they use. may play old-hob on the ISP that likes to use some other metric for accepting announcements, (e.g. RIR or other routing registry DB) and will no doubt increase the tension on justification of proxy announcements, but overall, this seems to be a good goal.
First, we do accept prefixes from most ASes based on RIR.
good traditional thinking .... requireing me to announce the whole /20 when all i'm using is a /27... after all, its just one routeing table entry... why should you care? :) (playing straightman)
Second, we don't simply assign address space sequentially from our assigned spaces. We have an addressing plan that leaves the assignments deliberately sparse to allow for better management and the ability to keep our PA assignments to a site contiguous. To only announce the active space would increase the number of routes we announce by about 80%. If everyone did this, the routing table would increase massively. So would the time to compute the routes which might lead to some really bad instability for some routers.
so -IF- everyone followed your internal address assignment policies, scattering used space in a sparse matrix throughout the allocated pool, then announing a single prefix (the aggregate) makes sense. Of course this leaves you w/ lots of space that is useful for forging as valid source addresses. (we'll even leave DHCP pools out of the discussion - for now) so it would make sense for you to announce the aggregate - since you use "random" bits throughout (nice marbling!) but for those folks who use a more compact internal representation, is there a good reason to reject their /27 instead of the larger /20 that has been allocated?
thanks for letting me rant. :)
Any time, Bill.
I'll try and use it wisely --bill
On Wed, Oct 13, 2004 at 08:24:09PM +0000, bmanning@vacation.karoshi.com wrote:
On Wed, Oct 13, 2004 at 12:54:44PM -0700, Kevin Oberman wrote:
Date: Wed, 13 Oct 2004 18:43:45 +0000 From: bmanning@vacation.karoshi.com
seems like a good lead in, so yes - i advocate folks only announce what they use. may play old-hob on the ISP that likes to use some other metric for accepting announcements, (e.g. RIR or other routing registry DB) and will no doubt increase the tension on justification of proxy announcements, but overall, this seems to be a good goal.
Why? This is a serious question, as the bulk of network architectures with which I'm familiar have a wall between IGP and EGP, where the EGP is focused on _stable_ reachability & the IGP is concerned with optimal forwarding path within the AS. Sinking (or darknet detecting or...) unused space and happily shuffling between used & unused without having to worry how it affects your reachability [or its stability] is a generally a Good Thing.
Second, we don't simply assign address space sequentially from our assigned spaces. We have an addressing plan that leaves the assignments [snip] so -IF- everyone followed your internal address assignment policies, scattering used space in a sparse matrix throughout the allocated pool, then announing a single prefix (the aggregate) makes sense. Of [snip]
"There are more internal policies on Teh Intarweb than are dreamt of in your philosophy." Internal policies are *internal*, and the usual wall between internal and external policies means that maximal reachibility can be guarenteed with minimal hassle and zero need to have to pick up the phone and convince some random remote AS to change their policies to accept your topology-of-the-week. Seems that any actions that are recommended should be internal policy/topology neutral, no? Should I mention publishing the only the deaggregates-in-use essentially rolls out the red carpet to hijackers letting them know prefixes you *will*not*notice* that they borrow, except when the abuse calls and subpoenas come rolling in? Oh no, that's for later in the thread. Sorry. Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE
The second is a harder problem, because of the business decisions of some providers to source packets from prefixes that they do not announce.
i presume you are not intending to recommend that i drop packets that multi-homed customers hand me when they have also asked me to de-pref the prefix from which they come? i might be their backup for inbound, but they need to balance their outbound. randy
The second is a harder problem, because of the business decisions of some providers to source packets from prefixes that they do not announce.
i presume you are not intending to recommend that i drop packets that multi-homed customers hand me when they have also asked me to de-pref the prefix from which they come? i might be their backup for inbound, but they need to balance their outbound.
No, I'm not recommending that. In the example that you give, the prefix from which the packets in question will be sourced *is* offered as a destination? Assuming yes, then regardless of whether that offer is selected as the best to use to reach the destination or not, the edge router that needs to make the decision to accept or reject packets sourced in that prefix can use its knowledge that the offer was made to accept packets. The problem is differentiating these two cases: 1. the offer of a route to a prefix is not made but packets sourced in that prefix are legitimate and are expected to be forwarded; the reverse path is only available through a different AS 2. the source address is spoofed; packets are not legitimate and should be dropped Once upon a time, I tried to enable loose-mode uRPF on a peering interface, effectively treating #1 as #2. The complaints were relatively instantaneous (at 2am local for me, a traffic-low time), and not localized to a specific source prefix (the majority were residential broadband users); I wound up turning the loose-mode uRPF check off in fairly short order. Attempts to discover why #1 was happening ended with technical people shrugging their shoulders and saying that the money people made them do it. Stephen
The second is a harder problem, because of the business decisions of some providers to source packets from prefixes that they do not announce. i presume you are not intending to recommend that i drop packets that multi-homed customers hand me when they have also asked me to de-pref the prefix from which they come? i might be their backup for inbound, but they need to balance their outbound. No, I'm not recommending that.
so i suspected <g>
In the example that you give, the prefix from which the packets in question will be sourced *is* offered as a destination?
not by me. but by one or more of the originator's other upstreams. i suspect this is not uncommon.
The problem is differentiating these two cases: 1. the offer of a route to a prefix is not made but packets sourced in that prefix are legitimate and are expected to be forwarded; the reverse path is only available through a different AS
2. the source address is spoofed; packets are not legitimate and should be dropped
Once upon a time, I tried to enable loose-mode uRPF on a peering interface, effectively treating #1 as #2. The complaints were relatively instantaneous (at 2am local for me, a traffic-low time), and not localized to a specific source prefix (the majority were residential broadband users); I wound up turning the loose-mode uRPF check off in fairly short order.
Attempts to discover why #1 was happening ended with technical people shrugging their shoulders and saying that the money people made them do it.
yep. some times it is even less intentional and less understood; see tim's paper on bgp wedgies. and the "management made me do it," is a bit disingenuous. it's part of what it means to have customers. randy
yep. some times it is even less intentional and less understood; see tim's paper on bgp wedgies. and the "management made me do it," is a bit disingenuous. it's part of what it means to have customers.
My customers, back when I had them, must have been better-behaved than most.
yep. some times it is even less intentional and less understood; see tim's paper on bgp wedgies. and the "management made me do it," is a bit disingenuous. it's part of what it means to have customers. My customers, back when I had them, must have been better-behaved than most.
then why did you get those calls you mention in your last message? randy
yep. some times it is even less intentional and less understood; see tim's paper on bgp wedgies. and the "management made me do it," is a bit disingenuous. it's part of what it means to have customers. My customers, back when I had them, must have been better-behaved than most.
then why did you get those calls you mention in your last message?
They weren't from customers, they were from users connected to Other People's Networks who couldn't reach my customers. I took trouble tickets from them, too.
yep. some times it is even less intentional and less understood; see tim's paper on bgp wedgies.
http://ispcolumn.isoc.org/2004-09/bgp-wedgies.html Nice article. Too bad we don't see more stuff written up like this in such a clear way. http://www.potaroo.net/ietf/idref/draft-griffin-bgp-wedgies/ --Michael Dillon
On Wed, 13 Oct 2004, Randy Bush wrote:
The second is a harder problem, because of the business decisions of some providers to source packets from prefixes that they do not announce.
i presume you are not intending to recommend that i drop packets that multi-homed customers hand me when they have also asked me to de-pref the prefix from which they come? i might be their backup for inbound, but they need to balance their outbound.
FWIW, (you probably know this, but most on nanog maybe don't), If you do 'feasible path strict uRPF' as described in BCP84 (I don't know if others than Juniper are providing that), you can enable strict uRPF toward those customers, still de-pref them, and accept the packets with correct source addresses. That's what we do with our customers whether multihomed or not. One can also achieve the same without 'feasible path' by operationally adjusting the weight or preference for the advertisement you receive w/ eBGP and the advertisement you send in iBGP (so that only that one router would send its traffic over that link), but that's likely a bit more work and operational complexity. -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
On Thu, Oct 14, 2004 at 08:05:50AM +0300, Pekka Savola wrote:
If you do 'feasible path strict uRPF' as described in BCP84 (I don't know if others than Juniper are providing that), you can enable strict uRPF toward those customers, still de-pref them, and accept the packets with correct source addresses.
That's what we do with our customers whether multihomed or not.
And what do you do with a BGP customer which sends you traffic from prefixes he doesn't want to announce to you? There are such customers. Fail filter ACL? Best regards, Daniel (who wishes active/feasible would be switchable per interface, not only per router) -- CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
On Thu, 14 Oct 2004, Daniel Roesen wrote:
On Thu, Oct 14, 2004 at 08:05:50AM +0300, Pekka Savola wrote:
If you do 'feasible path strict uRPF' as described in BCP84 (I don't know if others than Juniper are providing that), you can enable strict uRPF toward those customers, still de-pref them, and accept the packets with correct source addresses.
That's what we do with our customers whether multihomed or not.
And what do you do with a BGP customer which sends you traffic from prefixes he doesn't want to announce to you? There are such customers. Fail filter ACL?
Good point. It could be doable with fail-filter ACL, but we don't have any of these, so it'd be just a silent discard. Honestly, I fail to see this as a big problem. If they don't want to announce the prefix to us, why would they want to source traffic from that prefix to us? The inbound traffic engineering is the more tricky business, not the outbound. If they want to keep the link usage low, they could just send it with no-export or no-advertise, or suitably prepended. Except for really wacky asymmetric multihoming cases, I'd expect that some customers might actually want 'restricted' or 'internal' traffic to be discarded (compare to RFC1918 sourced traffic from enterprises, because they use RFC1918 but don't set up the discarding ACLs on their own). -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
On Thu, Oct 14, 2004 at 06:24:21PM +0300, Pekka Savola wrote:
Honestly, I fail to see this as a big problem. If they don't want to announce the prefix to us, why would they want to source traffic from that prefix to us?
I could delve in some exceptionally ugly examples of peering politics but refrain to do that. Let's just say that it's one way to make things "look wrong" to The Uninitiated and easy to point complainers to the other party.
The inbound traffic engineering is the more tricky business, not the outbound.
Well, but outbound TE based on _source_ needs policy routing. With some vendor's gear this is not a too big problem, but with some other vendor's gear this comes at a much higher price.
If they want to keep the link usage low, they could just send it with no-export or no-advertise, or suitably prepended.
Prepend doesn't work as you as upstream normally localpref your customers above all others (you have to by default, otherwise there are setups in which $customer might be without _any_ upstream although the link to you is still fine). no-export doesn't work as $customer doesn't want ANY traffic from your other customers to him directly for certain prefixes. no-advertise is ugly++. Best regards, Daniel -- CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
On 14-okt-04, at 16:41, Daniel Roesen wrote:
And what do you do with a BGP customer which sends you traffic from prefixes he doesn't want to announce to you? There are such customers.
The whole point of BCP38 is that this isn't supposed to happen. Yes, these restrictions are a huge pain in the rear end but a denial of service without even the possibility to tell where the packets come from is MUCH worse.
On Thu, Oct 14, 2004 at 08:35:50PM +0200, Iljitsch van Beijnum wrote:
And what do you do with a BGP customer which sends you traffic from prefixes he doesn't want to announce to you? There are such customers.
The whole point of BCP38 is that this isn't supposed to happen.
Unfortunately we are living in reality.
Yes, these restrictions are a huge pain in the rear end but a denial of service without even the possibility to tell where the packets come from is MUCH worse.
What you actually want to know is what the ingress interfaces for the flows are. And if the ingress interface is not a p2p interface, from which peer. For both problems quite effective solutions do exist (ok, not really for the latter, but this is highly vendor specific). Given that most DDoSses are mounted via huge zombie collections, there is not much point in knowing the real source IPs. Best regards, Daniel -- CLUE-RIPE -- Jabber: dr@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
On Oct 14, 2004, at 4:27 PM, Daniel Roesen wrote:
Yes, these restrictions are a huge pain in the rear end but a denial of service without even the possibility to tell where the packets come from is MUCH worse.
What you actually want to know is what the ingress interfaces for the flows are. And if the ingress interface is not a p2p interface, from which peer. For both problems quite effective solutions do exist (ok, not really for the latter, but this is highly vendor specific).
No, what I really want to know is the source IP.
Given that most DDoSses are mounted via huge zombie collections, there is not much point in knowing the real source IPs.
Didn't we cover this? Yes, there are zombie armies launching DDoS from "real" IP addresses. But that does not mean there are no spoofed-source attacks any more. -- TTFN, patrick
On 14-okt-04, at 22:27, Daniel Roesen wrote:
And what do you do with a BGP customer which sends you traffic from prefixes he doesn't want to announce to you? There are such customers.
The whole point of BCP38 is that this isn't supposed to happen.
Unfortunately we are living in reality.
Tell that to the customers with the unrealistic wishes.
Yes, these restrictions are a huge pain in the rear end but a denial of service without even the possibility to tell where the packets come from is MUCH worse.
What you actually want to know is what the ingress interfaces for the flows are.
For me, this has never been a big problem. (Not saying it isn't for anyone else.)
And if the ingress interface is not a p2p interface, from which peer.
Sure, that helps, but it doesn't shut up the packets. With real addresses you can build filters and/or contact the source. Yes, both are hard to do. But with spoofed sources there is pretty much nothing you can do except hope that your transit choices were good ones and they'll investigate.
Given that most DDoSses are mounted via huge zombie collections, there is not much point in knowing the real source IPs.
If the bottleneck isn't your ingress it is possible to filter tens of thousands of real sources. If the sources are fake you need to do much more destructive filtering.
And what do you do with a BGP customer which sends you traffic from prefixes he doesn't want to announce to you? There are such customers.
The whole point of BCP38 is that this isn't supposed to happen.
no. the whole point of BCP38 is that if this is supposed to happen, you (the upstream) should have to take explicit, non-default action which would probably include a source-address ACL, or static routes, or something. -- Paul Vixie
On Thu, 14 Oct 2004, Daniel Roesen wrote:
On Thu, Oct 14, 2004 at 08:05:50AM +0300, Pekka Savola wrote:
know if others than Juniper are providing that), you can enable strict uRPF toward those customers, still de-pref them, and accept the packets with correct source addresses.
That's what we do with our customers whether multihomed or not.
And what do you do with a BGP customer which sends you traffic from prefixes he doesn't want to announce to you? There are such customers. Fail filter ACL?
This has been my question with uRPF from the beginning. You can solve this on for some networks, but it doesn't scale very well. Especially where you really don't know that your customer's customer is doing this.
And what do you do with a BGP customer which sends you traffic from prefixes he doesn't want to announce to you? There are such customers. Fail filter ACL?
This has been my question with uRPF from the beginning. You can solve this on for some networks, but it doesn't scale very well. Especially where you really don't know that your customer's customer is doing this.
It's 2004, and so, your customers who want to do this have to explain why, and you have to maintain extra-ordinary filters for such customers, at either your cost or the customer's cost. -- Paul Vixie
On Fri, 15 Oct 2004, Paul Vixie wrote:
And what do you do with a BGP customer which sends you traffic from prefixes he doesn't want to announce to you? There are such customers. Fail filter ACL?
This has been my question with uRPF from the beginning. You can solve this on for some networks, but it doesn't scale very well. Especially where you really don't know that your customer's customer is doing this.
It's 2004, and so, your customers who want to do this have to explain why, and you have to maintain extra-ordinary filters for such customers, at either your cost or the customer's cost.
ah-ha! Patriot-Act!
participants (30)
-
alex＠yuriev.com -
bmanning＠vacation.karoshi.com -
Bora Akyol -
Christopher L. Morrow -
Dan Hollis -
Daniel Roesen -
Daniel Senie -
David G. Andersen -
Edward B. Dreger -
Fergie (Paul Ferguson) -
Fred Baker -
Hank Nussbacher -
Iljitsch van Beijnum -
Joe Abley -
Joe Provo -
Jon Lewis -
joshua sahala -
JP Velders -
Kevin Oberman -
Michael.Dillon＠radianz.com -
Niels Bakker -
Patrick W Gilmore -
Paul Vixie -
Pekka Savola -
Randy Bush -
Richard A Steenbergen -
Stephen J. Wilcox -
Stephen Stuart -
Steven Champeon -
Suresh Ramasubramanian