RE: Where are static bogon filters appropriate? was: 96.2.0.0/16 Bogons
No, the SP can't be the 'Internet firewall' for customers,
They can if the SP supplies and manages the CPE device. Nowadays, a lot of functionality could potentially be provided in a CPE device. Hardware cost and hardware capabilities are no longer barriers to doing this. There is still software cost, of course, but that can get amortized over many years and many, many subscribers.
but protecting one's customers from being spammed/packeted from purported source addresses which are by definition nonsensical (as well as protecting everyone else's customers from same) doesn't seem much to ask.
What's needed here are better/easier/less brittle mechanisms for same. Until such time as they're invented and deployed, let's not make the perfect the enemy of the merely good, yes?
Less brittle mechanisms don't need to be invented. There is nothing hard about putting an operational process in place to manage bogon filtering. There is also nothing hard about writing some software to update bogon filters. People do this kind of thing every day in network operations without much fanfare precisely because it is nothing special. It's just good operational practice in a company that does not expect its vendors to spoonfeed them everything. There's kind of a disease in the telecomms business. Many years ago someone got the idea that developing telecomms devices and systems was one business, and operating those devices and systems was another. Over time this crystalized into the idea that an operator bought platforms from vendors and then operated those platforms according to the vendor's instructions. This may be good for the capital investment industry (banks etc.) but it fails when there is a need for creativity in the telecomms operator. Sometimes, network operators have to take the bull by the horns and develop their own systems to do a job that vendors simply don't understand. --Michael Dillon
On Mar 2, 2007, at 7:31 AM, <michael.dillon@bt.com> wrote:
Sometimes, network operators have to take the bull by the horns and develop their own systems to do a job that vendors simply don't understand.
Concur - but it seems that many seem to be looking for someone else to do this for them (or, perhaps, the lack of someone to do it for them as an excuse to do nothing at all). ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice The telephone demands complete participation. -- Marshall McLuhan
On Fri, 2 Mar 2007, Roland Dobbins wrote:
Sometimes, network operators have to take the bull by the horns and develop their own systems to do a job that vendors simply don't understand.
Concur - but it seems that many seem to be looking for someone else to do this for them (or, perhaps, the lack of someone to do it for them as an excuse to do nothing at all).
How much of a problem is traffic from unallocated addresses? Backbone operators probably have NetFlow data which they could mine to find out. On the other hand, how much of a problem is obsolete bogon filters causing everytime IANA delegates another block to an RIR? Or by the way, how much spoofed traffic uses allocated addresses?
I think Sean raises a good point. I guess the larger picture is what are we trying to protect and what are trying to protect that from. -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Sean Donelan Sent: Friday, March 02, 2007 3:19 PM To: Roland Dobbins Cc: NANOG Subject: Re: Where are static bogon filters appropriate? was: 96.2.0.0/16 Bogons On Fri, 2 Mar 2007, Roland Dobbins wrote:
Sometimes, network operators have to take the bull by the horns and develop their own systems to do a job that vendors simply don't understand.
Concur - but it seems that many seem to be looking for someone else to do this for them (or, perhaps, the lack of someone to do it for them as an excuse to do nothing at all).
How much of a problem is traffic from unallocated addresses? Backbone operators probably have NetFlow data which they could mine to find out. On the other hand, how much of a problem is obsolete bogon filters causing everytime IANA delegates another block to an RIR? Or by the way, how much spoofed traffic uses allocated addresses?
On Fri, 2 Mar 2007 15:37:01 -0600 "Eric Ortega" <eric.ortega@midco.net> wrote:
I think Sean raises a good point. I guess the larger picture is what are we trying to protect and what are trying to protect that from.
Bingo. The problem isn't with "security people", it's with "security people who use checklists -- and old ones at that -- in preference to thinking". Droids exist in every field.... --Steve Bellovin, http://www.cs.columbia.edu/~smb
At 04:18 PM 3/2/2007, Sean Donelan wrote:
On Fri, 2 Mar 2007, Roland Dobbins wrote:
Sometimes, network operators have to take the bull by the horns and develop their own systems to do a job that vendors simply don't understand.
Concur - but it seems that many seem to be looking for someone else to do this for them (or, perhaps, the lack of someone to do it for them as an excuse to do nothing at all).
How much of a problem is traffic from unallocated addresses? Backbone operators probably have NetFlow data which they could mine to find out. On the other hand, how much of a problem is obsolete bogon filters causing everytime IANA delegates another block to an RIR?
Or by the way, how much spoofed traffic uses allocated addresses?
How do you know, if you're the one being attacked and you have no idea if the originating network or their immediate upstream implemented BCP38? Shall we just discard ingress filtering? If few attacks are using it today, should we declare it no longer relevant? At the same time we should ask if we should be x-raying shoes at the airport, since there's only been one guy who tried to blow up his shoes. The larger security question is, "do you stop looking for old threats simply because they're not the most common threats?" How many CodeRed packets flow over the Internet on a typical day? I assure you it's not zero. The initial drafts of the document that became BCP38 were written 10 1/2 years ago, triggered by a serious problem of spoof-based attacks that were causing serious problems including serious interruption of services. The problem had a solution, but one that required cooperation among networks. The operation of the entire Internet required cooperation among networks. I don't know to what degree any sense of cooperation is left these days. Probably won't matter when Google or ATT take over the whole thing. In the mean time, the presence of an ACL line or two at the border of each edge network is NOT a significant burden. Yes, Cisco and others have implemented uRPF that can do the same thing with a bit less typing in some cases. I really don't care which mechanism is used. I do care when my network is hammered with packets. When I send reports to other networks and they can't be sure the packets came from their networks, that's not helpful. So there, that's my rant about why we might all want to try and keep the 'net a cooperative place, and a bit about how ingress filtering continues to play a part in that cooperation. This is pretty far from the topic of the bogon list issue with 96/8 though.
On Fri, 2 Mar 2007, Daniel Senie wrote:
How do you know, if you're the one being attacked and you have no idea if the originating network or their immediate upstream implemented BCP38? Shall we just discard ingress filtering? If few attacks are using it today, should we declare it no longer relevant? At the same time we should ask if we should be x-raying shoes at the airport, since there's only been one guy who tried to blow up his shoes. The larger security question is, "do you stop looking for old threats simply because they're not the most common threats?" How many CodeRed packets flow over the Internet on a typical day? I assure you it's not zero.
Show me the data. How many CodeRed packets originate from unallocated addresses? Is the proposal actually effective at detecting or protecting against the threat? Or is it just a wasted effort for show? http://www.tsa.gov/press/happenings/kip_hawley_x-ray_remarks.shtm Instead of dropping packets with unallocated sources addresses, perhaps backbones should shutdown interfaces they receive packets from unallocated address space. Would this be more effective at both stopping the sources of unallocated addresses; as well as sources that spoof other addresses because the best way to prevent your interface from being shutdown by backbone operators is to be certain you only transmit packets with your source addresses.
http://www.completewhois.com/hijacked/files/203.27.251.0.txt http://www.completewhois.com/hijacked/index.htm This can proof the opposite. Malware comes from redirected allocated blocks, not from bogons. Kind regards Peter and Karin Sean Donelan wrote:
On Fri, 2 Mar 2007, Daniel Senie wrote:
How do you know, if you're the one being attacked and you have no idea if the originating network or their immediate upstream implemented BCP38? Shall we just discard ingress filtering? If few attacks are using it today, should we declare it no longer relevant? At the same time we should ask if we should be x-raying shoes at the airport, since there's only been one guy who tried to blow up his shoes. The larger security question is, "do you stop looking for old threats simply because they're not the most common threats?" How many CodeRed packets flow over the Internet on a typical day? I assure you it's not zero.
Show me the data.
How many CodeRed packets originate from unallocated addresses?
Is the proposal actually effective at detecting or protecting against the threat? Or is it just a wasted effort for show?
http://www.tsa.gov/press/happenings/kip_hawley_x-ray_remarks.shtm
Instead of dropping packets with unallocated sources addresses, perhaps backbones should shutdown interfaces they receive packets from unallocated address space. Would this be more effective at both stopping the sources of unallocated addresses; as well as sources that spoof other addresses because the best way to prevent your interface from being shutdown by backbone operators is to be certain you only transmit packets with your source addresses.
-- Peter and Karin Dambier Cesidian Root - Radice Cesidiana Rimbacher-Strasse 16 D-69509 Moerlenbach-Bonsweiher +49(6209)795-816 (Telekom) +49(6252)750-308 (VoIP: sipgate.de) mail: peter@peter-dambier.de mail: peter@echnaton.serveftp.com http://iason.site.voila.fr/ https://sourceforge.net/projects/iason/ http://www.cesidianroot.com/
http://www.completewhois.com/hijacked/files/203.27.251.0.txt
http://www.completewhois.com/hijacked/index.htm
This can proof the opposite.
Malware comes from redirected allocated blocks, not from bogons.
I don't think this is proof. The haphazard way that BCP38 and ingress prefix filtering of Bogon/DUSA make 'spoofing' from these Bogon/DUSA blocks unprofitable to a miscreant and forces them to work too hard. What this data does demonstrate is that hijacking of valid prefixes has not been mitigated. And, there is most likely an economic motivation to use the hijacked prefixes. In other words, the miscreants can use the technique over and over - not get caught - not work too hard - and make money (the first three and most important principles of miscreants). This data points to another problem - where SPs are not putting ingress prefix filters on their BGP speaking customers. That is another area where you have a lot of operational entropy issues. OPEX is increased on the building of the prefix provision tool, the maintenance of the policy, synchronization of that policy with the peer ingress filters, and customers calls required when ever the customer gets prefix updates. Hence many (most) operators rather not do the prefix filters on their customers (usually 2 to 4 lines of policy on a J & C router). For many, the OPEX is just too high.
On Sat, 3 Mar 2007, Sean Donelan wrote:
Instead of dropping packets with unallocated sources addresses, perhaps backbones should shutdown interfaces they receive packets from unallocated address space. Would this be more effective at both stopping the sources of unallocated addresses; as well as sources that spoof other addresses because the best way to prevent your interface from being shutdown by backbone operators is to be certain you only transmit packets with your source addresses.
uRPF or plain source-based filtering for the IP blocks allocated to the customer is enough. Shutting it down doesn't make any commercial sense, customers wont buy your service if their port is going to be shut down due to a single packet. They'll (likely) understand if you won't forward a packet from them which has a source address not not belonging to them, though. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Sun, 4 Mar 2007, Mikael Abrahamsson wrote:
Instead of dropping packets with unallocated sources addresses, perhaps backbones should shutdown interfaces they receive packets from unallocated address space. Would this be more effective at both stopping the sources of unallocated addresses; as well as sources that spoof other addresses because the best way to prevent your interface from being shutdown by backbone operators is to be certain you only transmit packets with your source addresses.
uRPF or plain source-based filtering for the IP blocks allocated to the customer is enough. Shutting it down doesn't make any commercial sense, customers wont buy your service if their port is going to be shut down due to a single packet. They'll (likely) understand if you won't forward a packet from them which has a source address not not belonging to them, though.
When customers misconfigure their router, e.g. wrong BGP neighbor or ASN, wrong interface IP address, exceed max prefix limit, etc; don't they lose Internet connectivity until they fix it? A properly configure router should never forward even a single bad packet. If it does, isn't it likely to have configuration problems so why continue to keep misconfigured routers connected? Customers are unlikely to fix problems which don't cause them to lose service.
On Sun, 4 Mar 2007, Sean Donelan wrote:
When customers misconfigure their router, e.g. wrong BGP neighbor or ASN, wrong interface IP address, exceed max prefix limit, etc; don't they lose Internet connectivity until they fix it?
A properly configure router should never forward even a single bad packet. If it does, isn't it likely to have configuration problems so why continue to keep misconfigured routers connected?
Customers are unlikely to fix problems which don't cause them to lose service.
Even though the BOFH in me agrees with you, I also know that every cent on my paycheck comes from the customers, so I prefer not to treat them like crap. If I can protect the internet from my customers by doing uRPF or source IP based filtering, I achieve the same thing as you but with less customer impact. Also, all the examples you give implies a BGP transit customer. I am imagining all kinds of customers, from colo customers where I am their default gateway, to residential customers where it's the same way. Disabling their port and punting them to customer support is NOT a cost efficient way of dealing with the problems, at least not in the market I am in. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Tue, 6 Mar 2007, Mikael Abrahamsson wrote:
Also, all the examples you give implies a BGP transit customer. I am imagining all kinds of customers, from colo customers where I am their default gateway, to residential customers where it's the same way.
I tried to give examples upstream of a router, not a bridged/direct connection which may have all sorts of unroutable junk which a router should not (and mostly doesn't) forward. Although spoofing MAC addresses is probably suspicious behaivor in most bridged networks too.
Disabling their port and punting them to customer support is NOT a cost efficient way of dealing with the problems, at least not in the market I am in.
Isn't this true of everything (bad source addresses, worms, abuse, etc). Does hiding/ignoring the problem just makes it worse because there is no incentive to fix the problem while it is still a small problem? If it isn't important enough to bother the customer, why bother to fix it? How you stop forwarding bad stuff is a local decision. As long as you stop it, no one will turn off your interface. If your network is forwarding so many packets with false source addresses that it would be a major customer support cost issue to fix, your network probably has other configuration problems. You are probably just deferring those customer service costs until an unpredictable time in the future when those misconfigurations disrupt other parts of your network.
On Tue, 6 Mar 2007, Sean Donelan wrote:
Isn't this true of everything (bad source addresses, worms, abuse, etc). Does hiding/ignoring the problem just makes it worse because there is no incentive to fix the problem while it is still a small problem? If it isn't important enough to bother the customer, why bother to fix it?
Let's take a concrete example: Customer gets hacked, one of their boxen starts spewing traffic with spoofed addresses. The way I understand your solution is to automatically shut their port and disrupt all their traffic, and have them call customer support to get any further. Do you really think this is a good solution? I don't see any customer with a choice continuing having a relationship with me if I treat them like that. It will cost me and them too much. So instead I just drop their spoofed traffic and if they call and say that their line is slow, I'll just say it's full and they can themselves track down the offending machine and shut it off to solve the problem. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Tue, 06 Mar 2007 21:54:06 +0100, Mikael Abrahamsson said:
So instead I just drop their spoofed traffic and if they call and say that their line is slow, I'll just say it's full and they can themselves track down the offending machine and shut it off to solve the problem.
This doesn't sound very scalable. You're almost certainly overcommitted on the upstream side and likely looking at congestion if many customers are spewing. What do you tell the customer who calls and complains that *he* isn't a major traffic source, but he's seeing dropped packets and delays on your upstream link? Do you tell him its full and they can track down which other customer is the offender?
On Tue, 6 Mar 2007, Valdis.Kletnieks@vt.edu wrote:
On Tue, 06 Mar 2007 21:54:06 +0100, Mikael Abrahamsson said:
So instead I just drop their spoofed traffic and if they call and say that their line is slow, I'll just say it's full and they can themselves track down the offending machine and shut it off to solve the problem.
This doesn't sound very scalable. You're almost certainly overcommitted on the upstream side and likely looking at congestion if many customers are spewing.
If they're spewing spoofed traffic I'm dropping it, so that's not a problem.
What do you tell the customer who calls and complains that *he* isn't a major traffic source, but he's seeing dropped packets and delays on your upstream link? Do you tell him its full and they can track down which other customer is the offender?
Do you usually design networks that can't handle customers using what they have paid for? I don't. (for any reasonable amount of statistical oversubscripion of course) -- Mikael Abrahamsson email: swmike@swm.pp.se
Mikael Abrahamsson wrote:
On Tue, 6 Mar 2007, Sean Donelan wrote:
Isn't this true of everything (bad source addresses, worms, abuse, etc). Does hiding/ignoring the problem just makes it worse because there is no incentive to fix the problem while it is still a small problem? If it isn't important enough to bother the customer, why bother to fix it?
Let's take a concrete example:
Customer gets hacked, one of their boxen starts spewing traffic with spoofed addresses. The way I understand your solution is to automatically shut their port and disrupt all their traffic, and have them call customer support to get any further.
Do you really think this is a good solution?
I don't see any customer with a choice continuing having a relationship with me if I treat them like that. It will cost me and them too much.
So instead I just drop their spoofed traffic and if they call and say that their line is slow, I'll just say it's full and they can themselves track down the offending machine and shut it off to solve the problem.
Neither one is really all that good but both have merit - some compromises are in order. We shut them off only if it's causing serious problems. If we can mitigate the problem without shutting them off completely we will. The usual example is customers spewing spam on port 25. We block port 25 at the customers CPE and notify them as to why and how to work around the block (use webmail or submission) while they fix the problem. It's amazing how many customers are just plain OK with that and never do get around to fixing the machine - but at least they know that we blocked something for a reason. Anything you do silently tends to cause customers to decide 'you suck' and go elsewhere. Line is slow 'cause there machine is beating it to death? Just get a new provider. When the new one also sucks they either shrug and decide that's the way it is or finally fix the problem. Either way the customer is lost to you 'cause they won't come back even after they figure out it was their problem in the first place. Shutting them off causes churn, leaving problems silently in place also causes churn. The middle road mitigates damage and still manages to keep the customers happy (well.. that might be stretching it a bit...happier?). -- Mark Radabaugh Amplex mark@amplex.net 419.837.5015
On Tue, 6 Mar 2007, Mikael Abrahamsson wrote:
Customer gets hacked, one of their boxen starts spewing traffic with spoofed addresses. The way I understand your solution is to automatically shut their port and disrupt all their traffic, and have them call customer support to get any further.
Do you really think this is a good solution?
I don't see any customer with a choice continuing having a relationship with me if I treat them like that. It will cost me and them too much.
So instead I just drop their spoofed traffic and if they call and say that their line is slow, I'll just say it's full and they can themselves track down the offending machine and shut it off to solve the problem.
Compromised systems rarely have one thing wrong with them, and delaying the pain just makes things worse. Drop spoofed traffic, and they send non-spoofed packets. Block port 25, and they send slammer on port 1434 Block messenger port 1025, and they send DNS DOS on port 53 Block irc bots port 6667, and they send VOIP spam port 5060 and so on and so on. <http://www.washingtonpost.com/wp-dyn/content/article/2007/03/08/AR2007030802012.html> The fast-spreading virus infected as many as 200 county computers Wednesday, and technicians shut down the entire network for Anne Arundel offices for more than 24 hours. http://msmvps.com/blogs/donna/archive/2006/02/12/83332.aspx One day last year, things started going haywire at Northwest Hospital and Medical Center. Key cards would no longer open the operating-room doors; computers in the intensive-care unit shut down; doctors' pagers wouldn't work. It turns out the Seattle hospital's computers . along with up to 50,000 others across the country . had been turned into an army of robots controlled by 20-year-old Caused by "known" vulnerabilities with patches available, but the customers decided it wasn't "important" enough to take action before they lost everything. Is it really customer service to avoid the issue?
On Mar 2, 2007, at 1:18 PM, Sean Donelan wrote:
How much of a problem is traffic from unallocated addresses? Backbone operators probably have NetFlow data which they could mine to find out. On the other hand, how much of a problem is obsolete bogon filters causing everytime IANA delegates another block to an RIR?
Or by the way, how much spoofed traffic uses allocated addresses?
No one has done the digging required to answer any of these questions, unfortunately. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice The telephone demands complete participation. -- Marshall McLuhan
Speaking of bogons and more practical daily operation issues, perhaps you guys can help reaching the fine folks at AS7643 or maybe their upstream provider can be kind enough to filter out the following: BGP routing table entry for 123.0.0.0/8, version 14613827 Paths: (1 available, best #1, not advertised outside local AS) Not advertised to any peer 3561 3491 7643 7643 7643 7643 Thanks. --- William Leibzon Elan Networks william@elan.net
On 3/2/07, Roland Dobbins <rdobbins@cisco.com> wrote:
No one has done the digging required to answer any of these questions, unfortunately.
Can you get a valid answer to this based on the existence of BCP38? What I mean is, if your upstream is filtering bogons, you can't get a good read on the amount of "bad" traffic sourcing from "illegal" addresses. However, I'm sure it's there. If we stop filtering so-called "bad" addresses, I'm sure that the attacks from those addresses will increase when it's realized that the filters are gone. I agree with others in that you can't stop looking for old attacks just because they don't happen much anymore. But we can improve the ways we look. uRPF is definitely a dynamic option, but as I understood it, there were issues with using it on multi-homed networks with asynchronous routing. Granted, it has been some time since I've looked at uRPF. I think something like the Cymru bogon route server is great, but I'm not a very trusting person when it comes to something like that. I don't like giving up that level of control. Of course, at some point, I suppose have to trust something... I definitely believe in filtering both bogons and RFC 1918 space, it's just a management issue that has to be dealt with.
----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice
-- Jason 'XenoPhage' Frisvold XenoPhage0@gmail.com http://blog.godshell.com
participants (13)
-
Barry Greene (bgreene) -
Daniel Senie -
Eric Ortega -
Jason Frisvold -
Mark Radabaugh -
michael.dillon＠bt.com -
Mikael Abrahamsson -
Peter Dambier -
Roland Dobbins -
Sean Donelan -
Steven M. Bellovin -
Valdis.Kletnieks＠vt.edu -
william(at)elan.net