IAB concerns against permanent deployment of edge-based filtering
IAB concerns against permanent deployment of edge-based filtering The IAB notes that there ISPs/ASes undertaking permanent deployment of edge-based protocol number/port number packet filtering on traffic received from eBGP peers. As a short term response to security incidents this is a prudent operational measure that limits the spread of various forms of attack, and also mitigates some level of risk associated with network vulnerabilities. For example, many ISPs installed temporary filters in response to a July 2003 security advisory for CISCO routers ( http://www.cert.org/advisories/CA-2003-15.html). In the case of this incident PIM (protocol # 103) and mobile-ip4 (55) packets could trigger the vulnerability. The operational community responded with widespread deployment of filters at AS borders for these protocol numbers. Because of this, PIM and mobile-ip4 no longer work across such AS borders. The IAB is concerned about the practice of the permanent deployment of such traffic filters, since this could block the operation of certain applications in current use, as well as limiting the potential for deployment of future applications. Such filters ultimately limit extensibility of the Internet protocol as well as the Internet itself. It is an entirely appropriate and operationally prudent response to filter at the AS border as a short term mitigation of various network vulnerabilities. However, filters at AS borders do not provide any more than a relatively short term mitigation, and certainly do not solve the real problem of eliminating all forms of exploitation of such vulnerabilities. Over time knowledge of a vulnerability spreads across the network and potential exploiters of a vulnerability will be within an ISP/AS as well as being on the outside. The only stable and appropriate longer term operational response is to upgrade network equipment to eliminate the vulnerability, rather than attempting to configure packet filters intended to prevent externally located third parties from exploiting it. While short term traffic filters are deployed, the appropriate recommended longer term action is to: - To install filters to detect packets that are directed to the router itself to protect the router. (do not filter traffic that goes through the routers). - To update router firmware to a version known to eliminate the vulnerability Regards, Jun-ichiro itojun Hagino, on behalf of IAB (iab@ietf.org)
Jun-ichiro itojun Hagino wrote:
While short term traffic filters are deployed, the appropriate recommended longer term action is to:
Edge networks have a lot more to upgrade than backbone networks. Obtaining IOS code that works for all the different types of routers and meets the ISP's policy is not an easy task. Some files had to be specifically requested from the vendor as they no longer supported the version and didn't have pre-compiles. In addition, there are "other" bugs in IOS which must be considered for each application of a router deployment. This must be tested and monitored at one application locale and once verified can be deployed for all similar applications. In some cases,there were no alternatives, and this forced hardware upgrades at various locations (ie, memory). Such upgrades require money and time on the part of the edge customers. The peering blocks allow for limited protection of a majority of the customers while they continue to repair their networks. RPC blocks will be much worse, as the storm is still pretty loud, and low bandwidth customers cannot handle the extra noise. -Jack
On Sat, 18 Oct 2003, Jack Bates wrote:
Jun-ichiro itojun Hagino wrote:
While short term traffic filters are deployed, the appropriate recommended longer term action is to:
Edge networks have a lot more to upgrade than backbone networks. [...]
Agreed, but when an edge network fails to upgrade, it just harms itself. Backbone networks harms everyone concerned. It's good to remember who bears the pain for (in)action in whichever case. -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
why the heck does the IAB think they should tell me how to run my network?
i don't think expressing concerns id telling you what you MUST do. but all in all i suspect the iab's motivations were because your network (btw, which one is that?) is part of the INTERnet, and we would like it all to interoperate end to end. you have been here long enough to remember when the internet was a cooperation between operators, yes? randy
In a message written on Sat, Oct 18, 2003 at 07:39:37AM -0700, bmanning@karoshi.com wrote:
why the heck does the IAB think they should tell me how to run my network?
I think the IAB has a legitimate point. Network operators rely today on the fact that different services use different ports, so they can block particular types of access/behavior by blocking ports. However, this behavior has already started to change how applications work. We've all seen the streaming media clients, or IM programs that will use port 80 to get past a firewall, even though they are not http traffic. Virus writers have done the same thing. New VPN technologies use SSL, on the SSL web server port, but then send IP packets over them, not web requests. There is a real danger that long-term continued blocking will lead to "everything on one port" (probably 80). This will have the end result that ISP's will be unable to filter out the bad traffic anymore by using a port based filter, nor will they be able to collect statistics or other usage data. Additionally, this moves the problem up the stack as if everything runs on port 80 some "intelligent" demuxer will be needed at a higher layer for a box that wants to run multiple services. I'm not saying ISP's shouldn't filter, but the long term filtering is a problem. It will cause application developers to do things that will make long term filtering not work, in the end. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
I think the IAB has a legitimate point.
Network operators rely today on the fact that different services use different ports, so they can block particular types of access/behavior by blocking ports.
I think the IAB has a legitimate point and I agree with it 100%. Unfortunately, I also think it lacks a certain amount of practicality. When/if I remove the microsoft port filters (for example) from the interface between my campus and my eBGP peers or between segments of our campus network, my network starts to melt down because of the sudden influx of virus probes and traffic related to the spike in infections on our 10k - 20k hosts. If the recommendation is to remove these low-cost protections, is there a recommendation on how I can prevent the subsequent and severe instability on my network?
There is a real danger that long-term continued blocking will lead to "everything on one port" (probably 80).
Or port 443. If the traffic is on port 80, most signature systems can determine that its not necessarily a standard HTTP interaction. If its on port 443 and has a basic level of encryption, the signature-based systems fail.
I'm not saying ISP's shouldn't filter, but the long term filtering is a problem. It will cause application developers to do things that will make long term filtering not work, in the end.
I absolutely agree. Its the same argument that we made to our administration regarding why we shouldn't block outright peer-to-peer applications. First, the applications themselves aren't a problem, aren't illegal and, given that we are a University, we should try not to stifle their developement. Just as important, if we blocked them outright, our community would likley shift to applications that are more effective at hiding themselves from us. Since the only drawback to allowing them is that they increase the average bandwidth demand per user, something we plan for anyway, we chose not to filter. Unfortunately, I can't make the same argument about the edge port filters we have in place for security reasons. Though there is a general benifit gained by allowing application development, the overwhelming cost that we'd incur dealing with the compromised hosts themselves, the substantial increase in network traffic and network attacks that they generate, etc., makes removing these protections cost prohibitive. Again, I definitely agree with the IAB's recommendation. However, its difficult to defend this point of view in practice since most of the equipment does basic packet filtering in hardware or with minimal cost to peformance. So, I just can't figure out how to sit in front of our administration and justify the replacement of a zero-cost solution with the cost of added staff and equipment to mitigate these security risks, especially when the up side is just not "limiting the potential for deployment of future applications". Eric :)
In a message written on Sat, Oct 18, 2003 at 12:26:21PM -0400, Eric Gauthier wrote:
Again, I definitely agree with the IAB's recommendation. However, its difficult to defend this point of view in practice since most of the equipment does basic packet filtering in hardware or with minimal cost to peformance. So, I just can't figure out how to sit in front of our administration and justify the replacement of a zero-cost solution with the cost of added staff and equipment to mitigate these security risks, especially when the up side is just not "limiting the potential for deployment of future applications".
Well, but you've hit the nail on the head. The fitler solution is _NOT_ zero cost, it is deferred cost. I suggest you phrase it that way. It's a way of deferring the cost to later, with interest. The longer you use it, the higher that interest payment will be, in the form of new and different attacks you can't block. Phrasing it to the bean counters that it is deferring the cost, with interest, and suggesting that simultaneously some money be spent on user education, better software, or whatever is appropriate to insure you don't have a "huge baloon payment" later might help put it in terms they can understand. Similar parallels can be drawn to antibiotics -- the over use will eventually render them ineffective. It's a very similar situation, and sometimes you have to just invest in not getting sick in the first place (wash your hands...patch your system). -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
I think the IAB has a legitimate point.
perhaps. but last I checked, it was the Internet Architecture Board not the Internet Operations Board. So form an architectural purity perspective, sure, don't filter (and by extention, pull out firewalls and NATS.... :)
There is a real danger that long-term continued blocking will lead to "everything on one port"
fair amount of handwaving there. prudent/paranoid folk over the years have persuaded me that it makes the best sense to only run those applications/services that I need to and shut off everything else - until/unless there is a demonstrated need for it. --bill
On Sat, 18 Oct 2003 11:14:42 PDT, bmanning@karoshi.com said:
There is a real danger that long-term continued blocking will lead to "everything on one port"
fair amount of handwaving there.
Question: Why was RFC3093 published? (Think(*) for a bit here...) About a month later, there was a *major* flame-fest on the IETF list due to this message: http://www.ietf.org/mail-archive/ietf/Current/msg11918.html Yes, the basic reason for this proposal was because many firewalls will pass HTTP but not BEEP. What major P2P applications have included a "run over port 80" option to let themselves through firewalls? It's not just handwaving. (*) Remember - satire isn't funny if it isn't about something recognizable...
There is a real danger that long-term continued blocking will lead to "everything on one port" fair amount of handwaving there.
Question: Why was RFC3093 published? (Think(*) for a bit here...) About a month later, there was a *major* flame-fest on the IETF list due to this message: http://www.ietf.org/mail-archive/ietf/Current/msg11918.html
What major P2P applications have included a "run over port 80" option to let themselves through firewalls?
lots.
It's not just handwaving.
the handwaving is based on some presumption about what is on the other side of the "port 80" spiggot. what other services are enabled on your systems that listen to port 80? do you have systems that don't speak/listen on port 80?
(*) Remember - satire isn't funny if it isn't about something recognizable...
... to someone. :) --bill
Valdis hits the nail on the head. And this boils down to something that I believe is attributable to someone commenting on the old FSP protocol, perhaps Erik Fair: The Internet routes around damage. Damage can take the form of a broken link, or it can take the form of an access-list. In the early '90s, NASA attempted to protect its links from "unauthorized use" (which in this particular case was porn). That caused a whole protocol to be developed (proving the old adage). Well, nowadays you don't even need to build a whole protocol- you can just use HTTP. And that was the point of Keith's & Ned's RFC on HTTP as a substrate. Excessive restrictions in firewalls bring about this use, and that makes the HTTP implementations fairly complex, and it will subvert the intentions of network administrators. So as a temporary measure during an active attack, access-lists make sense. Over the long haul, however, unless you're going to block downstream TCP packets with SYN only and ALL OTHER TRAFFIC, IP can run on just about anything. Eliot Valdis.Kletnieks@vt.edu wrote:
On Sat, 18 Oct 2003 11:14:42 PDT, bmanning@karoshi.com said:
There is a real danger that long-term continued blocking will lead to "everything on one port"
fair amount of handwaving there.
Question: Why was RFC3093 published? (Think(*) for a bit here...)
About a month later, there was a *major* flame-fest on the IETF list due to this message:
http://www.ietf.org/mail-archive/ietf/Current/msg11918.html
Yes, the basic reason for this proposal was because many firewalls will pass HTTP but not BEEP.
What major P2P applications have included a "run over port 80" option to let themselves through firewalls?
It's not just handwaving.
(*) Remember - satire isn't funny if it isn't about something recognizable...
Date: Sat, 18 Oct 2003 11:14:42 -0700 (PDT) From: bmanning@...
perhaps. but last I checked, it was the Internet Architecture Board not the Internet Operations Board. So form an architectural purity perspective, sure, don't filter (and by extention, pull out firewalls and NATS.... :)
Ports < 1024 are "privileged" and tend not to be used as a source port for outgoing packets. This in turn affects packet filters. Life might be easier if a port range had been reserved for passive FTP connections. It would seem architecture and operations are at least somewhat coupled. Should there not be interaction between the two? "Here is what we built; deal with it!" doesn't appeal to me. (Judging from the wildcard threads, it doesn't seem to appeal to others, either.) I'd like the arch folks to listen to the ops crowd, and I see no reason why it shouldn't go the other way too. Eddy -- Brotsman & Dreger, Inc. - EverQuick Internet Division Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _________________________________________________________________ DO NOT send mail to the following addresses : blacklist@brics.com -or- alfra@intc.net -or- curbjmp@intc.net Sending mail to spambait addresses is a great way to get blocked.
prudent/paranoid folk over the years have persuaded me that it makes the best sense to only run those applications/services that I need to and shut off everything else - until/unless there is a demonstrated need for it.
very true for a host, even somewhat true for a site. very untrue for a backbone. randy
prudent/paranoid folk over the years have persuaded me that it makes the best sense to only run those applications/services that I need to and shut off everything else - until/unless there is a demonstrated need for it.
very true for a host, even somewhat true for a site. very untrue for a backbone.
randy
there appears to be a disconnect in the wording of the IAB document: it starts: ---- IAB concerns against permanent deployment of edge-based filtering The IAB notes that there ISPs/ASes undertaking permanent deployment of edge-based protocol number/port number packet filtering on traffic received from eBGP peers. ---- it can be viewed from the perspective of a transit provider looking toward its edges, the clients. it can be viewed from the perspective of a multihomed client looking toward its edges, the transit providers. which one you take depends on where you start... :) then there is the idea of "permanent" deployment ... little is permanent in networking. the hard problem is when vendors put filters in silicon. :( --bill
prudent/paranoid folk over the years have persuaded me that it makes the best sense to only run those applications/services that I need to and shut off everything else - until/unless there is a demonstrated need for it. very true for a host, even somewhat true for a site. very untrue for a backbone. there appears to be a disconnect in the wording of the IAB document: it starts:
IAB concerns against permanent deployment of edge-based filtering
The IAB notes that there ISPs/ASes undertaking permanent deployment of edge-based protocol number/port number packet filtering on traffic received from eBGP peers. ---- it can be viewed from the perspective of a transit provider looking toward its edges, the clients.
it can be viewed from the perspective of a multihomed client looking toward its edges, the transit providers.
which one you take depends on where you start... :)
then there is the idea of "permanent" deployment ... little is permanent in networking. the hard problem is when vendors put filters in silicon. :(
i have been assuming, possibly quite incorrectly, that the iab concern was with backbone providers. possibly this is due to my perspective. imiho, backbones move packets, and the more we muck with them the less happier our customers are. but i filter like hell at my personal site edge, and do try to keep unwanted things off my hosts. randy
On Mon, Oct 20, 2003 at 05:00:58AM -0700, bmanning@karoshi.com <bmanning@karoshi.com> wrote a message of 35 lines which said:
then there is the idea of "permanent" deployment ... little is permanent in networking. the hard problem is when vendors put filters in silicon. :(
I worked in network service company ("Setting up a firewall in one morning on a router you do not know"). My definition of "permanent" is "something installed by someone who left just after".
OK... I've been lurking for a while. I think the definition IAB intended to express concern about was: Backbones (transit providers) deploying [permanent] filtration on their connections with other ISPs. I would like to propose the following terminology definitions FOR THIS EMAIL message and ask that my following comments be viewed with these definitions in mind: "Edge Network" A network which does not provide transit between multiple BGP speaking neighboring ASs. "End Network" Or "End User Network" a network which has may or may not speak BGP, but, provides services to a single organization and has a single administrative control at it's border(s). (e.g. Sun Microsystems, Tellme, HP, etc., not MSN, C&W, Verio, etc.) "Transit Network" A network which does not meet the definitions of Edge Network or End Network. I think given those terms, there is generally agreement that the following are good operational practice: 1. Edge Networks and End Networks should not emit packets containing source address specifications outside of their assigned (or allocated) ranges. They should employ filters at their peering-points to prevent this. 2. Transit Networks should _NOT_ permanently (or quasi-permanently) block traffic to other transit networks other than to the minimal extent necessary to meet operational considerations around attacks. The general deployment of such filters would in itself be a form of denial of service. 3. End networks should accept and emit traffic related to their desired service profile (what internet features they want to take advantage of) and block others. 4. Any network connecting to an Edge or End network may (and in some cases should) cooperate in filtering traffic at conneciton-points to said Edge or End networks in a manner consistent with the desires of the Edge or End network in question. To do so contrary to the wishes of the Edge or End network in question is a form of denial of service. 5. It is generally good practice for any network providing services to Edge or End networks to have a published AUP and to disconnect customers which violate the AUP. This is not contrary to the wishes of the client, or they should not have signed the AUP. Having said that, I don't think IAB is trying to tell people how to run their networks. I do think IAB has a point that if I'm connected to an ISP which is a customer of UUNET, and I want to exchange traffic of some unpopular service with another host that is a connected via an ISP that is a customer of C&W, it is a bad thing if C&W and UUNET block that traffic at their peering point(s). If my ISP blocks it or the ISP that connects the other host blocks it, then, presumably, I (or the person at the other end of the connection) have some ability to address it with the service provider we are paying. Having UUNET or C&W block it at some arbitrary point in between is: 1. Hard to isolate. 2. Hard to troubleshoot. 3. Unexpected damage 4. Not a good idea in most cases. Assuming that this is what IAB was attempting to address, I agree it should be addressed. The fact that I need to make this assumption should, IMHO, also be addressed by IAB and they should clarify what their concern is. Owen --On Monday, October 20, 2003 05:00:58 AM -0700 bmanning@karoshi.com wrote:
prudent/paranoid folk over the years have persuaded me that it makes the best sense to only run those applications/services that I need to and shut off everything else - until/unless there is a demonstrated need for it.
very true for a host, even somewhat true for a site. very untrue for a backbone.
randy
there appears to be a disconnect in the wording of the IAB document: it starts: ---- IAB concerns against permanent deployment of edge-based filtering
The IAB notes that there ISPs/ASes undertaking permanent deployment of edge-based protocol number/port number packet filtering on traffic received from eBGP peers. ---- it can be viewed from the perspective of a transit provider looking toward its edges, the clients.
it can be viewed from the perspective of a multihomed client looking toward its edges, the transit providers.
which one you take depends on where you start... :)
then there is the idea of "permanent" deployment ... little is permanent in networking. the hard problem is when vendors put filters in silicon. :(
--bill
At 10:57 AM -0700 10/20/03, Owen DeLong wrote:
OK... I've been lurking for a while.
I think the definition IAB intended to express concern about was:
Backbones (transit providers) deploying [permanent] filtration on their connections with other ISPs.
I would like to propose the following terminology definitions FOR THIS EMAIL message and ask that my following comments be viewed with these definitions in mind:
What you're doing here reminds me of what we did in the BGP Convergence Technology draft, http://www.ietf.org/draft-ietf-bmwg-conterm-05.txt (now with the IESG). We made what we felt was a useful but informal distinction between customer edge routers at the POP, and interprovider edge routers. Randy Bush, as the advisor, pointed out these definitions are not rigorous, and indeed might be considered a research problem. We modified our language to reflect his concerns, and that we didn't expect the definitions to be normative. Nevertheless, there seems a practical need, which you put as a policy consideration here, to distinguish between routers that connect an ISP to transit and nontransit peers. The nontransit peers often will not be taking full routes.
"Edge Network" A network which does not provide transit between multiple BGP speaking neighboring ASs.
"End Network" Or "End User Network" a network which has may or may not speak BGP, but, provides services to a single organization and has a single administrative control at it's border(s). (e.g. Sun Microsystems, Tellme, HP, etc., not MSN, C&W, Verio, etc.)
"Transit Network" A network which does not meet the definitions of Edge Network or End Network.
I think given those terms, there is generally agreement that the following are good operational practice:
1. Edge Networks and End Networks should not emit packets containing source address specifications outside of their assigned (or allocated) ranges. They should employ filters at their peering-points to prevent this.
2. Transit Networks should _NOT_ permanently (or quasi-permanently) block traffic to other transit networks other than to the minimal extent necessary to meet operational considerations around attacks. The general deployment of such filters would in itself be a form of denial of service.
3. End networks should accept and emit traffic related to their desired service profile (what internet features they want to take advantage of) and block others.
4. Any network connecting to an Edge or End network may (and in some cases should) cooperate in filtering traffic at conneciton-points to said Edge or End networks in a manner consistent with the desires of the Edge or End network in question. To do so contrary to the wishes of the Edge or End network in question is a form of denial of service.
5. It is generally good practice for any network providing services to Edge or End networks to have a published AUP and to disconnect customers which violate the AUP. This is not contrary to the wishes of the client, or they should not have signed the AUP.
Having said that, I don't think IAB is trying to tell people how to run their networks. I do think IAB has a point that if I'm connected to an ISP which is a customer of UUNET, and I want to exchange traffic of some unpopular service with another host that is a connected via an ISP that is a customer of C&W, it is a bad thing if C&W and UUNET block that traffic at their peering point(s). If my ISP blocks it or the ISP that connects the other host blocks it, then, presumably, I (or the person at the other end of the connection) have some ability to address it with the service provider we are paying. Having UUNET or C&W block it at some arbitrary point in between is:
1. Hard to isolate. 2. Hard to troubleshoot. 3. Unexpected damage 4. Not a good idea in most cases.
Assuming that this is what IAB was attempting to address, I agree it should be addressed. The fact that I need to make this assumption should, IMHO, also be addressed by IAB and they should clarify what their concern is.
Owen
--On Monday, October 20, 2003 05:00:58 AM -0700 bmanning@karoshi.com wrote:
prudent/paranoid folk over the years have persuaded me that it makes the best sense to only run those applications/services that I need to and shut off everything else - until/unless there is a demonstrated need for it.
very true for a host, even somewhat true for a site. very untrue for a backbone.
randy
there appears to be a disconnect in the wording of the IAB document: it starts: ---- IAB concerns against permanent deployment of edge-based filtering
The IAB notes that there ISPs/ASes undertaking permanent deployment of edge-based protocol number/port number packet filtering on traffic received from eBGP peers. ---- it can be viewed from the perspective of a transit provider looking toward its edges, the clients.
it can be viewed from the perspective of a multihomed client looking toward its edges, the transit providers.
which one you take depends on where you start... :)
then there is the idea of "permanent" deployment ... little is permanent in networking. the hard problem is when vendors put filters in silicon. :(
--bill
participants (13)
-
bmanning＠karoshi.com -
E.B. Dreger -
Eliot Lear -
Eric Gauthier -
Howard C. Berkowitz -
Jack Bates -
Jun-ichiro itojun Hagino -
Leo Bicknell -
Owen DeLong -
Pekka Savola -
Randy Bush -
Stephane Bortzmeyer -
Valdis.Kletnieks＠vt.edu