A smaller North American network provider, with a modest North American backbone, numbers their internal routers on public IP space that they do not announce to the world. One of the largest North American network providers filters/drops ICMP messages so that they only pass those with a source IP address that appears in their routing table. As a result, traceroutes from big.net into small.net have numerous hops that time out. Traceroutes from elsewhere that go into small.net but return on big.net also have numerous hops that time out. We do all still think that traceroute is important, don't we? If so, which of these two nets is unreasonable in their actions/policies? Please note that we're not talking about RFC1918 space, or reserved IP space of any kind. Also, think about the scenario where some failure happens leaving big.net with an incomplete routing table, thus breaking traceroute when it is perhaps most needed. Thanks, -mark
On Sun, 24 Sep 2006 14:59:50 -0700 (PDT) Mark Kent <mark@noc.mainstreet.net> wrote:
If so, which of these two nets is unreasonable in their actions/policies?
The non-announcers, because they're also breaking PMTUD. Regards, Mark. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
The non-announcers, because they're also breaking PMTUD.
If you're not sure what benefits PMTUD gives, you might want to review this page: http://www.psc.edu/~mathis/MTU/index.html --Michael Dillon
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Kent wrote:
A smaller North American network provider, with a modest North American backbone, numbers their internal routers on public IP space that they do not announce to the world.
One of the largest North American network providers filters/drops ICMP messages so that they only pass those with a source IP address that appears in their routing table.
As a result, traceroutes from big.net into small.net have numerous hops that time out.
Traceroutes from elsewhere that go into small.net but return on big.net also have numerous hops that time out.
We do all still think that traceroute is important, don't we?
If so, which of these two nets is unreasonable in their actions/policies?
Please note that we're not talking about RFC1918 space, or reserved IP space of any kind. Also, think about the scenario where some failure happens leaving big.net with an incomplete routing table, thus breaking traceroute when it is perhaps most needed.
Thanks, -mark
This is yet another reason one shouldn't rely on pings & traceroutes to perform reachability analysis. regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFFxP+pbZvCIJx1bcRAnN8AJ0VqiwhNkxUm5MxG8p/hLptiJ1IdQCg7wIB nx2woHkYDzu1+7MBdnOZaEw= =mlPK -----END PGP SIGNATURE-----
virendra rode wrote:
This is yet another reason one shouldn't rely on pings & traceroutes to perform reachability analysis.
So, you're in the "traceroute is not important" camp? (you'll note that in my email I did ask whether we think traceroute is important) Mark Smith wrote:
The non-announcers, because they're also breaking PMTUD.
Really? How? Remember, we're not talking about RFC1918 space, where there is a BCP that says we should filter it at the edge. We're talking about public IP space, that just doesn't happen to be announced outside of a particular AS. Thanks, -mark
On Sep 24, 2006, at 4:33 PM, Mark Kent wrote:
Remember, we're not talking about RFC1918 space, where there is a BCP that says we should filter it at the edge. We're talking about public IP space, that just doesn't happen to be announced outside of a particular AS.
If the intent is to prevent folks from reaching out and touching random network infrastructure devices directly whilst still allowing traceroute to work, iACLs and/or using IS-IS as one's IGP and null- routing the infrastructure blocks at one's various edges achieves the same effect with less potential for breakage: http://www.nanog.org/mtg-0405/mcdowell.html Note that a good infrastructure addressing plan is a prerequisite for both of these methods. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice Any information security mechanism, process, or procedure which can be consistently defeated by the successful application of a single class of attacks must be considered fatally flawed. -- The Lucy Van Pelt Principle of Secure Systems Design
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mark Kent wrote:
virendra rode wrote:
This is yet another reason one shouldn't rely on pings & traceroutes to perform reachability analysis.
So, you're in the "traceroute is not important" camp? (you'll note that in my email I did ask whether we think traceroute is important)
I'm sure its important. All I'm saying is, icmp can get rate-limited (many times it does) which could possibly lead to packet loss and even drops while traversing hops. regards, /virendra
Mark Smith wrote:
The non-announcers, because they're also breaking PMTUD.
Really? How? Remember, we're not talking about RFC1918 space, where there is a BCP that says we should filter it at the edge. We're talking about public IP space, that just doesn't happen to be announced outside of a particular AS.
Thanks, -mark
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFFyejpbZvCIJx1bcRAsFXAKDokAbujtIiuvGDXss2Tt5U3CXElQCgkpKG UaS6MDxtWKjdbiLewujDs/Q= =qgo2 -----END PGP SIGNATURE-----
Hi Mark, On Sun, 24 Sep 2006 16:33:30 -0700 (PDT) Mark Kent <mark@noc.mainstreet.net> wrote:
Mark Smith wrote:
The non-announcers, because they're also breaking PMTUD.
Really? How? Remember, we're not talking about RFC1918 space, where there is a BCP that says we should filter it at the edge. We're talking about public IP space, that just doesn't happen to be announced outside of a particular AS.
When a router that can't shove a DF'd packet down a link because the MTU is too small needs to create a ICMP Destination Unreachable, Packet Too Big, Fragmentation Required, it needs to pick a source IP address to use for that ICMP packet, which will be one of those assigned to the router with the MTU problem (I'm fairly sure it's the IP address assigned to the outgoing interface for this ICMP packet, although I don't think it probably matters much). If an upstream router, i.e. on the way back to the sender who needs to resend with a smaller packet, is dropping these packets because they fail RPF, then PMTUD breaks. The result might be connection timeouts at the sender, or possibly after quite a while the sender might try smaller packets and eventually they'll get through (I think Windows might do this). Either way, bad end-user experience. PMTUD as it currently works isn't ideal, as of course there isn't any guarantee that these ICMP Dest Unreachables will get there even in a "good" network. However, most of the time it works, where as in the scenario you're presenting, it definately won't. Regards, Mark. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
In response to this:
Mark Smith wrote:
The non-announcers, because they're also breaking PMTUD.
Really? How?
Mark Smith replied with two paragraphs, but it's not 100% clear to me that he got the reason why I asked. I asked because his initial statement boiled down to "numbering on un-announced space breaks PMTUD"... but it doesn't, not by itself (which he later expanded). It only does so in the presence of filtering. I think this is an important point to make because of my interaction with small.net. When I pointed out the timeouts they said that it was because they don't announce the router IP addresses, which is true but not the whole story. I mentioned that some providers in the past numbered on rfc1918 space and traceroute still worked, so that alone was not enough. Then they said "it's because of the asymmetric path," and that also is true, but again not enough. A large proportion of traffic is asymmetric, but traceroute still works. Then they gave me an explanation that rested on the fact that the routers will not respond to pings because they are unannounced outside of their world. That too is true, but irrelevant and I told them how Jacobson's traceroute works and told them that *someone* was dropping/filtering the return packets and I'ld like to know who/why. They somewhat implied that it was my fault, and this situation was unique to my net, so I used the big.net looking glass to show how the same things happens from space not associated with my network. (Yes, I should have done this from the outset.) With that they asked big.net, and big.net said they filtered, and that's where we are. My point here is that it took me ten (10) emails with small.net to get this information partly because the small.net support staff had notions in their head premised on too simplistic statements like "numbering on un-announced space breaks PMTUD." I wanted to clear this up because this list is likely read by support people at various networks, and it's pretty clear that not all of them are well versed even on something as thoroughly discussed over the ages as traceroute. Thanks, -mark
Once upon a time, Mark Kent <mark@noc.mainstreet.net> said:
I think this is an important point to make because of my interaction with small.net. When I pointed out the timeouts they said that it was because they don't announce the router IP addresses, which is true but not the whole story. I mentioned that some providers in the past numbered on rfc1918 space and traceroute still worked, so that alone was not enough.
Not announcing their router interface IP space is not any type of security. Anyone directly connected to them (customer or peer) could if they wish statically route that IP space, and any such security would be gone. Unless it is otherwise filtered, any customer with a default route can reach their routers. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
On Mon, 25 Sep 2006, Chris Adams wrote:
Once upon a time, Mark Kent <mark@noc.mainstreet.net> said:
I think this is an important point to make because of my interaction with small.net. When I pointed out the timeouts they said that it was because they don't announce the router IP addresses, which is true but not the whole story. I mentioned that some providers in the past numbered on rfc1918 space and traceroute still worked, so that alone was not enough.
Not announcing their router interface IP space is not any type of security. Anyone directly connected to them (customer or peer) could if they wish statically route that IP space, and any such security would be gone. Unless it is otherwise filtered, any customer with a default route can reach their routers.
Nevertheless putting router interface ip address for your network in one specific block is very effective as way to quickly get rid of DoS attack on the router - you simply stop announcing that block but everything else on the network still works. And doing tricks like having primary ip address which not important at all (except for logging traffic actually destined to it) while secondary ip on the same interface is really the one used for inter-connection also works quite well. -- William Leibzon Elan Networks william@elan.net
On Monday, 2006-09-25 at 10:09 MST, Mark Kent <mark@noc.mainstreet.net> wrote:
Mark Smith replied with two paragraphs, but it's not 100% clear to me that he got the reason why I asked. I asked because his initial statement boiled down to "numbering on un-announced space breaks PMTUD"... but it doesn't, not by itself (which he later expanded).
It only does so in the presence of filtering.
Which is exactly what one might expect to happen. At least it seems to me that RFC 3704 (BCP 84, http://www.ietf.org/rfc/rfc3704.txt) applies. When your traffic is sourced with dubious addresses, you should expect much of it to disappear. And when this happens, you're hurting your customers and your customers' customers (okay, sometimes it's "just" your peer's customers - still a concern in my opinion). -- Tony Rall
On Tue, Sep 26, 2006 at 12:17:27AM -0700, Tony Rall wrote:
On Monday, 2006-09-25 at 10:09 MST, Mark Kent <mark@noc.mainstreet.net> wrote:
Mark Smith replied with two paragraphs, but it's not 100% clear to me that he got the reason why I asked. I asked because his initial statement boiled down to "numbering on un-announced space breaks PMTUD"... but it doesn't, not by itself (which he later expanded).
It only does so in the presence of filtering.
Which is exactly what one might expect to happen. At least it seems to me that RFC 3704 (BCP 84, http://www.ietf.org/rfc/rfc3704.txt) applies.
When your traffic is sourced with dubious addresses, you should expect much of it to disappear. And when this happens, you're hurting your customers and your customers' customers (okay, sometimes it's "just" your peer's customers - still a concern in my opinion).
I think this is the critical point, dubious ip sources have been used/abused in the past and those at "big.net" that have done something to mitigate the risk to the world from their customers and their customers from the world are doing a "community service" imnsho ;). I don't see anyone here really advocating spoofed sources (except for perhaps the mobile-ip users out there ;) How many people here have rpf enabled by default on their customer CPE devices they ship? (in your template or whatnot) Do you u-rpf your dsl/cable/dial/fract-t1/t1 customers that are not doing bgp? It's hard to get this implemented across an entire network. At one time I seem to recall someone saying that 7018 was fully bcp-38 compliant, but I could be wrong. While doing u-rpf on your customers won't mitigate attacks against them, it will help mitigate the costs of tracking spoofed attacks across your network infrastructure (which is harder if you're doing mpls) that you incur. Now, i'm guessing i may be the one responsible for the practices of "big.net", but i do encourage other big.nets to enable u-rpf strict or loose wherever possible based on their equipment capabilities. Every little bit will help. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Possible approach for small.net - ok, you know that big.net will drop any packets sourced from x.x.x.x if there's no route there (loose uRPF for downstream ISPs like small.net, strict uRPF for end-users.) So give them a route. Either give them a route on one of your direct interfaces to them, and then get rid of the packets by ACL or by null-routing it, or if that causes too much trouble, get yourself a 56kbps line from a spare router and advertise it from there.
[Can we all have a moment of silence for a useful, interesting, and on-topic post?] On Sep 24, 2006, at 5:59 PM, Mark Kent wrote:
A smaller North American network provider, with a modest North American backbone, numbers their internal routers on public IP space that they do not announce to the world.
One of the largest North American network providers filters/drops ICMP messages so that they only pass those with a source IP address that appears in their routing table.
As a result, traceroutes from big.net into small.net have numerous hops that time out.
Traceroutes from elsewhere that go into small.net but return on big.net also have numerous hops that time out.
We do all still think that traceroute is important, don't we?
If so, which of these two nets is unreasonable in their actions/ policies?
Who said either was? First: Your network, your rules. Don't expect others to play by your rules. But more importantly, there is nothing that says two perfectly reasonable, rational "rules" cannot create a problem when intersecting in interesting ways. But if forced, I'd say Small.Net gets my vote for needing correction. I see less "wrongness" in a networking running what is essentially loose RPF than a network who expects supposedly bogon- sourced packets to be forwarded. (One could argue that non-announced space is bogus.) Just remember, I would only say that if pushed. Normally I would say neither is wrong.
Please note that we're not talking about RFC1918 space, or reserved IP space of any kind. Also, think about the scenario where some failure happens leaving big.net with an incomplete routing table, thus breaking traceroute when it is perhaps most needed.
In such an instance, I would suggest Big.Net will have far, far larger problems than whether pings get returned from prefixes it can't reach anyway. -- TTFN, patrick
[ Quotations have been reordered for clarity in the reply ] On 24 Sep 2006, at 22:59, Mark Kent wrote:
If so, which of these two nets is unreasonable in their actions/ policies?
I don't think either are *unreasonable* in what they've done. Both actions are prima facie reasonable but have an unforeseen synthesis that is undesirable.
One of the largest North American network providers filters/drops ICMP messages so that they only pass those with a source IP address that appears in their routing table.
This is clearly reasonable as part of an effort to mitigate ICMP based network abuse. In fact, I'd argue that it would probably be reasonable to drop any packet, not just ICMP, based on its absence from the routing table - conditioned on having a full, stable routing table.
A smaller North American network provider, with a modest North American backbone, numbers their internal routers on public IP space that they do not announce to the world.
Several people have mooted this as good practice on the basis routers do not need to be reachable (as an end system) except by legitimate managers of those routers (i.e. within the AS in question).
As a result, traceroutes from big.net into small.net have numerous hops that time out.
Traceroutes from elsewhere that go into small.net but return on big.net also have numerous hops that time out.
We do all still think that traceroute is important, don't we?
On balance, it's small.net that will have to change to rectify this. My argument would be: 1) Big.net's approach is wholly legitimate - deny spoofed packets transit. 2) Small.net's intentions are good, but they are pseudo-spoofing packets. ICMP packets will, by design, originate from the incoming interface used by the packet that triggers the ICMP packet. Thus giving an interface an address is implicitly giving that interface the ability to source packets with that address to potential anywhere in the Internet. If you don't legitimately announce address space then sourcing packets with addresses in that space is (one definition of) spoofing. On balance both are acting with good intent, but small.net haven't fully seen the consequences for ICMP in their scheme.
Please note that we're not talking about RFC1918 space, or reserved IP space of any kind. Also, think about the scenario where some failure happens leaving big.net with an incomplete routing table, thus breaking traceroute when it is perhaps most needed.
Filtering ICMP is always dangerous. If you are going to do it you *must* understand the consequences both to yourself and to others, and also understand the consequences in both normal situations and all possible failure modes. (If I had a penny for every broken PMTU detection I'd seen because of someone's over eager filtering of ICMP...)
Thanks, -mark
On Sep 25, 2006, at 9:06 AM, Ian Mason wrote:
ICMP packets will, by design, originate from the incoming interface used by the packet that triggers the ICMP packet. Thus giving an interface an address is implicitly giving that interface the ability to source packets with that address to potential anywhere in the Internet. If you don't legitimately announce address space then sourcing packets with addresses in that space is (one definition of) spoofing.
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router? For instance, you could have a "loopback99" which is in an announced block, but filtered at all your borders. Then set "ip icmp error source-interface loopback99" or something. All error messages from a router would come from this address, regardless of the incoming or outgoing interface. Things like PMTUD would still work, and your / 30s could be in private space or non-announced space or even imaginary^Wv6 space. :) Note I said "error messages", so things like TTL Expired, Port Unreachable, and Can't Fragment would come from here, but things like ICMP Echo Request / Reply pairs would not. Perhaps that should be considered as well, but it is not what I am suggesting here. Obviously there's lots of side effects, and probably unintended consequences I have not considered, but I think the good might out- weigh the bad. Or not. Which is why I'm offering it up for suggestion. (Unless, of course, I get 726384 "you are off-topic" replies, in which case I withdraw the suggestion.) -- TTFN, patrick
Patrick W. Gilmore wrote:
On Sep 25, 2006, at 9:06 AM, Ian Mason wrote:
ICMP packets will, by design, originate from the incoming interface used by the packet that triggers the ICMP packet. Thus giving an interface an address is implicitly giving that interface the ability to source packets with that address to potential anywhere in the Internet. If you don't legitimately announce address space then sourcing packets with addresses in that space is (one definition of) spoofing.
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router?
I do. I have suggested much the same in the past.
On Mon, 25 Sep 2006 09:22:34 -0400 "Patrick W. Gilmore" <patrick@ianai.net> wrote:
On Sep 25, 2006, at 9:06 AM, Ian Mason wrote:
ICMP packets will, by design, originate from the incoming interface used by the packet that triggers the ICMP packet. Thus giving an interface an address is implicitly giving that interface the ability to source packets with that address to potential anywhere in the Internet. If you don't legitimately announce address space then sourcing packets with addresses in that space is (one definition of) spoofing.
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router?
I do. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
Might this not be a bad idea if the router has interfaces on multiple, separate paths? Such a case may be where one customer or set of traffic routes over a link to ISP A, and other traffic over a link to ISP B, and not all related addresses are portable. In that case the loopback address for the ICMP errors might show from an address that seems to belong to a block from ISP A, but is really traffic that was transported on ISP B. Just trying to come up with possible issues, not saying that's a best practice or anything... -Scott -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Patrick W. Gilmore Sent: Monday, September 25, 2006 9:23 AM To: nanog@merit.edu Cc: Patrick W. Gilmore Subject: New router feature - icmp error source-interface [was: icmp rpf] On Sep 25, 2006, at 9:06 AM, Ian Mason wrote:
ICMP packets will, by design, originate from the incoming interface used by the packet that triggers the ICMP packet. Thus giving an interface an address is implicitly giving that interface the ability to source packets with that address to potential anywhere in the Internet. If you don't legitimately announce address space then sourcing packets with addresses in that space is (one definition of) spoofing.
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router? For instance, you could have a "loopback99" which is in an announced block, but filtered at all your borders. Then set "ip icmp error source-interface loopback99" or something. All error messages from a router would come from this address, regardless of the incoming or outgoing interface. Things like PMTUD would still work, and your / 30s could be in private space or non-announced space or even imaginary^Wv6 space. :) Note I said "error messages", so things like TTL Expired, Port Unreachable, and Can't Fragment would come from here, but things like ICMP Echo Request / Reply pairs would not. Perhaps that should be considered as well, but it is not what I am suggesting here. Obviously there's lots of side effects, and probably unintended consequences I have not considered, but I think the good might out- weigh the bad. Or not. Which is why I'm offering it up for suggestion. (Unless, of course, I get 726384 "you are off-topic" replies, in which case I withdraw the suggestion.) -- TTFN, patrick
On Sep 25, 2006, at 5:26 PM, Berkman, Scott wrote:
Might this not be a bad idea if the router has interfaces on multiple, separate paths? Such a case may be where one customer or set of traffic routes over a link to ISP A, and other traffic over a link to ISP B, and not all related addresses are portable. In that case the loopback address for the ICMP errors might show from an address that seems to belong to a block from ISP A, but is really traffic that was transported on ISP B.
Just trying to come up with possible issues, not saying that's a best practice or anything...
I doubt it is possible to make a rule / knob / idea / feature / whatever that cannot be misused, or that is applicable to all corner cases. That's why it's a knob. :) If it is applicable to your situation, you should use it. If not, not. But if the knob has enough use in enough situations, then it is probably something we want to push the vendors to implement. -- TTFN, patrick
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Patrick W. Gilmore Sent: Monday, September 25, 2006 9:23 AM To: nanog@merit.edu Cc: Patrick W. Gilmore Subject: New router feature - icmp error source-interface [was: icmp rpf]
On Sep 25, 2006, at 9:06 AM, Ian Mason wrote:
ICMP packets will, by design, originate from the incoming interface used by the packet that triggers the ICMP packet. Thus giving an interface an address is implicitly giving that interface the ability to source packets with that address to potential anywhere in the Internet. If you don't legitimately announce address space then sourcing packets with addresses in that space is (one definition of) spoofing.
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router?
For instance, you could have a "loopback99" which is in an announced block, but filtered at all your borders. Then set "ip icmp error source-interface loopback99" or something. All error messages from a router would come from this address, regardless of the incoming or outgoing interface. Things like PMTUD would still work, and your / 30s could be in private space or non-announced space or even imaginary^Wv6 space. :)
Note I said "error messages", so things like TTL Expired, Port Unreachable, and Can't Fragment would come from here, but things like ICMP Echo Request / Reply pairs would not. Perhaps that should be considered as well, but it is not what I am suggesting here.
Obviously there's lots of side effects, and probably unintended consequences I have not considered, but I think the good might out- weigh the bad. Or not. Which is why I'm offering it up for suggestion.
(Unless, of course, I get 726384 "you are off-topic" replies, in which case I withdraw the suggestion.)
-- TTFN, patrick
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Patrick W. Gilmore Sent: Monday, September 25, 2006 5:31 PM To: nanog@merit.edu Cc: Patrick W. Gilmore Subject: Re: New router feature - icmp error source-interface [was: icmp rpf]
On Sep 25, 2006, at 5:26 PM, Berkman, Scott wrote:
Might this not be a bad idea if the router has interfaces on multiple, separate paths? Such a case may be where one customer or set of traffic routes over a link to ISP A, and other traffic over a link to ISP B, and not all related addresses are portable. In that case the loopback address for the ICMP errors might show from an address that seems to belong to a block from ISP A, but is really traffic that was transported on ISP B.
Just trying to come up with possible issues, not saying that's a best practice or anything...
I doubt it is possible to make a rule / knob / idea / feature / whatever that cannot be misused, or that is applicable to all corner cases.
That's why it's a knob. :) If it is applicable to your situation, you should use it. If not, not.
But if the knob has enough use in enough situations, then it is probably something we want to push the vendors to implement.
-- TTFN, patrick
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Patrick W. Gilmore Sent: Monday, September 25, 2006 9:23 AM To: nanog@merit.edu Cc: Patrick W. Gilmore Subject: New router feature - icmp error source-interface [was: icmp rpf]
On Sep 25, 2006, at 9:06 AM, Ian Mason wrote:
ICMP packets will, by design, originate from the incoming interface used by the packet that triggers the ICMP packet. Thus giving an interface an address is implicitly giving that interface the ability to source packets with that address to potential anywhere in the Internet. If you don't legitimately announce address space then sourcing packets with addresses in that space is (one definition of) spoofing.
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router?
For instance, you could have a "loopback99" which is in an announced block, but filtered at all your borders. Then set "ip icmp error source-interface loopback99" or something. All error messages from a router would come from this address, regardless of the incoming or outgoing interface. Things like PMTUD would still work, and your / 30s could be in private space or non-announced space or even imaginary^Wv6 space. :)
Note I said "error messages", so things like TTL Expired, Port Unreachable, and Can't Fragment would come from here, but things like ICMP Echo Request / Reply pairs would not. Perhaps that should be considered as well, but it is not what I am suggesting here.
Obviously there's lots of side effects, and probably unintended consequences I have not considered, but I think the good might out- weigh the bad. Or not. Which is why I'm offering it up for suggestion.
(Unless, of course, I get 726384 "you are off-topic" replies, in which case I withdraw the suggestion.)
-- TTFN, patrick
C and J both already have a similar feature, however I'm not sure whether or not they apply to ICMP. They both support PBR for locally originated packets - which, should include if the thought process is correct, ICMP. Perhaps someone with some time to waste can verify this in a lab. http://www.cisco.com/en/US/customer/products/sw/iosswrel/ps1828/products _configuration_guide_chapter09186a00800ca590.html#5406 -Dave
On Mon, Sep 25, 2006 at 04:33:18PM -0700, David Temkin wrote:
C and J both already have a similar feature, however I'm not sure whether or not they apply to ICMP. They both support PBR for locally originated packets - which, should include if the thought process is correct, ICMP. Perhaps someone with some time to waste can verify this in a lab.
http://www.cisco.com/en/US/customer/products/sw/iosswrel/ps1828/products _configuration_guide_chapter09186a00800ca590.html#5406
The actual path taken for the ICMP generated by the router does not matter, we're just talking about the source address selected by the router. The only reasons that the source address (which reveals a real IP address on a router) matters at all for ICMP error responses are: * So traceroute works (current industry standard behavior is to use the ingress interface IP so you see the forward path in traceroute, not the reverse path, which may be asymmetric. * So your replies don't get thwacked by people doing uRPF strict (i.e. they must come from announced IPs or people doing strict strict with no exception filtering capabilities will block the traceroute responses). * Optionally, allowing naive tools like MTR to ping the IP they discover via traceroute, lest weenies flood your noc with "I'm pinging 10lolz!" emails. Revealing your interface IPs carries all kinds of DoS/security risks with it, since there are a great many routers out there without good control plane policing functionality (and even some of those that have it, don't really have it :P). Since there is really no legitimate need for people from the outside world to ever communicate with your real interface IPs at all (with the exception of some rate limited ICMP echo/reply due to aforementioned mtr weenies), having the option to hide those real addresses completely in ICMP source address selection is a very good thing for enhancing network security. As I said you can accomplish part of this hack with primary/secondary IPs on interfaces. You can also accomplish some level of filtering by numbering your interfaces out of common blocks which are filtered at your various borders/edges. It's still a pain in the !(*#&*, especially if you number your links out of any "regional blocks" to cut down on asymmetric routing confusion, or have any number of peers who provide /30s from their own IP space. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Mon, Sep 25, 2006 at 09:22:34AM -0400, Patrick W. Gilmore wrote:
On Sep 25, 2006, at 9:06 AM, Ian Mason wrote:
ICMP packets will, by design, originate from the incoming interface used by the packet that triggers the ICMP packet. Thus giving an interface an address is implicitly giving that interface the ability to source packets with that address to potential anywhere in the Internet. If you don't legitimately announce address space then sourcing packets with addresses in that space is (one definition of) spoofing.
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router?
You know I was just having this discussion with someone else a couple days ago. It turns out, much to my surprise, that the RFC actually calls for the ICMP error-message packet (as you said, the things that aren't ping etc which require a specific source-address) to originate from the OUTGOING interface used to return the ICMP message to the original sender. After much googling, I can't find any document where this has ever been officially updated either. The defacto industry standard on the other hand has been to use the primary address of the inbound interface, which serves exactly one function: it makes traceroute work. The hack that people use to simulate this functionality normally is: ip address the.fake.ip.here 255.255.255.252 ip address the.real.ip.here 255.255.255.252 secondary (FYI side note the Juniper equivilent is... confusing or non-working, hard to tell which. The tag "primary" is defined as the source address for local broadcast/multicast, "preferred" is defined as the source when you have multiple IPs within a subnet. Neither one should work for what we're talking about according to the docs, but if you actually try it... sometimes it works, sometimes it won't, and sometimes the behavior is different if you include only one but not the other :P). This works well for simple external-facing interfaces, things that speak BGP for example, but can confuse things like OSPF etc when you use secondaries on internal interfaces. FWIW I've been asking for more people to implement exactly what you're talking about for years (specifically setting ONLY the ICMP error source interface without risk of screwing up the interface in other ways). You can debate over exactly how to do it (global or per interface, icmp source-interface lo99 vs icmp source-ip 1.2.3.4, etc), but I agree wholeheatedly it should be done. It should be really simple too!
(Unless, of course, I get 726384 "you are off-topic" replies, in which case I withdraw the suggestion.)
Please stop talking about networking on NANOG, you're confusing people. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Sep 25, 2006, at 5:40 PM, Richard A Steenbergen wrote:
On Mon, Sep 25, 2006 at 09:22:34AM -0400, Patrick W. Gilmore wrote:
On Sep 25, 2006, at 9:06 AM, Ian Mason wrote:
ICMP packets will, by design, originate from the incoming interface used by the packet that triggers the ICMP packet. Thus giving an interface an address is implicitly giving that interface the ability to source packets with that address to potential anywhere in the Internet. If you don't legitimately announce address space then sourcing packets with addresses in that space is (one definition of) spoofing.
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router?
You know I was just having this discussion with someone else a couple days ago. It turns out, much to my surprise, that the RFC actually calls for the ICMP error-message packet (as you said, the things that aren't ping etc which require a specific source-address) to originate from the OUTGOING interface used to return the ICMP message to the original sender. After much googling, I can't find any document where this has ever been officially updated either. The defacto industry standard on the other hand has been to use the primary address of the inbound interface, which serves exactly one function: it makes traceroute work.
I have not read the RFC in full, but after chatting with Daniel offline (see, some people actually do talk without posting! :), I believe this only applies to packets addressed to the router. Since packets going -through- the router have absolutely no guarantee what source will be used coming back, I don't seen an issue here. Just change the idea such that it only is used for error messages to packets where the dest addy is not an interface on the router. Also, this makes traceroute -easier- to use. Suddenly all interfaces on the same router have the same IP address, thereby making it easy to tell if two traceroutes intersect, even if they use different interfaces. Oh, and who said RFCs can't be updated? :-)
(Unless, of course, I get 726384 "you are off-topic" replies, in which case I withdraw the suggestion.)
Please stop talking about networking on NANOG, you're confusing people. :)
I knew someone would flame me for it. :) -- TTFN, patrick
Can someone from comcast contact me off list please ? Thanks, Ansh Kanwar Lead Network Engineer -- Citrix Online (AS16815) 5385 Hollister Avenue Santa Barbara, CA 93111 USA --
Anshuman: A good place to start for operational contacts is both the puck.nether.net site and the www.peeringdb.com. i found this: http://puck.nether.net/netops/nocs.cgi?ispname=comcast and this: (you can log in as a guest)... https://www.peeringdb.com/private/participant_view.php?id=822 now go get them!!!! peter cohen. On 9/25/06, Anshuman Kanwar <Anshuman.Kanwar@citrix.com> wrote:
Can someone from comcast contact me off list please ?
Thanks,
Ansh Kanwar Lead Network Engineer -- Citrix Online (AS16815) 5385 Hollister Avenue Santa Barbara, CA 93111 USA --
At 9:22 AM -0400 9/25/06, Patrick W. Gilmore wrote:
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router?
It certainly would beat the alternative of no response at all, but one would hope it wouldn't become common practice since it reduces the information returned (e.g. during a traceroute, you'd lose the sometimes useful information from in-addr about what particular interface was involved). /John
On Mon, Sep 25, 2006 at 08:45:49PM -0400, John Curran wrote:
At 9:22 AM -0400 9/25/06, Patrick W. Gilmore wrote:
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router?
It certainly would beat the alternative of no response at all, but one would hope it wouldn't become common practice since it reduces the information returned (e.g. during a traceroute, you'd lose the sometimes useful information from in-addr about what particular interface was involved).
Personally I'd hope that if it was implemented, it would support mapping on a per-interface basis (especially for NSP use). That should in theory lead to even more accurate information, since each network would be capable of easily renumbering without impact, and managing their own DNS for every interface. Currently a great many PTRs are out of date because IP blocks supplied by peers, exchange points, or transit providers, are too much of a pain to keep updated when interfaces move etc. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Mon, Sep 25, 2006 at 09:22:34AM -0400, Patrick W. Gilmore wrote: ...
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router? ...
I've sometimes thought it would be useful when I wanted to hide a route. But security via obscurity just makes it that much harder to fix something. Many more times than this would have been useful, I've been able to identify at which router a problem was by a 'traceroute' that told me into which router by which interface I was going. When the owner of the router might not even have known. Or I have had attempts to do this foiled by routers that used an internal loopback IP address. On the whole, then, I guess I would vote, "no". -- Joe Yao ----------------------------------------------------------------------- This message is not an official statement of OSIS Center policies.
On Mon, 25 Sep 2006, Joseph S D Yao wrote:
On Mon, Sep 25, 2006 at 09:22:34AM -0400, Patrick W. Gilmore wrote: ...
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router? ...
I've sometimes thought it would be useful when I wanted to hide a route. But security via obscurity just makes it that much harder to fix
I think in the original poster's scenario one network was looking to protect their resources/equipment from a majority of the network's ills. It's not unreasonable... atleast not in my mind. It's also not 'security through obscurity' since one of the parties is/was leaking their information OUT, just not 'in' :)
something. Many more times than this would have been useful, I've been able to identify at which router a problem was by a 'traceroute' that
What's interesting is that today, in many networks, the usefulness of traceeroute has bee degraded by other non-ip issues (<cough>mpls</cough>) not in ALL cases, but certainly in many you are not seeing quite what you'd expect from the traceroute :(
At 10:29 PM 9/25/2006, Chris L. Morrow wrote:
On Mon, 25 Sep 2006, Joseph S D Yao wrote:
On Mon, Sep 25, 2006 at 09:22:34AM -0400, Patrick W. Gilmore wrote: ...
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router? ...
I've sometimes thought it would be useful when I wanted to hide a route. But security via obscurity just makes it that much harder to fix
I think in the original poster's scenario one network was looking to protect their resources/equipment from a majority of the network's ills. It's not unreasonable... atleast not in my mind. It's also not 'security through obscurity' since one of the parties is/was leaking their information OUT, just not 'in' :)
The issue is that the world has changed. Some networks actually expect not only the use of public addresses on router interfaces, but addresses from advertised blocks. Why has the world changed? Because not everyone is friendly on the Internet. There were issues raised in Mobile IP by the drafts that predated the publication of RFC 2267. Why? Well, Mobile IP WG had made the assumption that all IPv4 packet forwarding would be done, without filtering, based solely on the destination IP address. The down side was it made some of the traffic look spoofed. The world changed, people started abusing the Internet in new ways, and the old assumptions no longer held. Mobile IP WG adapted to the new reality that was coming. The longevity of the TCP/IP protocol suite is attributable to the continued effort of many people, not to savant qualities of those who first wrote specifications.
something. Many more times than this would have been useful, I've been able to identify at which router a problem was by a 'traceroute' that
What's interesting is that today, in many networks, the usefulness of traceeroute has bee degraded by other non-ip issues (<cough>mpls</cough>) not in ALL cases, but certainly in many you are not seeing quite what you'd expect from the traceroute :(
There's no more degradation of usefulness from MPLS than there was from ATM or Frame Relay. Pick your poison for virtualized link layer, and you'll have some degree of difficulty determining and debugging the underlying network without tools to peer under the hood.
Joseph S D Yao wrote:
On Mon, Sep 25, 2006 at 09:22:34AM -0400, Patrick W. Gilmore wrote: ...
Who thinks it would be a "good idea" to have a knob such that ICMP error messages are always source from a certain IP address on a router?
...
I've sometimes thought it would be useful when I wanted to hide a route. But security via obscurity just makes it that much harder to fix something. Many more times than this would have been useful, I've been able to identify at which router a problem was by a 'traceroute' that told me into which router by which interface I was going. When the owner of the router might not even have known. Or I have had attempts to do this foiled by routers that used an internal loopback IP address. On the whole, then, I guess I would vote, "no".
Why not just do a show ip route? since you can actually verify the information against your routing table. This way you can see when the route was learned, where was it learned from and how long ago it was last updated... the problem is that too many people "engineers" rely on traceroute... sure traceroute is a wonderful tool, however it is meant to assist you in "tracking down" the problem. I've seen far too many "you are filtering, investigate please" when all that has been done is implementing acls and rate limiting. IMO, If you want to implement a non-routable ip space to protect your backbone... go for it if you want to icmp rate limit *i know level3 does this out of both nyc and la* which causes mass threads of "we are getting packet loss, please investigate" go for it .. if your network engineers are not equipped with the information to how to fully diagnose a network/problem.... you should think about new hires. Cheers, Payam
On Mon, Sep 25, 2006, Ian Mason wrote:
Filtering ICMP is always dangerous. If you are going to do it you *must* understand the consequences both to yourself and to others, and also understand the consequences in both normal situations and all possible failure modes. (If I had a penny for every broken PMTU detection I'd seen because of someone's over eager filtering of ICMP...)
Is there a BCP for "handling ICMP?" I'm walking the Cisco certification path and they're quite vocal about ICMP rate limiting over any kind of filtering on routers/switches. I haven't read their firewall documentation so I'm not sure what they're preaching for PIX/ASA. (Yup, if I had a penny for every PMTU fix-by-unbreaking-ICMP-filtering I've repaired over the last 10 years..) Adrian
At 10:06 25/09/2006, Ian Mason wrote:
One of the largest North American network providers filters/drops ICMP messages so that they only pass those with a source IP address that appears in their routing table.
This is clearly reasonable as part of an effort to mitigate ICMP based network abuse.
As a matter of fact, most ICMP-based attacks don't require spoofing of the source IP address. You do have to spoof the addresses in the "original datagram" included in the ICMP payload, though. Kindest regards, -- Fernando Gont e-mail: fernando@gont.com.ar || fgont@acm.org PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
On Sun, Sep 24, 2006 at 02:59:50PM -0700, Mark Kent wrote:
A smaller North American network provider, with a modest North American backbone, numbers their internal routers on public IP space that they do not announce to the world.
One of the largest North American network providers filters/drops ICMP messages so that they only pass those with a source IP address that appears in their routing table.
I would hope they're doing it for more than just ICMP packets. There are numerous nefarious uses of the network with unrouted/spoofed addres space. Various hosts have done bad things (in the past) if they get something like a SYN that appears to be from themselves. Protecting ones customers from spoofed address DoS attacks and leaking of unrouted IP space (1918 or otherwise) that isn't globally reachable I would argue should be, or is a current best practice. The "good" packets that are dropped in this scenario are sufficent limited (yes, pmtu and these cases of traceroutes, etc..) but there are also well known solutions and workarounds to this as well. It's still hard to get people to fix their "deny all icmp" policies that some companies have that create troubles for others. I've had issues accessing my own bank website in the past due to p-mtu issues. These aren't places that are easily approachable to resolve the problem in most cases.
As a result, traceroutes from big.net into small.net have numerous hops that time out.
Others have pointed out how this can be resolved by by using different techniques and still protect the infrastructure. It may be of value for small.net to look at it and see what applies to them.
Traceroutes from elsewhere that go into small.net but return on big.net also have numerous hops that time out.
We do all still think that traceroute is important, don't we?
I agree traceroute is important and valuable. It's one of the things I have asked people to send me in the past for debugging, but isn't the sole source of debugging available. Other techniques can be applied. Did big.net just turn this on, or has it been on for months/years now? - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Jared Mauch wrote:
I would hope they're doing it for more than just ICMP packets.
yes, loose RPF, but I just care about ICMP.
I would argue should be, or is a current best practice.
OK, so I must have missed the memo :-) Who among AS1239, AS701, AS3356, AS7018, AS209 does loose RPF (not just strict RPF on single-homed customers)?
Did big.net just turn this on, or has it been on for months/years now?
I'm pretty sure it's months and not years. I've noticed it for a while, but it just recently drove me to the point where I'd complain about it. Thanks, -mark
On Sep 25, 2006, at 12:22 PM, Mark Kent wrote:
Jared Mauch wrote:
I would hope they're doing it for more than just ICMP packets.
yes, loose RPF, but I just care about ICMP.
I would argue should be, or is a current best practice.
OK, so I must have missed the memo :-)
It's been all the rage. :)
Who among AS1239, AS701, AS3356, AS7018, AS209 does loose RPF (not just strict RPF on single-homed customers)?
I'm wondering why that is relevant. If all those ASNs only filtered on ASN instead of prefix for customer announcements (for instance), would that mean no one should? -- TTFN, patrick
participants (25)
-
Adrian Chadd -
Anshuman Kanwar -
Berkman, Scott -
Bill Stewart -
Chris Adams -
Chris L. Morrow -
Daniel Senie -
David Temkin -
Fernando Gont -
Ian Mason -
Jared Mauch -
Joe Maimon -
John Curran -
Joseph S D Yao -
Mark Kent -
Mark Smith -
Michael.Dillonļ¼ btradianz.com -
Patrick W. Gilmore -
Payam Tarverdyan Chychi -
Peter Cohen -
Richard A Steenbergen -
Roland Dobbins -
Tony Rall -
virendra rode // -
william(at)elan.net