How long is reasonable to fix a routing issue in IPv6?
The below has been on going for over a week (yes it has been reported to HE) and the IETF). Not returning time exceeded really make debugging routing problems hard. Mark traceroute6 -lI www.ietf.org traceroute6 to www.ietf.org (2001:1890:1112:1::1e) from 2001:470:1f00:ffff::5a1, 64 hops max, 16 byte packets 1 dviscorg.tunnel.tserv1.fmt.ipv6.he.net (2001:470:1f00:ffff::5a0) 172.669 ms 182.219 ms 170.424 ms 2 2001:470:0:1f::1 (2001:470:0:1f::1) 178.128 ms 171.926 ms 174.877 ms 3 gige-g4-8.core1.fmt2.he.net (2001:470:0:2d::2) 172.193 ms 171.265 ms 171.221 ms 4 10gigabitethernet6-4.core1.lax1.he.net (2001:470:0:18d::2) 198.338 ms 190.680 ms 182.413 ms 5 10gigabitethernet1-3.core1.lax2.he.net (2001:470:0:72::2) 178.924 ms 189.431 ms 180.538 ms 6 2001:470:0:1e6::2 (2001:470:0:1e6::2) 181.137 ms 179.374 ms 182.321 ms 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 2001:1890:1fff:ffff::1 (2001:1890:1fff:ffff::1) 262.713 ms * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
If those interim hops are IPv4 only nodes part of 6/PE there could be a few things going on here. 1) Juniper u-RPF for family inet6 drops the mapped-v4-v6 source packets generated by a P/PE router 2) FreeBSD (8.2 at least) doesn't seem to like mapped-v4-v6 source packets with its default traceroute (same for mtr on FreeBSD) (tcpdump will show you the packets coming back, the freebsd traceroute6 seems to have a few unresolved bugs.. you need to force -w 1 as well likely) 3) If end-to-end connectivity works, Workarounds: the IPv4 only P/PE device should have some sort of IPv6 address placed on transit interfaces to allow TTL expired to be sourced from something capable (this IP doesn't need to be able to be reached/routed to that interface, just exist). I spent a lot of time looking at a similar problem and it ended up being a combination of #1 & #2 above. You will see this problem across The AT&T and Cogent networks in my experience. - Jared On Jul 6, 2011, at 10:39 PM, Mark Andrews wrote:
The below has been on going for over a week (yes it has been reported to HE) and the IETF).
Not returning time exceeded really make debugging routing problems hard.
Mark
traceroute6 -lI www.ietf.org traceroute6 to www.ietf.org (2001:1890:1112:1::1e) from 2001:470:1f00:ffff::5a1, 64 hops max, 16 byte packets 1 dviscorg.tunnel.tserv1.fmt.ipv6.he.net (2001:470:1f00:ffff::5a0) 172.669 ms 182.219 ms 170.424 ms 2 2001:470:0:1f::1 (2001:470:0:1f::1) 178.128 ms 171.926 ms 174.877 ms 3 gige-g4-8.core1.fmt2.he.net (2001:470:0:2d::2) 172.193 ms 171.265 ms 171.221 ms 4 10gigabitethernet6-4.core1.lax1.he.net (2001:470:0:18d::2) 198.338 ms 190.680 ms 182.413 ms 5 10gigabitethernet1-3.core1.lax2.he.net (2001:470:0:72::2) 178.924 ms 189.431 ms 180.538 ms 6 2001:470:0:1e6::2 (2001:470:0:1e6::2) 181.137 ms 179.374 ms 182.321 ms 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 2001:1890:1fff:ffff::1 (2001:1890:1fff:ffff::1) 262.713 ms * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * *
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
In message <9634AA8C-C8FD-4AFE-A888-82C9C326636D@puck.nether.net>, Jared Mauch w rites:
If those interim hops are IPv4 only nodes part of 6/PE there could be a = few things going on here.
1) Juniper u-RPF for family inet6 drops the mapped-v4-v6 source packets = generated by a P/PE router 2) FreeBSD (8.2 at least) doesn't seem to like mapped-v4-v6 source = packets with its default traceroute (same for mtr on FreeBSD) (tcpdump will show you the packets coming back, the freebsd traceroute6 = seems to have a few unresolved bugs.. you need to force -w 1 as well = likely)
I see nothing in tcpdump other than the outgoing traffic and the replies already noted.
3) If end-to-end connectivity works,=20
Workarounds:
the IPv4 only P/PE device should have some sort of IPv6 address placed = on transit interfaces to allow TTL expired to be sourced from something = capable (this IP doesn't need to be able to be reached/routed to that = interface, just exist).
I spent a lot of time looking at a similar problem and it ended up being = a combination of #1 & #2 above. You will see this problem across The = AT&T and Cogent networks in my experience.
The path is going through AT&T.
- Jared
On Jul 6, 2011, at 10:39 PM, Mark Andrews wrote:
=20 The below has been on going for over a week (yes it has been reported = to HE) and the IETF). =20 Not returning time exceeded really make debugging routing problems = hard. =20 Mark =20 traceroute6 -lI www.ietf.org traceroute6 to www.ietf.org (2001:1890:1112:1::1e) from = 2001:470:1f00:ffff::5a1, 64 hops max, 16 byte packets 1 dviscorg.tunnel.tserv1.fmt.ipv6.he.net (2001:470:1f00:ffff::5a0) = 172.669 ms 182.219 ms 170.424 ms 2 2001:470:0:1f::1 (2001:470:0:1f::1) 178.128 ms 171.926 ms = 174.877 ms 3 gige-g4-8.core1.fmt2.he.net (2001:470:0:2d::2) 172.193 ms 171.265 = ms 171.221 ms 4 10gigabitethernet6-4.core1.lax1.he.net (2001:470:0:18d::2) 198.338 = ms 190.680 ms 182.413 ms 5 10gigabitethernet1-3.core1.lax2.he.net (2001:470:0:72::2) 178.924 = ms 189.431 ms 180.538 ms 6 2001:470:0:1e6::2 (2001:470:0:1e6::2) 181.137 ms 179.374 ms = 182.321 ms 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 2001:1890:1fff:ffff::1 (2001:1890:1fff:ffff::1) 262.713 ms * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * =20 --=20 Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Jul 7, 2011, at 2:14 AM, Mark Andrews wrote:
3) If end-to-end connectivity works,=20
Workarounds:
the IPv4 only P/PE device should have some sort of IPv6 address placed = on transit interfaces to allow TTL expired to be sourced from something = capable (this IP doesn't need to be able to be reached/routed to that = interface, just exist).
I spent a lot of time looking at a similar problem and it ended up being = a combination of #1 & #2 above. You will see this problem across The = AT&T and Cogent networks in my experience.
The path is going through AT&T.
Yes. AT&T and Cogent are aware of this. I think there may be an IETF draft out there that talks about this as well but don't have a pointer to it. It is true that if an MPLS-LSR/P device does not understand the IPv6 frame you will see no response for that TTL. It's only the P devices that do understand an IPv6 frame, but decide to put a mapped-v4 address in the ttl expired message. The real questions are: 1) Is hurricane electric doing loose-rpf for ipv6/inet6 and dropping these packets? (and if they are, are you requesting they make this change?) 2) is a mapped-v4 address a valid *source* address on the wire even if it's not a valid dest? 3) should operators of IPv6 capable equipment be running them in an MPLS-LSR/P role be assigning an IPv6 address on interfaces to provide a valid source-address even if they are not reachable in return? Should the vendor provide a knob to generate the ttl expiry messages from some other source address, obscuring the interface IPs involved (such as a loopback)? Mark, it may also be valuable to see testing from a server at ISC that doesn't transit HE to reach the ATT network. While you still can't see who is dropping your packets, you may find someone who doesn't have loose-rpf enabled and observe some of the other behaviors noted. - Jared (BTW, 2914 does do loose-rpf on inet6 to drop unrouted space on Juniper devices)
On 7/7/11 6:20 AM, Jared Mauch wrote:
On Jul 7, 2011, at 2:14 AM, Mark Andrews wrote:
3) If end-to-end connectivity works,=20
Workarounds:
the IPv4 only P/PE device should have some sort of IPv6 address placed = on transit interfaces to allow TTL expired to be sourced from something = capable (this IP doesn't need to be able to be reached/routed to that = interface, just exist).
I spent a lot of time looking at a similar problem and it ended up being = a combination of #1& #2 above. You will see this problem across The = AT&T and Cogent networks in my experience. The path is going through AT&T. Yes. AT&T and Cogent are aware of this. I think there may be an IETF draft out there that talks about this as well but don't have a pointer to it. It is true that if an MPLS-LSR/P device does not understand the IPv6 frame you will see no response for that TTL. It's only the P devices that do understand an IPv6 frame, but decide to put a mapped-v4 address in the ttl expired message.
The real questions are:
1) Is hurricane electric doing loose-rpf for ipv6/inet6 and dropping these packets? (and if they are, are you requesting they make this change?)
We aren't doing loose-rpf on ISC's transit session (for the reason you stated and others).
2) is a mapped-v4 address a valid *source* address on the wire even if it's not a valid dest? 3) should operators of IPv6 capable equipment be running them in an MPLS-LSR/P role be assigning an IPv6 address on interfaces to provide a valid source-address even if they are not reachable in return? Should the vendor provide a knob to generate the ttl expiry messages from some other source address, obscuring the interface IPs involved (such as a loopback)?
Mark, it may also be valuable to see testing from a server at ISC that doesn't transit HE to reach the ATT network. While you still can't see who is dropping your packets, you may find someone who doesn't have loose-rpf enabled and observe some of the other behaviors noted.
The problem observed also happens for native IPv6 packets sent to 2001:1890:1112:1::1e We can confirm the packets leave our network and are handed off to the correct party. We've sent emails regarding this to the other parties involved with no response so far. I'll make sure people start getting phone calls. Mike.
- Jared
(BTW, 2914 does do loose-rpf on inet6 to drop unrouted space on Juniper devices)
In message <4E15DFD7.3090802@he.net>, Mike Leber writes:
On 7/7/11 6:20 AM, Jared Mauch wrote:
On Jul 7, 2011, at 2:14 AM, Mark Andrews wrote:
3) If end-to-end connectivity works,=20
Workarounds:
the IPv4 only P/PE device should have some sort of IPv6 address placed = on transit interfaces to allow TTL expired to be sourced from something = capable (this IP doesn't need to be able to be reached/routed to that = interface, just exist).
I spent a lot of time looking at a similar problem and it ended up being = a combination of #1& #2 above. You will see this problem across The = AT&T and Cogent networks in my experience. The path is going through AT&T. Yes. AT&T and Cogent are aware of this. I think there may be an IETF draft out there that talks about this as well but don't have a pointer to it. It i s true that if an MPLS-LSR/P device does not understand the IPv6 frame you wil l see no response for that TTL. It's only the P devices that do understand an IPv6 frame, but decide to put a mapped-v4 address in the ttl expired message.
The real questions are:
1) Is hurricane electric doing loose-rpf for ipv6/inet6 and dropping these p ackets? (and if they are, are you requesting they make this change?)
We aren't doing loose-rpf on ISC's transit session (for the reason you stated and others).
2) is a mapped-v4 address a valid *source* address on the wire even if it's not a valid dest? 3) should operators of IPv6 capable equipment be running them in an MPLS-LSR /P role be assigning an IPv6 address on interfaces to provide a valid source-a ddress even if they are not reachable in return? Should the vendor provide a knob to generate the ttl expiry messages from some other source address, obscu ring the interface IPs involved (such as a loopback)?
Mark, it may also be valuable to see testing from a server at ISC that doesn 't transit HE to reach the ATT network. While you still can't see who is drop ping your packets, you may find someone who doesn't have loose-rpf enabled and observe some of the other behaviors noted.
The problem observed also happens for native IPv6 packets sent to 2001:1890:1112:1::1e
We can confirm the packets leave our network and are handed off to the correct party.
We've sent emails regarding this to the other parties involved with no response so far. I'll make sure people start getting phone calls.
Mike.
Thanks. I wasn't trying to complain about HE's support, which has always been good. I was trying to point out the not having working time exceeded messages makes everybody's job harder. That it is time to take IPv6 seriously and part of doing that means making sure that error reporting works. Mark
- Jared
(BTW, 2914 does do loose-rpf on inet6 to drop unrouted space on Juniper devi ces)
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Fri, Jul 08, 2011 at 10:21:13PM +0200, Florian Weimer wrote:
* Jared Mauch:
2) is a mapped-v4 address a valid *source* address on the wire even if it's not a valid dest?
By the way, has the analogous issue involving v4 addresses from RFC 1918 space ever been settled?
define "valid"? mapped-v4 and 1918 addresses are valid in that both meet the address format constraints. technically they work as both source and destination - the restriction is administrative. /bill
On Fri, Jul 08, 2011 at 10:21:13PM +0200, Florian Weimer wrote:
* Jared Mauch:
2) is a mapped-v4 address a valid *source* address on the wire even if it's not a valid dest?
By the way, has the analogous issue involving v4 addresses from RFC 1918 space ever been settled?
define "valid"?
"Exempt from BCP 38 filters". I thought that was clear enough, sorry. 8-)
On Jul 7, 2011, at 1:20 PM, Jared Mauch wrote:
2) is a mapped-v4 address a valid *source* address on the wire even if it's not a valid dest?
The world is split on that and why wouldn't it in theory be a valid dest? It would be easier if this had made it past draft state: http://tools.ietf.org/html/draft-itojun-v6ops-v4mapped-harmful -- Bjoern A. Zeeb You have to have visions! Stop bit received. Insert coin for new address family.
participants (6)
-
Bjoern A. Zeeb -
bmanning＠vacation.karoshi.com -
Florian Weimer -
Jared Mauch -
Mark Andrews -
Mike Leber