FYI - thought this would be of interest to some of you, there will be more news on this front shortly. http://www.comcast6.net/ 6to4 Relays Activated Tuesday, August 17, 2010 As we started our IPv6 trials, we began to observe an increase in 6to4 relay traffic. 6to4 is a transition mechanism built into some operating systems and home gateways. While it is not a transition technology that Comcast planned to invest in due to limitations related to performance, we did observe poor performance when 6to4 was used by our customers. In many cases, these customers were not even aware that 6to4 was enabled by default or that their device or operating system was attempting to use 6to4 to communicate with IPv6 resources on the Internet. In most cases, we observed that 6to4-enabled operating systems and devices were attempting to use a 6to4 relay infrastructure hosted by a midwestern university. In order to improve the Internet experience for Comcast customers who are using 6to4, whether knowingly or not, we have decided to operate 6to4 relays on a temporary, trial basis. Comcast has decided to deploy 6to4 relays in five locations around our network to improve performance and predictability, as compared to operating relays from a single location. These 6to4 relays are available via the standard 6to4 Anycast IP address, according to RFC 3068, which is 192.88.99.1. Devices attempting to use 6to4 within our network should automatically discover and utilize these new 6to4 relays, without end user intervention or configuration. The first pair of these relays was activated today. We plan to activate the remaining three within the next seven to ten days. We plan to monitor the performance of the 6to4 relays, to measure any beneficial effects resulting from adding these elements to our network. As our IPv6 trials evolve and we develop our plans for 2011 and beyond, we will assess our plans to support 6to4 moving forward. John ========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com w) http://www.comcast6.net =========================================
These are good news. However, if Comcast provides native IPv6 to their customers, then the IPv6 native customers don't need these 6to4 relays? Airport Extreme, Linksys and other user equipment, enable IPv6 by doing 6to4 tunnels, so what this press release says, is that there are many users who are already on IPv6 via Comcast network but not native? Providing relays close to them, is a good transition move. Alternatively, the measurement of this 6to4 bandwidth on IPv4 may give you an idea of the demand for IPv6 from your customers? May be you detected a non null number here? I'm just trying to understand more IPv6 by the examples. I'm personally using 6to4 at home, and experiencing some MTU issues, which seems related to some PTB packets suppressed on the way between some end points, and that can depend on which 6to4 relay I'm using. Still trying to debug this (I'm not too fanatic about it, but work on it when I have a bit of time). I thought I would mention that. The WAND people have done some good studies: http://www.ripe.net/ripe/meetings/ripe-60/presentations/Stasiewicz-Measureme... At the office, I have a more classical tunnel with he.net and do not have any issue there. ----- Original Message ----- From: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com> To: "NANOG" <nanog@nanog.org> Sent: Sunday, 29 August, 2010 5:49:30 AM Subject: Comcast enables 6to4 relays FYI - thought this would be of interest to some of you, there will be more news on this front shortly. http://www.comcast6.net/ 6to4 Relays Activated Tuesday, August 17, 2010 As we started our IPv6 trials, we began to observe an increase in 6to4 relay traffic. 6to4 is a transition mechanism built into some operating systems and home gateways. While it is not a transition technology that Comcast planned to invest in due to limitations related to performance, we did observe poor performance when 6to4 was used by our customers. In many cases, these customers were not even aware that 6to4 was enabled by default or that their device or operating system was attempting to use 6to4 to communicate with IPv6 resources on the Internet.
Franck, As you know 6to4 is enabled by default in many cases and is used perhaps more than folks realize. Because of this and other observations we decided to deploy our own relays. This does not alter our plans for our native dual stack trials, in fact, I hope to have more news on this front soon. It is true that 6to4 has challenges, some of these may be related to how 6to4 relays have been deployed. Others may be related to the protocol itself. Either way, by deploying our own we observed an improvement, we hope others have as well. John On 8/28/10 6:06 PM, "Franck Martin" <franck@genius.com> wrote:
These are good news.
However, if Comcast provides native IPv6 to their customers, then the IPv6 native customers don't need these 6to4 relays?
Airport Extreme, Linksys and other user equipment, enable IPv6 by doing 6to4 tunnels, so what this press release says, is that there are many users who are already on IPv6 via Comcast network but not native? Providing relays close to them, is a good transition move. Alternatively, the measurement of this 6to4 bandwidth on IPv4 may give you an idea of the demand for IPv6 from your customers? May be you detected a non null number here?
I'm just trying to understand more IPv6 by the examples.
I'm personally using 6to4 at home, and experiencing some MTU issues, which seems related to some PTB packets suppressed on the way between some end points, and that can depend on which 6to4 relay I'm using. Still trying to debug this (I'm not too fanatic about it, but work on it when I have a bit of time). I thought I would mention that. The WAND people have done some good studies: http://www.ripe.net/ripe/meetings/ripe-60/presentations/Stasiewicz-Measureme... s_of_IPv6_Path_MTU_Discovery_Behaviour.pdf
At the office, I have a more classical tunnel with he.net and do not have any issue there.
----- Original Message ----- From: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com> To: "NANOG" <nanog@nanog.org> Sent: Sunday, 29 August, 2010 5:49:30 AM Subject: Comcast enables 6to4 relays
FYI - thought this would be of interest to some of you, there will be more news on this front shortly.
6to4 Relays Activated Tuesday, August 17, 2010
As we started our IPv6 trials, we began to observe an increase in 6to4 relay traffic. 6to4 is a transition mechanism built into some operating systems and home gateways. While it is not a transition technology that Comcast planned to invest in due to limitations related to performance, we did observe poor performance when 6to4 was used by our customers. In many cases, these customers were not even aware that 6to4 was enabled by default or that their device or operating system was attempting to use 6to4 to communicate with IPv6 resources on the Internet.
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
John Jason Brzozowski <john_brzozowski@cable.comcast.com> writes:
This does not alter our plans for our native dual stack trials, in fact, I hope to have more news on this front soon.
comcast native dual stack is working fine at my house. "traceroute6 -q1 mol.redbarn.org" shows details.
On 8/29/10 6:25 AM, John Jason Brzozowski wrote:
Franck,
As you know 6to4 is enabled by default in many cases and is used perhaps more than folks realize. Because of this and other observations we decided to deploy our own relays.
Right prior to this the nearest 6to4 relay router from the vantage-point of comcast customers was at AMSIX. It's a given that you're going to have path asymmetry, in this case however it was frequently worse in one direction than in the other. This ought greatly improve the performance of existing devices located at comcast's customers.
This does not alter our plans for our native dual stack trials, in fact, I hope to have more news on this front soon.
It is true that 6to4 has challenges, some of these may be related to how 6to4 relays have been deployed. Others may be related to the protocol itself. Either way, by deploying our own we observed an improvement, we hope others have as well.
John
On 8/28/10 6:06 PM, "Franck Martin" <franck@genius.com> wrote:
These are good news.
However, if Comcast provides native IPv6 to their customers, then the IPv6 native customers don't need these 6to4 relays?
Airport Extreme, Linksys and other user equipment, enable IPv6 by doing 6to4 tunnels, so what this press release says, is that there are many users who are already on IPv6 via Comcast network but not native? Providing relays close to them, is a good transition move. Alternatively, the measurement of this 6to4 bandwidth on IPv4 may give you an idea of the demand for IPv6 from your customers? May be you detected a non null number here?
I'm just trying to understand more IPv6 by the examples.
I'm personally using 6to4 at home, and experiencing some MTU issues, which seems related to some PTB packets suppressed on the way between some end points, and that can depend on which 6to4 relay I'm using. Still trying to debug this (I'm not too fanatic about it, but work on it when I have a bit of time). I thought I would mention that. The WAND people have done some good studies: http://www.ripe.net/ripe/meetings/ripe-60/presentations/Stasiewicz-Measureme... s_of_IPv6_Path_MTU_Discovery_Behaviour.pdf
At the office, I have a more classical tunnel with he.net and do not have any issue there.
----- Original Message ----- From: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com> To: "NANOG" <nanog@nanog.org> Sent: Sunday, 29 August, 2010 5:49:30 AM Subject: Comcast enables 6to4 relays
FYI - thought this would be of interest to some of you, there will be more news on this front shortly.
6to4 Relays Activated Tuesday, August 17, 2010
As we started our IPv6 trials, we began to observe an increase in 6to4 relay traffic. 6to4 is a transition mechanism built into some operating systems and home gateways. While it is not a transition technology that Comcast planned to invest in due to limitations related to performance, we did observe poor performance when 6to4 was used by our customers. In many cases, these customers were not even aware that 6to4 was enabled by default or that their device or operating system was attempting to use 6to4 to communicate with IPv6 resources on the Internet.
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
Before we turned up our own relays the closest 6to4 relay was a single relay hosted by a mid-western university. Regardless where the next closest relays are located deploying our own resulted in improvements (as you pointed out below). John On 8/29/10 12:24 PM, "Joel Jaeggli" <joelja@bogus.com> wrote:
On 8/29/10 6:25 AM, John Jason Brzozowski wrote:
Franck,
As you know 6to4 is enabled by default in many cases and is used perhaps more than folks realize. Because of this and other observations we decided to deploy our own relays.
Right prior to this the nearest 6to4 relay router from the vantage-point of comcast customers was at AMSIX. It's a given that you're going to have path asymmetry, in this case however it was frequently worse in one direction than in the other.
This ought greatly improve the performance of existing devices located at comcast's customers.
This does not alter our plans for our native dual stack trials, in fact, I hope to have more news on this front soon.
It is true that 6to4 has challenges, some of these may be related to how 6to4 relays have been deployed. Others may be related to the protocol itself. Either way, by deploying our own we observed an improvement, we hope others have as well.
John
On 8/28/10 6:06 PM, "Franck Martin" <franck@genius.com> wrote:
These are good news.
However, if Comcast provides native IPv6 to their customers, then the IPv6 native customers don't need these 6to4 relays?
Airport Extreme, Linksys and other user equipment, enable IPv6 by doing 6to4 tunnels, so what this press release says, is that there are many users who are already on IPv6 via Comcast network but not native? Providing relays close to them, is a good transition move. Alternatively, the measurement of this 6to4 bandwidth on IPv4 may give you an idea of the demand for IPv6 from your customers? May be you detected a non null number here?
I'm just trying to understand more IPv6 by the examples.
I'm personally using 6to4 at home, and experiencing some MTU issues, which seems related to some PTB packets suppressed on the way between some end points, and that can depend on which 6to4 relay I'm using. Still trying to debug this (I'm not too fanatic about it, but work on it when I have a bit of time). I thought I would mention that. The WAND people have done some good studies: http://www.ripe.net/ripe/meetings/ripe-60/presentations/Stasiewicz-Measureme nt s_of_IPv6_Path_MTU_Discovery_Behaviour.pdf
At the office, I have a more classical tunnel with he.net and do not have any issue there.
----- Original Message ----- From: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com> To: "NANOG" <nanog@nanog.org> Sent: Sunday, 29 August, 2010 5:49:30 AM Subject: Comcast enables 6to4 relays
FYI - thought this would be of interest to some of you, there will be more news on this front shortly.
6to4 Relays Activated Tuesday, August 17, 2010
As we started our IPv6 trials, we began to observe an increase in 6to4 relay traffic. 6to4 is a transition mechanism built into some operating systems and home gateways. While it is not a transition technology that Comcast planned to invest in due to limitations related to performance, we did observe poor performance when 6to4 was used by our customers. In many cases, these customers were not even aware that 6to4 was enabled by default or that their device or operating system was attempting to use 6to4 to communicate with IPv6 resources on the Internet.
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
Is there a list of 6to4 relays? I'm curious. Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
Franck Martin wrote:
Is there a list of 6to4 relays?
I'm curious.
Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
Sprint has one which is absolutely horrible (or was a year or two ago). I'd recommend any and every ISP to setup a 6to4, even if it runs over a v6 tunnel to HE. Accidentally getting one from someone else can give you exceptionally broken 6to4 connectivity. Being anycast, I'd say routeviews might be a good place to check for some, but often times they are hidden within the networks they serve. That being said, 6to4 itself is often horrible. It works fine if you are talking 6to4 direct to the remote site (vs using a relay), but relays often break and are hard to troubleshoot due to their nature. In the last year, my 6to4 tunnel has peaked at 6.44mb/s (1day average) but more common peaks are 150-250kb/s (5min average). My tunnel to he.net was running 4-8 times that including the 6to4 relay serving all my customers + native traffic for 15 hosts and 2 servers. It's hard to get accurate on recent traffic loads as much of my v6 traffic shifted to dual stacked peers and I don't have a method of separating the v4/v6 traffic in the graphs (me thinks it's time to test ipv6 over mpls). Jack
Well I found my 6to4 gateway: traceroute to 192.88.99.1 (192.88.99.1), 30 hops max, 40 byte packets 10 te3-3.ccr01.ind01.atlas.cogentco.com (154.54.3.30) [AS174] 244.965 ms 244.964 ms 244.952 ms 11 38.20.52.226 (38.20.52.226) [AS174] 244.336 ms 38.20.52.222 (38.20.52.222) [AS174] 244.300 ms 266.250 ms 12 38.104.214.6 (38.104.214.6) [AS174] 326.141 ms 326.139 ms 376.926 ms 13 ge-7-0-0.103.rtr.ictc.indiana.gigapop.net (149.165.254.142) [AS10680] 612.357 ms 612.367 ms 612.358 ms 14 rtr3.ul.indiana.gigapop.net (149.165.255.129) [AS10680] 376.777 ms 319.641 ms * and I have so much issues with 6to4 that I have decided to disable it at home (airport extreme). I found out PTB was not transmitted and using scamper and the help of Matthew Luckie there is an odd MTU of 1422 from Internet to me. I suspect the 6to4 relay did not put the MTU to 1280 to be on the safe side... (I saw a recommendation like that on the net). I have seen that the new hardware of the airport extreme has a new firmware that does more IPv6 magic, but old hardware are not yet benefiting this new firmware... Once a new firmware comes I'll re-enable and see... PS: I found scamper to be a very good troubleshooting tool (once you know what to do) and wish it would be on all OS, like traceroute and now tracepath is... ----- Original Message ----- From: "Jack Bates" <jbates@brightok.net> To: "Franck Martin" <franck@genius.com> Cc: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com>, "NANOG" <nanog@nanog.org> Sent: Tuesday, 31 August, 2010 9:44:11 AM Subject: Re: Comcast enables 6to4 relays Franck Martin wrote:
Is there a list of 6to4 relays?
I'm curious.
Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
Sprint has one which is absolutely horrible (or was a year or two ago). I'd recommend any and every ISP to setup a 6to4, even if it runs over a v6 tunnel to HE. Accidentally getting one from someone else can give you exceptionally broken 6to4 connectivity. Being anycast, I'd say routeviews might be a good place to check for some, but often times they are hidden within the networks they serve. That being said, 6to4 itself is often horrible. It works fine if you are talking 6to4 direct to the remote site (vs using a relay), but relays often break and are hard to troubleshoot due to their nature.
Franck Martin wrote:
Well I found my 6to4 gateway: <snip trace> and I have so much issues with 6to4 that I have decided to disable it at home (airport extreme). I found out PTB was not transmitted and using scamper and the help of Matthew Luckie there is an odd MTU of 1422 from Internet to me. I suspect the 6to4 relay did not put the MTU to 1280 to be on the safe side... (I saw a recommendation like that on the net).
Actually, their MTU won't matter. 6to4 is EXTREMELY asymmetric when using relays. The return traffic to you will be via the first router than can support 6to4, not the relay you sent packets to. This is why troubleshooting 6to4 is such a PITA. If your airport supports static tunnels, set one up at tunnel brokers and be done with it. Jack
if only I had a static IPv4 address, but who gives such luxury for free nowadays to a end user? I'm in an end user configuration here The office configuration is working perfectly fine. I'm talking about my experience as a "end user" just enabling IPv6 on my airport extreme and trying to figure out what is happening.... On my return path I got: traceroute from 2001:470:1:74::3 to 2002:7114:4a9d:0:xxxxxxxx 1 2001:470:1:74::1 5.582 ms [mtu: 1500] 2 2001:470:0:2d::2 0.483 ms [mtu: 1500] 3 2001:470:0:30::2 173.747 ms [mtu: 1500] 4 2001:470:0:13b::2 0.848 ms [mtu: 1500] 5 2002:7114:4a9d::1 274.299 ms [mtu: 1480] 6 2002:7114:4a9d:0:xxxxxxxx 299.939 ms [*mtu: 1422] So I suspect on return path I use a HE.Net relay? And yes I agree it is such a PITA to troubleshoot, I cannot pinpoint the issue. Nor I cannot see if I'm the only one with this kind of troubles, I suspect most just give up immediately... ----- Original Message ----- From: "Jack Bates" <jbates@brightok.net> To: "Franck Martin" <franck@genius.com> Cc: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com>, "NANOG" <nanog@nanog.org> Sent: Tuesday, 31 August, 2010 10:14:39 AM Subject: Re: Comcast enables 6to4 relays Franck Martin wrote:
Well I found my 6to4 gateway: <snip trace> and I have so much issues with 6to4 that I have decided to disable it at home (airport extreme). I found out PTB was not transmitted and using scamper and the help of Matthew Luckie there is an odd MTU of 1422 from Internet to me. I suspect the 6to4 relay did not put the MTU to 1280 to be on the safe side... (I saw a recommendation like that on the net).
Actually, their MTU won't matter. 6to4 is EXTREMELY asymmetric when using relays. The return traffic to you will be via the first router than can support 6to4, not the relay you sent packets to. This is why troubleshooting 6to4 is such a PITA. If your airport supports static tunnels, set one up at tunnel brokers and be done with it. Jack
Others may correct me, but... Franck Martin wrote:
5 2002:7114:4a9d::1 274.299 ms [mtu: 1480] 6 2002:7114:4a9d:0:xxxxxxxx 299.939 ms [*mtu: 1422]
So I suspect on return path I use a HE.Net relay?
Yes, and it appears that your host is replying back to the office.
And yes I agree it is such a PITA to troubleshoot, I cannot pinpoint the issue. Nor I cannot see if I'm the only one with this kind of troubles, I suspect most just give up immediately...
Given that you seem to complete the trace one direction, but not the other, I can think of 2 things, 1) Your originating host may be breaking PMTU (so the packet you send is too large and doesn't make it, you never resend a smaller packet, but it works when tracerouting from the other side due to PMTU working in that direction and you are responding with the same size packet). 2) If hosts in the middle on your traceroute to the destination don't have access to 6to4 relay, they won't be able to reply back to you, so you may end up with timeout hops before getting to the endpoint. Jack
Jack Bates writes:
1) Your originating host may be breaking PMTU (so the packet you send is too large and doesn't make it, you never resend a smaller packet, but it works when tracerouting from the other side due to PMTU working in that direction and you are responding with the same size packet).
Your mentioning PMTU discovery issues in connection with 6to4 prompts me to confess how our open 6to4 relay has probably contributed to the perception of brokenness of 6to4 for quite a while *blush*. The relay runs on a Cisco 7600 with PFC3 - btw. this is an excellent platform to run an 6to4 relay on, because it can do the encap/decap in hardware if configured correctly. At some point of the relay becoming popular (load currently fluctuates between 80 Mb/s and 200 Mb/s), I noticed that our router very often failed to send ICMPv6 messages such as "packet too big". First I suspected our control-plane rate-limit (CoPP) configuration, but couldn't find anything there. Finally I found that I had to configure a generous "ipv6 icmp error-interval"[1], because the (invisible) default configuration will only permit one such ICMPv6 message to be generated every 100 milliseconds, and that's WAY insufficient for a popular router. We currently use ipv6 icmp error-interval 2 100 (max. steady state rate 500 ICMPv6s/second - one every 2 milliseonds - with bursts up to 100) with no ill effects. Note that the same rate-limit will also cause stars in IPv6 traceroutes through popular routers if the default setting is used. The issue is probably not restricted to Cisco, as the ICMPv6 standard (RFC 4443) mandates that ICMPv6 error messages be rate limited. It even has good (if hand-wavy) guidance on how to arrive at defaults - the values used on our Cisco 7600 (and possibly all other IOS devices?) correspond to the RFC's suggestion for "a small/mid-size device" *hrmpf* (yes Randy, I know I should get real routers :-). Anybody knows which defaults are used by other devices/vendors? In general, rate limits are very useful for protecting routers' notoriously underpowered control planes, but (1) it's hard to come up with reasonable defaults, and (2) I suspect that not most people don't monitor them (because that's often hard), and thus won't notice when normal traffic levels trip these limits. -- Simon. [1] See http://www.cisco.com/en/US/docs/ios/ipv6/command/reference/ipv6_06.html#wp21...
On Wed, 01 Sep 2010 23:18:55 +0200 Simon Leinen <simon.leinen@switch.ch> wrote:
Jack Bates writes:
1) Your originating host may be breaking PMTU (so the packet you send is too large and doesn't make it, you never resend a smaller packet, but it works when tracerouting from the other side due to PMTU working in that direction and you are responding with the same size packet).
Your mentioning PMTU discovery issues in connection with 6to4 prompts me to confess how our open 6to4 relay has probably contributed to the perception of brokenness of 6to4 for quite a while *blush*.
The relay runs on a Cisco 7600 with PFC3 - btw. this is an excellent platform to run an 6to4 relay on, because it can do the encap/decap in hardware if configured correctly.
At some point of the relay becoming popular (load currently fluctuates between 80 Mb/s and 200 Mb/s), I noticed that our router very often failed to send ICMPv6 messages such as "packet too big".
That potentially starts to explain why I haven't noticed PMTUD issues on my 6to4 tunnel that I've been running for a number of years, and have been quite surprised be and somewhat doubtful of people to say there are issues. I've pumped the MTU up to 1472 on it to suit my PPPoE 1492 MTU, instead of leaving it at 1280, and haven't had any issues that have made me suspect PMTUD. I'm in Asia Pacific so I've probably either been using 6to4 gateways that are lightly used, or ones that have had this parameter changed.
First I suspected our control-plane rate-limit (CoPP) configuration, but couldn't find anything there.
Finally I found that I had to configure a generous "ipv6 icmp error-interval"[1], because the (invisible) default configuration will only permit one such ICMPv6 message to be generated every 100 milliseconds, and that's WAY insufficient for a popular router. We currently use
ipv6 icmp error-interval 2 100
(max. steady state rate 500 ICMPv6s/second - one every 2 milliseonds - with bursts up to 100) with no ill effects.
Note that the same rate-limit will also cause stars in IPv6 traceroutes through popular routers if the default setting is used.
The issue is probably not restricted to Cisco, as the ICMPv6 standard (RFC 4443) mandates that ICMPv6 error messages be rate limited. It even has good (if hand-wavy) guidance on how to arrive at defaults - the values used on our Cisco 7600 (and possibly all other IOS devices?) correspond to the RFC's suggestion for "a small/mid-size device" *hrmpf* (yes Randy, I know I should get real routers :-).
Anybody knows which defaults are used by other devices/vendors?
In general, rate limits are very useful for protecting routers' notoriously underpowered control planes, but (1) it's hard to come up with reasonable defaults, and (2) I suspect that not most people don't monitor them (because that's often hard), and thus won't notice when normal traffic levels trip these limits. -- Simon. [1] See http://www.cisco.com/en/US/docs/ios/ipv6/command/reference/ipv6_06.html#wp21...
----- Original Message -----
From: "Mark Smith" <nanog@85d5b20a518b8f6864949bd940457dc124746ddc.nosense.org> To: "Simon Leinen" <simon.leinen@switch.ch> Cc: "Brzozowski" <john_brzozowski@cable.comcast.com>, "NANOG" <nanog@nanog.org>, John Sent: Thursday, 2 September, 2010 9:50:28 AM Subject: Re: ICMPv6 rate limits breaking PMTUD (and traceroute) [Re: Comcast enables 6to4 relays] On Wed, 01 Sep 2010 23:18:55 +0200 Simon Leinen <simon.leinen@switch.ch> wrote:
Jack Bates writes:
1) Your originating host may be breaking PMTU (so the packet you send is too large and doesn't make it, you never resend a smaller packet, but it works when tracerouting from the other side due to PMTU working in that direction and you are responding with the same size packet).
Your mentioning PMTU discovery issues in connection with 6to4 prompts me to confess how our open 6to4 relay has probably contributed to the perception of brokenness of 6to4 for quite a while *blush*.
The relay runs on a Cisco 7600 with PFC3 - btw. this is an excellent platform to run an 6to4 relay on, because it can do the encap/decap in hardware if configured correctly.
At some point of the relay becoming popular (load currently fluctuates between 80 Mb/s and 200 Mb/s), I noticed that our router very often failed to send ICMPv6 messages such as "packet too big".
That potentially starts to explain why I haven't noticed PMTUD issues on my 6to4 tunnel that I've been running for a number of years, and have been quite surprised be and somewhat doubtful of people to say there are issues. I've pumped the MTU up to 1472 on it to suit my PPPoE 1492 MTU, instead of leaving it at 1280, and haven't had any issues that have made me suspect PMTUD. I'm in Asia Pacific so I've probably either been using 6to4 gateways that are lightly used, or ones that have had this parameter changed.
Well I have an issue with a MTU of 1434 on a 6to4 link, but on my return path (I'm a client using an airport). The MTU seems real odd, and it fails all the time, but then I notice, some time of the day it is ok, so may be it coincide with load on the relay? I have the strong feeling this is exactly what I'm experiencing... I can provide extensive debug information if interested, but do not want to bore the list with details...
On Wed, 1 Sep 2010, Simon Leinen wrote:
Your mentioning PMTU discovery issues in connection with 6to4 prompts me to confess how our open 6to4 relay has probably contributed to the perception of brokenness of 6to4 for quite a while *blush*.
We're also doing the same thing, 6to4 on 7600. Could you please share your configuration you're using on the tunnel interfaces? I thought 6to4 traffic was only supposed to be 1280MTU? -- Mikael Abrahamsson email: swmike@swm.pp.se
On Wed, 1 Sep 2010, Simon Leinen wrote:
Note that the same rate-limit will also cause stars in IPv6 traceroutes through popular routers if the default setting is used. ... Anybody knows which defaults are used by other devices/vendors?
I've noticed 6to4 relay rate-limiter blackholes before (e.g. in Your.org relay in AMS, got quickly fixed once I reported it). FWIW, Linux default is 1000pps and BSD has 100pps which is too low for a popular relay. In our relays we've used 1000-3000pps. The majority of ICMPv6's is caused by windows boxes testing the relay's liveness. Depending on the MTU configuration of the relay's tunnel interface (there isn't a BCP on this I think), you will also get more issues if you run the relay at MTU=1280 rather than (say) 1480. But using 1480 may result in an IPv4 blackhole if you source packets from an anycast address and your destination is e.g. behind PPPoE, so... -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
I actually agree with the below. Using whatever you learn "today" via BGP does not appear to be a good plan. 6to4 in particular becomes very unpredictable and does in fact contribute to brokenness. I am not saying deploying your own will make 6to4 good or great, it will however, help to make it less broken and more controllable. If operators deployed their own it would likely be beneficial to their subscribers, particularly if there is a lot of 6to4. I would also go out on a limb and say that absent and poorly deployed 6to4 relays collectively play hugely into the perception of brokenness. There was a noticeable difference deploying our own 6to4 relays compared to using the next or first available. I am not saying the other relay was poorly run, just saying it is different having one on-net versus off-net. John On 8/30/10 5:44 PM, "Jack Bates" <jbates@brightok.net> wrote:
Franck Martin wrote:
Is there a list of 6to4 relays?
I'm curious.
Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
Sprint has one which is absolutely horrible (or was a year or two ago). I'd recommend any and every ISP to setup a 6to4, even if it runs over a v6 tunnel to HE. Accidentally getting one from someone else can give you exceptionally broken 6to4 connectivity. Being anycast, I'd say routeviews might be a good place to check for some, but often times they are hidden within the networks they serve.
That being said, 6to4 itself is often horrible. It works fine if you are talking 6to4 direct to the remote site (vs using a relay), but relays often break and are hard to troubleshoot due to their nature.
In the last year, my 6to4 tunnel has peaked at 6.44mb/s (1day average) but more common peaks are 150-250kb/s (5min average).
My tunnel to he.net was running 4-8 times that including the 6to4 relay serving all my customers + native traffic for 15 hosts and 2 servers. It's hard to get accurate on recent traffic loads as much of my v6 traffic shifted to dual stacked peers and I don't have a method of separating the v4/v6 traffic in the graphs (me thinks it's time to test ipv6 over mpls).
Jack
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
found it: http://www.bgpmon.net/6to4.php?week=4 Not what I call a big list, considering... ----- Original Message ----- From: "Franck Martin" <franck@genius.com> To: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com> Cc: "NANOG" <nanog@nanog.org> Sent: Tuesday, 31 August, 2010 9:21:58 AM Subject: Re: Comcast enables 6to4 relays Is there a list of 6to4 relays? I'm curious. Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
In a message written on Tue, Aug 31, 2010 at 09:47:14AM +1200, Franck Martin wrote:
found it:
http://www.bgpmon.net/6to4.php?week=4
Not what I call a big list, considering...
Note that these are people willing to provide a 6to4 relay free to the entire Internet. There are plenty of people who offer 6to4 inside their own network for their own customers, but never advertise the prefix to world+dog. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
The Comcast 6to4 relays are not on this list, perhaps this is a list of open ones? John On 8/30/10 5:47 PM, "Franck Martin" <franck@genius.com> wrote:
found it:
http://www.bgpmon.net/6to4.php?week=4
Not what I call a big list, considering...
----- Original Message ----- From: "Franck Martin" <franck@genius.com> To: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com> Cc: "NANOG" <nanog@nanog.org> Sent: Tuesday, 31 August, 2010 9:21:58 AM Subject: Re: Comcast enables 6to4 relays
Is there a list of 6to4 relays?
I'm curious.
Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
Yes this is the list of visible relays as seen from the BGP backbone monitoring... If you don't offer your relays to the rest of the world, they won't show up there... ----- Original Message ----- From: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com> To: "Franck Martin" <franck@genius.com> Cc: "NANOG" <nanog@nanog.org> Sent: Tuesday, 31 August, 2010 10:09:17 AM Subject: Re: Comcast enables 6to4 relays The Comcast 6to4 relays are not on this list, perhaps this is a list of open ones? John On 8/30/10 5:47 PM, "Franck Martin" <franck@genius.com> wrote:
found it:
http://www.bgpmon.net/6to4.php?week=4
Not what I call a big list, considering...
----- Original Message ----- From: "Franck Martin" <franck@genius.com> To: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com> Cc: "NANOG" <nanog@nanog.org> Sent: Tuesday, 31 August, 2010 9:21:58 AM Subject: Re: Comcast enables 6to4 relays
Is there a list of 6to4 relays?
I'm curious.
Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
So the biggest problem with 6to4 relays is that they are not ubiquitous and/or well run. Does offering your relays to the world, thereby improving the odds of off-net traffic returning through them >0, actually offer an improvement to your own users' experience with 6to4? Joe Franck Martin wrote:
Yes this is the list of visible relays as seen from the BGP backbone monitoring...
If you don't offer your relays to the rest of the world, they won't show up there...
----- Original Message ----- From: "John Jason Brzozowski"<john_brzozowski@cable.comcast.com> To: "Franck Martin"<franck@genius.com> Cc: "NANOG"<nanog@nanog.org> Sent: Tuesday, 31 August, 2010 10:09:17 AM Subject: Re: Comcast enables 6to4 relays
The Comcast 6to4 relays are not on this list, perhaps this is a list of open ones?
John
On 8/30/10 5:47 PM, "Franck Martin"<franck@genius.com> wrote:
found it:
http://www.bgpmon.net/6to4.php?week=4
Not what I call a big list, considering...
----- Original Message ----- From: "Franck Martin"<franck@genius.com> To: "John Jason Brzozowski"<john_brzozowski@cable.comcast.com> Cc: "NANOG"<nanog@nanog.org> Sent: Tuesday, 31 August, 2010 9:21:58 AM Subject: Re: Comcast enables 6to4 relays
Is there a list of 6to4 relays?
I'm curious.
Also, I'm also curious to know if ISPs in Europe (which are more advanced in IPv6 deployment) have experienced the same issues?
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
On 30/08/2010 23:47, Franck Martin wrote:
found it:
http://www.bgpmon.net/6to4.php?week=4
Not what I call a big list, considering...
The list seems to be showing relays that announce both the IPv4 and the IPv6 anycast prefixes. I have noticed a number of deployments that announce the (in)famous IPv4 prefix and then consider their deployment complete. I suspect that there is a lack of 2002::/16 announcements and this would be contributing to the regular problems with return paths. Obviously the IPv6 content networks benefit the most from having a relay translating back to IPv4. Anyone have experience with this? -- Graham Beneke
The list seems to be showing relays that announce both the IPv4 and the IPv6 anycast prefixes.
I have noticed a number of deployments that announce the (in)famous IPv4 prefix and then consider their deployment complete. I suspect that there is a lack of 2002::/16 announcements and this would be contributing to the regular problems with return paths.
Obviously the IPv6 content networks benefit the most from having a relay translating back to IPv4.
Anyone have experience with this?
-- Graham Beneke
Is there a reason not to advertise more specific prefixes from 2002::/16 to ensure that traffic for your v4 routes comes back to your own 6to4 router? If for example all my users have v4 addresses in 192.0.2.0/24, I could advertise 2002:C002:0000::/40 instead of or in addition to the full 2002::/16. Cheers. Mitchell
On 2010-08-31 08:22, Mitchell Warden wrote: [..]
Is there a reason not to advertise more specific prefixes from 2002::/16 to ensure that traffic for your v4 routes comes back to your own 6to4 router?
If for example all my users have v4 addresses in 192.0.2.0/24, I could advertise 2002:C002:0000::/40 instead of or in addition to the full 2002::/16.
The RFC forbids that with a good reason, as then we'll end up importing the IPv4 BGP table into IPv6... not something we want to see (unless one loves to import 300k routes in there, I guess people will really start whining about that though ;). Greets, Jeroen
I think this http://www.gossamer-threads.com/lists/nsp/ipv6/13537 may answer some of the questions on how to make it work correctly. I like the fact the 6to4 gateway should be on a separate machine that BGP with the main router. If the gateway dies, the routes are withdrawn and clients go and look for another gateway somewhere else.... ----- Original Message ----- From: "Jeroen Massar" <jeroen@unfix.org> To: "Mitchell Warden" <wardenm@wardenm.net> Cc: nanog@nanog.org Sent: Tuesday, 31 August, 2010 6:46:52 PM Subject: Re: Comcast enables 6to4 relays On 2010-08-31 08:22, Mitchell Warden wrote: [..]
Is there a reason not to advertise more specific prefixes from 2002::/16 to ensure that traffic for your v4 routes comes back to your own 6to4 router?
If for example all my users have v4 addresses in 192.0.2.0/24, I could advertise 2002:C002:0000::/40 instead of or in addition to the full 2002::/16.
The RFC forbids that with a good reason, as then we'll end up importing the IPv4 BGP table into IPv6... not something we want to see (unless one loves to import 300k routes in there, I guess people will really start whining about that though ;). Greets, Jeroen
In message <20100831062203.be89ed60@mail.wardenm.net>, "Mitchell Warden" writes :
The list seems to be showing relays that announce both the IPv4 and the IPv6 anycast prefixes.
I have noticed a number of deployments that announce the (in)famous IPv4 prefix and then consider their deployment complete. I suspect that there is a lack of 2002::/16 announcements and this would be contributing to the regular problems with return paths.
Obviously the IPv6 content networks benefit the most from having a relay translating back to IPv4.
Anyone have experience with this?
-- Graham Beneke
Is there a reason not to advertise more specific prefixes from 2002::/16 to ensure that traffic for your v4 routes comes back to your own 6to4 router?
If for example all my users have v4 addresses in 192.0.2.0/24, I could advertise 2002:C002:0000::/40 instead of or in addition to the full 2002::/16.
Cheers. Mitchell
Which would end up with the entire set of IPv4 routes in IPv6. This is not a good idea. Just do you part and encapuslate/decapsulate as soon as possible and let the other end do the same. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
As the 6to4 is an "default" option on Apple Airport Extreme to enable ipv6, I would have thought that Apple would have provided a few gateways? Same for Microsoft that has it in its OS? Reminds me of the ntp servers issue built in on some devices...
On Sat, 28 Aug 2010, John Jason Brzozowski wrote:
In most cases, we observed that 6to4-enabled operating systems and devices were attempting to use a 6to4 relay infrastructure hosted by a midwestern university.
Before that they used our (Tele2) 6to4 relays in Amsterdam and Paris. I think this was discussed here 1-2 years back. I couldn't find it, but <http://gpshead.blogspot.com/2009/01/consumer-router-ipv6-firewall-fail.html> says the same thing. I urge more people to look up what 6to4 relays you're using and set up your own if needed. People *are* using it and you not doing it is making things worse for your customers. Yes, 6to4 is generally bad but it's out there. Everybody needs to think about it. -- Mikael Abrahamsson email: swmike@swm.pp.se
Mikael, I agree with your points and echoed them in my earlier reply. 6to4 is out there and is likely not to go away any time soon. Folks should definitely see what 6to4 relay they default to, you might be surprised (or not). FWIW - I updated ARIN's wiki for 6to4 relay deployment with some generic updates. I will add some more text shortly that folks may find useful if they decide to deploy their own 6to4 relays. John On 8/29/10 3:19 AM, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
On Sat, 28 Aug 2010, John Jason Brzozowski wrote:
In most cases, we observed that 6to4-enabled operating systems and devices were attempting to use a 6to4 relay infrastructure hosted by a midwestern university.
Before that they used our (Tele2) 6to4 relays in Amsterdam and Paris. I think this was discussed here 1-2 years back.
I couldn't find it, but <http://gpshead.blogspot.com/2009/01/consumer-router-ipv6-firewall-fail.html> says the same thing.
I urge more people to look up what 6to4 relays you're using and set up your own if needed. People *are* using it and you not doing it is making things worse for your customers. Yes, 6to4 is generally bad but it's out there. Everybody needs to think about it.
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
On Sat, Aug 28, 2010 at 10:49, John Jason Brzozowski <john_brzozowski@cable.comcast.com> wrote:
As we started our IPv6 trials, we began to observe an increase in 6to4 relay traffic. 6to4 is a transition mechanism built into some operating systems and home gateways. While it is not a transition technology that Comcast planned to invest in due to limitations related to performance, we did observe poor performance when 6to4 was used by our customers. In many cases, these customers were not even aware that 6to4 was enabled by default or that their device or operating system was attempting to use 6to4 to communicate with IPv6 resources on the Internet.
In most cases, we observed that 6to4-enabled operating systems and devices were attempting to use a 6to4 relay infrastructure hosted by a midwestern university. In order to improve the Internet experience for Comcast customers who are using 6to4, whether knowingly or not, we have decided to operate 6to4 relays on a temporary, trial basis.
Comcast has decided to deploy 6to4 relays in five locations around our network to improve performance and predictability, as compared to operating relays from a single location. These 6to4 relays are available via the standard 6to4 Anycast IP address, according to RFC 3068, which is 192.88.99.1. Devices attempting to use 6to4 within our network should automatically discover and utilize these new 6to4 relays, without end user intervention or configuration.
The first pair of these relays was activated today. We plan to activate the remaining three within the next seven to ten days. We plan to monitor the performance of the 6to4 relays, to measure any beneficial effects resulting from adding these elements to our network. As our IPv6 trials evolve and we develop our plans for 2011 and beyond, we will assess our plans to support 6to4 moving forward.
Hi John, First of all, that's great news -- I think it will help a lot. Have you also considered deploying Teredo relays? I'm guessing that there are quite a few Windows Vista+ systems that could benefit from having a few closer Teredo relays and it's probably a similar amount of traffic that you're seeing compared to 6to4 tunnels. Best, Bill Fehring
Hey Bill, No plans for Teredo at this time. John On 8/30/10 5:57 PM, "Bill Fehring" <lists@billfehring.com> wrote:
On Sat, Aug 28, 2010 at 10:49, John Jason Brzozowski <john_brzozowski@cable.comcast.com> wrote:
As we started our IPv6 trials, we began to observe an increase in 6to4 relay traffic. 6to4 is a transition mechanism built into some operating systems and home gateways. While it is not a transition technology that Comcast planned to invest in due to limitations related to performance, we did observe poor performance when 6to4 was used by our customers. In many cases, these customers were not even aware that 6to4 was enabled by default or that their device or operating system was attempting to use 6to4 to communicate with IPv6 resources on the Internet.
In most cases, we observed that 6to4-enabled operating systems and devices were attempting to use a 6to4 relay infrastructure hosted by a midwestern university. In order to improve the Internet experience for Comcast customers who are using 6to4, whether knowingly or not, we have decided to operate 6to4 relays on a temporary, trial basis.
Comcast has decided to deploy 6to4 relays in five locations around our network to improve performance and predictability, as compared to operating relays from a single location. These 6to4 relays are available via the standard 6to4 Anycast IP address, according to RFC 3068, which is 192.88.99.1. Devices attempting to use 6to4 within our network should automatically discover and utilize these new 6to4 relays, without end user intervention or configuration.
The first pair of these relays was activated today. We plan to activate the remaining three within the next seven to ten days. We plan to monitor the performance of the 6to4 relays, to measure any beneficial effects resulting from adding these elements to our network. As our IPv6 trials evolve and we develop our plans for 2011 and beyond, we will assess our plans to support 6to4 moving forward.
Hi John,
First of all, that's great news -- I think it will help a lot.
Have you also considered deploying Teredo relays? I'm guessing that there are quite a few Windows Vista+ systems that could benefit from having a few closer Teredo relays and it's probably a similar amount of traffic that you're seeing compared to 6to4 tunnels.
Best,
Bill Fehring
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
On Mon, Aug 30, 2010 at 3:34 PM, Jack Bates <jbates@brightok.net> wrote:
John Jason Brzozowski wrote:
Hey Bill,
No plans for Teredo at this time.
I'm sure, like us, you looked at what was involved and said, "eh, easier to just provide native v6 than deal with that mess." 6to4 is definitely a more friendly protocol for the network engineer.
100% agree about deploying native IPv6 being better than any "transition mechanism". Time spent deploying IPv6 native is time well spent with real long term payback (ROI ...). Any significant amount of time spent on any transition mechanism that is not strategic to your business, especially unmanaged off-network tunnels, will be painful. Better to work on the piloting a long term solution that will scale and fits the long term reality of living in a IPv4-only + IPv6-only + dual-stack internet. Although i think 6to4 has hurt IPv6 adoption more than help it, i believe comcast is doing the right thing given the fact that many people are running 6to4 and don't even know it.... and without on-net relays, the experience leaves a lot to be desired.. Regards, Cameron ======== http://groups.google.com/group/tmoipv6beta ========
On Mon, 30 Aug 2010, Jack Bates wrote:
I'm sure, like us, you looked at what was involved and said, "eh, easier to just provide native v6 than deal with that mess." 6to4 is definitely a more friendly protocol for the network engineer.
End users are using 6to4 and Teredo, if an ISP isn't providing their own relays, someone else is and the performance might be good or bad. Same logic applies to Teredo as to 6to4 and why if you're an ISP who cares, you should run your own. Your customers are using both, whether they know it or not. -- Mikael Abrahamsson email: swmike@swm.pp.se
Enabled two more 6to4 relays this morning. :) John On 8/31/10 1:14 AM, "Mikael Abrahamsson" <swmike@swm.pp.se> wrote:
On Mon, 30 Aug 2010, Jack Bates wrote:
I'm sure, like us, you looked at what was involved and said, "eh, easier to just provide native v6 than deal with that mess." 6to4 is definitely a more friendly protocol for the network engineer.
End users are using 6to4 and Teredo, if an ISP isn't providing their own relays, someone else is and the performance might be good or bad.
Same logic applies to Teredo as to 6to4 and why if you're an ISP who cares, you should run your own. Your customers are using both, whether they know it or not.
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
Way to go! more! more! ;) ----- Original Message ----- From: "John Jason Brzozowski" <john_brzozowski@cable.comcast.com> To: "NANOG" <nanog@nanog.org> Sent: Tuesday, 31 August, 2010 6:18:21 PM Subject: UPDATED - Comcast enables 6to4 relays Enabled two more 6to4 relays this morning. :) John
On Tue, 31 Aug 2010, John Jason Brzozowski wrote:
Enabled two more 6to4 relays this morning. :)
Out of curiousity, what is the aggregate Mbps load on the relays? Related question is the platform on which these are run. -- Pekka Savola "You each name yourselves king, yet the Netcore Oy kingdom bleeds." Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
On 8/31/10 7:36 AM, "Pekka Savola" <pekkas@netcore.fi> wrote:
On Tue, 31 Aug 2010, John Jason Brzozowski wrote:
Enabled two more 6to4 relays this morning. :)
Out of curiousity, what is the aggregate Mbps load on the relays? Related question is the platform on which these are run. [jjmb] for now I can say the traffic doubled after the first pair of relays were deployed. They are Linux based.
========================================= John Jason Brzozowski Comcast Cable e) mailto:john_brzozowski@cable.comcast.com o) 609-377-6594 m) 484-962-0060 w) http://www.comcast6.net =========================================
Mikael Abrahamsson wrote:
End users are using 6to4 and Teredo, if an ISP isn't providing their own relays, someone else is and the performance might be good or bad.
Teredo usage isn't common enough on our network to warrant the work. Very few apps will activate it is my guess.
Same logic applies to Teredo as to 6to4 and why if you're an ISP who cares, you should run your own. Your customers are using both, whether they know it or not.
A customer is more likely (not always) to know when teredo has been activated. I've considered putting it in, but it is not friendly in many ways. 6to4 is usually running on routers in various pops. Teredo, I'd have to back feed to a server farm. This doesn't make for ideal traffic patterns even with bandwidth being so low. Then there is the "customer is unaware" fact. If the customer is unaware that their NAT is being pierced for IPv6 communication, then we have contributed to decreasing their security. For this reason, it might not be completely unwarranted for an ISP to block teredo all together. 6to4 doesn't suffer from this as there is no NAT traversal. Jack
On Tue, 31 Aug 2010, Jack Bates wrote:
Teredo usage isn't common enough on our network to warrant the work. Very few apps will activate it is my guess.
<http://ipv6.tele2.net/teredo_stats.php> As I stated, either your users are using your Teredo server, or they're using someone elses. Not running one yourself doesn't mean your users aren't running Teredo.
A customer is more likely (not always) to know when teredo has been activated. I've considered putting it in, but it is not friendly in many ways. 6to4 is usually running on routers in various pops. Teredo, I'd have to back feed to a server farm. This doesn't make for ideal traffic patterns even with bandwidth being so low.
Then the traffic is going to someone elses, how is that more optimal?
Then there is the "customer is unaware" fact. If the customer is unaware that their NAT is being pierced for IPv6 communication, then we have contributed to decreasing their security. For this reason, it might not be completely unwarranted for an ISP to block teredo all together. 6to4 doesn't suffer from this as there is no NAT traversal.
Blocking Teredo completely is a whole other discussion. Also, some NAT gateways will support a single device behind it doing Proto 41, so saying 6to4 has no NAT traversal and thus won't work beind NAT isn't true in all cases. -- Mikael Abrahamsson email: swmike@swm.pp.se
On 2010-08-31 16:54, Mikael Abrahamsson wrote:
On Tue, 31 Aug 2010, Jack Bates wrote:
Teredo usage isn't common enough on our network to warrant the work. Very few apps will activate it is my guess.
<http://ipv6.tele2.net/teredo_stats.php>
As I stated, either your users are using your Teredo server, or they're using someone elses. Not running one yourself doesn't mean your users aren't running Teredo.
psssst it's relay not server :) I guess everybody mixes that up one day or another, it is also a reason why just having Microsoft's default server is not a huge issue. [..]
Then there is the "customer is unaware" fact. If the customer is unaware that their NAT is being pierced for IPv6 communication, then we have contributed to decreasing their security. For this reason, it might not be completely unwarranted for an ISP to block teredo all together. 6to4 doesn't suffer from this as there is no NAT traversal.
Jack: there are a lot more methods to infect a host than this as there are lots and lots of p2p protocols which are being used by C&C botnets. And never forgot about this very simple protocol called HTTP(S).
Blocking Teredo completely is a whole other discussion.
Also, some NAT gateways will support a single device behind it doing Proto 41, so saying 6to4 has no NAT traversal and thus won't work beind NAT isn't true in all cases.
Flaky but it works. Generally they just tag 'oh protocol 41 has to go to host X' thus when you enable a second all traffic either moves there or sticks at the first. It's the reason Teredo/AYIYA/etc exist ;) Greets, Jeroen
Jeroen Massar wrote:
Jack: there are a lot more methods to infect a host than this as there are lots and lots of p2p protocols which are being used by C&C botnets. And never forgot about this very simple protocol called HTTP(S).
I agree, though let's consider HTTP. If a firewall is set to filter it, yet you are tunneling through with IPv6, you've bypassed your HTTP filters which may, among other things, provide AV protection. I recognize that there are plenty of ways to infect a machine. My concern is that teredo can bypass firewall security and relies upon host security to protect the computer. Unfortunately, not everyone utilizes host security and has dependence on network firewalls. Jack
On 2010-08-31 18:07, Jack Bates wrote:
Jeroen Massar wrote:
Jack: there are a lot more methods to infect a host than this as there are lots and lots of p2p protocols which are being used by C&C botnets. And never forgot about this very simple protocol called HTTP(S).
I agree, though let's consider HTTP. If a firewall is set to filter it, yet you are tunneling through with IPv6, you've bypassed your HTTP filters which may, among other things, provide AV protection. I recognize that there are plenty of ways to infect a machine. My concern is that teredo can bypass firewall security and relies upon host security to protect the computer. Unfortunately, not everyone utilizes host security and has dependence on network firewalls.
If you have a "firewall" which only blocks things it knows you don't have a proper firewall. The only 'firewall' that makes sense anyway is the one which is unplugged. There is always a way out of the network as long as you can have a controlled box on the outside that you can send packets to and from. Network firewalls are great for 'centralized' mitigation and trying to at least cut out most of the wrong stuff you don't want to see as an administrator, but if you are truly serious about it then you should be deploying monitoring on the hosts that are attached to your network too, just remember that a lot of people have VPN software, connect from home to that VPN and do other weird setups (Skype for instance, BitTorrent) where there are possibilities to bypass your "firewall". And indeed, there is no proper solution for that unless you create a walled garden and allow people to only connect to known services and only allow them to send minimal messages, no flash, no other cruft like images. Steganography is also so much fun, Too many ways, even per default and also if someone really wants. Only thing you can do is keep your eyes wide open and of course define what you are really trying to protect against, as one can just as well just use sneakernet to move data around. Greets, Jeroen
Jeroen Massar wrote:
just remember that a lot of people have VPN software, connect from home to that VPN and do other weird setups (Skype for instance, BitTorrent) where there are possibilities to bypass your "firewall".
I agree. My concern here is that we are dealing with improper firewalls. We are dealing with ignorance, and we have M$ enabling teredo by default (though not active until they install the appropriate app). Creating what is essentially a public vpn through a firewall without the user being aware of it is insecure. For all the wonderful popups that vista+ gives, it amazes me that teredo isn't one of them. 6to4 doesn't suffer the same issues. Primarily because RFC1918 addressing can't be used in 6to4. This means that at a minimum, the router has to participate or the host behind it must be manually configured with a 6to4 address (for the proto 41 pass through to work). Neither is an automatic traversal of the router's policies without user knowledge. Jack
On Tue, 31 Aug 2010 12:02:56 CDT, Jack Bates said:
6to4 doesn't suffer the same issues. Primarily because RFC1918 addressing can't be used in 6to4. This means that at a minimum, the router has to participate or the host behind it must be manually configured with a 6to4 address (for the proto 41 pass through to work). Neither is an automatic traversal of the router's policies without user knowledge.
I'm sure that somewhere, somebody has managed to configure their gear such that one of those two things can happen without user knowledge. :)
On 2010-08-31 19:02, Jack Bates wrote:
Jeroen Massar wrote:
just remember that a lot of people have VPN software, connect from home to that VPN and do other weird setups (Skype for instance, BitTorrent) where there are possibilities to bypass your "firewall".
I agree. My concern here is that we are dealing with improper firewalls.
Then fix your firewall, next to those administrators. You seem to love managing things centrally, but you forget that if you do things the MS way: Active Directory / Domains, that Teredo&6to4 are automatically turned off unless you turn the policy switch on. MS thus takes care of this.
We are dealing with ignorance, and we have M$ enabling teredo by default (though not active until they install the appropriate app).
Creating what is essentially a public vpn through a firewall without the user being aware of it is insecure. For all the wonderful popups
No, Teredo & 6to4 (and ISATAP) are enabled per default on Vista/Win7 and also XP if you install IPv6, if the host has native it will use that, if is in non-RFC1918 space it will try 6to4, if it is in RFC1918 space it will try Teredo. This is great for getting IPv6 connectivity going. It is 'bad' for a corporate network. You can work around it two ways: enable native IPv6 or use active directory and voila the moment that a host is in the domain it does not do this automatically. If you do not administer the hosts then you don't have anything that you can do anyway as there will be software on those hosts which you will not like and which will easily pierce through your puny firewall. DNS tunnels near always work for that matter. that vista+
gives, it amazes me that teredo isn't one of them.
As there are no listening ports being opened and only outbound traffic is permitted, just the same as the IPv4 adapter, how is this 'dangerous' ? (unless the IPv6 stack is breakable)
6to4 doesn't suffer the same issues. Primarily because RFC1918 addressing can't be used in 6to4. This means that at a minimum, the router has to participate or the host behind it must be manually configured with a 6to4 address (for the proto 41 pass through to work). Neither is an automatic traversal of the router's policies without user knowledge.
If you have one person setting up ICS on their machine and they have enabled IPv6 voila the whole network gets IPv6, that thus does not solve your problem either. Or are you monitoring IPv6 RAs etc? I think you have to move to better analyzing & monitoring your network and more control over the hosts which participate in that network. Greets, Jeroen
Jeroen Massar wrote:
If you have one person setting up ICS on their machine and they have enabled IPv6 voila the whole network gets IPv6, that thus does not solve your problem either. Or are you monitoring IPv6 RAs etc?
Setting up ICS with IPv6 is user knowledge in my opinion. In addition, the ICS will handle the firewall rules unless the user chooses to turn it off.
I think you have to move to better analyzing & monitoring your network and more control over the hosts which participate in that network.
My concern is as an ISP that has customers who are unaware that their little routers aren't filtering all of their packets. There are a million ways they might get infected or have security problems. However, teredo is obviously a circumvention of protection they *think* they have. Corporate networks can secure their own networks (or not, but they are held to a higher standard than average home user and failure to protect is their own fault). Jack
On 2010-08-31 19:32, Jack Bates wrote:
Jeroen Massar wrote:
If you have one person setting up ICS on their machine and they have enabled IPv6 voila the whole network gets IPv6, that thus does not solve your problem either. Or are you monitoring IPv6 RAs etc?
Setting up ICS with IPv6 is user knowledge in my opinion. In addition, the ICS will handle the firewall rules unless the user chooses to turn it off.
I think you have to move to better analyzing & monitoring your network and more control over the hosts which participate in that network.
My concern is as an ISP that has customers who are unaware that their little routers aren't filtering all of their packets. There are a million ways they might get infected or have security problems. However, teredo is obviously a circumvention of protection they *think* they have.
There is no circumvention here. Teredo is the same as having a P2P app (take Skype as a random example) that connects to an outside host and uses that to relay messages to something else. Allowing outside hosts to use that network to connect to your inbound host. Teredo does not enable more inbound connections than before, unless a an App supports IPv6, but then that app was installed by the user thus they want it to run. Also note that XP/2k3/Vista/Seven/2k8 all have firewalls per default that support IPv6 and that handle IPv4 and IPv6 exactly the same: ask the user with an annoying popup. Vista/Seven/2k8 even (can) do that for outbound connections. The only thing you can do to help your users is to provide them with proper education and to explain them to keep up to date and run the right tools and not click anywhere they can.... and that is a mission which is near impossible. Teredo though is far from your worst worry. Just check how many "Teredo", or heck, IPv6 related infections you have and how many you have who have autodialers and the gazillion of other botnets on their hosts. You can sleep very tight over your perceived "Teredo" problem ;) Greets, Jeroen
The only thing you can do to help your users is to provide them with proper education and to explain them to keep up to date and run the right tools and not click anywhere they can.... and that is a mission which is near impossible.
I thought user education in threat management was long ago abandoned as a realistic defense mechanism. Don't get me wrong, I loved my users when I was supporting a desktop fleet; but the key to their survival was always policy implementation through Active Directory; back in the day, blocking executable files in email prevented a lot more problems than training users not to open them did. Don't get me wrong, every little bit helps. But when you consider your security with a scrutinous eye, you should always ignore the question 'how educated are my users'. It's irrelevant.
On 2010-08-31 19:58, Nathan Eisenberg wrote:
The only thing you can do to help your users is to provide them with proper education and to explain them to keep up to date and run the right tools and not click anywhere they can.... and that is a mission which is near impossible.
I thought user education in threat management was long ago abandoned as a realistic defense mechanism. Don't get me wrong, I loved my users when I was supporting a desktop fleet; but the key to their survival was always policy implementation through Active Directory; back in the day, blocking executable files in email prevented a lot more problems than training users not to open them did.
When you control the hosts in your network then indeed that works quite well and is most very likely the best approach, though it fails miserably again when users don't want to be part of your control. If you are an ISP then you don't control the hosts of your users and then the only thing left is to educate... which is near impossible as you state.
Don't get me wrong, every little bit helps. But when you consider your security with a scrutinous eye, you should always ignore the question 'how educated are my users'. It's irrelevant.
As long as you check the PDF viewer version of the ladies at the HR department ;) Greets, Jeroen
1. I completely agree with Jeroen 2. Jack, if you have specific concerns that Jeroen hasn't answered, feel free to ping me off line. I own Teredo in Windows. Sean from "M$" -----Original Message----- From: Jeroen Massar [mailto:jeroen@unfix.org] Sent: Tuesday, August 31, 2010 10:40 AM To: Jack Bates Cc: NANOG Subject: Re: Teredo and 'firewalls' (Re: Comcast enables 6to4 relays) On 2010-08-31 19:32, Jack Bates wrote:
Jeroen Massar wrote:
If you have one person setting up ICS on their machine and they have enabled IPv6 voila the whole network gets IPv6, that thus does not solve your problem either. Or are you monitoring IPv6 RAs etc?
Setting up ICS with IPv6 is user knowledge in my opinion. In addition, the ICS will handle the firewall rules unless the user chooses to turn it off.
I think you have to move to better analyzing & monitoring your network and more control over the hosts which participate in that network.
My concern is as an ISP that has customers who are unaware that their little routers aren't filtering all of their packets. There are a million ways they might get infected or have security problems. However, teredo is obviously a circumvention of protection they *think* they have.
There is no circumvention here. Teredo is the same as having a P2P app (take Skype as a random example) that connects to an outside host and uses that to relay messages to something else. Allowing outside hosts to use that network to connect to your inbound host. Teredo does not enable more inbound connections than before, unless a an App supports IPv6, but then that app was installed by the user thus they want it to run. Also note that XP/2k3/Vista/Seven/2k8 all have firewalls per default that support IPv6 and that handle IPv4 and IPv6 exactly the same: ask the user with an annoying popup. Vista/Seven/2k8 even (can) do that for outbound connections. The only thing you can do to help your users is to provide them with proper education and to explain them to keep up to date and run the right tools and not click anywhere they can.... and that is a mission which is near impossible. Teredo though is far from your worst worry. Just check how many "Teredo", or heck, IPv6 related infections you have and how many you have who have autodialers and the gazillion of other botnets on their hosts. You can sleep very tight over your perceived "Teredo" problem ;) Greets, Jeroen
participants (20)
-
Bill Fehring -
Cameron Byrne -
Franck Martin -
Graham Beneke -
Jack Bates -
Jeroen Massar -
Joe Maimon -
Joel Jaeggli -
John Jason Brzozowski -
Leo Bicknell -
Mark Andrews -
Mark Smith -
Mikael Abrahamsson -
Mitchell Warden -
Nathan Eisenberg -
Paul Vixie -
Pekka Savola -
Sean Siler -
Simon Leinen -
Valdis.Kletnieks＠vt.edu