On Mon, 13 Oct 2008, Nathan Ward wrote:

And using a relay at Tele2 AB (Sweden).

So really Max, all your outbound IPv6 packets are going via Sweden, even if they're going to elsewhere in the US.

Well, actually, Tele2 has 6to4 relays in Stockholm, Paris, Amsterdam and Frankfurt. But correct, it's a relay in Europe that is being used.

In order to get around this, encourage your ISP to build a 6to4 relay, which is a couple of commands on a spare Cisco router. For extra points, get them to build out a Teredo relay as well, which is a few commands on a spare Linux box.

We have very good experience with 7600 for 6to4 relay usage. This brings up an interesting question, should we stop announcing our 6to4 relays outside of Europe? Is there consensus in the business how this should be done? I have heard opinions both ways. -- Mikael Abrahamsson email: swmike@swm.pp.se

On 13/10/2008, at 9:53 AM, Stephen Sprunk wrote:

Mikael Abrahamsson wrote:

...

This brings up an interesting question, should we stop announcing our 6to4 relays outside of Europe? Is there consensus in the business how this should be done? I have heard opinions both ways.

I can understand why some folks would say stop, but unfortunately Europe has the closest public 6to4 relays to the US, and our own providers don't seem to want to put any up. That means 6to4 will break for a great many folks who _are_ trying to use IPv6 (like developers trying to get ahead of the curve and make sure their apps don't break when the transition finally happens) but whose providers haven't clued in yet.

I'm sure I sound like a broken record to some, but whenever I see these comments I feel the need to step up and correct them, until I don't see them anymore. By far the biggest end users of IPv6 are non-experimenters. Real end users, many of whom do not know what an IP address is. 6to4 is enabled by default in Vista - any Vista machine with a non- RFC1918 address will use 6to4. It is also available in some linksys routers, and is enabled by default in Apple Airport Extreme.

(My traceroutes to 192.88.99.1 have a next-to-last hop in Amsterdam, and I'm on one of the largest ISPs in the US, which apparently hasn't figured out 6to4, much less native IPv6.)

There are public 6to4 relays in the US, I guess your provider just has a shorter ASPATH to somewhere in Europe. Unfortunately BGP had no idea of geography :-) Re. whether to advertise outside your continent, it really depends whether you're trying to achieve 'good enough' connectivity for 100% of people, or really good connectivity for 95% of people. Perhaps a good way to do it is advertise outside Europe, but have the providers that get your advertisement out there prepend their AS a few times as it leaves. That way, US providers will still prefer US 6to4 relays (ie lower latency) but any who don't get a 192.88.99.0/24 route from the US will us your relay in Europe. Kinda gets you best of both worlds. -- Nathan Ward

On 13/10/2008, at 3:46 PM, Daniel Senie wrote:

At 06:05 PM 10/12/2008, Nathan Ward wrote:

...

On 13/10/2008, at 9:53 AM, Stephen Sprunk wrote:

...

Mikael Abrahamsson wrote:

...

This brings up an interesting question, should we stop announcing our 6to4 relays outside of Europe? Is there consensus in the business how this should be done? I have heard opinions both ways.

I can understand why some folks would say stop, but unfortunately Europe has the closest public 6to4 relays to the US, and our own providers don't seem to want to put any up. That means 6to4 will break for a great many folks who _are_ trying to use IPv6 (like developers trying to get ahead of the curve and make sure their apps don't break when the transition finally happens) but whose providers haven't clued in yet.

I'm sure I sound like a broken record to some, but whenever I see these comments I feel the need to step up and correct them, until I don't see them anymore.

By far the biggest end users of IPv6 are non-experimenters. Real end users, many of whom do not know what an IP address is.

6to4 is enabled by default in Vista - any Vista machine with a non- RFC1918 address will use 6to4. It is also available in some linksys routers, and is enabled by default in Apple Airport Extreme.

Not to rain on anyone's parade, but it'd be interesting (and difficult, unfortunately) to know how many Vista machines are actually on non-RFC1918 addresses. Corporate users are in many cases staying with XP for a while, but they're more likely to have public space than most. A great many home users have a cheap NAT box that provides RFC1918 addresses.

I do wonder whether where the Vista machines on public IPs really are. I also have to wonder if performance is really better when those users are routed over 6to4 in Europe from, say California, or whether they'd actually get better performance if they stuck in a NAT box, resulting in their using IPv4 instead?

Don't worry, you're not raining on my parade if that's what you're concerned about. I don't like Vista/XPSP2 having 6to4, Teredo is the protocol designed to connect end hosts to the IPv6 network. That works through NAT, and is enabled by default on Vista. 6to4 should existing in CPE devices, etc. not in end hosts. Cue religious war. Also, Windows boxes that are part of a domain will only try ISATAP and native IPv6 - they will not attempt to tunnel IPv6 over IPv4 using public relays (ISATAP is an internal thing). I did a bit of stats, and roughly 95% of packets leaving an ISP's aggregation layer were from hosts behind NAT (look at TTL, make assumptions based on initial TTL). So, 6to4 is only on 5% of customers, assuming that % of packets and % of customers are roughly equal. Here's a mini-rant I had about Teredo traffic offlist when someone said they had very little 6to4 traffic. I thought it was on-list. begin. I suspect you'll find that Teredo contributes to a very large amount of it, but you won't be seeing it as you don't have a local Teredo relay (in my understanding of your network, anyway :-) Even then you won't see Teredo<->Teredo, or Teredo<->NonTeredo when NonTeredo is on another network. An interesting way to get a rough idea of how much Teredo<->NotTeredo is going on is to look at the packets going to teredo.ipv6.microsoft.com port 3544/UDP. Every Vista/XPSP2 Teredo client will send a UDP packet there every 30 seconds (IIRC), and then another packet for every new NonTeredo host it wants to talk to. Source UDP port is generally static and unique for each client host, so you can get an idea for unique number of hosts. The periodic packets are going to be 68b (of IPv4+UDP+IPv6 = 68b), whereas the new-connection packets are going to be at least 76b (IPv4+UDP+IPv6+ICMPv6+Echo Request = 76b, then there's also the ICMPv6 Echo Request payload). Obviously you want to add 14b if you've got ethernet headers and what not. If you have netflow anywhere, you should be able to ask it an appropriate question with the above info. That'll tell you number of end-to-end connections there are which may give you some insight there. If you've got a netflow exporter, I'd be more than happy to run stats over the data to figure out what amount of Teredo there is. end. -- Nathan Ward

On 13/10/2008, at 7:24 PM, Mikael Abrahamsson wrote:

On Sun, 12 Oct 2008, Daniel Senie wrote:

...

I do wonder whether where the Vista machines on public IPs really are. I also have to wonder if performance is really better when those users are routed over 6to4 in Europe from, say California, or whether they'd actually get better performance if they stuck in a NAT box, resulting in their using IPv4 instead?

I'd say it's very rare where IPv6 will give you better performance than IPv4 right now.

Regarding where they are, I'd say all over the place. It's very common in my regional market to hand out one or more public IPs, and if the customer doesn't put their own NAT box there, then their Vista computer(s) will have public IPs and will use 6to4.

Regarding 6to4 or Teredo, I've done some testing of my own and the statelessness of 6to4 makes it avoid some of the session setup/NAT travesal magic of Teredo that slows Teredo down. I'd much rather see the NAT boxes do 6to4 and run native on their local LAN segment, than having end hosts do Teredo to get thru the NAT. It'll give the end user a better IPv6 experience.

Long term I agree, but short term I prefer Teredo for regular end users' experience. Where regular end user means an end user communicates with a relatively small number of remote hosts. Several reasons: 1) 6to4 currently lacks a testing mechanism to ensure that it is functioning at startup, and that it is still functioning. Packets are sent and blackholed by the network, resulting in a 90s timeout waiting for a response to the three SYN packets in a TCP connection set up. 90s is a long time for users today, and my experience shows that they consider a service to be 'broken' before they wait for the timeout to expire. 2) If Teredo relays are deployed close to the service (ie. content, etc.) then performance is almost equivalent to IPv4. 6to4 relies on relays being close to both the client and the server, which requires end users' ISPs to build at least *some* IPv6 infrastructure, maintain transit, etc. When you consider that this infrastructure and transit is quite likely to be over long tunnels to weird parts of the world, this is a bad thing. Putting relays close to the content helps for the reverse path (ie. content -> client), however the forward path (client -> content) is likely to perform poorly. -- Nathan Ward

On 10/23/08 6:39 PM, "Tony Hain" <alh-ietf@tndh.net> wrote:

A properly implemented client will do the longest prefix match against that set, so a 6to4 client will go directly to the content provider's 6to4 router, while a native client will take the direct path.

Not quite. Say the server has native IPv6 address 2001::1 and 6to4 IPv6 2002::X. Say the client has native IPv6 address 2003::1 and 6to4 IPv6 2002::Y. Longest prefix match will choose 6to4 over native IPv6. Not good. - Alain.

On 23 Oct 2008, at 18:46, Alain Durand wrote:

On 10/23/08 6:39 PM, "Tony Hain" <alh-ietf@tndh.net> wrote:

...

A properly implemented client will do the longest prefix match against that set, so a 6to4 client will go directly to the content provider's 6to4 router, while a native client will take the direct path.

Not quite. Say the server has native IPv6 address 2001::1 and 6to4 IPv6 2002::X. Say the client has native IPv6 address 2003::1 and 6to4 IPv6 2002::Y. Longest prefix match will choose 6to4 over native IPv6. Not good.

Surely the longest prefix only wins over prefixes that cover the long prefix. What you seem to be suggesting is that a client will somehow be able to deduce the prefix length for two bare v6 addresses received from the DNS, and will choose the one with the longest prefix first. That seems unlikely to be the case in general. Joe

On Mon, 13 Oct 2008, Mikael Abrahamsson wrote:

On Sun, 12 Oct 2008, Daniel Senie wrote:

...

I do wonder whether where the Vista machines on public IPs really are. I also have to wonder if performance is really better when those users are routed over 6to4 in Europe from, say California, or whether they'd actually get better performance if they stuck in a NAT box, resulting in their using IPv4 instead?

I'd say it's very rare where IPv6 will give you better performance than IPv4 right now.

Rare = Absolutely Yes. Impossible = No :-)

Regarding where they are, I'd say all over the place. It's very common in my regional market to hand out one or more public IPs, and if the customer doesn't put their own NAT box there, then their Vista computer(s) will have public IPs and will use 6to4.

Regarding 6to4 or Teredo, I've done some testing of my own and the statelessness of 6to4 makes it avoid some of the session setup/NAT travesal magic of Teredo that slows Teredo down. I'd much rather see the NAT boxes do 6to4 and run native on their local LAN segment, than having end hosts do Teredo to get thru the NAT. It'll give the end user a better IPv6 experience.

Fully agree. Unfortunately not every NATbox/cheap-consumer-router is happy to pass on 6to4 packets to its next hop :-(

-- Mikael Abrahamsson email: swmike@swm.pp.se

Cheers, ------------------------------------------------------------------------- Carlos Friac,as See: Wide Area Network Working Group (WAN) www.gigapix.pt FCCN - Fundacao para a Computacao Cientifica Nacional www.6deploy.org Av. do Brasil, n.101 www.ipv6.eu 1700-066 Lisboa, Portugal, Europe Tel: +351 218440100 Fax: +351 218472167 www.fccn.pt ------------------------------------------------------------------------- The end is near........ see http://ipv4.potaroo.net "Internet is just routes (282391/1511), naming (billions) and... people!" Esta mensagem foi enviada de: / This message was sent from: 2001:690:2080:8004:250:daff:fe3b:2830 Aviso de Confidencialidade Esta mensagem e' exclusivamente destinada ao seu destinatario, podendo conter informacao CONFIDENCIAL, cuja divulgacao esta' expressamente vedada nos termos da lei. Caso tenha recepcionado indevidamente esta mensagem, solicitamos-lhe que nos comunique esse mesmo facto por esta via ou para o telefone +351 218440100 devendo apagar o seu conteudo de imediato. Warning This message is intended exclusively for its addressee. It may contain CONFIDENTIAL information protected by law. If this message has been received due to any error, please notify us via e-mail or by telephone +351 218440100 and delete it immediately.

On 13/10/2008, at 7:18 PM, Mikael Abrahamsson wrote:

On Mon, 13 Oct 2008, Nathan Ward wrote:

...

6to4 is enabled by default in Vista - any Vista machine with a non- RFC1918 address will use 6to4. It is also available in some linksys routers, and is enabled by default in Apple Airport Extreme.

I've been told there is a difference between OEM and non-OEM Vista machines when it comes to Teredo being activated or not.

I've not heard that, I'll be interested if someone can confirm? Perhaps certain vendors disable IPv6 because of the lack of relays etc. in the network causing performance problems.

...

Perhaps a good way to do it is advertise outside Europe, but have the providers that get your advertisement out there prepend their AS a few times as it leaves. That way, US providers will still prefer US 6to4 relays (ie lower latency) but any who don't get a 192.88.99.0/24 route from the US will us your relay in Europe. Kinda gets you best of both worlds.

Yeah, that's been one option as well, prepending 2-3 times to our peers/transit in the US is probably a good middle way.

Regarding some numbers on 6to4 and Teredo usage, I'd like to point people to this thread on the ipv6ops IETF list:

<http://www.ops.ietf.org/lists/v6ops/v6ops.2008/msg01582.html>

If someone has some nice code that'll take a list of IPv6 addresses and break it down to geographical distribution of native/teredo/ 6to4, I'd be more than happy to run it on my data.

Teredo and 6to4 is easy - translate the addresses to IPv4 and geolocate with maxmind geoip or something. IPv6 is harder, you have to build a geolocation database, which I suppose you'd have to build from either origin AS location, or whatever location data the RIR has, for the prefix an address is in. I have been intending on building a map from Teredo and 6to4 using IPv6 addresses from my bittorrent population stuff, and have that as one output of a periodic study. I'm not sure how to do non-Teredo/6to4 addresses though, so if you've got some ideas there I'll whip something up, the Teredo/6to4 stuff is very simple. -- Nathan Ward -- Nathan Ward

On Oct 12, 2008, at 3:53 PM, Stephen Sprunk wrote:

I can understand why some folks would say stop, but unfortunately Europe has the closest public 6to4 relays to the US, and our own providers don't seem to want to put any up.

There are quite a few 6to4 relays outside Europe. I think the problem more is that a lot of providers haven't gotten the hang of routing to an anycasted prefix that so many people are announcing. Or making sure 6to4 is fast is so far down their priority list they just don't care. Either way, peering seems to be a lot more dense in Europe, so they're more likely to be peering with someone announcing a 6to4 relay there. Prefer sending traffic to someone you're peering with (even if it's further away) to someone local that you have to pay for, and you don't always get sensible routing on a prefix like this. From the best that I can tell, here's a reasonably complete list of who are running 6to4 relays... Its not easy to tell what country someone's relay is in, but this is probably close. Oceania/Asia: Australia: 1221 Telstra Korea: 17832 NISA Europe: Denmark: 1835 FSK Net Estonia: 3327 Linxtelecom Finland: 1741 FUNET Germany: 286 kpn.de 5430 Freenet 8767 m-net.de 12816 mwn 15598 IP Exchange 20640 Titan 29259 IABG Teleport 35244 kms.de Italy: 12779 itgate.net Netherlands: 1101 SURFNet 8954 InTouch 26943 Your.Org 31383 Computel Portugal: 1930 FCCN Spain: 16206 Abared Sweden: 1257 Tele2 16150 GlobalTransit Switzerland: 559 switch.ch United Kingdom: 5400 BT North America: US: 59 University of Wisconsin 109 Cisco 1239 Sprint 3344 Kewlio 5050 Pittsburgh Supercomputing Center 6175 Sprint 7019 NTT 10533 Ottawa Internet Exchange 19255 Your.Org 19782 Indiana University 25795 ARP Networks