In message <CAD6AjGT8GZuS5_pu4+z2UcqOMFOn8k_i8GdtQ0bwmvy3Un1EOg@mail.gmail.com> , Ca By writes:

On Wed, Jun 10, 2015 at 1:22 PM, Nicholas Warren <nwarren@barryelectric.com> wrote:

...

Sincere apologies if this e-mail is inappropriate for this audience, We are (going to be) a startup ISP building a new network from the ground up. I was hoping I could get an opinion, or two, on how everyone feels about 464XLAT. I saw what everyone was saying about it in the 'Android doesn't support DHCPv6' discussion, but what about in the wireline side of things? The main reason we are even considering 464XLAT as opposed to dual-stack (the latter is, in my ignorant opinion, the better option.) is the fear of IPv4 depletion that we think might hit ARIN between now and the start of next year; causing us to pay a premium for IPv4 in the gray market. So I guess the real question here would be: is our fear real, or is it just bug on the wall? If our fear is real, what should we implement so that our users can still get to the v4 internet, are we even thinking soberly by suggesting 464XLAT? Thanks, - Nich

Yes, your fears about IPv4 are correct.

If you have a look at ARIN PPML lately, you can see some pretty intense "discussion" about companies exporting ARIN addresses to CCNIC and so on.

As a greenfield, you should definitely be focused on IPv6-only to the edge solutions. DS-lite, MAP-E, and 464XLAT come to mind.

DS-lite is the oldest and most common in wireline. 464XLAT is more common in mobile. MAP-E and MAP-T have not yet been deployed at the same scale as DS-lite and 464XLAT yet AFAIK, not sure if they will be.

You could also simply do dual-stack with private space and CGN to the end user using RFC1918 (10.0.0.0/8, 100.64.0.0/10)

Regards, CB

464XLAT is a abomination that grew from NAT64/DNS64 despite DNS64 not working with DNSSEC. NAT64/DNS64 was pushed as a "short term solution" as DNS64 cuts off IPv4 access if there is a IPv6 address for the remote site as the client only asks for AAAA addresses. In practice the address selection rules move most traffic from IPv4 to IPv6 anyway so there is no need to have DNS64. -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

A network needs users or it is useless. I am curious as to how your native IPv6 network communicated with (if at all) the v4 world. Has anyone confronted you about your network being IPv6? I might have problems with reading comprehension, but in your statement " So you might position to pitch upfront your new world Internet service from day one.", do you mean pitch as in, setup; or pitch as, into the trash. Thank you, - Nich Warren -----Original Message----- From: Bob Evans [mailto:bob@FiberInternetCenter.com] Sent: Thursday, June 11, 2015 9:20 AM To: Nicholas Warren Cc: nanog@nanog.org Subject: Re: Greenfield 464XLAT (In January) Actually , there is no better audience that I know of to ask this question. And my information might be more marketing related and hardware skeptical. My IPv6 direction choice was much easier than yours. You need to figure out how to build an IPv4 network today from scratch in a world where the IPv4 bus ride seats have largely assigned. When we setup our IPv6 ability, I chose to build a native IPv6 network. Tunneling and translation devices left me wondering about packet flow at those gateway points. Aside from verbal sales assurances, I still had the feeling that under loads these devices would break momentarily or cause latency issues. For web and email services it's not a big issue. Sure everyone could show me a twitch game playing well or a video conference call, but what happens when the device is under load or attacked ? Will service latency be detected by a cleaver well known gamer ? One that points to the issue as a flaw that makes others think our network is unusable for all kinds of services ? Overcome issues like "this ISP forces you to use IPv6" ? The hardware costs can be small compared to consumer perceptions marketing dollars. So you might position to pitch upfront your new world Internet service from day one. European and Comcast has been implementing NAT 6 related things for years. My son made me move his connection to the smallest bandwidth DSL on ATT for his games. However, our Comcast has been fine perfectly for watching Amazon and Netflix streaming (most of the time). Thank You Bob Evans CTO

Sincere apologies if this e-mail is inappropriate for this audience, We are (going to be) a startup ISP building a new network from the ground up. I was hoping I could get an opinion, or two, on how everyone feels about 464XLAT. I saw what everyone was saying about it in the 'Android doesn't support DHCPv6' discussion, but what about in the wireline side of things? The main reason we are even considering 464XLAT as opposed to dual-stack (the latter is, in my ignorant opinion, the better option.) is the fear of IPv4 depletion that we think might hit ARIN between now and the start of next year; causing us to pay a premium for IPv4 in the gray market. So I guess the real question here would be: is our fear real, or is it just bug on the wall? If our fear is real, what should we implement so that our users can still get to the v4 internet, are we even thinking soberly by suggesting 464XLAT? Thanks, - Nich

On Thu, Jun 11, 2015 at 7:19 AM, Bob Evans <bob@fiberinternetcenter.com> wrote:

Actually , there is no better audience that I know of to ask this question. And my information might be more marketing related and hardware skeptical.

My IPv6 direction choice was much easier than yours. You need to figure out how to build an IPv4 network today from scratch in a world where the IPv4 bus ride seats have largely assigned.

When we setup our IPv6 ability, I chose to build a native IPv6 network. Tunneling and translation devices left me wondering about packet flow at those gateway points. Aside from verbal sales assurances, I still had the feeling that under loads these devices would break momentarily or cause latency issues. For web and email services it's not a big issue. Sure everyone could show me a twitch game playing well or a video conference call, but what happens when the device is under load or attacked ? Will service latency be detected by a cleaver well known gamer ? One that points to the issue as a flaw that makes others think our network is unusable for all kinds of services ? Overcome issues like "this ISP forces you to use IPv6" ? The hardware costs can be small compared to consumer perceptions marketing dollars. So you might position to pitch upfront your new world Internet service from day one.

European and Comcast has been implementing NAT 6 related things for years. My son made me move his connection to the smallest bandwidth DSL on ATT for his games. However, our Comcast has been fine perfectly for watching Amazon and Netflix streaming (most of the time).

I have been running across more and more and more people that are actually doing that - using two isps - one for games and voice, and the other for streaming and web. I had no idea how prevalent it was. Still don't, all my evidence is anecdotal (and it all points at bufferbloat related problems).

Thank You Bob Evans CTO

...

Sincere apologies if this e-mail is inappropriate for this audience, We are (going to be) a startup ISP building a new network from the ground up. I was hoping I could get an opinion, or two, on how everyone feels about 464XLAT. I saw what everyone was saying about it in the 'Android doesn't support DHCPv6' discussion, but what about in the wireline side of things? The main reason we are even considering 464XLAT as opposed to dual-stack (the latter is, in my ignorant opinion, the better option.) is the fear of IPv4 depletion that we think might hit ARIN between now and the start of next year; causing us to pay a premium for IPv4 in the gray market. So I guess the real question here would be: is our fear real, or is it just bug on the wall? If our fear is real, what should we implement so that our users can still get to the v4 internet, are we even thinking soberly by suggesting 464XLAT? Thanks, - Nich

I so would not want to be a new entrant in this market. When last I tried various ipv4 over ipv6 tunnelling methods (in nicaragua 2009) it was very hard to get devices that could do it at all. I think common tools I use now (like openwrt) are really far along on every possible encapsulation method, but the state of other gear still lagging.

...

-- Dave Täht What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast

I figured that duel-stack would be the way to go, but I worry that ARIN might not give us space for duel stack out of their reserved pool (https://www.arin.net/policy/nrpm.html#four10), and that this .13 of a /8 won't make it to next year. I suppose that would be a question for the ARIN mailing list? Thank you, - Nich Warren -----Original Message----- From: William Herrin [mailto:bill@herrin.us] Sent: Thursday, June 11, 2015 12:13 PM To: Nicholas Warren Cc: nanog@nanog.org Subject: Re: Greenfield 464XLAT (In January) On Wed, Jun 10, 2015 at 4:22 PM, Nicholas Warren <nwarren@barryelectric.com> wrote:

Sincere apologies if this e-mail is inappropriate for this audience,

Hi Nich, Looks like the correct audience to me.

We are (going to be) a startup ISP building a new network from the ground up. [...] The main reason we are even considering 464XLAT as opposed to dual-stack (the latter is, in my ignorant opinion, the better option.) is the fear of IPv4 depletion that we think might hit ARIN between now and the start of next year; causing us to pay a premium for IPv4 in the gray market.

Your customers will require end-to-end IPv4 for the foreseeable future. 464XLAT can provide natted IPv4 using an internal IPv6 infrastructure in special circumstances. Specifically: you must have sufficient control of the customer equipment to compel it to employ 464XLAT to provide IPv4 services to the customer. If your customers lease phones from you and your phone vendors build in 464XLAT support, T-Mobile has demonstrated that this is practical. If your customers bring generic Macs and PCs with the odd Linux user in the mix (their equipment, not yours), you may be asking for extensive support headaches with 464XLAT. Dual stack with carrier NAT would also handle your IPv4 needs. You'll have an additional expense maintaining both protocols within your infrastructure. Nevertheless, this approach alleviates the need to control the customer premises equipment. Regardless of your approach, DS+NAT or 464XLAT, you will require a comparable number of global IPv4 addresses. Neither technology eliminates your need for IPv4 addresses facing the public Internet. Regardless of your approach, you will need to make provisions to support customers who require a global and/or static IPv4 address without NAT. It need not be part of your basic package, but if it's unavailable at any price you can be sure of getting a PR black eye at some point. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

I am thinking now that our best option would be to go duel-stack lite (RFC6333), after reading what you fellows have to say about 464XLAT. I feel as though I should add that our peer networks (one was started at the end of 2013) are implementing IPv4 only networks; they are pressuring management into thinking that IPv6 is too experimental to deploy, and that IPv4 (only) is the only way to go. Thank you, - Nich Warren -----Original Message----- From: William Herrin [mailto:bill@herrin.us] Sent: Thursday, June 11, 2015 12:13 PM To: Nicholas Warren Cc: nanog@nanog.org Subject: Re: Greenfield 464XLAT (In January) On Wed, Jun 10, 2015 at 4:22 PM, Nicholas Warren <nwarren@barryelectric.com> wrote:

Sincere apologies if this e-mail is inappropriate for this audience,

Hi Nich, Looks like the correct audience to me.

We are (going to be) a startup ISP building a new network from the ground up. [...] The main reason we are even considering 464XLAT as opposed to dual-stack (the latter is, in my ignorant opinion, the better option.) is the fear of IPv4 depletion that we think might hit ARIN between now and the start of next year; causing us to pay a premium for IPv4 in the gray market.

Your customers will require end-to-end IPv4 for the foreseeable future. 464XLAT can provide natted IPv4 using an internal IPv6 infrastructure in special circumstances. Specifically: you must have sufficient control of the customer equipment to compel it to employ 464XLAT to provide IPv4 services to the customer. If your customers lease phones from you and your phone vendors build in 464XLAT support, T-Mobile has demonstrated that this is practical. If your customers bring generic Macs and PCs with the odd Linux user in the mix (their equipment, not yours), you may be asking for extensive support headaches with 464XLAT. Dual stack with carrier NAT would also handle your IPv4 needs. You'll have an additional expense maintaining both protocols within your infrastructure. Nevertheless, this approach alleviates the need to control the customer premises equipment. Regardless of your approach, DS+NAT or 464XLAT, you will require a comparable number of global IPv4 addresses. Neither technology eliminates your need for IPv4 addresses facing the public Internet. Regardless of your approach, you will need to make provisions to support customers who require a global and/or static IPv4 address without NAT. It need not be part of your basic package, but if it's unavailable at any price you can be sure of getting a PR black eye at some point. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

* Baldur Norddahl <baldur.norddahl@gmail.com>

The high tech solution is stuff like MAP where you move the cost out to the CPE. But then you need to control the CPE - if you have that then great. You would still want to sell a non-NAT (and MAP is NAT) to users that require a public IPv4 address, so you still need to go dual stack or use some tunnelling for that.

Hi Baldur, MAP is *not* NAT; that's what's so neat about it. The users do get a public IPv4 address (or prefix!) routed to their CPE's WAN interface, towards which they can accept inbound unsolicited connections. The public IPv4 address could be port-restricted if the operator wants address sharing, but it does not have to be. You could do both at the same time, e.g., giving your "premium" users a /32 or /28, while the standard subscription includes a /32 with 4k ports. I will grant you that MAP-T performs NAT (i.e., protocol translation) internally, but the translations that happens when a packet enters the MAP domain are reversed when it exits. So the IPv4 addresses are transparent end-to-end. MAP-E (and lw4o6 for that matter), on the other hand, has no form of NAT anywhere. (Unless you count the NAPT44 that sits between the subscriber's RFC1918 LAN segment and the CPE's WAN interface, but that's not exactly something that's unique to MAP.) Nicholas: If I were you, before going down the 464XLAT route, I'd first look closely at these technologies, in the order given: 1) MAP (because it is fully stateless) 2) lw4o6 (because it is mostly stateless, i.e., no session tracking) 3) DS-Lite (which, like 464XLAT, is stateful, but you'll have way more CPEs to choose from than with 464XLAT, which is mostly for mobile) Tore