Owen, On Sat, Sep 18, 2021 at 23:51 Owen DeLong <owen@delong.com> wrote:
On Sep 18, 2021, at 12:34 , Victor Kuarsingh <victor@jvknet.com> wrote:
On Sat, Sep 18, 2021 at 2:39 PM John Levine <johnl@iecc.com> wrote:
It appears that Owen DeLong via NANOG <owen@delong.com> said:
Glad you noted this. Thinking this was/is purely a hardware cycle problem related to normal/forced upgrade strategies. On that point, most hardware I know of from 2004 in larger networks is long fully depreciated and sweating assets 15+ years can happen, but I don't personally think this is the biggest issue.
It’s certainly not purely hardware, but it also doesn’t require solving the entire problem across the entire backend.
What is urgently needed is to fix the customer-facing front end so that it speaks both protocols (or at least speaks IPv6).
This is a great question. So when we approached this (going back a while now) this is what I thought too. But I was responsible to solve this end to end. So just updating front end (CPEs, network routers, DHCP, provisioning, IP address planning, security, peering/transit, staff training, test harnesses/plans, etc) was not sufficient to turn on IPv6. The high cost and time challenge issues were not upgrading backend servers to IPv6, but backend provisioning systems, tools, customer support tools, their training, and related items. These systems were far more complex and costly to address (especially when considering opportunity cost). Without all of this in place, we could not actually deploy IPv6 for production use (it’s not just Turing it on, its our ability to operate it down to the person answer the phone, the technician that installs and/or fixes/replaces items in home). Also at that time vendor CPE hardware was not as far along on IPv6 readiness (approx mid-decade 2000s). Getting reliable code at that time was hard with a lot of brokenness (which also could not go into production until it was reliable). Perhaps this a non-issue today, but it certainly was not a something that was “ready to go” even 10-15 years. This (IPv6 readiness in CPE code) was also competing with other priorities often blowing up timelines which meant it had to wait as not releasing code into production on a strict timeline was not an option for business reasons. All said and done, we got it completed, but it far most costly than we first thought, and IPv4 costs did not go away after we were completed. Sure it’s possible to reduce those costs with an aggressive program, but it is not as simple as many think.
As you noted John, its the plethora of software, support systems, tooling, and most important in many environments - legacy customer management and provisioning systems that can be the limiting factor. I recall looking back when leading IPv6 turn-up, those were the biger problems to solve for. Often such systems are extremely expensive to touch and working on them required prioritization against direct revenue generating projects (opportunity cost) . Replacing routers was just a money problem.
Why do those systems have to be updated in order to deliver the service to the customer in both protocols?
Addressed in above comment. When it comes to turning on IPv6 and reducing your dependency on IPv4, your mileage may vary (in terms of real costs, complexities and deployment). Regards, Victor K
I mean sure, you’ll probably need to fix some logging databases that think that a customers address can be logged as a uint32_t, but that’s a relatively small number of systems that need to be updated in that case and it’s not like expanding the field to uint128_t in the database is incompatible with the old software during the upgrade process.
I am by far not saying I agree with choices made by hold-outs, but I also understand this is for many, not just an engineering problem to solve.
I agree there are some systems that may make this more complex in some environments, but I don’t agree that it’s impossible (or even particularly difficult) for a content provider to deliver their content on both protocols today even if they don’t upgrade their back-end systems.
Owen