Multiple vendors' IPv6 issues
Hi NANOG, The company I work for has no business case for being on the IPv6-Internet. However, I am an inquisitive person and I am always looking to learn new things, so about 3 years ago I started down the IPv6 path. This was early 2012. Fast forward to today. We have a /44 presence for our company's multiple sites; All our desktop computers have been on the IPv6 Internet since June, 2012 and we have a few AAAAs in our external DNS for some key services — and, there have been bugs. *Lots* of bugs. Now, maybe (_maybe_) I can have some sympathy for smaller network companies (like Arista Networks at the time) to not quite have their act together as far as IPv6 goes, but for larger, well-established companies to still have critical IPv6 bugs is just inexcusable! This month has just been the most disheartening time working with IPv6. Vendor 1: Aruba Networks. Upon adding an IPv6 address to start managing our WiFi controller over IPv6, I receive a call from our Telecom Lead saying that or WiFi VoIP phones have just gone offline. WHAT? All I did was add an IPv6 address to a management interface which has *nothing* to do with our VoIP system or SSID, ACLs, policies, roles, etc. Vendor 2: Palo Alto Networks: After upgrading our firewalls from a version which has a nasty bug where the IPv6 neighbor table wasn't being cleaned up properly (which would overflow the table and break IPv6), we now have a *new* IPv6 neighbor discovery bug where one of our V6-enabled DMZ hosts just falls of the IPv6 network. The only solution: clear the neighbor table on the Palo Alto or the client (linux) host. Vendor 3: Arista Networks: We are seeing a very similar ND bug with Arista. This one is slightly more interesting because it only started after upgrading our Arista EOS code — and it only appears to affect Virtual Machines which are behind our RedHat Enterprise Virtualization cluster. None of the hundreds of VMware-connected hosts are affected. The symptom is basically the same as the Palo Alto bug. Neighbor table gets in some weird state where ND breaks and the host is unreachable until the neighbor table is cleared. Oh, and the final straw today, which is *almost* leading me to throw in the IPv6 towel completely (for now): On certain hosts (VMs), scp'ing a file over the [Arista] LAN (10 gigabit LAN) takes 5 minutes over IPv6 and <1 second over IPv4. What happened? It really saddens me that it is still not receiving anywhere near the kind of QA (partly as a result of lack of adoption) that IPv4 has. Oh, and let's not forget everybody's "favorite" vendor, Cisco. Why is it, Cisco, that I have to restart my IPv6 OSPF3 process on my ASA every time my Palo Alto firewall crashes and fails over, otherwise none of my VPN clients can connect via IPv6? Why do you hurt me so, IPv6? I just wanted to be friends, and now I just want to break up with you. Maybe we can try to be friends again when your vendors get their shit together. -David
On Tuesday, May 26, 2015, David Sotnick <sotnickd-nanog@ddv.com> wrote:
Hi NANOG,
The company I work for has no business case for being on the IPv6-Internet. However, I am an inquisitive person and I am always looking to learn new things, so about 3 years ago I started down the IPv6 path. This was early 2012.
Fast forward to today. We have a /44 presence for our company's multiple sites; All our desktop computers have been on the IPv6 Internet since June, 2012 and we have a few AAAAs in our external DNS for some key services — and, there have been bugs. *Lots* of bugs.
Now, maybe (_maybe_) I can have some sympathy for smaller network companies (like Arista Networks at the time) to not quite have their act together as far as IPv6 goes, but for larger, well-established companies to still have critical IPv6 bugs is just inexcusable!
This month has just been the most disheartening time working with IPv6.
Vendor 1:
Aruba Networks. Upon adding an IPv6 address to start managing our WiFi controller over IPv6, I receive a call from our Telecom Lead saying that or WiFi VoIP phones have just gone offline. WHAT? All I did was add an IPv6 address to a management interface which has *nothing* to do with our VoIP system or SSID, ACLs, policies, roles, etc.
Vendor 2:
Palo Alto Networks: After upgrading our firewalls from a version which has a nasty bug where the IPv6 neighbor table wasn't being cleaned up properly (which would overflow the table and break IPv6), we now have a *new* IPv6 neighbor discovery bug where one of our V6-enabled DMZ hosts just falls of the IPv6 network. The only solution: clear the neighbor table on the Palo Alto or the client (linux) host.
Vendor 3:
Arista Networks: We are seeing a very similar ND bug with Arista. This one is slightly more interesting because it only started after upgrading our Arista EOS code — and it only appears to affect Virtual Machines which are behind our RedHat Enterprise Virtualization cluster. None of the hundreds of VMware-connected hosts are affected. The symptom is basically the same as the Palo Alto bug. Neighbor table gets in some weird state where ND breaks and the host is unreachable until the neighbor table is cleared.
Oh, and the final straw today, which is *almost* leading me to throw in the IPv6 towel completely (for now): On certain hosts (VMs), scp'ing a file over the [Arista] LAN (10 gigabit LAN) takes 5 minutes over IPv6 and <1 second over IPv4. What happened?
It really saddens me that it is still not receiving anywhere near the kind of QA (partly as a result of lack of adoption) that IPv4 has.
Oh, and let's not forget everybody's "favorite" vendor, Cisco. Why is it, Cisco, that I have to restart my IPv6 OSPF3 process on my ASA every time my Palo Alto firewall crashes and fails over, otherwise none of my VPN clients can connect via IPv6?
Why do you hurt me so, IPv6? I just wanted to be friends, and now I just want to break up with you. Maybe we can try to be friends again when your vendors get their shit together.
-David
Had ipv4 ever hurt you ? Me too. CB
On 27/May/15 01:27, Ca By wrote:
Had ipv4 ever hurt you ?
Me too.
IPv4 still hurts me (in some ways, worse than IPv6), and it's 2015. Figures... You just need to open cases with your vendors and help them fix these issues. Sadly, no way around this. Software is not perfect. The humans that write it, even less so. Mark.
David, While I agree with you that there is no excuse for the general IPv6 brokenness across all vendors, they are just doing what participants on lists like this one tell them. Name&Shame may help a little, but until a large number of people get serious and stop prioritizing IPv4 in their purchasing demands, the vendors are not going to prioritize IPv6. Until the vendors clearly hear a collective "we are not buying this product because IPv6 is broken", everyone will get exactly the behavior you are witnessing. While I appreciate the challenges you are facing, it is likely that you will be helped by documenting the percentage of IPv6 traffic you see when things do work. While it may not be much now, that can change quickly and will provide internal ammunition when you try to take a stand about refusing to use a product. If your IPv6 percentage grows anywhere near the 2x/yr rate that Google has been seeing it won't take long before IPv6 is the driving protocol. For fun, project this http://www.google.com/intl/en/ipv6/statistics.html forward 4 years and hand it to the vendors that can't get their IPv6 act together. Then ask them how they plan to still be in business at that point ...... Tony
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Sotnick Sent: Tuesday, May 26, 2015 4:19 PM To: NANOG Subject: Multiple vendors' IPv6 issues
Hi NANOG,
The company I work for has no business case for being on the IPv6-Internet. However, I am an inquisitive person and I am always looking to learn new things, so about 3 years ago I started down the IPv6 path. This was early 2012.
Fast forward to today. We have a /44 presence for our company's multiple sites; All our desktop computers have been on the IPv6 Internet since June, 2012 and we have a few AAAAs in our external DNS for some key services — and, there have been bugs. *Lots* of bugs.
Now, maybe (_maybe_) I can have some sympathy for smaller network companies (like Arista Networks at the time) to not quite have their act together as far as IPv6 goes, but for larger, well-established companies to still have critical IPv6 bugs is just inexcusable!
This month has just been the most disheartening time working with IPv6.
Vendor 1:
Aruba Networks. Upon adding an IPv6 address to start managing our WiFi controller over IPv6, I receive a call from our Telecom Lead saying that or WiFi VoIP phones have just gone offline. WHAT? All I did was add an IPv6 address to a management interface which has *nothing* to do with our VoIP system or SSID, ACLs, policies, roles, etc.
Vendor 2:
Palo Alto Networks: After upgrading our firewalls from a version which has a nasty bug where the IPv6 neighbor table wasn't being cleaned up properly (which would overflow the table and break IPv6), we now have a *new* IPv6 neighbor discovery bug where one of our V6-enabled DMZ hosts just falls of the IPv6 network. The only solution: clear the neighbor table on the Palo Alto or the client (linux) host.
Vendor 3:
Arista Networks: We are seeing a very similar ND bug with Arista. This one is slightly more interesting because it only started after upgrading our Arista EOS code — and it only appears to affect Virtual Machines which are behind our RedHat Enterprise Virtualization cluster. None of the hundreds of VMware-connected hosts are affected. The symptom is basically the same as the Palo Alto bug. Neighbor table gets in some weird state where ND breaks and the host is unreachable until the neighbor table is cleared.
Oh, and the final straw today, which is *almost* leading me to throw in the IPv6 towel completely (for now): On certain hosts (VMs), scp'ing a file over the [Arista] LAN (10 gigabit LAN) takes 5 minutes over IPv6 and <1 second over IPv4. What happened?
It really saddens me that it is still not receiving anywhere near the kind of QA (partly as a result of lack of adoption) that IPv4 has.
Oh, and let's not forget everybody's "favorite" vendor, Cisco. Why is it, Cisco, that I have to restart my IPv6 OSPF3 process on my ASA every time my Palo Alto firewall crashes and fails over, otherwise none of my VPN clients can connect via IPv6?
Why do you hurt me so, IPv6? I just wanted to be friends, and now I just want to break up with you. Maybe we can try to be friends again when your vendors get their shit together.
-David
On May 27, 2015, at 1:59 AM, Tony Hain <alh-ietf@tndh.net> wrote:
David,
While I agree with you that there is no excuse for the general IPv6 brokenness across all vendors, they are just doing what participants on lists like this one tell them. Name&Shame may help a little, but until a large number of people get serious and stop prioritizing IPv4 in their purchasing demands, the vendors are not going to prioritize IPv6. Until the vendors clearly hear a collective "we are not buying this product because IPv6 is broken", everyone will get exactly the behavior you are witnessing.
While I appreciate the challenges you are facing, it is likely that you will be helped by documenting the percentage of IPv6 traffic you see when things do work. While it may not be much now, that can change quickly and will provide internal ammunition when you try to take a stand about refusing to use a product. If your IPv6 percentage grows anywhere near the 2x/yr rate that Google has been seeing it won't take long before IPv6 is the driving protocol. For fun, project this http://www.google.com/intl/en/ipv6/statistics.html forward 4 years and hand it to the vendors that can't get their IPv6 act together. Then ask them how they plan to still be in business at that point ......
Tony
I like this page even better for that purpose. It does the forward projecting for you and projects 33% in one year and above 90% in 4 years. https://www.vyncke.org/ipv6status/project.php?metric=q&country=us This says that 45% of web pages viewed by people worldwide are available via IPv6 (It does not say that 45% of web pages are available via IPv6, it says that since Facebook and others, which are IPv6 enabled, have more page views than some less popular sites that are IPv4 only and that results in 45% of web pages viewed being available via IPv6.) http://6lab.cisco.com/stats/ http://6lab.cisco.com/stats/information.php#content It is also interesting to sort this page by IPv6 percent. http://www.stateoftheinternet.com/trends-visualizations-ipv6-adoption-ipv4-e...
-----Original Message-----
From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of David Sotnick Sent: Tuesday, May 26, 2015 4:19 PM To: NANOG Subject: Multiple vendors' IPv6 issues
Hi NANOG,
The company I work for has no business case for being on the IPv6-Internet. However, I am an inquisitive person and I am always looking to learn new things, so about 3 years ago I started down the IPv6 path. This was early 2012.
Fast forward to today. We have a /44 presence for our company's multiple sites; All our desktop computers have been on the IPv6 Internet since June, 2012 and we have a few AAAAs in our external DNS for some key services — and, there have been bugs. *Lots* of bugs.
Now, maybe (_maybe_) I can have some sympathy for smaller network companies (like Arista Networks at the time) to not quite have their act together as far as IPv6 goes, but for larger, well-established companies to still have critical IPv6 bugs is just inexcusable!
This month has just been the most disheartening time working with IPv6.
Vendor 1:
Aruba Networks. Upon adding an IPv6 address to start managing our WiFi controller over IPv6, I receive a call from our Telecom Lead saying that or WiFi VoIP phones have just gone offline. WHAT? All I did was add an IPv6 address to a management interface which has *nothing* to do with our VoIP system or SSID, ACLs, policies, roles, etc.
Vendor 2:
Palo Alto Networks: After upgrading our firewalls from a version which has a nasty bug where the IPv6 neighbor table wasn't being cleaned up properly (which would overflow the table and break IPv6), we now have a *new* IPv6 neighbor discovery bug where one of our V6-enabled DMZ hosts just falls of the IPv6 network. The only solution: clear the neighbor table on the Palo Alto or the client (linux) host.
Vendor 3:
Arista Networks: We are seeing a very similar ND bug with Arista. This one is slightly more interesting because it only started after upgrading our Arista EOS code — and it only appears to affect Virtual Machines which are behind our RedHat Enterprise Virtualization cluster. None of the hundreds of VMware-connected hosts are affected. The symptom is basically the same as the Palo Alto bug. Neighbor table gets in some weird state where ND breaks and the host is unreachable until the neighbor table is cleared.
Oh, and the final straw today, which is *almost* leading me to throw in the IPv6 towel completely (for now): On certain hosts (VMs), scp'ing a file over the [Arista] LAN (10 gigabit LAN) takes 5 minutes over IPv6 and <1 second over IPv4. What happened?
It really saddens me that it is still not receiving anywhere near the kind of QA (partly as a result of lack of adoption) that IPv4 has.
Oh, and let's not forget everybody's "favorite" vendor, Cisco. Why is it, Cisco, that I have to restart my IPv6 OSPF3 process on my ASA every time my Palo Alto firewall crashes and fails over, otherwise none of my VPN clients can connect via IPv6?
Why do you hurt me so, IPv6? I just wanted to be friends, and now I just want to break up with you. Maybe we can try to be friends again when your vendors get their shit together.
-David
--- Bruce Curtis bruce.curtis@ndsu.edu Certified NetAnalyst II 701-231-8527 North Dakota State University
On Tue, 2015-05-26 at 23:59 -0700, Tony Hain wrote:
(...) For fun, project this http://www.google.com/intl/en/ipv6/statistics.html (...)
Hi, If someone from google is listening it would be really nice to spend a few minutes t oavoid flash for displaying this graph, it doesn't work on my Google Nexus 4 and my flash-less chrome/chromium desktops :). Sincerely, Laurent
On Tue, 26 May 2015, David Sotnick wrote:
Arista EOS code — and it only appears to affect Virtual Machines which are behind our RedHat Enterprise Virtualization cluster. None of the hundreds of VMware-connected hosts are affected. The symptom is basically the same as the Palo Alto bug. Neighbor table gets in some weird state where ND
Is VMWare contributing somehow to the problem? Marcin
On Tue, May 26, 2015 at 04:19:25PM -0700, David Sotnick wrote:
Hi NANOG,
The company I work for has no business case for being on the IPv6-Internet. However, I am an inquisitive person and I am always looking to learn new things, so about 3 years ago I started down the IPv6 path. This was early 2012.
Fast forward to today. We have a /44 presence for our company's multiple sites; All our desktop computers have been on the IPv6 Internet since June, 2012 and we have a few AAAAs in our external DNS for some key services — and, there have been bugs. *Lots* of bugs.
Now, maybe (_maybe_) I can have some sympathy for smaller network companies (like Arista Networks at the time) to not quite have their act together as far as IPv6 goes, but for larger, well-established companies to still have critical IPv6 bugs is just inexcusable!
My current favorites are: https://tools.cisco.com/bugsearch/bug/CSCut62344 Which doesn't allow you to see the neighbors on an interface. this is fun when diagnosing qemu/kvm issues with the macvtap and hosts with ipv6. turns out you to 'fix it' you need to make the macvtap interface promisc as the icmpv6 messages don't make it through the macvtap driver to the VM breaking neighbor discovery. You can guess how we saw the first bug with the second one. This isn't as bad as a colleague who told me he is taking classes at a university whose professor said that a /20 is neither a class A or class B allocation but in the middle, not knowing that CIDR has existed for the past 20 years. Turns out we need a few more SMEs to teach people about CIDR and IPv6 addressing to prevent univeristy professors from teaching the next generation something that doesn't apply anymore. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On 5/27/2015 3:20 PM, Jared Mauch wrote:
On Tue, May 26, 2015 at 04:19:25PM -0700, David Sotnick wrote:
Hi NANOG,
The company I work for has no business case for being on the IPv6-Internet. However, I am an inquisitive person and I am always looking to learn new things, so about 3 years ago I started down the IPv6 path. This was early 2012.
Fast forward to today. We have a /44 presence for our company's multiple sites; All our desktop computers have been on the IPv6 Internet since June, 2012 and we have a few AAAAs in our external DNS for some key services — and, there have been bugs. *Lots* of bugs.
Now, maybe (_maybe_) I can have some sympathy for smaller network companies (like Arista Networks at the time) to not quite have their act together as far as IPv6 goes, but for larger, well-established companies to still have critical IPv6 bugs is just inexcusable! My current favorites are:
https://tools.cisco.com/bugsearch/bug/CSCut62344
Which doesn't allow you to see the neighbors on an interface. this is fun when diagnosing qemu/kvm issues with the macvtap and hosts with ipv6. turns out you to 'fix it' you need to make the macvtap interface promisc as the icmpv6 messages don't make it through the macvtap driver to the VM breaking neighbor discovery. You don't need full promisc mode, just the (poorly documented) allmulticast option (ip link set dev $macvtap allmulticast on)
On 27/05/2015 20:35, Brian Rak wrote:
You don't need full promisc mode, just the (poorly documented) allmulticast option (ip link set dev $macvtap allmulticast on)
...And poorly supported on some real hardware (notably Wi-Fi adapters), where the hash filter on each NIC's MAC is not guaranteed to support "ALLMULTI". It's a prerequisite for software multicast forwarding, and, it might be argued, a good litmus test for IPv6 readiness.
participants (10)
-
Brian Rak -
Bruce Curtis -
Bruce Simpson -
Ca By -
David Sotnick -
Jared Mauch -
Laurent GUERBY -
Marcin Cieslak -
Mark Tinka -
Tony Hain