-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Following up on a two year old thread, one of my clients just hit this problem. The failure is not that www.pay.gov is not reachable over ipv6 (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 connection, but the connection then hangs waiting for the TLS handshake. openssl s_client -connect www.pay.gov:443 openssl s_client -servername www.pay.gov -connect 199.169.192.21:443 Browsers (at least firefox) see that as a very slow site, and it does not trigger their happy eyeballs fast failover to ipv4. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEAREKAAYFAlgrjDEACgkQL6j7milTFsG8OwCgh5yRxxZHskjL4HVhzxIEmenA LQgAniRMcYf/DIcg+8ve55MxUgrUbmzC =MS8j -----END PGP SIGNATURE-----
In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byington writes :
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Following up on a two year old thread, one of my clients just hit this problem. The failure is not that www.pay.gov is not reachable over ipv6 (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 connection, but the connection then hangs waiting for the TLS handshake.
openssl s_client -connect www.pay.gov:443
openssl s_client -servername www.pay.gov -connect 199.169.192.21:443
Browsers (at least firefox) see that as a very slow site, and it does not trigger their happy eyeballs fast failover to ipv4.
Happy eyeballs is about making the connection not whether TCP connections work after the initial packet exchange. I would send a physical letter to the relevent Inspector General requesting that they ensure all web sites under their juristiction that are supposed to be reachable from the public net get audited regularly to ensure that IPv6 connections work from public IP space. While you are sending the letter can you also ask why pay.gov's DNS servers are broken. Checking: 'pay.gov' as at 2016-11-16T20:21:28Z pay.gov @199.169.194.28 (ns1.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok pay.gov @2605:3100:fffc:100::7 (ns1.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok pay.gov @199.169.192.28 (ns2.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok pay.gov @2605:3100:fffd:100::7 (ns2.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok Mark
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux)
iEYEAREKAAYFAlgrjDEACgkQL6j7milTFsG8OwCgh5yRxxZHskjL4HVhzxIEmenA LQgAniRMcYf/DIcg+8ve55MxUgrUbmzC =MS8j -----END PGP SIGNATURE-----
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
I fixed it (and Netflix) by turning off IPv6 for all my users... but any chance this is a path MTU issue causing the apparent hang? Matthew Kaufman On Wed, Nov 16, 2016 at 12:26 PM Mark Andrews <marka@isc.org> wrote:
In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byington writes :
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Following up on a two year old thread, one of my clients just hit this problem. The failure is not that www.pay.gov is not reachable over ipv6 (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 connection, but the connection then hangs waiting for the TLS handshake.
openssl s_client -connect www.pay.gov:443
openssl s_client -servername www.pay.gov -connect 199.169.192.21:443
Browsers (at least firefox) see that as a very slow site, and it does not trigger their happy eyeballs fast failover to ipv4.
Happy eyeballs is about making the connection not whether TCP connections work after the initial packet exchange.
I would send a physical letter to the relevent Inspector General requesting that they ensure all web sites under their juristiction that are supposed to be reachable from the public net get audited regularly to ensure that IPv6 connections work from public IP space.
While you are sending the letter can you also ask why pay.gov's DNS servers are broken.
Checking: 'pay.gov' as at 2016-11-16T20:21:28Z
pay.gov @199.169.194.28 (ns1.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok pay.gov @2605:3100:fffc:100::7 (ns1.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok pay.gov @199.169.192.28 (ns2.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok pay.gov @2605:3100:fffd:100::7 (ns2.twai.gov.): edns=ok edns1=timeout edns@512=noopt ednsopt=ok edns1opt=timeout do=ok ednsflags=ok docookie=ok edns@512tcp=ok optlist=ok
Mark
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux)
iEYEAREKAAYFAlgrjDEACgkQL6j7milTFsG8OwCgh5yRxxZHskjL4HVhzxIEmenA LQgAniRMcYf/DIcg+8ve55MxUgrUbmzC =MS8j -----END PGP SIGNATURE-----
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On Wed, 2016-11-16 at 20:59 +0000, Matthew Kaufman wrote:
I fixed it (and Netflix) by turning off IPv6 for all my users... but any chance this is a path MTU issue causing the apparent hang?
I fixed it by using the rpz feature of bind to disable the AAAA record for www.pay.gov. I lookup the real A record, and then put www.pay.gov IN A %s into the local rpz zone. That suppresses the AAAA record, so local clients are forced into IPv4 for that site. That allows them to use IPv6 for other sites. path MTU - hm, I need to check that. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEAREKAAYFAlgsz0YACgkQL6j7milTFsEbpwCgiJwZm3R/0VowqNFu4afHwPRq siwAmwdAj2YCLnlNQAs5Q5E5hcthaoiP =yqXb -----END PGP SIGNATURE-----
On 11/16/16, Mark Andrews <marka@isc.org> wrote:
In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byington writes :
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Following up on a two year old thread, one of my clients just hit this problem. The failure is not that www.pay.gov is not reachable over ipv6 (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 connection, but the connection then hangs waiting for the TLS handshake.
openssl s_client -connect www.pay.gov:443
openssl s_client -servername www.pay.gov -connect 199.169.192.21:443
Browsers (at least firefox) see that as a very slow site, and it does not trigger their happy eyeballs fast failover to ipv4.
Happy eyeballs is about making the connection not whether TCP connections work after the initial packet exchange.
I would send a physical letter to the relevent Inspector General requesting that they ensure all web sites under their juristiction that are supposed to be reachable from the public net get audited regularly to ensure that IPv6 connections work from public IP space.
That will absolutely work. NIST is still monitoring ipv6 .gov sites https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov so the IG isn't going to do anything there & pay.gov has a contact us page https://pay.gov/public/home/contact that I'd bet works much better than a letter to the IG Regards, Lee
In message <CAD8GWsvetSmn1ssFk_AdTtKheog0e1ZfXRLd11FpkbPJGHM6hw@mail.gmail.com> , Lee writes:
On 11/16/16, Mark Andrews <marka@isc.org> wrote:
In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byington writes :
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Following up on a two year old thread, one of my clients just hit this problem. The failure is not that www.pay.gov is not reachable over ipv6 (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 connection, but the connection then hangs waiting for the TLS handshake.
openssl s_client -connect www.pay.gov:443
openssl s_client -servername www.pay.gov -connect 199.169.192.21:443
Browsers (at least firefox) see that as a very slow site, and it does not trigger their happy eyeballs fast failover to ipv4.
Happy eyeballs is about making the connection not whether TCP connections work after the initial packet exchange.
I would send a physical letter to the relevent Inspector General requesting that they ensure all web sites under their juristiction that are supposed to be reachable from the public net get audited regularly to ensure that IPv6 connections work from public IP space.
That will absolutely work.
NIST is still monitoring ipv6 .gov sites https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov
Which show green which means that the tests they are doing are not sufficient. They need to test from behind a 1280 mtu link. The DNSSEC testing is also insufficient. 9-11commission.gov shows green for example but if you use DNS COOKIES (which BIND 9.10.4 and BIND 9.11.0 do) then servers barf and return BADVERS and validation fails. QWEST you have been informed of this already. Why the hell should validating resolver have to work around the crap you guys are using? DO YOUR JOBS which is to use RFC COMPLIANT servers. You get PAID to do DNS because people think you are compentent to do the job. Evidence shows otherwise. https://ednscomp.isc.org/compliance/gov-full-report.html show the broken servers for .gov. It isn't hard to check.
so the IG isn't going to do anything there & pay.gov has a contact us page https://pay.gov/public/home/contact that I'd bet works much better than a letter to the IG
You have to be able to get to https://pay.gov/public/home/contact to use it. Most people don't have the skill set to force the use of IPv4. If it is production it should work. It is the I-G's role to ensure this happens. Butts need to kicked. Mark
Regards, Lee -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
I think it is not just a matter of testing behind a 1280 MTU, but about making sure that PMTUD is not broken, so it just works in any circumstances. Regards, Jordi -----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de Mark Andrews <marka@isc.org> Responder a: <marka@isc.org> Fecha: jueves, 17 de noviembre de 2016, 9:26 Para: Lee <ler762@gmail.com> CC: <nanog@nanog.org> Asunto: Re: pay.gov and IPv6 In message <CAD8GWsvetSmn1ssFk_AdTtKheog0e1ZfXRLd11FpkbPJGHM6hw@mail.gmail.com> , Lee writes: > On 11/16/16, Mark Andrews <marka@isc.org> wrote: > > > > In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byington > > writes > > : > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA512 > >> > >> Following up on a two year old thread, one of my clients just hit this > >> problem. The failure is not that www.pay.gov is not reachable over ipv6 > >> (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 > >> connection, but the connection then hangs waiting for the TLS handshake. > >> > >> openssl s_client -connect www.pay.gov:443 > >> > >> openssl s_client -servername www.pay.gov -connect 199.169.192.21:443 > >> > >> Browsers (at least firefox) see that as a very slow site, and it does > >> not trigger their happy eyeballs fast failover to ipv4. > > > > Happy eyeballs is about making the connection not whether TCP > > connections work after the initial packet exchange. > > > > I would send a physical letter to the relevent Inspector General > > requesting that they ensure all web sites under their juristiction > > that are supposed to be reachable from the public net get audited > > regularly to ensure that IPv6 connections work from public IP space. > > That will absolutely work. > > NIST is still monitoring ipv6 .gov sites > https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov Which show green which means that the tests they are doing are not sufficient. They need to test from behind a 1280 mtu link. The DNSSEC testing is also insufficient. 9-11commission.gov shows green for example but if you use DNS COOKIES (which BIND 9.10.4 and BIND 9.11.0 do) then servers barf and return BADVERS and validation fails. QWEST you have been informed of this already. Why the hell should validating resolver have to work around the crap you guys are using? DO YOUR JOBS which is to use RFC COMPLIANT servers. You get PAID to do DNS because people think you are compentent to do the job. Evidence shows otherwise. https://ednscomp.isc.org/compliance/gov-full-report.html show the broken servers for .gov. It isn't hard to check. > so the IG isn't going to do anything there & pay.gov has a contact us page > https://pay.gov/public/home/contact > that I'd bet works much better than a letter to the IG You have to be able to get to https://pay.gov/public/home/contact to use it. Most people don't have the skill set to force the use of IPv4. If it is production it should work. It is the I-G's role to ensure this happens. Butts need to kicked. Mark > Regards, > Lee -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org ********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.
In message <CC8936B2-1396-4375-85AA-A0247FD78012@consulintel.es>, JORDI PALET M ARTINEZ writes:
I think it is not just a matter of testing behind a 1280 MTU, but about makin g sure that PMTUD is not broken, so it just works in any circumstances.
Regards, Jordi
If you don't do MSS fix up a 1280 link in the middle will find PMTUD issues provided the testing host has a MTU > 1280. Mark
-----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de Mark Andrews <marka@isc.org> Responder a: <marka@isc.org> Fecha: jueves, 17 de noviembre de 2016, 9:26 Para: Lee <ler762@gmail.com> CC: <nanog@nanog.org> Asunto: Re: pay.gov and IPv6
In message <CAD8GWsvetSmn1ssFk_AdTtKheog0e1ZfXRLd11FpkbPJGHM6hw@mail.gmai l.com> , Lee writes: > On 11/16/16, Mark Andrews <marka@isc.org> wrote: > > > > In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byingto n > > writes > > : > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA512 > >> > >> Following up on a two year old thread, one of my clients just hit th is > >> problem. The failure is not that www.pay.gov is not reachable over i pv6 > >> (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 > >> connection, but the connection then hangs waiting for the TLS handsh ake. > >> > >> openssl s_client -connect www.pay.gov:443 > >> > >> openssl s_client -servername www.pay.gov -connect 199.169.192.21:443 > >> > >> Browsers (at least firefox) see that as a very slow site, and it doe s > >> not trigger their happy eyeballs fast failover to ipv4. > > > > Happy eyeballs is about making the connection not whether TCP > > connections work after the initial packet exchange. > > > > I would send a physical letter to the relevent Inspector General > > requesting that they ensure all web sites under their juristiction > > that are supposed to be reachable from the public net get audited > > regularly to ensure that IPv6 connections work from public IP space. > > That will absolutely work. > > NIST is still monitoring ipv6 .gov sites > https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov
Which show green which means that the tests they are doing are not sufficient. They need to test from behind a 1280 mtu link.
The DNSSEC testing is also insufficient. 9-11commission.gov shows green for example but if you use DNS COOKIES (which BIND 9.10.4 and BIND 9.11.0 do) then servers barf and return BADVERS and validation fails. QWEST you have been informed of this already.
Why the hell should validating resolver have to work around the crap you guys are using? DO YOUR JOBS which is to use RFC COMPLIANT servers. You get PAID to do DNS because people think you are compentent to do the job. Evidence shows otherwise.
https://ednscomp.isc.org/compliance/gov-full-report.html show the broken servers for .gov. It isn't hard to check.
> so the IG isn't going to do anything there & pay.gov has a contact us p age > https://pay.gov/public/home/contact > that I'd bet works much better than a letter to the IG
You have to be able to get to https://pay.gov/public/home/contact to use it. Most people don't have the skill set to force the use of IPv4.
If it is production it should work. It is the I-G's role to ensure this happens. Butts need to kicked.
Mark
> Regards, > Lee -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company
This electronic message contains information which may be privileged or confi dential. The information is intended to be for the use of the individual(s) n amed above. If you are not the intended recipient be aware that any disclosur e, copying, distribution or use of the contents of this information, includin g attached files, is prohibited.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
The good news is that I reported this particular site as a problem two and three years ago, both, and it isn't any worse. Matthew Kaufman On Wed, Nov 16, 2016 at 6:29 PM Mark Andrews <marka@isc.org> wrote:
In message <CC8936B2-1396-4375-85AA-A0247FD78012@consulintel.es>, JORDI PALET M ARTINEZ writes:
I think it is not just a matter of testing behind a 1280 MTU, but about makin g sure that PMTUD is not broken, so it just works in any circumstances.
Regards, Jordi
If you don't do MSS fix up a 1280 link in the middle will find PMTUD issues provided the testing host has a MTU > 1280.
Mark
-----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de Mark Andrews < marka@isc.org> Responder a: <marka@isc.org> Fecha: jueves, 17 de noviembre de 2016, 9:26 Para: Lee <ler762@gmail.com> CC: <nanog@nanog.org> Asunto: Re: pay.gov and IPv6
In message <CAD8GWsvetSmn1ssFk_AdTtKheog0e1ZfXRLd11FpkbPJGHM6hw@mail.gmai l.com> , Lee writes: > On 11/16/16, Mark Andrews <marka@isc.org> wrote: > > > > In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byingto n > > writes > > : > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA512 > >> > >> Following up on a two year old thread, one of my clients just hit th is > >> problem. The failure is not that www.pay.gov is not reachable over i pv6 > >> (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 > >> connection, but the connection then hangs waiting for the TLS handsh ake. > >> > >> openssl s_client -connect www.pay.gov:443 > >> > >> openssl s_client -servername www.pay.gov -connect 199.169.192.21:443 > >> > >> Browsers (at least firefox) see that as a very slow site, and it doe s > >> not trigger their happy eyeballs fast failover to ipv4. > > > > Happy eyeballs is about making the connection not whether TCP > > connections work after the initial packet exchange. > > > > I would send a physical letter to the relevent Inspector General > > requesting that they ensure all web sites under their juristiction > > that are supposed to be reachable from the public net get audited > > regularly to ensure that IPv6 connections work from public IP space. > > That will absolutely work. > > NIST is still monitoring ipv6 .gov sites > https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov
Which show green which means that the tests they are doing are not sufficient. They need to test from behind a 1280 mtu link.
The DNSSEC testing is also insufficient. 9-11commission.gov shows green for example but if you use DNS COOKIES (which BIND 9.10.4 and BIND 9.11.0 do) then servers barf and return BADVERS and validation fails. QWEST you have been informed of this already.
Why the hell should validating resolver have to work around the crap you guys are using? DO YOUR JOBS which is to use RFC COMPLIANT servers. You get PAID to do DNS because people think you are compentent to do the job. Evidence shows otherwise.
https://ednscomp.isc.org/compliance/gov-full-report.html show the broken servers for .gov. It isn't hard to check.
> so the IG isn't going to do anything there & pay.gov has a contact us p age > https://pay.gov/public/home/contact > that I'd bet works much better than a letter to the IG
You have to be able to get to https://pay.gov/public/home/contact to use it. Most people don't have the skill set to force the use of IPv4.
If it is production it should work. It is the I-G's role to ensure this happens. Butts need to kicked.
Mark
> Regards, > Lee -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company
This electronic message contains information which may be privileged or confi dential. The information is intended to be for the use of the individual(s) n amed above. If you are not the intended recipient be aware that any disclosur e, copying, distribution or use of the contents of this information, includin g attached files, is prohibited.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 11/16/16, Matthew Kaufman <matthew@matthew.at> wrote:
The good news is that I reported this particular site as a problem two and three years ago, both, and it isn't any worse.
did you contact Pay.gov Customer Service at: 800-624-1373 (Toll free, Option #2) or send an email to pay.gov.clev@clev.frb.org I just called, but I can't duplicate the problem and they need to work with someone that is having a problem reaching the site. Regards, Lee
Matthew Kaufman On Wed, Nov 16, 2016 at 6:29 PM Mark Andrews <marka@isc.org> wrote:
In message <CC8936B2-1396-4375-85AA-A0247FD78012@consulintel.es>, JORDI PALET M ARTINEZ writes:
I think it is not just a matter of testing behind a 1280 MTU, but about makin g sure that PMTUD is not broken, so it just works in any circumstances.
Regards, Jordi
If you don't do MSS fix up a 1280 link in the middle will find PMTUD issues provided the testing host has a MTU > 1280.
Mark
-----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de Mark Andrews < marka@isc.org> Responder a: <marka@isc.org> Fecha: jueves, 17 de noviembre de 2016, 9:26 Para: Lee <ler762@gmail.com> CC: <nanog@nanog.org> Asunto: Re: pay.gov and IPv6
In message <CAD8GWsvetSmn1ssFk_AdTtKheog0e1ZfXRLd11FpkbPJGHM6hw@mail.gmai l.com> , Lee writes: > On 11/16/16, Mark Andrews <marka@isc.org> wrote: > > > > In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byingto n > > writes > > : > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA512 > >> > >> Following up on a two year old thread, one of my clients just hit th is > >> problem. The failure is not that www.pay.gov is not reachable over i pv6 > >> (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 > >> connection, but the connection then hangs waiting for the TLS handsh ake. > >> > >> openssl s_client -connect www.pay.gov:443 > >> > >> openssl s_client -servername www.pay.gov -connect 199.169.192.21:443 > >> > >> Browsers (at least firefox) see that as a very slow site, and it doe s > >> not trigger their happy eyeballs fast failover to ipv4. > > > > Happy eyeballs is about making the connection not whether TCP > > connections work after the initial packet exchange. > > > > I would send a physical letter to the relevent Inspector General > > requesting that they ensure all web sites under their juristiction > > that are supposed to be reachable from the public net get audited > > regularly to ensure that IPv6 connections work from public IP space. > > That will absolutely work. > > NIST is still monitoring ipv6 .gov sites > https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov
Which show green which means that the tests they are doing are not sufficient. They need to test from behind a 1280 mtu link.
The DNSSEC testing is also insufficient. 9-11commission.gov shows green for example but if you use DNS COOKIES (which BIND 9.10.4 and BIND 9.11.0 do) then servers barf and return BADVERS and validation fails. QWEST you have been informed of this already.
Why the hell should validating resolver have to work around the crap you guys are using? DO YOUR JOBS which is to use RFC COMPLIANT servers. You get PAID to do DNS because people think you are compentent to do the job. Evidence shows otherwise.
https://ednscomp.isc.org/compliance/gov-full-report.html show the broken servers for .gov. It isn't hard to check.
> so the IG isn't going to do anything there & pay.gov has a contact us p age > https://pay.gov/public/home/contact > that I'd bet works much better than a letter to the IG
You have to be able to get to https://pay.gov/public/home/contact to use it. Most people don't have the skill set to force the use of IPv4.
If it is production it should work. It is the I-G's role to ensure this happens. Butts need to kicked.
Mark
> Regards, > Lee -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company
This electronic message contains information which may be privileged or confi dential. The information is intended to be for the use of the individual(s) n amed above. If you are not the intended recipient be aware that any disclosur e, copying, distribution or use of the contents of this information, includin g attached files, is prohibited.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
I sent email there and to another contact I had at the time. And I'm not going to break my users by turning IPv6 back on, so someone else will need to work with them. Matthew Kaufman On Thu, Nov 17, 2016 at 9:48 AM Lee <ler762@gmail.com> wrote:
On 11/16/16, Matthew Kaufman <matthew@matthew.at> wrote:
The good news is that I reported this particular site as a problem two and three years ago, both, and it isn't any worse.
did you contact Pay.gov Customer Service at: 800-624-1373 <(800)%20624-1373> (Toll free, Option #2) or send an email to pay.gov.clev@clev.frb.org
I just called, but I can't duplicate the problem and they need to work with someone that is having a problem reaching the site.
Regards, Lee
Matthew Kaufman On Wed, Nov 16, 2016 at 6:29 PM Mark Andrews <marka@isc.org> wrote:
In message <CC8936B2-1396-4375-85AA-A0247FD78012@consulintel.es>, JORDI PALET M ARTINEZ writes:
I think it is not just a matter of testing behind a 1280 MTU, but
makin
g sure that PMTUD is not broken, so it just works in any circumstances.
Regards, Jordi
If you don't do MSS fix up a 1280 link in the middle will find PMTUD issues provided the testing host has a MTU > 1280.
Mark
-----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de Mark Andrews < marka@isc.org> Responder a: <marka@isc.org> Fecha: jueves, 17 de noviembre de 2016, 9:26 Para: Lee <ler762@gmail.com> CC: <nanog@nanog.org> Asunto: Re: pay.gov and IPv6
In message <CAD8GWsvetSmn1ssFk_AdTtKheog0e1ZfXRLd11FpkbPJGHM6hw@mail.gmai l.com> , Lee writes: > On 11/16/16, Mark Andrews <marka@isc.org> wrote: > > > > In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byingto n > > writes > > : > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA512 > >> > >> Following up on a two year old thread, one of my clients just hit th is > >> problem. The failure is not that www.pay.gov is not reachable over i pv6 > >> (2605:3100:fffd:100::15). They accept (TCP handshake) the
about port
443
> >> connection, but the connection then hangs waiting for the TLS
ake. > >> > >> openssl s_client -connect www.pay.gov:443 > >> > >> openssl s_client -servername www.pay.gov -connect 199.169.192.21:443 > >> > >> Browsers (at least firefox) see that as a very slow site, and it doe s > >> not trigger their happy eyeballs fast failover to ipv4. > > > > Happy eyeballs is about making the connection not whether TCP > > connections work after the initial packet exchange. > > > > I would send a physical letter to the relevent Inspector General > > requesting that they ensure all web sites under their juristiction > > that are supposed to be reachable from the public net get audited > > regularly to ensure that IPv6 connections work from public IP space. > > That will absolutely work. > > NIST is still monitoring ipv6 .gov sites > https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov
Which show green which means that the tests they are doing are not sufficient. They need to test from behind a 1280 mtu link.
The DNSSEC testing is also insufficient. 9-11commission.gov shows green for example but if you use DNS COOKIES (which BIND 9.10.4 and BIND 9.11.0 do) then servers barf and return BADVERS and validation fails. QWEST you have been informed of this already.
Why the hell should validating resolver have to work around the crap you guys are using? DO YOUR JOBS which is to use RFC COMPLIANT servers. You get PAID to do DNS because people think you are compentent to do the job. Evidence shows otherwise.
https://ednscomp.isc.org/compliance/gov-full-report.html show the broken servers for .gov. It isn't hard to check.
> so the IG isn't going to do anything there & pay.gov has a contact us p age > https://pay.gov/public/home/contact > that I'd bet works much better than a letter to the IG
You have to be able to get to https://pay.gov/public/home/contact to use it. Most people don't have the skill set to force the use of IPv4.
If it is production it should work. It is the I-G's role to ensure
handsh this
happens. Butts need to kicked.
Mark
> Regards, > Lee -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 <+61%202%209871%204742>
INTERNET: marka@isc.org
********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company
This electronic message contains information which may be privileged
or confi
dential. The information is intended to be for the use of the individual(s) n amed above. If you are not the intended recipient be aware that any disclosur e, copying, distribution or use of the contents of this information, includin g attached files, is prohibited.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 <+61%202%209871%204742> INTERNET: marka@isc.org
On 11/17/16, Matthew Kaufman <matthew@matthew.at> wrote:
I sent email there and to another contact I had at the time.
and the response was?
And I'm not going to break my users by turning IPv6 back on, so someone else will need to work with them.
That's fine, but until someone is willing to work with them don't expect it to get fixed. Regards, Lee
Matthew Kaufman
On Thu, Nov 17, 2016 at 9:48 AM Lee <ler762@gmail.com> wrote:
On 11/16/16, Matthew Kaufman <matthew@matthew.at> wrote:
The good news is that I reported this particular site as a problem two and three years ago, both, and it isn't any worse.
did you contact Pay.gov Customer Service at: 800-624-1373 <(800)%20624-1373> (Toll free, Option #2) or send an email to pay.gov.clev@clev.frb.org
I just called, but I can't duplicate the problem and they need to work with someone that is having a problem reaching the site.
Regards, Lee
Matthew Kaufman On Wed, Nov 16, 2016 at 6:29 PM Mark Andrews <marka@isc.org> wrote:
In message <CC8936B2-1396-4375-85AA-A0247FD78012@consulintel.es>, JORDI PALET M ARTINEZ writes:
I think it is not just a matter of testing behind a 1280 MTU, but
makin
g sure that PMTUD is not broken, so it just works in any circumstances.
Regards, Jordi
If you don't do MSS fix up a 1280 link in the middle will find PMTUD issues provided the testing host has a MTU > 1280.
Mark
-----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de Mark Andrews < marka@isc.org> Responder a: <marka@isc.org> Fecha: jueves, 17 de noviembre de 2016, 9:26 Para: Lee <ler762@gmail.com> CC: <nanog@nanog.org> Asunto: Re: pay.gov and IPv6
In message <CAD8GWsvetSmn1ssFk_AdTtKheog0e1ZfXRLd11FpkbPJGHM6hw@mail.gmai l.com> , Lee writes: > On 11/16/16, Mark Andrews <marka@isc.org> wrote: > > > > In message <1479249003.3937.6.camel@ns.five-ten-sg.com>, Carl Byingto n > > writes > > : > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA512 > >> > >> Following up on a two year old thread, one of my clients just hit th is > >> problem. The failure is not that www.pay.gov is not reachable over i pv6 > >> (2605:3100:fffd:100::15). They accept (TCP handshake) the
about port
443
> >> connection, but the connection then hangs waiting for the TLS
ake. > >> > >> openssl s_client -connect www.pay.gov:443 > >> > >> openssl s_client -servername www.pay.gov -connect 199.169.192.21:443 > >> > >> Browsers (at least firefox) see that as a very slow site, and it doe s > >> not trigger their happy eyeballs fast failover to ipv4. > > > > Happy eyeballs is about making the connection not whether TCP > > connections work after the initial packet exchange. > > > > I would send a physical letter to the relevent Inspector General > > requesting that they ensure all web sites under their juristiction > > that are supposed to be reachable from the public net get audited > > regularly to ensure that IPv6 connections work from public IP space. > > That will absolutely work. > > NIST is still monitoring ipv6 .gov sites > https://usgv6-deploymon.antd.nist.gov/cgi-bin/generate-gov
Which show green which means that the tests they are doing are not sufficient. They need to test from behind a 1280 mtu link.
The DNSSEC testing is also insufficient. 9-11commission.gov shows green for example but if you use DNS COOKIES (which BIND 9.10.4 and BIND 9.11.0 do) then servers barf and return BADVERS and validation fails. QWEST you have been informed of this already.
Why the hell should validating resolver have to work around the crap you guys are using? DO YOUR JOBS which is to use RFC COMPLIANT servers. You get PAID to do DNS because people think you are compentent to do the job. Evidence shows otherwise.
https://ednscomp.isc.org/compliance/gov-full-report.html show the broken servers for .gov. It isn't hard to check.
> so the IG isn't going to do anything there & pay.gov has a contact us p age > https://pay.gov/public/home/contact > that I'd bet works much better than a letter to the IG
You have to be able to get to https://pay.gov/public/home/contact to use it. Most people don't have the skill set to force the use of IPv4.
If it is production it should work. It is the I-G's role to ensure
handsh this
happens. Butts need to kicked.
Mark
> Regards, > Lee -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 <+61%202%209871%204742>
INTERNET: marka@isc.org
********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company
This electronic message contains information which may be privileged
or confi
dential. The information is intended to be for the use of the individual(s) n amed above. If you are not the intended recipient be aware that any disclosur e, copying, distribution or use of the contents of this information, includin g attached files, is prohibited.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 <+61%202%209871%204742> INTERNET: marka@isc.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On Thu, 2016-11-17 at 15:32 -0500, Lee wrote:
That's fine, but until someone is willing to work with them don't expect it to get fixed.
I am working with pay.gov.clev@clev.frb.org, trying to explain the problem. They seem to think I should provide "application name and ID" before they can research this. I responded as below. I will let you know if there is any progress on this. It is 100% reproducible here - and should be reproducible from anywhere with a slightly smaller than normal MTU. We try to get to https://www.pay.gov/public/home - which fails. I presume that is before there is any application name or ID. Try to reach that page from a browser on a machine with ipv6 connectivity that goes thru a tunnel with a 1300 byte MTU. It will fail. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEAREKAAYFAlguP8oACgkQL6j7milTFsFgrQCfY2QLE0njiSRIILTzR4Fjpk3c F3AAnjZj8+3W51Qr9e1oGWAxrUC8Pnf3 =UpN0 -----END PGP SIGNATURE-----
On 11/17/16, Carl Byington <carl@five-ten-sg.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On Thu, 2016-11-17 at 15:32 -0500, Lee wrote:
That's fine, but until someone is willing to work with them don't expect it to get fixed.
I am working with pay.gov.clev@clev.frb.org, trying to explain the problem. They seem to think I should provide "application name and ID" before they can research this.
They probably want that so they know where to route the ticket. I pointed them at https://tools.ietf.org/html/rfc4890#page-14 and suggested they have someone on the network team use ipv6 to connect to pay.gov over a 1280 byte MTU link. Hopefully that's enough to get the ticket routed to the correct group.
I responded as below. I will let you know if there is any progress on this.
I'd appreciate that. And thanks for being willing to work with them to fix the problem. Lee
It is 100% reproducible here - and should be reproducible from anywhere with a slightly smaller than normal MTU.
We try to get to https://www.pay.gov/public/home - which fails. I presume that is before there is any application name or ID.
Try to reach that page from a browser on a machine with ipv6 connectivity that goes thru a tunnel with a 1300 byte MTU. It will fail.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux)
iEYEAREKAAYFAlguP8oACgkQL6j7milTFsFgrQCfY2QLE0njiSRIILTzR4Fjpk3c F3AAnjZj8+3W51Qr9e1oGWAxrUC8Pnf3 =UpN0 -----END PGP SIGNATURE-----
I am working with pay.gov.clev@clev.frb.org, trying to explain the problem.
The intersection of government bureaucracy and technical issues is frustrating to say the least. I just sent the message below, but have no expectation that it will change anything. ============== On Fri, 2016-11-18 at 12:39 +0000, CLEV Pay Gov wrote:
It would be best to discuss this via phone. Please contact our help desk at the number below and we could see if there's anything we could do over the phone to help troubleshoot.
That is hopeless. Verbal technical discussions rarely work unless both sides can see the same text. Have you ever tried (while talking on the phone) to get someone to type in clev.frb.org without making a bunch of mistakes in the spelling?? Anyway, just for my amusement, I did call 800-624-1373, Option #2, and am on the line now, trying to explain this. 10 minutes and counting. Ok, there does not seem to be any overall ticket for "pay.gov does not work at all". They refuse to open a tech support ticket.
If not, we may need to open a ticket for our technical support.
Please open a ticket, and attach the following text for your tech support folks. Alternatively, have them look at the "pay.gov and ipv6" thread on nanog: http://mailman.nanog.org/pipermail/nanog/2016-November/thread.html www.pay.gov has an IPv6 address of 2605:3100:fffd:100::15, but that machine or its upstream routers are filtering icmpv6 messages. That web site is not accessible from systems with an MTU of 1280 bytes. The test case is: echo -e 'GET /public/home HTTP/1.0\n' | \ openssl s_client -servername www.pay.gov -ign_eof -connect \ '[2605:3100:fffd:100::15]:443' Run that (or just use a browser to try https://www.pay.gov) from a system with a 1500 byte MTU, and it works. Run it from a system with upstream connectivity via a tunnel, so the path MTU is smaller, and it fails. Such tunnels are common for IPv6. Please stop filtering icmpv6.
I tested from my home and happy eyeballs is not falling back to IPv4. So, I tend to suspect that is not ICMPv6 filtering, but something else, such as wrong load balancer or ECMP configuration. Regards, Jordi -----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de Carl Byington <carl@five-ten-sg.com> Responder a: <carl@five-ten-sg.com> Fecha: sábado, 19 de noviembre de 2016, 3:22 Para: <nanog@nanog.org> Asunto: Re: pay.gov and IPv6 > > I am working with pay.gov.clev@clev.frb.org, trying to explain the > problem. The intersection of government bureaucracy and technical issues is frustrating to say the least. I just sent the message below, but have no expectation that it will change anything. ============== On Fri, 2016-11-18 at 12:39 +0000, CLEV Pay Gov wrote: > It would be best to discuss this via phone. Please contact our help > desk at the number below and we could see if there's anything we could > do over the phone to help troubleshoot. That is hopeless. Verbal technical discussions rarely work unless both sides can see the same text. Have you ever tried (while talking on the phone) to get someone to type in clev.frb.org without making a bunch of mistakes in the spelling?? Anyway, just for my amusement, I did call 800-624-1373, Option #2, and am on the line now, trying to explain this. 10 minutes and counting. Ok, there does not seem to be any overall ticket for "pay.gov does not work at all". They refuse to open a tech support ticket. > If not, we may need to open a ticket for our technical support. Please open a ticket, and attach the following text for your tech support folks. Alternatively, have them look at the "pay.gov and ipv6" thread on nanog: http://mailman.nanog.org/pipermail/nanog/2016-November/thread.html www.pay.gov has an IPv6 address of 2605:3100:fffd:100::15, but that machine or its upstream routers are filtering icmpv6 messages. That web site is not accessible from systems with an MTU of 1280 bytes. The test case is: echo -e 'GET /public/home HTTP/1.0\n' | \ openssl s_client -servername www.pay.gov -ign_eof -connect \ '[2605:3100:fffd:100::15]:443' Run that (or just use a browser to try https://www.pay.gov) from a system with a 1500 byte MTU, and it works. Run it from a system with upstream connectivity via a tunnel, so the path MTU is smaller, and it fails. Such tunnels are common for IPv6. Please stop filtering icmpv6. ********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.
Somebody pointed to me that even happy eyeballs will not fall back to IPv4 when PMTUD is blocked … This is a big issue, many folks are deploying IPv6 web sites, and not double-checking this. Actually, this is VERY BIG issue with all the 1&1 sites. I tried to contact them many times for more than a year, and they seem to not care, so clearly not a recommended hosting provider, as they don’t care about the quality of service that their customers have. I will change my mind if someone from 1&1 is finally responding, in case they are in this list … For example, you will not get this working if you have a lower MTU than 1.500, which is quite normal, not just for tunnels, but also because the PPP/others encapsulation in many access links: http://diskmakerx.com/ Furthermore, I mention this filtering problem in the article about the IPv6 survey results, for those that don’t follow RIPE LABS site: https://labs.ripe.net/Members/jordipaletm/results-of-the-ipv6-deployment-sur... Regards, Jordi -----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de JORDI PALET MARTINEZ <jordi.palet@consulintel.es> Responder a: <jordi.palet@consulintel.es> Fecha: viernes, 18 de noviembre de 2016, 21:05 Para: <nanog@nanog.org> Asunto: Re: pay.gov and IPv6 I tested from my home and happy eyeballs is not falling back to IPv4. So, I tend to suspect that is not ICMPv6 filtering, but something else, such as wrong load balancer or ECMP configuration. Regards, Jordi -----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de Carl Byington <carl@five-ten-sg.com> Responder a: <carl@five-ten-sg.com> Fecha: sábado, 19 de noviembre de 2016, 3:22 Para: <nanog@nanog.org> Asunto: Re: pay.gov and IPv6 > > I am working with pay.gov.clev@clev.frb.org, trying to explain the > problem. The intersection of government bureaucracy and technical issues is frustrating to say the least. I just sent the message below, but have no expectation that it will change anything. ============== On Fri, 2016-11-18 at 12:39 +0000, CLEV Pay Gov wrote: > It would be best to discuss this via phone. Please contact our help > desk at the number below and we could see if there's anything we could > do over the phone to help troubleshoot. That is hopeless. Verbal technical discussions rarely work unless both sides can see the same text. Have you ever tried (while talking on the phone) to get someone to type in clev.frb.org without making a bunch of mistakes in the spelling?? Anyway, just for my amusement, I did call 800-624-1373, Option #2, and am on the line now, trying to explain this. 10 minutes and counting. Ok, there does not seem to be any overall ticket for "pay.gov does not work at all". They refuse to open a tech support ticket. > If not, we may need to open a ticket for our technical support. Please open a ticket, and attach the following text for your tech support folks. Alternatively, have them look at the "pay.gov and ipv6" thread on nanog: http://mailman.nanog.org/pipermail/nanog/2016-November/thread.html www.pay.gov has an IPv6 address of 2605:3100:fffd:100::15, but that machine or its upstream routers are filtering icmpv6 messages. That web site is not accessible from systems with an MTU of 1280 bytes. The test case is: echo -e 'GET /public/home HTTP/1.0\n' | \ openssl s_client -servername www.pay.gov -ign_eof -connect \ '[2605:3100:fffd:100::15]:443' Run that (or just use a browser to try https://www.pay.gov) from a system with a 1500 byte MTU, and it works. Run it from a system with upstream connectivity via a tunnel, so the path MTU is smaller, and it fails. Such tunnels are common for IPv6. Please stop filtering icmpv6. ********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited. ********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited. ********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On Sun, 2016-11-20 at 10:51 +0100, JORDI PALET MARTINEZ wrote:
For example, you will not get this working if you have a lower MTU than 1.500, which is quite normal, not just for tunnels, but also because the PPP/others encapsulation in many access links:
curl -6 -v http://diskmakerx.com/ That works here via an he.net tunnel. Perhaps 1and1 fixed something. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEAREKAAYFAlgyOp4ACgkQL6j7milTFsFslACfaRbslKuUHrxGJyuI+j8X/Sle AZ8AoIg2wF/GWBdDtphQSuD9/gGMDfOA =4nAp -----END PGP SIGNATURE-----
In message <1479686835.13553.4.camel@ns.five-ten-sg.com>, Carl Byington writes:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On Sun, 2016-11-20 at 10:51 +0100, JORDI PALET MARTINEZ wrote:
For example, you will not get this working if you have a lower MTU than 1.500, which is quite normal, not just for tunnels, but also because the PPP/others encapsulation in many access links:
curl -6 -v http://diskmakerx.com/
That works here via an he.net tunnel. Perhaps 1and1 fixed something.
And the advertised MSS was what? On my box I'm seeing 1220 for IPv6 compared with 1460 for IPv4. 1220 shouldn't see PMTU problems. 1460 on the other hand will cause problems as more clients are forced behind IPv4 as a service links. 11:14:50.056215 IP 172.30.42.121.52280 > 217.160.0.214.80: Flags [S], seq 2115844511, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 538455715 ecr 0,sackOK,eol], length 0 11:14:50.137770 IP6 2001:470:a001:4:c00d:3d8c:14e7:6de3.52282 > 2001:8d8:100f:f000::2d5.80: Flags [S], seq 4181372812, win 65535, options [mss 1220,nop,wscale 4,nop,nop,TS val 538455793 ecr 0,sackOK,eol], length 0 Mark
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux)
iEYEAREKAAYFAlgyOp4ACgkQL6j7milTFsFslACfaRbslKuUHrxGJyuI+j8X/Sle AZ8AoIg2wF/GWBdDtphQSuD9/gGMDfOA =4nAp -----END PGP SIGNATURE-----
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On Mon, 2016-11-21 at 11:26 +1100, Mark Andrews wrote:
And the advertised MSS was what? On my box I'm seeing 1220 for IPv6 compared with 1460 for IPv4. 1220 shouldn't see PMTU problems.
--> 2001:8d8:100f:f000::2d5 syn w/ mss 1440 <-- 2001:8d8:100f:f000::2d5 syn,ack w/ mss 1420 --> 2001:8d8:100f:f000::2d5 ack --> 2001:8d8:100f:f000::2d5 GET / HTTP/1.1 <-- 2001:8d8:100f:f000::2d5 ack <-- 2001:8d8:100f:f000::2d5 data... data is a 1480 byte ipv6 packet: ip6.header.payload.length = 1440 which is the tcp packet tcp.header.length = 32 bytes tcp.data.size = 1408 bytes (http response) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEAREKAAYFAlgyRUYACgkQL6j7milTFsH8tgCeLB9E9C2pjFqgp1w2YpipmvzZ ib4Ani/cQmAgEo3QPfA9hMntGq4VLoO/ =mmCW -----END PGP SIGNATURE-----
Tested again from four different networks. Not working for me. Same in other web sites hosted by 1&1 (for example www.legalveritas.es). All their IPv6 web sites are broken, every time I need to access their web sites, need to disable IPv6, I know how to do that, but regular folks not. tbit from 2001:df0:4:4000::1:115 to 2001:8d8:100f:f000::2d5 server-mss 1420, result: pmtud-fail app: http, url: http://diskmakerx.com/ [ 0.010] TX SYN 64 seq = 0:0 [ 0.286] RX SYN/ACK 64 seq = 0:1 [ 0.286] TX 60 seq = 1:1 [ 0.297] TX 233 seq = 1:1(173) [ 0.573] RX 60 seq = 1:174 [ 0.799] RX 1480 seq = 1:174(1420) [ 0.799] RX 73 seq = 1421:174(13) [ 0.799] RX 1480 seq = 1434:174(1420) [ 0.799] RX 1480 seq = 2854:174(1420) [ 0.799] TX PTB 1280 mtu = 1280 [ 0.799] RX 1480 seq = 4274:174(1420) [ 0.799] RX 1480 seq = 5694:174(1420) [ 0.799] RX 1480 seq = 7114:174(1420) [ 0.801] RX 1480 seq = 8534:174(1420) [ 0.809] TX 60 seq = 174:1 [ 0.821] RX 1480 seq = 9954:174(1420) [ 0.824] RX 1480 seq = 11374:174(1420) [ 1.628] RX 1480 seq = 1:174(1420) [ 1.629] TX PTB 1280 mtu = 1280 [ 3.288] RX 1480 seq = 1:174(1420) [ 3.288] TX PTB 1280 mtu = 1280 [ 6.612] RX 1480 seq = 1:174(1420) [ 6.612] TX PTB 1280 mtu = 1280 [ 13.252] RX 1480 seq = 1:174(1420) Regards, Jordi -----Mensaje original----- De: Mark Andrews <marka@isc.org> Responder a: <marka@isc.org> Fecha: lunes, 21 de noviembre de 2016, 1:26 Para: Carl Byington <carl@five-ten-sg.com> CC: <jordi.palet@consulintel.es>, <nanog@nanog.org> Asunto: Re: pay.gov and IPv6 In message <1479686835.13553.4.camel@ns.five-ten-sg.com>, Carl Byington writes: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > On Sun, 2016-11-20 at 10:51 +0100, JORDI PALET MARTINEZ wrote: > > For example, you will not get this working if you have a lower MTU > > than 1.500, which is quite normal, not just for tunnels, but also > > because the PPP/others encapsulation in many access links: > > > http://diskmakerx.com/ > > curl -6 -v http://diskmakerx.com/ > > That works here via an he.net tunnel. Perhaps 1and1 fixed something. And the advertised MSS was what? On my box I'm seeing 1220 for IPv6 compared with 1460 for IPv4. 1220 shouldn't see PMTU problems. 1460 on the other hand will cause problems as more clients are forced behind IPv4 as a service links. 11:14:50.056215 IP 172.30.42.121.52280 > 217.160.0.214.80: Flags [S], seq 2115844511, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 538455715 ecr 0,sackOK,eol], length 0 11:14:50.137770 IP6 2001:470:a001:4:c00d:3d8c:14e7:6de3.52282 > 2001:8d8:100f:f000::2d5.80: Flags [S], seq 4181372812, win 65535, options [mss 1220,nop,wscale 4,nop,nop,TS val 538455793 ecr 0,sackOK,eol], length 0 Mark > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.14 (GNU/Linux) > > iEYEAREKAAYFAlgyOp4ACgkQL6j7milTFsFslACfaRbslKuUHrxGJyuI+j8X/Sle > AZ8AoIg2wF/GWBdDtphQSuD9/gGMDfOA > =4nAp > -----END PGP SIGNATURE----- > > -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org ********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.
Tested again from four different networks. Not working for me. > > Same in other web sites hosted by 1&1 (for example www.legalveritas.es). All their IPv6 web sites are broken, every time I need to access their web sites, need to disable IPv6, I know how to do
[ 0.286] TX 60 seq = 1:1 > [ 0.297] TX 233 seq = 1:1(173) > [ 0.573] RX 60 seq = 1:174 > [ 0.799] RX 1480 seq = 1:174(1420) > [ 0.799] RX 73 seq = 1421:174(13) > [ 0.799] RX 1480 seq = 1434:174(1420) [ 0.799] RX 1480 seq = 2854:174(1420) > [ 0.799] TX PTB 1280 mtu = 1280 > [ 0.799] RX 1480 seq = 4274:174(1420) > [ 0.799] RX 1480 seq = 5694:174(1420) [ 0.799] RX 1480 seq = 7114:174(1420) > [ 0.801] RX 1480 seq = 8534:174(1420) > [ 0.809] TX 60 seq = 174:1 > [ 0.821] RX 1480 seq = 9954:174(1420) > [ 0.824] RX 1480 seq = 11374:174(1420) > [ 1.628] RX 1480 seq = 1:174(1420) [ 1.629] TX PTB 1280 mtu = 1280 > [ 3.288] RX 1480 seq = 1:174(1420) > [ 3.288] TX PTB 1280 mtu = 1280 > [ 6.612] RX 1480 seq = 1:174(1420) [ 6.612] TX PTB 1280 mtu = 1280 > [ 13.252] RX 1480 seq = 1:174(1420) > > > > Regards, > Jordi > > -----Mensaje original----- > De: Mark Andrews <marka@isc.org> > Responder a: <marka@isc.org> > Fecha: lunes, 21 de noviembre de 2016, 1:26 > Para: Carl Byington <carl@five-ten-sg.com> > CC: <jordi.palet@consulintel.es>, <nanog@nanog.org> > Asunto: Re: pay.gov and IPv6 > > > In message <1479686835.13553.4.camel@ns.five-ten-sg.com>, Carl Byington writes: On Sun, 2016-11-20 at 10:51 +0100, JORDI PALET MARTINEZ wrote:
For example, you will not get this working if you have a lower MTU than 1.500, which is quite normal, not just for tunnels, but also because the PPP/others encapsulation in many access links:
curl -6 -v http://diskmakerx.com/
That works here via an he.net tunnel. Perhaps 1and1 fixed something.
And the advertised MSS was what? On my box I'm seeing 1220 for IPv6 compared with 1460 for IPv4. 1220 shouldn't see PMTU problems.
1460 on the other hand will cause problems as more clients are forced behind IPv4 as a service links.
11:14:50.056215 IP 172.30.42.121.52280 > 217.160.0.214.80: Flags
[S], seq 2115844511, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 538455715 ecr 0,sackOK,eol], length 0
11:14:50.137770 IP6 2001:470:a001:4:c00d:3d8c:14e7:6de3.52282 >
2001:8d8:100f:f000::2d5.80: Flags [S], seq 4181372812, win 65535, options [mss 1220,nop,wscale 4,nop,nop,TS val 538455793 ecr 0,sackOK,eol], length 0
Mark
> > > > -- > Mark Andrews, ISC > 1 Seymour St., Dundas Valley, NSW 2117, Australia > PHONE: +61 2 9871 4742 INTERNET: marka@isc.org > > > > > > ********************************************** > IPv4 is over > Are you ready for the new Internet ? > http://www.consulintel.es > The IPv6 Company > > This electronic message contains information which may be
00:02:02.758900 IP6 2601:647:4201:XXXX.60962 > 2605:3100:fffd:100::15.443: Flags [S], seq 2375673666, win 65535, options [mss 1440,nop,wscale 5,nop,nop,TS val 568401205 ecr 0,sackOK,eol], length 0 00:02:02.811619 IP6 2605:3100:fffd:100::15.443 > 2601:647:4201:XXXX.60962: Flags [S.], seq 2570148804, ack 2375673667, win 4320, options [mss 1440,nop,nop,TS val 2246573166 ecr 568401205,sackOK,eol], length 0 it's happy to do 1440 which is unsprising, as1239 is probably not therefore the source of the pmtud hole. On 11/20/16 11:51 PM, JORDI PALET MARTINEZ wrote: that, but regular folks not. > > tbit from 2001:df0:4:4000::1:115 to 2001:8d8:100f:f000::2d5 > server-mss 1420, result: pmtud-fail > app: http, url: http://diskmakerx.com/ > [ 0.010] TX SYN 64 seq = 0:0 > [ 0.286] RX SYN/ACK 64 seq = 0:1 privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited. > > >
On Thu, 17 Nov 2016, Mark Andrews wrote:
Why the hell should validating resolver have to work around the crap you guys are using? DO YOUR JOBS which is to use RFC COMPLIANT servers. You get PAID to do DNS because people think you are compentent to do the job. Evidence shows otherwise.
https://ednscomp.isc.org/compliance/gov-full-report.html show the broken servers for .gov. It isn't hard to check.
Engineer complains the universe doesn't match his assumptions.
* Mark Andrews:
The DNSSEC testing is also insufficient. 9-11commission.gov shows green for example but if you use DNS COOKIES (which BIND 9.10.4 and BIND 9.11.0 do) then servers barf and return BADVERS and validation fails. QWEST you have been informed of this already.
Why the hell should validating resolver have to work around the crap you guys are using?
The protocol doesn't have proper version negotation, and again and again, implementers have tried to force backwards-incompatible implementations on the Internet at large.
In message <87twb4slol.fsf@mid.deneb.enyo.de>, Florian Weimer writes:
* Mark Andrews:
The DNSSEC testing is also insufficient. 9-11commission.gov shows green for example but if you use DNS COOKIES (which BIND 9.10.4 and BIND 9.11.0 do) then servers barf and return BADVERS and validation fails. QWEST you have been informed of this already.
Why the hell should validating resolver have to work around the crap you guys are using?
The protocol doesn't have proper version negotation, and again and again, implementers have tried to force backwards-incompatible implementations on the Internet at large.
The servers talk EDNS. They server signed zones which require EDNS support. These are EDNS version 0 queries. BADVERS is the response code for when you receive a EDNS version field higher than the one you implement and it requires that you use EDNS to send the rcode. The query has a EDNS version field of 0. The response has a EDNS version field of 0. The response has a rcode of BADVERS. Yes, the behaviour of how to handle unknown options was clarified in RFC 6891. With RFC 2671 nameserver developer had a choice to make * BADVERS was never a sensible response however. BADVERS had explict instructions for when it should be sent and they didn't include "if there is a EDNS option you don't understand return BADVERS". If you thought the version field was wrong then that made it a malformed request which should elicit a FORMERR. * FORMERR also wasn't a good idea. RFC 2671 didn't say that no options could be added to a EDNS version 1 query but I can see if you thought EDNS version 0 was not allowed to have EDNS options that it could be seen as a malformed record. Unless you know the format of the option you can't sensibly return FORMERR on it. * NOTIMP would have made sense before RFC 6891 was published but we never saw a implementation that did this. * Echoing back the option made some sense, though sending a option that you don't understand is risky. * Ignoring the option also made sense. This is what RFC 6891 says to do and matches the unknown EDNS flags behaviour. * Ask the working group / IETF. I wouldn't be complaining about it if they were only supporting plain DNS. You can tell the difference between a server that supports EDNS and one that doesn't. They behave differently. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Nov 15, 2016, at 5:30 PM, Carl Byington <carl@five-ten-sg.com> wrote:
openssl s_client -connect www.pay.gov:443
I’m not seeing the issue here, but they do have some possible issues the way they’re setting cookies (See details below). What path are you seeing to them? I’m also not having the issue from the IETF97 network here in Seoul which has IPv6 as well. puck:~$ traceroute6 www.pay.gov. traceroute to www.pay.gov. (2605:3100:fffd:100::15), 30 hops max, 80 byte packets 1 ge-0-7-0-22.r05.chcgil09.us.bb.gin.ntt.net (2001:418:3f4::1) 0.751 ms 0.871 ms 0.994 ms 2 verio-gw.cgcil.ipv6.att.net (2001:1890:1fff:307:192:205:32:193) 2.008 ms 1.991 ms 2.837 ms 3 cgcil22crs.ipv6.att.net (2001:1890:ff:ffff:12:122:132:198) 27.333 ms 27.167 ms 27.070 ms 4 sl9mo22crs.ipv6.att.net (2001:1890:ff:ffff:12:122:2:178) 27.602 ms 27.646 ms 27.628 ms 5 sl9mo21crs.ipv6.att.net (2001:1890:ff:ffff:12:122:2:217) 30.055 ms 29.894 ms 29.855 ms 6 dlstx22crs.ipv6.att.net (2001:1890:ff:ffff:12:122:2:1) 28.888 ms 27.016 ms 26.933 ms 7 dlstx84crs.ipv6.att.net (2001:1890:ff:ffff:12:123:18:249) 28.126 ms 26.757 ms 26.645 ms 8 2001:1890:ff:ffff:12:122:124:141 (2001:1890:ff:ffff:12:122:124:141) 26.142 ms 26.269 ms 26.179 ms 9 2001:1890:c00:610b::1138:7d27 (2001:1890:c00:610b::1138:7d27) 27.273 ms 27.255 ms 27.544 ms 10 2001:1890:1c08:cf01::2 (2001:1890:1c08:cf01::2) 27.673 ms !X 27.559 ms !X 27.465 ms !X curl -v https://www.pay.gov/public/home * Trying 2605:3100:fffd:100::15... * TCP_NODELAY set * Connected to www.pay.gov (2605:3100:fffd:100::15) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * ALPN/NPN, server did not agree to a protocol * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=www.pay.gov,O=United States Department of Treasury,L=Washington,ST=District of Columbia,C=US * start date: May 28 14:58:43 2015 GMT * expire date: May 29 06:16:02 2018 GMT * common name: www.pay.gov * issuer: CN=Entrust Certification Authority - L1K,OU="(c) 2012 Entrust, Inc. - for authorized use only",OU=See www.entrust.net/legal-terms,O="Entrust, Inc.",C=US
GET /public/home HTTP/1.1 Host: www.pay.gov User-Agent: curl/7.51.0 Accept: */*
< HTTP/1.1 200 OK < Date: Wed, 16 Nov 2016 21:52:08 GMT < Content-type: text/html; charset=ISO-8859-1 < Strict-transport-security: max-age=31536001; includeSubDomains < Cache-Control: no-cache < Cache-Control: no-store < Pragma: no-cache < Expires: Thu, 01 Jan 1970 00:00:00 GMT < X-XSS-Protection: 1; mode=block < Strict-Transport-Security: max-age=31536000 < Set-Cookie: JSESSIONID=949QYsVLKQqBB42HTy91pJnGfnfJthLfQTv02CvDnt7rNQnpSvb1!1259175335!-1040755441!1479333128223; path=/public; secure; HttpOnly < Set-Cookie: JSESSIONID=949QYsVLKQqBB42HTy91pJnGfnfJthLfQTv02CvDnt7rNQnpSvb1!1259175335!-1040755441; path=/public; HttpOnly < Set-Cookie: ClientId=14793331282345260; path=/public; HttpOnly; secure < Set-Cookie: ClientId=1479333128244363; path=/public; HttpOnly; secure < X-FRAME-OPTIONS: DENY < Content-Language: en-US < X-Powered-By: Servlet/2.5 JSP/2.1 < Transfer-encoding: chunked
It happens too often, unfortunately. People deploying IPv6 at web sites and other services, don’t check if PMTUD is broken by filtering, ECMP, load balancers, etc. This is the case here: tbit from 2001:df0:4:4000::1:115 to 2605:3100:fffd:100::15 server-mss 1440, result: pmtud-fail app: http, url: https://www.pay.gov/ [ 0.009] TX SYN 64 seq = 0:0 [ 0.165] RX SYN/ACK 64 seq = 0:1 [ 0.166] TX 60 seq = 1:1 [ 0.166] TX 371 seq = 1:1(311) [ 0.325] RX 1500 seq = 1:312(1440) [ 0.325] RX 1500 seq = 1441:312(1440) [ 0.325] TX PTB 1280 mtu = 1280 [ 0.325] RX 1362 seq = 2881:312(1302) [ 3.325] RX 1500 seq = 1:312(1440) [ 3.325] TX PTB 1280 mtu = 1280 [ 9.326] RX 1500 seq = 1:312(1440) [ 9.326] TX PTB 1280 mtu = 1280 [ 21.325] RX 1500 seq = 1:312(1440) [ 21.325] TX PTB 1280 mtu = 1280 [ 45.325] RX 1500 seq = 1:312(1440) Regards, Jordi -----Mensaje original----- De: NANOG <nanog-bounces@nanog.org> en nombre de Carl Byington <carl@five-ten-sg.com> Responder a: <carl@five-ten-sg.com> Fecha: miércoles, 16 de noviembre de 2016, 7:30 Para: <nanog@nanog.org> Asunto: pay.gov and IPv6 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Following up on a two year old thread, one of my clients just hit this problem. The failure is not that www.pay.gov is not reachable over ipv6 (2605:3100:fffd:100::15). They accept (TCP handshake) the port 443 connection, but the connection then hangs waiting for the TLS handshake. openssl s_client -connect www.pay.gov:443 openssl s_client -servername www.pay.gov -connect 199.169.192.21:443 Browsers (at least firefox) see that as a very slow site, and it does not trigger their happy eyeballs fast failover to ipv4. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEAREKAAYFAlgrjDEACgkQL6j7milTFsG8OwCgh5yRxxZHskjL4HVhzxIEmenA LQgAniRMcYf/DIcg+8ve55MxUgrUbmzC =MS8j -----END PGP SIGNATURE----- ********************************************** IPv4 is over Are you ready for the new Internet ? http://www.consulintel.es The IPv6 Company This electronic message contains information which may be privileged or confidential. The information is intended to be for the use of the individual(s) named above. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, including attached files, is prohibited.
participants (9)
-
Carl Byington -
Florian Weimer -
Jared Mauch -
joel jaeggli -
JORDI PALET MARTINEZ -
Lee -
Mark Andrews -
Matthew Kaufman -
Sean Donelan