Quoting from the web site at https://dnsflagday.net/ What is happening? The current DNS is unnecessarily slow and suffers from inability to deploy new features. To remediate these problems, vendors of DNS software and also big public DNS providers are going to remove certain workarounds on February 1st, 2019. This change affects only sites which operate software which is not following published standards. Are you affected? On that web page, there is a Domain Owner's test. You can enter a domain name and click 'test' and shortly receive a report of what was found regarding your domain's DNS servers. I somehow managed to miss the announcement of this upcoming event, even though I read this mailing list fairly closely. Perhaps it was announced somewhere else instead. I think it needs to be mentioned here if it hasn't already been. - Brian
I’ve been complaining for YEARS about lack of EDNS compliance. If you run really old Windows DNS servers you are broken. If you run a firewall in front of your DNS server you may be broken. If you are QWEST you are broken. Mark
On 24 Jan 2019, at 11:10 am, Brian Kantor <Brian@ampr.org> wrote:
Quoting from the web site at https://dnsflagday.net/
What is happening?
The current DNS is unnecessarily slow and suffers from inability to deploy new features. To remediate these problems, vendors of DNS software and also big public DNS providers are going to remove certain workarounds on February 1st, 2019.
This change affects only sites which operate software which is not following published standards. Are you affected?
On that web page, there is a Domain Owner's test. You can enter a domain name and click 'test' and shortly receive a report of what was found regarding your domain's DNS servers.
I somehow managed to miss the announcement of this upcoming event, even though I read this mailing list fairly closely. Perhaps it was announced somewhere else instead. I think it needs to be mentioned here if it hasn't already been. - Brian
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Wed, Jan 23, 2019 at 5:27 PM Mark Andrews <marka@isc.org> wrote:
I’ve been complaining for YEARS about lack of EDNS compliance.
If you run really old Windows DNS servers you are broken.
If you run a firewall in front of your DNS server you may be broken.
If you are QWEST you are broken.
Mark
duckdns.org :(
On Thu, 24 Jan 2019 11:22:44 +1100, Mark Andrews <marka@isc.org> may have written:
If you run a firewall in front of your DNS server you may be broken.
If you run a firewall in front of your DNS server and the firewall breaks EDNS, then your firewall is broken. And has been a long, long time. I put a firewall in place back in 2004, and EDNS compliance was one of the tests back then. -- Mike Meredith, University of Portsmouth Chief Systems Engineer, Hostmaster, Security, and Timelord!
On 24 Jan 2019, at 9:02 pm, Mike Meredith <mike.meredith@port.ac.uk> wrote:
On Thu, 24 Jan 2019 11:22:44 +1100, Mark Andrews <marka@isc.org> may have written:
If you run a firewall in front of your DNS server you may be broken.
If you run a firewall in front of your DNS server and the firewall breaks EDNS, then your firewall is broken. And has been a long, long time. I put a firewall in place back in 2004, and EDNS compliance was one of the tests back then.
EDNS usage has changed since them. Back in 2004 there was zero use of EDNS options in queries. That is no longer true. NSID (RFC 5001) the first option to make it into main stream code was allocated in 2007 and that saw occasional use. DNS COOKIE has been in every query named has emitted since BIND 9.11.0 and in late BIND 9.10 versions. Lots of firewalls still reject it.
-- Mike Meredith, University of Portsmouth Chief Systems Engineer, Hostmaster, Security, and Timelord!
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
We see Juniper firewalls blocking EDNS(1) and NSID by default. We see Checkpoint firewalls blocking EDNS(1) and EDNS flags by default. There is a another vendor that blocks EDNS(1). Juniper and Checkpoint have newer code that doesn’t do this. The old firewalls are still out there however. You can see them easily when you are doing bulk testing and mark timeouts in a different colour. Please go look at the reports on https://ednscomp.isc.org to see how obvious they are. There were times in the last 4 years where over 50% of the tested servers were dropping EDNS(1) queries. With drop rates like that you limited the ability of the IETF to use EDNS(1) to fix issues with EDNS correctly. The RFC 6891 would have included a version bump except for these stupid firewalls. The clarification of unknown EDNS option behaviour warranted a version bump. Blocking any of the extension mechanisms (version, flag or option) isn’t doing anyone any benefit. If you have a firewall that does it please FIX IT.
On 24 Jan 2019, at 10:13 pm, Mark Andrews <marka@isc.org> wrote:
On 24 Jan 2019, at 9:02 pm, Mike Meredith <mike.meredith@port.ac.uk> wrote:
On Thu, 24 Jan 2019 11:22:44 +1100, Mark Andrews <marka@isc.org> may have written:
If you run a firewall in front of your DNS server you may be broken.
If you run a firewall in front of your DNS server and the firewall breaks EDNS, then your firewall is broken. And has been a long, long time. I put a firewall in place back in 2004, and EDNS compliance was one of the tests back then.
EDNS usage has changed since them. Back in 2004 there was zero use of EDNS options in queries. That is no longer true. NSID (RFC 5001) the first option to make it into main stream code was allocated in 2007 and that saw occasional use. DNS COOKIE has been in every query named has emitted since BIND 9.11.0 and in late BIND 9.10 versions. Lots of firewalls still reject it.
-- Mike Meredith, University of Portsmouth Chief Systems Engineer, Hostmaster, Security, and Timelord!
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Mark Andrews <marka@isc.org> writes:
I’ve been complaining for YEARS about lack of EDNS compliance.
Didn't help. bjorn@miraculix:~$ dig +edns=42 +noednsnegotiation @1.1 ; <<>> DiG 9.11.5-P1-1-Debian <<>> +edns=42 +noednsnegotiation @1.1 ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40525 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1452 ;; QUESTION SECTION: ;. IN NS ;; ANSWER SECTION: . 9060 IN NS d.root-servers.net. . 9060 IN NS e.root-servers.net. . 9060 IN NS f.root-servers.net. . 9060 IN NS g.root-servers.net. . 9060 IN NS h.root-servers.net. . 9060 IN NS i.root-servers.net. . 9060 IN NS j.root-servers.net. . 9060 IN NS k.root-servers.net. . 9060 IN NS l.root-servers.net. . 9060 IN NS m.root-servers.net. . 9060 IN NS a.root-servers.net. . 9060 IN NS b.root-servers.net. . 9060 IN NS c.root-servers.net. ;; Query time: 3 msec ;; SERVER: 1.0.0.1#53(1.0.0.1) ;; WHEN: Thu Jan 24 14:40:00 CET 2019 ;; MSG SIZE rcvd: 431 Bjørn
On 25 Jan 2019, at 12:50 am, Bjørn Mork <bjorn@mork.no> wrote:
Mark Andrews <marka@isc.org> writes:
I’ve been complaining for YEARS about lack of EDNS compliance.
Didn't help.
Perfect vs incremental improvement. Please go look at the graphs on ednscomp.isc.org. You will see the fixes being applied. You can see firewalls being removed. You can seen IPv6 being deployed on DNS server farms with broken DNS servers. You can see when name servers are fixed to remove implementation flaws.
bjorn@miraculix:~$ dig +edns=42 +noednsnegotiation @1.1
; <<>> DiG 9.11.5-P1-1-Debian <<>> +edns=42 +noednsnegotiation @1.1 ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40525 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1452 ;; QUESTION SECTION: ;. IN NS
;; ANSWER SECTION: . 9060 IN NS d.root-servers.net. . 9060 IN NS e.root-servers.net. . 9060 IN NS f.root-servers.net. . 9060 IN NS g.root-servers.net. . 9060 IN NS h.root-servers.net. . 9060 IN NS i.root-servers.net. . 9060 IN NS j.root-servers.net. . 9060 IN NS k.root-servers.net. . 9060 IN NS l.root-servers.net. . 9060 IN NS m.root-servers.net. . 9060 IN NS a.root-servers.net. . 9060 IN NS b.root-servers.net. . 9060 IN NS c.root-servers.net.
;; Query time: 3 msec ;; SERVER: 1.0.0.1#53(1.0.0.1) ;; WHEN: Thu Jan 24 14:40:00 CET 2019 ;; MSG SIZE rcvd: 431
Bjørn
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Wed, Jan 23, 2019 at 7:11 PM Brian Kantor <Brian@ampr.org> wrote:
Quoting from the web site at https://dnsflagday.net/
huh, from the 'dns illuminati' eh" DNS hosted by gandi.net? resolves to 3 /32's on 3 adjacent /24's.. in github's ip space, routed by fastly.com ... I'm sure glad the whois data for that domain is sensible too... :( none of that particularly leaves me feeling like I should go put any data at all into the site. -chris
Well you can go to https://ednscomp.isc.org and click on "Test Your Servers Here” which is what https://dnsflagday.net calls behind the scenes. You will just need to interpret the results as they apply to DNS flag day. If you don’t want to go there you can go to https://gitlab.isc.org and down load and compile the DNS compliance tester and then run “genreport -i bind11 -e”. which is the actual test code being run. But hey you did do proper acceptance testing when you installed your DNS servers and firewalls to ensure that they implemented the DNS protocol correctly and they your firewalls don’t block well formed DNS queries (lots of them do by default). Mark
On 24 Jan 2019, at 3:35 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 7:11 PM Brian Kantor <Brian@ampr.org> wrote: Quoting from the web site at https://dnsflagday.net/
huh, from the 'dns illuminati' eh"
DNS hosted by gandi.net? resolves to 3 /32's on 3 adjacent /24's.. in github's ip space, routed by fastly.com ... I'm sure glad the whois data for that domain is sensible too... :(
none of that particularly leaves me feeling like I should go put any data at all into the site.
-chris
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Wed, Jan 23, 2019 at 11:45 PM Mark Andrews <marka@isc.org> wrote:
Well you can go to https://ednscomp.isc.org and click on "Test Your Servers Here” which is what https://dnsflagday.net calls behind the scenes. You will just need to interpret the results as they apply to DNS flag day. If you don’t want to go there you can go to https://gitlab.isc.org and down load and compile the DNS compliance tester and then run “genreport -i bind11 -e”. which is the actual test code being run.
oh excellent, I'll do this version. thanks.
But hey you did do proper acceptance testing when you installed your DNS servers and firewalls to ensure that they implemented the DNS protocol correctly and they your firewalls don’t block well formed DNS queries (lots of them do by default).
I did, yes.
Mark
On 24 Jan 2019, at 3:35 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 7:11 PM Brian Kantor <Brian@ampr.org> wrote: Quoting from the web site at https://dnsflagday.net/
huh, from the 'dns illuminati' eh"
DNS hosted by gandi.net? resolves to 3 /32's on 3 adjacent /24's.. in github's ip space, routed by fastly.com ... I'm sure glad the whois data for that domain is sensible too... :(
none of that particularly leaves me feeling like I should go put any data at all into the site.
-chris
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Also as a lot of you use F5 servers here is information about DNS flag day fixes. https://support.f5.com/csp/article/K07808381?sf206085287=1
On 24 Jan 2019, at 3:51 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 11:45 PM Mark Andrews <marka@isc.org> wrote: Well you can go to https://ednscomp.isc.org and click on "Test Your Servers Here” which is what https://dnsflagday.net calls behind the scenes. You will just need to interpret the results as they apply to DNS flag day. If you don’t want to go there you can go to https://gitlab.isc.org and down load and compile the DNS compliance tester and then run “genreport -i bind11 -e”. which is the actual test code being run.
oh excellent, I'll do this version. thanks.
But hey you did do proper acceptance testing when you installed your DNS servers and firewalls to ensure that they implemented the DNS protocol correctly and they your firewalls don’t block well formed DNS queries (lots of them do by default).
I did, yes.
Mark
On 24 Jan 2019, at 3:35 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 7:11 PM Brian Kantor <Brian@ampr.org> wrote: Quoting from the web site at https://dnsflagday.net/
huh, from the 'dns illuminati' eh"
DNS hosted by gandi.net? resolves to 3 /32's on 3 adjacent /24's.. in github's ip space, routed by fastly.com ... I'm sure glad the whois data for that domain is sensible too... :(
none of that particularly leaves me feeling like I should go put any data at all into the site.
-chris
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
And if you don’t want to go to the web site you can still see the content here https://github.com/dns-violations/dnsflagday
On 24 Jan 2019, at 4:32 pm, Mark Andrews <marka@isc.org> wrote:
Also as a lot of you use F5 servers here is information about DNS flag day fixes.
https://support.f5.com/csp/article/K07808381?sf206085287=1
On 24 Jan 2019, at 3:51 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 11:45 PM Mark Andrews <marka@isc.org> wrote: Well you can go to https://ednscomp.isc.org and click on "Test Your Servers Here” which is what https://dnsflagday.net calls behind the scenes. You will just need to interpret the results as they apply to DNS flag day. If you don’t want to go there you can go to https://gitlab.isc.org and down load and compile the DNS compliance tester and then run “genreport -i bind11 -e”. which is the actual test code being run.
oh excellent, I'll do this version. thanks.
But hey you did do proper acceptance testing when you installed your DNS servers and firewalls to ensure that they implemented the DNS protocol correctly and they your firewalls don’t block well formed DNS queries (lots of them do by default).
I did, yes.
Mark
On 24 Jan 2019, at 3:35 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 7:11 PM Brian Kantor <Brian@ampr.org> wrote: Quoting from the web site at https://dnsflagday.net/
huh, from the 'dns illuminati' eh"
DNS hosted by gandi.net? resolves to 3 /32's on 3 adjacent /24's.. in github's ip space, routed by fastly.com ... I'm sure glad the whois data for that domain is sensible too... :(
none of that particularly leaves me feeling like I should go put any data at all into the site.
-chris
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Thu, Jan 24, 2019 at 12:35 AM Mark Andrews <marka@isc.org> wrote:
And if you don’t want to go to the web site you can still see the content here
I think part of my snark was lost as snark here... So, we're asking 'everyone' to do 'something' on behalf of their domains, their users and the rest of the internet... we can't seem to do that in a fashion that's traceable, clearly has ownership and doesn't look like every halfbaked spam campaign in the world. Yes I could go digging for the right starting point at ISC or github or .. what?? Why wasn't this pretty clearly owned by 'ICANN' or some organization like that? It's lovely that github, fastly, gandi and ISC want to help, but... somewhere here some legitimacy could have been injected into the process, right? "HI, we're ICANN we do dns thingies, and we'd like to help make you make things better. Please use the website (provided by our partner(s) X, Y, Z to do the following A, B, C things, and get guidance on repair for problems at site FOO, BAR or BAZ. If there are questions please see our FAQ ( https://www.icann.org/dnsfixin/faq) or email <support@icann.org>. Thanks for taking the time to make the world better?" it's not super hard to do this, it's also apparently super easy to look like a spam/malware campaign.
On 24 Jan 2019, at 4:32 pm, Mark Andrews <marka@isc.org> wrote:
Also as a lot of you use F5 servers here is information about DNS flag day fixes.
https://support.f5.com/csp/article/K07808381?sf206085287=1
On 24 Jan 2019, at 3:51 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 11:45 PM Mark Andrews <marka@isc.org> wrote: Well you can go to https://ednscomp.isc.org and click on "Test Your Servers Here” which is what https://dnsflagday.net calls behind the scenes. You will just need to interpret the results as they apply to DNS flag day. If you don’t want to go there you can go to https://gitlab.isc.org and down load and compile the DNS compliance tester and then run “genreport -i bind11 -e”. which is the actual test code being run.
oh excellent, I'll do this version. thanks.
But hey you did do proper acceptance testing when you installed your DNS servers and firewalls to ensure that they implemented the DNS protocol correctly and they your firewalls don’t block well formed DNS queries (lots of them do by default).
I did, yes.
Mark
On 24 Jan 2019, at 3:35 pm, Christopher Morrow < morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 7:11 PM Brian Kantor <Brian@ampr.org> wrote: Quoting from the web site at https://dnsflagday.net/
huh, from the 'dns illuminati' eh"
DNS hosted by gandi.net? resolves to 3 /32's on 3 adjacent /24's.. in github's ip space, routed by fastly.com ... I'm sure glad the whois data for that domain is sensible too... :(
none of that particularly leaves me feeling like I should go put any data at all into the site.
-chris
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 24 Jan 2019, at 4:45 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Thu, Jan 24, 2019 at 12:35 AM Mark Andrews <marka@isc.org> wrote: And if you don’t want to go to the web site you can still see the content here
https://github.com/dns-violations/dnsflagday
I think part of my snark was lost as snark here... So, we're asking 'everyone' to do 'something' on behalf of their domains, their users and the rest of the internet... we can't seem to do that in a fashion that's traceable, clearly has ownership and doesn't look like every halfbaked spam campaign in the world.
Yes I could go digging for the right starting point at ISC or github or .. what?? Why wasn't this pretty clearly owned by 'ICANN' or some organization like that?
Well the IETF doesn’t want to be “Protocol Police”. ICANN has enforced this on gTLD operators as they are contractually obligated to run protocol compliant servers. Almost all of the ccTLD servers are fixed as well. https://ednscomp.isc.org/compliance/ts/tld-graphs.html https://ednscomp.isc.org/compliance/tld-report.html I’ve argued for years that TLD operators should be doing tests like this on all delegated domains and delisting them until they have fixed the server after a initial grace period with multiple warnings. They are in a position to send warning to operators and owners of delegated zones. They just say “this is not our job” despite RFC 1033 actually listing removal of delegation as something that should be done by the parent zone (TLD) if other methods of fixing servers that don’t follow the specification fail. No one wanted own this. This is Open Source DNS vendors saying "Enough is Enough”. We are tired of having to write workarounds for all the broken servers out there especially as those workarounds impacts on sites that have protocol compliant servers. None of use could move alone as we would get “But it works with …”. This needed to be a collective move. The public DNS resolver operators are also on board. No system works if there are not checks and balances in place. The DNS still doesn’t have checks and balances in place. Mark
It's lovely that github, fastly, gandi and ISC want to help, but... somewhere here some legitimacy could have been injected into the process, right?
"HI, we're ICANN we do dns thingies, and we'd like to help make you make things better. Please use the website (provided by our partner(s) X, Y, Z to do the following A, B, C things, and get guidance on repair for problems at site FOO, BAR or BAZ. If there are questions please see our FAQ (https://www.icann.org/dnsfixin/faq) or email <support@icann.org>. Thanks for taking the time to make the world better?"
it's not super hard to do this, it's also apparently super easy to look like a spam/malware campaign.
On 24 Jan 2019, at 4:32 pm, Mark Andrews <marka@isc.org> wrote:
Also as a lot of you use F5 servers here is information about DNS flag day fixes.
https://support.f5.com/csp/article/K07808381?sf206085287=1
On 24 Jan 2019, at 3:51 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 11:45 PM Mark Andrews <marka@isc.org> wrote: Well you can go to https://ednscomp.isc.org and click on "Test Your Servers Here” which is what https://dnsflagday.net calls behind the scenes. You will just need to interpret the results as they apply to DNS flag day. If you don’t want to go there you can go to https://gitlab.isc.org and down load and compile the DNS compliance tester and then run “genreport -i bind11 -e”. which is the actual test code being run.
oh excellent, I'll do this version. thanks.
But hey you did do proper acceptance testing when you installed your DNS servers and firewalls to ensure that they implemented the DNS protocol correctly and they your firewalls don’t block well formed DNS queries (lots of them do by default).
I did, yes.
Mark
On 24 Jan 2019, at 3:35 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 23, 2019 at 7:11 PM Brian Kantor <Brian@ampr.org> wrote: Quoting from the web site at https://dnsflagday.net/
huh, from the 'dns illuminati' eh"
DNS hosted by gandi.net? resolves to 3 /32's on 3 adjacent /24's.. in github's ip space, routed by fastly.com ... I'm sure glad the whois data for that domain is sensible too... :(
none of that particularly leaves me feeling like I should go put any data at all into the site.
-chris
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
* morrowc.lists@gmail.com (Christopher Morrow) [Thu 24 Jan 2019, 06:46 CET]:
So, we're asking 'everyone' to do 'something' on behalf of their domains, their users and the rest of the internet... we can't seem to do that in a fashion that's traceable, clearly has ownership and doesn't look like every halfbaked spam campaign in the world.
Do those link to presentations given at major industry conferences like RIPE and DNS-OARC, with well known industry people getting on stage to talk about it? What more do you need? More recent additions, mayhaps? UKNOF? https://www.youtube.com/watch?v=e4KdsexpB7Q LACNIC? https://www.youtube.com/watch?v=_hCGucH0kRU SDNOG? https://www.youtube.com/watch?v=UksQZkxOC-Q Oh wait, you had to scroll down on the main dnsflagday.net page to see those links, which in this day of only the first page of Google results mattering, isn't going to happen, apparently. -- Niels.
you are seriously cracking me up... but. On Thu, Jan 24, 2019 at 6:05 AM Niels Bakker <niels=nanog@bakker.net> wrote:
* morrowc.lists@gmail.com (Christopher Morrow) [Thu 24 Jan 2019, 06:46 CET]:
So, we're asking 'everyone' to do 'something' on behalf of their domains, their users and the rest of the internet... we can't seem to do that in a fashion that's traceable, clearly has ownership and doesn't look like every halfbaked spam campaign in the world.
Oh wait, you had to scroll down on the main dnsflagday.net page to see
those links, which in this day of only the first page of Google results mattering, isn't going to happen, apparently.
someone sent a url in a mail message, you didn't seriously just click on it without any attempt to verify it wasn't bad/problematic/going-to-eat-your-brain ? (in this day and age, everyone apparently does just do that)
On 1/23/19 8:44 PM, Mark Andrews wrote:
and they your firewalls don’t block well formed DNS queries (lots of them do by default).
My edge routers block *all* inbound DNS requests -- I was being hit by a ton of them at one point. Cavaet: I don't run a DNS server that is a domain zone master -- I use a DNS service for that. I do have a DNS server inside, but only to handle recursive requests from inside my network. Outbound DNS requests? Lets them through, and responses too.
On 25 Jan 2019, at 2:14 am, Stephen Satchell <list@satchell.net> wrote:
On 1/23/19 8:44 PM, Mark Andrews wrote:
and they your firewalls don’t block well formed DNS queries (lots of them do by default).
My edge routers block *all* inbound DNS requests -- I was being hit by a ton of them at one point. Cavaet: I don't run a DNS server that is a domain zone master -- I use a DNS service for that. I do have a DNS server inside, but only to handle recursive requests from inside my network.
Outbound DNS requests? Lets them through, and responses too.
Well does your DNS service properly manage the firewall in front of their servers? Does the anti DoS scrubbing service they are using also pass the well formed packets to the DNS server they are advertising? This was about testing the servers YOU directly or indirectly advertise to the world. It also applies to any recursive servers. They too need properly handle EDNS queries in all their forms. The test tool has a recursive mode for testing them (genreport -R). Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 1/24/19 11:46 AM, Mark Andrews wrote:
On 25 Jan 2019, at 2:14 am, Stephen Satchell <list@satchell.net> wrote:
My edge routers block *all* inbound DNS requests -- I was being hit by a ton of them at one point. Cavaet: I don't run a DNS server that is a domain zone master -- I use a DNS service for that. I do have a DNS server inside, but only to handle recursive requests from inside my network.
Outbound DNS requests? Lets them through, and responses too.
Well does your DNS service properly manage the firewall in front of their servers? Does the anti DoS scrubbing service they are using also pass the well formed packets to the DNS server they are advertising?
I have domains on several DNS services. Most of the services work properly according to the ISC tests. Two of the services show failures. So I called support on the pair. One service says they are deploying updates before the 1 Feb 19 deadline to all their DNS servers. The other one (starts with an "A") doesn't know when they will be fully compliant "but your customers should have no difficulty with getting DNS answers on your domains." I had downloaded the tool, so I tested my inside DNS servers just for grins. Passed with flying colors -- I had used Centos 7 in those servers, updated on a regularly scheduled basis, so of course it flew with flying colors. (Or do you prefer colours?)
On Wed, Jan 23, 2019 at 4:12 PM Brian Kantor <Brian@ampr.org> wrote:
Quoting from the web site at https://dnsflagday.net/
[...]
The current DNS is unnecessarily slow and suffers from inability to deploy new features. To remediate these problems, vendors of DNS software and also big public DNS providers are going to remove certain workarounds on February 1st, 2019.
I would like to note that there is an entire segment of the population that does not interact with technology between sundown on Friday, all the way through Sunday morning. Choosing Friday as a day to carry out an operational change of this sort does not seem to have given thought that if things break, there is a possibility they will have to stay broken for at least a full day before the right people can be engaged to work on the issue. In the future, can we try to schedule such events with more consideration on which day the change will take place? I will also note that this weekend is the Superbowl in the US; one of the bigger advertising events of the year. Potentially breaking advertising systems that rely on DNS two days before a major, once-a-year advertising event is *also* somewhat inconsiderate. While I understand that no day will work for everyone, and at some point you just have to pick a day and go for it, I will note that picking the Friday before the Superbowl does seem like a very unfortunate random pick for a day on which to do it. Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort? Thanks! Matt
On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort?
IIRC, DNS Flag Day was announce way before last years Super Bowl... what did the people who aren't ready for DNS Flag Day do in the past 364 days that they need a few more days to get ready for? -Jim P.
On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG <nanog@nanog.org> wrote:
On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort?
IIRC, DNS Flag Day was announce way before last years Super Bowl... what did the people who aren't ready for DNS Flag Day do in the past 364 days that they need a few more days to get ready for?
Oh, so they had 365 days to plan the time of the event and still picked a friday for that event? https://www.opsview.com/resources/system-administrator/blog/three-reasons-wh... I see.
On January 31, 2019 1:55:26 AM UTC, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG <nanog@nanog.org> wrote:
On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort?
IIRC, DNS Flag Day was announce way before last years Super Bowl... what did the people who aren't ready for DNS Flag Day do in the past 364 days that they need a few more days to get ready for?
Oh, so they had 365 days to plan the time of the event and still picked a friday for that event?
https://www.opsview.com/resources/system-administrator/blog/three-reasons-wh...
I see.
Well, you are either ready or you're not ...doing it on a Tuesday morning is not going to make any difference at this point. -Jim P.
You do realise that when the day was chosen it was just the date after which new versions of name servers by the original group of Open Source DNS developers would not have the work arounds incorporated? For ISC that will be BIND 9.14.0 and no that will not be available Feb 1 but you can use the development version 9.13 which has had the code for a while now. Individual operators of resolvers will make their own decisions about when to deploy. -- Mark Andrews
On 31 Jan 2019, at 12:55, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG <nanog@nanog.org> wrote: On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort?
IIRC, DNS Flag Day was announce way before last years Super Bowl... what did the people who aren't ready for DNS Flag Day do in the past 364 days that they need a few more days to get ready for?
Oh, so they had 365 days to plan the time of the event and still picked a friday for that event?
https://www.opsview.com/resources/system-administrator/blog/three-reasons-wh...
I see.
On Wed, Jan 30, 2019 at 6:23 PM Mark Andrews <marka@isc.org> wrote:
You do realise that when the day was chosen it was just the date after which new versions of name servers by the original group of Open Source DNS
you do realize you are proposing to make a breaking change (breaking change to a global system) on a friday. delaying until the following monday would not have mattered to you, I'm sure it's going to matter to other folks though. thanks, -chris
developers would not have the work arounds incorporated?
For ISC that will be BIND 9.14.0 and no that will not be available Feb 1 but you can use the development version 9.13 which has had the code for a while now.
Individual operators of resolvers will make their own decisions about when to deploy. -- Mark Andrews
On 31 Jan 2019, at 12:55, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG <nanog@nanog.org> wrote:
On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort?
IIRC, DNS Flag Day was announce way before last years Super Bowl... what did the people who aren't ready for DNS Flag Day do in the past 364 days that they need a few more days to get ready for?
Oh, so they had 365 days to plan the time of the event and still picked a friday for that event?
https://www.opsview.com/resources/system-administrator/blog/three-reasons-wh...
I see.
On Wed, Jan 30, 2019 at 9:51 PM Christopher Morrow <morrowc.lists@gmail.com> wrote:
you do realize you are proposing to make a breaking change (breaking change to a global system) on a friday. delaying until the following monday would not have
There are reasons to prefer a Friday over other days as well, but the internet doesn't schedule around random participant's personal preferences. Besides, its a substantial misrepresentation of what the DNS Flag day is to describe it as a "breaking change" made on a certain date - changes won't finish in a week, changes won't finish in two weeks --- every day of the week may be affected until the gradual process of every OS and DNS vendor releasing and every end user upgrading finishes. Each software vendor and service provider will have their very own update schedules regarding on what exact date the next version release and every manager of a system with a DNS Resolver software installation will have their own choice on when they actually install the next update at their site. Just because all the major maintainers of DNS resolver software agree all releases after tomorrow will remove the workarounds for broken DNS servers/firewalls that silently discard queries does not mean every software vendor is shipping their new code to release on Feb 1, _and_ every end user is rushing to upgrade their DNS resolver to remove the workarounds. -- -JH
The only ones that could potentially make a “breaking change” on the Feb 1 are Google, Cloudflare and Quad9. They are the public DNS recursive server operators that have committed to removing work arounds. Google has already stated publicly that it will be introducing changes gradually and I expect the other to also do so. Name server developers DO NOT have that power. Google, Cloudflare and Quad9 are needed so the collectively we don’t need to deal with “but it works with …”. That also the reason for the rest of the vendors doing it in unison. What is needed next is for infrastructure zone operators to down load the compliance tools and run them on the servers listed in their zones daily and then inform the owners of those delegations that their zones are on non-compliant servers and give them a dead line to fix them (120 days should be enough time). If the servers aren’t fixed by the dead line the delegation is removed until the servers are fixed or replaced with compliant ones. This will catch operators who install out-of-compliance servers and firewalls. The reaction so far by DNS server operators to DNS flag day shows that this is not actually unreasonable to require. The fixed code is out there for both name servers and firewalls. Mark
On 31 Jan 2019, at 2:49 pm, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 30, 2019 at 6:23 PM Mark Andrews <marka@isc.org> wrote: You do realise that when the day was chosen it was just the date after which new versions of name servers by the original group of Open Source DNS
you do realize you are proposing to make a breaking change (breaking change to a global system) on a friday. delaying until the following monday would not have mattered to you, I'm sure it's going to matter to other folks though.
thanks, -chris
developers would not have the work arounds incorporated?
For ISC that will be BIND 9.14.0 and no that will not be available Feb 1 but you can use the development version 9.13 which has had the code for a while now.
Individual operators of resolvers will make their own decisions about when to deploy. -- Mark Andrews
On 31 Jan 2019, at 12:55, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG <nanog@nanog.org> wrote: On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort?
IIRC, DNS Flag Day was announce way before last years Super Bowl... what did the people who aren't ready for DNS Flag Day do in the past 364 days that they need a few more days to get ready for?
Oh, so they had 365 days to plan the time of the event and still picked a friday for that event?
https://www.opsview.com/resources/system-administrator/blog/three-reasons-wh...
I see.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Thu, Jan 31, 2019, at 03:24, Mark Andrews wrote:
You do realise that when the day was chosen it was just the date after which new versions of name servers by the original group of Open Source DNS developers would not have the work arounds incorporated?
I think it's pretty safe to say that the "DNS Flag day" is more like a date of "end of support" rather than an "service termination". My guess is that some uncompliant servers will be still running long after that date... -- R-A.F.
-- Mark Andrews
On 31 Jan 2019, at 20:25, Radu-Adrian Feurdean <nanog@radu-adrian.feurdean.net> wrote:
On Thu, Jan 31, 2019, at 03:24, Mark Andrews wrote: You do realise that when the day was chosen it was just the date after which new versions of name servers by the original group of Open Source DNS developers would not have the work arounds incorporated?
I think it's pretty safe to say that the "DNS Flag day" is more like a date of "end of support" rather than an "service termination". My guess is that some uncompliant servers will be still running long after that date...
-- R-A.F.
On Thu, Jan 31, 2019, 01:27 Radu-Adrian Feurdean < nanog@radu-adrian.feurdean.net wrote:
On Thu, Jan 31, 2019, at 03:24, Mark Andrews wrote:
You do realise that when the day was chosen it was just the date after which new versions of name servers by the original group of Open Source DNS developers would not have the work arounds incorporated?
I think it's pretty safe to say that the "DNS Flag day" is more like a date of "end of support" rather than an "service termination". My guess is that some uncompliant servers will be still running long after that date...
-- R-A.F.
(resending from correct address) Right. The concern is that it's *also* the date when all the major recursive lookup servers are changing their behaviour. New software availability date? Awesome, go for it. Google, Cloudflare, Quad9 all changing their codebase/response behaviour on a Friday before a major sporting and advertising event? Not sounding like a really great idea from this side of the table. Are we certain that the changes on the part of the big four recursive DNS operators won't cause downstream issues? As someone noted earlier, this mainly affects products from a specific company, Microsoft, and L7 load balancers like A10s. I'm going to hope legal teams from each of the major recursive providers were consulted ahead of time to vet the effort, and ensure there were no concerns about collusion or anticompetitive practices, right? I'm fine with rolling out software that stops supporting bad behaviour. What I find to be concerning is when supposedly competing entities all band together in a pact that largely holds the rest of the world hostage to their arbitrary timeline. Perhaps it's time to create a new recursive resolver service that explicitly *is not* part of the cabal... Matt (hoping and praying this weekend will go smoothly)
On Thu, Jan 31, 2019 at 6:01 AM Matthew Petach <mpetach@netflight.com> wrote:
Google, Cloudflare, Quad9 all changing their codebase/response behaviour on a Friday before a major sporting and advertising event? Not sounding like a really great idea from this side of the table.
If your DNS zone is hosted on Google or Cloudflare's servers, then you should have nothing to worry about, other than your end users having a broken firewall in between their DNS resolver and the internet, and then updating their resolver software.... Actually, none of those providers announced detailed plans, at least yet; and maybe they won't even bother announcing. they could update their software yesterday if they wanted, or they could wait until next week, and it would still be "On or Around Feb 1, 2019." The 'Flag Day' is not a specific moment at which all providers necessarily push a big red button at the same instant to remove their workaround for broken DNS servers discarding queries.
Are we certain that the changes on the part of the big four recursive DNS operators won't cause downstream issues?
Not downstream issues. They will break resolution of the domains which have authoritative DNS servers that discard or ignore DNS queries which comply with all the original DNS standards but contain EDNS attributes. The common cause for this was Authoritative DNS servers placed behind 3rd party Firewalls that tried to "inspect" DNS traffic and discard queries and responses with "unknown" properties or sizes larger than 512 --- there may also be an esoteric DNS proxy/ balancer implementation with bugs, or some broken authoritative server implementations running software that was more than 10 years past End of Support at this point. If your authoritative DNS service sits behind such a broken device or on such broken DNS server, then clients already have troubles resolving your domains, and some time after the DNS Flag day, it will likely stop working altogether.
As someone noted earlier, this mainly affects products from a specific company, Microsoft, and L7 load balancers like A10s. I'm going to hope legal teams from each of the major recursive providers were consulted ahead of time to vet the effort, and ensure there were no concerns about collusion or anticompetitive practices, right?
I wouldn't be too concerned. The operators of a recursive DNS service very likely won't have an agreement giving them a duty to Microsoft, A10, etc; If you have a software or service that you expect to interoperate with DNS resolvers, then its on you to make sure your implementation of the software or service complies with the agreed standards regarding its processing of compliant messages. -- -JH
After reading through the thread, this reminds me of the Y2K flap, that turned into a non-event. My checks of authoritative DNS servers for my domains show no issues now.
On 1 Feb 2019, at 1:21 am, Stephen Satchell <list@satchell.net> wrote:
After reading through the thread, this reminds me of the Y2K flap, that turned into a non-event. My checks of authoritative DNS servers for my domains show no issues now.
As did I, but if we didn’t try and give lots of notice people would have complained. That said a lot of servers have been fixed that where not fully compliant and firewalls that unnecessarily blocked well formed EDNS packets with one or more of the extension mechanism present have been turned off. The authoritative DNS server space is much cleaner now and we the data to show it. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 31 Jan 2019, at 10:59 pm, Matthew Petach <mpetach@netflight.com> wrote:
On Thu, Jan 31, 2019, 01:27 Radu-Adrian Feurdean <nanog@radu-adrian.feurdean.net wrote:
On Thu, Jan 31, 2019, at 03:24, Mark Andrews wrote:
You do realise that when the day was chosen it was just the date after which new versions of name servers by the original group of Open Source DNS developers would not have the work arounds incorporated?
I think it's pretty safe to say that the "DNS Flag day" is more like a date of "end of support" rather than an "service termination". My guess is that some uncompliant servers will be still running long after that date...
-- R-A.F.
(resending from correct address)
Right.
The concern is that it's *also* the date when all the major recursive lookup servers are changing their behaviour.
New software availability date? Awesome, go for it.
Google, Cloudflare, Quad9 all changing their codebase/response behaviour on a Friday before a major sporting and advertising event?
Not sounding like a really great idea from this side of the table.
Are we certain that the changes on the part of the big four recursive DNS operators won't cause downstream issues?
As someone noted earlier, this mainly affects products from a specific company, Microsoft, and L7 load balancers like A10s. I'm going to hope legal teams from each of the major recursive providers were consulted ahead of time to vet the effort, and ensure there were no concerns about collusion or anticompetitive practices, right?
I'm fine with rolling out software that stops supporting bad behaviour.
What I find to be concerning is when supposedly competing entities all band together in a pact that largely holds the rest of the world hostage to their arbitrary timeline.
Perhaps it's time to create a new recursive resolver service that explicitly *is not* part of the cabal...
Matt (hoping and praying this weekend will go smoothly)
So you are worrying about sites running Windows DNS from prior to Windows Server 2003 (this is where Microsoft added EDNS support) and sites that have a firewall that blocks all EDNS queries. The large recursive server farms don’t do DNS COOKIE so you don’t need to worry about that. Those Windows servers work most of the time even with a DNS servers that don’t fall back to plain DNS on timeout. And if you have installed a firewall that blocks EDNS you have shot yourself in the foot. We actually have a hard time finding zones where all the servers are broken enough to not work with servers that don’t fallback to plain DNS on timeout. We can find zones where some of the servers are like that, but there is usually one or more servers that do respond correctly. Of the datasets I’ve looked at that 1 in 10,000 to 1 in 100,000 zones will have problems with updated servers. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 2019-01-31 08:15, Mark Andrews wrote:
We actually have a hard time finding zones where all the servers are broken enough to not work with servers that don’t fallback to plain DNS on timeout. We can find zones where some of the servers are like that, but there is usually one or more servers that do respond correctly.
Of the datasets I’ve looked at that 1 in 10,000 to 1 in 100,000 zones will have problems with updated servers.
I think the advertised testing tool may be flawed as blocking TCP/53 is enough to receive a STOP from the dnsflagday web site. It's been my (possibly flawed) understanding that TCP/53 is an option for clients but primarily it is a mechanism for the *server* to request the client communicate using TCP/53 instead - this could be due to UDP response size, anti-spoofing controls, etc... So is the tool right in saying that TCP/53 is a absolute requirement of ENDS0 support for "DNS Flag Day"? If so, do we expect a dramatic increases in TCP/53 requests over UDP/53 queries? Or is the tool flawed in its' claim that the DNS Flag Day changes *require* TCP/53 in order to resolve ones domain? Based upon my reading, the big concern is a edns=timeout result (from the ISC tool) not a edns512tcp=timeout. While I applaud the effort, the message and delivery could have been better. My first read on DNS Flag Day was that this was going to be the day new software versions were to be released which deprecated EDNS0 fallback measures so the adoption would be a gradual process by updating the latest versions of several DNS servers, only to find out that many resolvers used by end users use will be upgrading on Feb 1rst. Now, it sounds like the rollout schedule is on their timetable and could happen over the next several weeks and months. So really not a Flag "Day" now is it? I guess that's also why the date being on a Friday also isn't really a concern... Finally, why has no one stepped up to setup an updated resolver prior to Feb 1rst for actual testing? Or are there instructions on how one could setup their own resolver setup prior to Feb 1rst? No offense, but I'm not jumping to a brand new train of code just to enforce DNS Flag Day but I might choose enforce now under *existing* code bases rather than waiting for everyone else using Cloudflare/Google/OpenDNS as resolver may see me after Feb 1rst? If anyone has such instructions, post them - or put them on dnsflagday.net for anyone else wanting to adopt/test. Just some thoughts... -- -James
On 31/Jan/19 18:32, James Stahr wrote:
I think the advertised testing tool may be flawed as blocking TCP/53 is enough to receive a STOP from the dnsflagday web site. It's been my (possibly flawed) understanding that TCP/53 is an option for clients but primarily it is a mechanism for the *server* to request the client communicate using TCP/53 instead - this could be due to UDP response size, anti-spoofing controls, etc...
On a similar note, we tested for all our self-hosted zones OK 2 weeks ago. However, in previous days, the summary result said "NO GOOD, THINGS WILL BE SLOW COME FLAG DAY". The detailed test showed IPv4 tested perfect, but IPv6 probes timed out. The issue turned out to be an internal IPv6 routing/forwarding issue for our service within Century Link's (Level(3)'s) network. CL finally fixed that issue today and the flag day test tool is happy again. Some of our partners/customers were concerned our name servers were not ready, based purely on the summary result of the test tool. Perhaps adding some intelligence about whether the issue is the name server or the transport may be helpful. Mark.
TIMEOUT is TIMEOUT. The whole point of flag day is that you can’t tell whether TIMEOUT is broken routing, packet loss or badly configured firewall. The DNS flag day site assumes the latter as does the old resolver code. We are moving to a state where resolvers assume the former. You get a report with Red or Orange. Red reports have things which need to be fixed NOW be it a routing issue or packet loss or a badly configured firewall. Mark
On 1 Feb 2019, at 4:04 am, Mark Tinka <mark.tinka@seacom.mu> wrote:
On 31/Jan/19 18:32, James Stahr wrote:
I think the advertised testing tool may be flawed as blocking TCP/53 is enough to receive a STOP from the dnsflagday web site. It's been my (possibly flawed) understanding that TCP/53 is an option for clients but primarily it is a mechanism for the *server* to request the client communicate using TCP/53 instead - this could be due to UDP response size, anti-spoofing controls, etc...
On a similar note, we tested for all our self-hosted zones OK 2 weeks ago. However, in previous days, the summary result said "NO GOOD, THINGS WILL BE SLOW COME FLAG DAY". The detailed test showed IPv4 tested perfect, but IPv6 probes timed out.
The issue turned out to be an internal IPv6 routing/forwarding issue for our service within Century Link's (Level(3)'s) network. CL finally fixed that issue today and the flag day test tool is happy again.
Some of our partners/customers were concerned our name servers were not ready, based purely on the summary result of the test tool. Perhaps adding some intelligence about whether the issue is the name server or the transport may be helpful.
Mark.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Google has started their rollout. https://groups.google.com/forum/#!msg/public-dns-announce/-qaRKDV9InA/tExCFr... -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On 3 Feb 2019, at 2:01 am, Stephen Satchell <list@satchell.net> wrote:
On 2/1/19 1:23 PM, Mark Andrews wrote:
Google has started their rollout.
So has Red Hat (RHEL and Centos). I woke up to a rather large update this morning.
RedHat or third party RPM’s you have chosen to run on RedHat? RedHat is notorious for not updating packages they include in the base system and to the best of my knowledge they haven’t dug these changes out of BIND 9.13 (which is a development series) and back ported them. BIND 9.14 is yet to be released. Now PowerDNS have released their new recursive server and if you happen to have installed that on RedHat it may have been what you saw. https://blog.powerdns.com/2019/02/01/changes-in-the-powerdns-recursor-4-2-0/ Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Thu, Jan 31, 2019 at 10:33 AM James Stahr <stahr@mailbag.com> wrote: [snip]
So is the tool right in saying that TCP/53 is a absolute requirement of ENDS0 support for "DNS Flag Day"? If so, do we expect a dramatic increases in TCP/53 requests over UDP/53 queries? Or is the tool flawed [snip]
Their test tool will obviously alert on more error conditions than what the Flag Day is specifically about -- One or more of your DNS servers not responding [OR] TCP/53 not working are still broken configurations, But the brokenness is unrelated to what the flag day is about - In the first case, better safe than sorry, I suppose: Inability to complete one or more of the tests because of brokenness definitely means that things are broken. TCP/53 is a fairly strong requirement, except if you are supporting an authoritative-only server with no record labels that could result in a >512byte response, plus no DNSSEC-secured zones, and even then the AXFR protocol for replication to secondary servers calls for TCP. EDNS support is not required. Authoritative servers that don't support EDNS and are also compliant with the original DNS standard continue to work after the workarounds are removed. The relevant standard does not allow for silently ignoring requests that contain the EDNS option; patching the bug in a broken server does not necessarily entail the larger task of adding EDNS support -- achieving consistence compliance with either the DNS standard before EDNS, or the DNS standard after EDNS, is the requirement. There are two ways for a DNS server to relay the DNS responses larger than 512 bytes.... 1. The server replies with a truncated message with the truncate bit set, and the client connects to you over TCP to make the DNS request, OR The client provided the EDNS option with a larger packet size, and you support that, so you send a large UDP reply instead. A DNS server must support the first of these methods (The second is preferable but optional, and support for the first method over TCP is mandatory) if you could ever be required to handle a DNS message whose reply is larger than 512 bytes: All recursive resolvers fall into this category, and with DNSSEC + IPv6, many common queries will result in an answer longer than the original 512 byte limit of UDP. -- -JH
On 2019-01-31 08:32, James Stahr wrote:
I think the advertised testing tool may be flawed as blocking TCP/53 is enough to receive a STOP from the dnsflagday web site. It's been my (possibly flawed) understanding that TCP/53 is an option for clients but primarily it is a mechanism for the *server* to request the client communicate using TCP/53 instead - this could be due to UDP response size, anti-spoofing controls, etc...
That understanding is flawed, but understandable given how deeply ingrained this misinformation has become in the zeitgeist. Sections 4.2 and 4.2.2 of 1035 clearly state that TCP is an expected channel, not optional. This is relevant operationally for two reasons. First while most folks make an effort to keep answers under 512 bytes for response time reasons, you cannot guarantee that someone, somewhere in your org won't add a record that overflows. Also, you are guaranteed to overflow at some point once you roll out DNSSEC. I've even seen seemingly mundane things like SRV records overflow 512 bytes. The other reason it's relevant operationally is that there are more and more experimental projects in development now that involve using TCP, even without the need for truncation, as a way to have more assurance that a response is not spoofed. There are also efforts under way to evaluate whether "pipelining" DNS requests to servers that you are already sending a lot of requests to is ultimately more efficient than UDP at high volumes. So, like lack of EDNS compliance, lack of TCP "compliance" here is going to be a limiting factor for the growth of new features, at minimum; and could result in actual breakage. hope this helps, Doug
On 1 Feb 2019, at 3:32 am, James Stahr <stahr@mailbag.com> wrote:
On 2019-01-31 08:15, Mark Andrews wrote:
We actually have a hard time finding zones where all the servers are broken enough to not work with servers that don’t fallback to plain DNS on timeout. We can find zones where some of the servers are like that, but there is usually one or more servers that do respond correctly. Of the datasets I’ve looked at that 1 in 10,000 to 1 in 100,000 zones will have problems with updated servers.
I think the advertised testing tool may be flawed as blocking TCP/53 is enough to receive a STOP from the dnsflagday web site. It's been my (possibly flawed) understanding that TCP/53 is an option for clients but primarily it is a mechanism for the *server* to request the client communicate using TCP/53 instead - this could be due to UDP response size, anti-spoofing controls, etc…
DNS is defined to be both TCP and UDP. Blocking TCP has no real security benefit and comes from the MYTH that TCP is ONLY used for zone transfers. Way too many times people have blocked TCP then have those same servers return TC=1 which say USE TCP as the answer is TOO LARGE TO FIT IN UDP. Foot, Gun, Shoot. If you have a DNSSEC zone then TCP is effectively MANDATORY as clients still need to be able to get answers from behind a firewall that is enforcing a 512 byte limit. If you are running a recursive server TCP is effectively MANDATORY as you have no control over the RRset size. Lots of referrals REQUIRE TCP as the GLUE records are not OPTIONAL in a referral. Yes, BIND has been setting TC=1 on referrals for MANY years now when it is required, it should be setting it on many more. So if you want WORKING DNS you do not block TCP. Other DNS vendors also set TC=1 on some referrals. This means if you are hosting third party content then TCP is effectively MANDATORY as you have no content control. RFC 1123 said that DNS/TCP is a SHOULD where SHOULD is defined as * "SHOULD" This word or the adjective "RECOMMENDED" means that there may exist valid reasons in particular circumstances to ignore this item, but the full implications should be understood and the case carefully weighed before choosing a different course. I’ve yet to see anyone who has TCP blocked actually know all the places in the DNS protocol where doing so will cause things to break. None of them have done the necessary homework to actually exercise the SHOULD. There are lots of lemmings when it comes to DNS practices. All implementations of DNS are REQUIRED to support DNS/TCP. The DNS flag day site flags servers without TCP as broken which they *are*. Whether it should be red or orange is a matter for debate. A referral that in bigger than 512 bytes without involving DNSSEC. [beetle:~/git/bind9] marka% dig example.net @a.root-servers.net ; <<>> DiG 9.13.1+hotspot+add-prefetch+marka <<>> example.net @a.root-servers.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54567 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1472 ;; QUESTION SECTION: ;example.net. IN A ;; AUTHORITY SECTION: net. 172800 IN NS a.gtld-servers.net. net. 172800 IN NS b.gtld-servers.net. net. 172800 IN NS c.gtld-servers.net. net. 172800 IN NS d.gtld-servers.net. net. 172800 IN NS e.gtld-servers.net. net. 172800 IN NS f.gtld-servers.net. net. 172800 IN NS g.gtld-servers.net. net. 172800 IN NS h.gtld-servers.net. net. 172800 IN NS i.gtld-servers.net. net. 172800 IN NS j.gtld-servers.net. net. 172800 IN NS k.gtld-servers.net. net. 172800 IN NS l.gtld-servers.net. net. 172800 IN NS m.gtld-servers.net. ;; ADDITIONAL SECTION: a.gtld-servers.net. 172800 IN A 192.5.6.30 b.gtld-servers.net. 172800 IN A 192.33.14.30 c.gtld-servers.net. 172800 IN A 192.26.92.30 d.gtld-servers.net. 172800 IN A 192.31.80.30 e.gtld-servers.net. 172800 IN A 192.12.94.30 f.gtld-servers.net. 172800 IN A 192.35.51.30 g.gtld-servers.net. 172800 IN A 192.42.93.30 h.gtld-servers.net. 172800 IN A 192.54.112.30 i.gtld-servers.net. 172800 IN A 192.43.172.30 j.gtld-servers.net. 172800 IN A 192.48.79.30 k.gtld-servers.net. 172800 IN A 192.52.178.30 l.gtld-servers.net. 172800 IN A 192.41.162.30 m.gtld-servers.net. 172800 IN A 192.55.83.30 a.gtld-servers.net. 172800 IN AAAA 2001:503:a83e::2:30 b.gtld-servers.net. 172800 IN AAAA 2001:503:231d::2:30 c.gtld-servers.net. 172800 IN AAAA 2001:503:83eb::30 d.gtld-servers.net. 172800 IN AAAA 2001:500:856e::30 e.gtld-servers.net. 172800 IN AAAA 2001:502:1ca1::30 f.gtld-servers.net. 172800 IN AAAA 2001:503:d414::30 g.gtld-servers.net. 172800 IN AAAA 2001:503:eea3::30 h.gtld-servers.net. 172800 IN AAAA 2001:502:8cc::30 i.gtld-servers.net. 172800 IN AAAA 2001:503:39c1::30 j.gtld-servers.net. 172800 IN AAAA 2001:502:7094::30 k.gtld-servers.net. 172800 IN AAAA 2001:503:d2d::30 l.gtld-servers.net. 172800 IN AAAA 2001:500:d937::30 m.gtld-servers.net. 172800 IN AAAA 2001:501:b1f9::30 ;; Query time: 375 msec ;; SERVER: 198.41.0.4#53(198.41.0.4) ;; WHEN: Fri Feb 01 07:54:44 AEDT 2019 ;; MSG SIZE rcvd: 833 [beetle:~/git/bind9] marka%
So is the tool right in saying that TCP/53 is a absolute requirement of ENDS0 support for "DNS Flag Day"? If so, do we expect a dramatic increases in TCP/53 requests over UDP/53 queries? Or is the tool flawed in its' claim that the DNS Flag Day changes *require* TCP/53 in order to resolve ones domain? Based upon my reading, the big concern is a edns=timeout result (from the ISC tool) not a edns512tcp=timeout.
While I applaud the effort, the message and delivery could have been better. My first read on DNS Flag Day was that this was going to be the day new software versions were to be released which deprecated EDNS0 fallback measures so the adoption would be a gradual process by updating the latest versions of several DNS servers, only to find out that many resolvers used by end users use will be upgrading on Feb 1rst. Now, it sounds like the rollout schedule is on their timetable and could happen over the next several weeks and months. So really not a Flag "Day" now is it? I guess that's also why the date being on a Friday also isn't really a concern...
Finally, why has no one stepped up to setup an updated resolver prior to Feb 1rst for actual testing? Or are there instructions on how one could setup their own resolver setup prior to Feb 1rst? No offense, but I'm not jumping to a brand new train of code just to enforce DNS Flag Day but I might choose enforce now under *existing* code bases rather than waiting for everyone else using Cloudflare/Google/OpenDNS as resolver may see me after Feb 1rst? If anyone has such instructions, post them - or put them on dnsflagday.net for anyone else wanting to adopt/test. Just some thoughts…
From the DNS flag day site: DNS resolver operators On or around Feb 1st, 2019, major open source resolver vendors will release updates that will stop accommodating non-standard responses. This change will affect authoritative servers which do not comply either with the original DNS standard from 1987 (RFC1035) or the newer EDNS standards from 1999 (RFC2671 and RFC6891). Major public DNS resolver operators listed below are also removing accommodations so this change will also impact Internet users and providers who use these public DNS services. Sites hosted on incompatible authoritative servers may become unreachable through updated resolvers. The web form above diagnostic tool may be helpful while investigating problems with a particular domain. Domains which repeatedly fail the test above have problems with either their DNS software or their firewall configuration and cannot be fixed by DNS resolver operators. The following versions of DNS resolvers will not accommodate EDNS non-compliant responses: • BIND 9.13.3 (development) and 9.14.0 (production) • Knot Resolver has already implemented stricter EDNS handling in all current versions • PowerDNS Recursor 4.2.0 • Unbound 1.9.0 Now BIND 9.13.3 became public on 2018-09-19 and the Knot Resolver are already public. You can lookup PowerDNS and Unbound to see if they are public.
-- -James
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
The best time is usually a Wednesday at Noon or 11:00 in the impacted timezone. Of course, if the impact is worldwide then that would probably be UT1 :) --- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Christopher Morrow Sent: Wednesday, 30 January, 2019 18:55 To: Jim Popovitch Cc: nanog Subject: Re: DNS Flag Day, Friday, Feb 1st, 2019
On Wed, Jan 30, 2019 at 5:41 PM Jim Popovitch via NANOG <nanog@nanog.org> wrote:
On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort?
IIRC, DNS Flag Day was announce way before last years Super Bowl... what did the people who aren't ready for DNS Flag Day do in the past 364 days that they need a few more days to get ready for?
Oh, so they had 365 days to plan the time of the event and still picked a friday for that event?
https://www.opsview.com/resources/system-administrator/blog/three- reasons-why-not-make-major-it-changes-fridays
I see.
On Jan 30, 2019, at 17:40 , Jim Popovitch via NANOG <nanog@nanog.org> wrote:
On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort?
IIRC, DNS Flag Day was announce way before last years Super Bowl... what did the people who aren't ready for DNS Flag Day do in the past 364 days that they need a few more days to get ready for?
-Jim P.
Consider this… Sometimes you are responsible for fielding the calls and explaining problems that occur on systems that aren’t entirely within your control. A business class ISP, for example, would have a hard time proactively fixing all of their customer’s DNS resolvers and clients. Nonetheless, you can be assured that their call center will get the calls when the behavior of DNS changes in a way that negatively impacts some fraction of those clients. In my estimation, the most likely impact of this event will be on the enterprise, not the ISP or residential communities. The ISP community is either aware of and/or dealt with it in the normal course of business or they have had their head so deep in the sand that I don’t have much sympathy for what happens to them. The residential end user doesn’t run name servers for the most part, so, unlikely to be much impact there. The ones that do (such as myself) are likely technical enough and likely sufficiently involved in the ISP community to have heard about this issue and taken appropriate action. In my experience, enterprise IT, OTOH, is widely variable in its attentiveness to changes on the internet until after they have occurred. Network focused enterprises (e.g. Akamai, DropBox, etc.) are unlikely to be impacted. Other kinds of enterprises for whom the internet is more of a utility than a core function, OTOH, may have less awareness ahead of time (e.g. Chevron, GM, your local dive shop, the bodega on the corner, etc.). The larger enterprises probably have someone paying some attention. I suspect most of the casualties in this event will be in the Small to Medium business community. Owen
This basically affects sites using really old Windows DNS servers (Microsoft decided to make them only respond once with FORMERR so if that message is lost they appear to be dead until the timer clears) and those using firewalls that block EDNS queries. If you use such firewalls they are really doing nothing useful. Most of the other errors reported are benign as far as DNS flag day is concerned. Also apart from the public DNS resolvers people need to install updated software that has the work arounds removed. Mark -- Mark Andrews
On 31 Jan 2019, at 12:22, Matthew Petach <mpetach@netflight.com> wrote:
On Wed, Jan 23, 2019 at 4:12 PM Brian Kantor <Brian@ampr.org> wrote: Quoting from the web site at https://dnsflagday.net/ [...] The current DNS is unnecessarily slow and suffers from inability to deploy new features. To remediate these problems, vendors of DNS software and also big public DNS providers are going to remove certain workarounds on February 1st, 2019.
I would like to note that there is an entire segment of the population that does not interact with technology between sundown on Friday, all the way through Sunday morning.
Choosing Friday as a day to carry out an operational change of this sort does not seem to have given thought that if things break, there is a possibility they will have to stay broken for at least a full day before the right people can be engaged to work on the issue.
In the future, can we try to schedule such events with more consideration on which day the change will take place?
I will also note that this weekend is the Superbowl in the US; one of the bigger advertising events of the year. Potentially breaking advertising systems that rely on DNS two days before a major, once-a-year advertising event is *also* somewhat inconsiderate.
While I understand that no day will work for everyone, and at some point you just have to pick a day and go for it, I will note that picking the Friday before the Superbowl does seem like a very unfortunate random pick for a day on which to do it.
Any chance this could wait until say the Tuesday *after* the Superbowl, when we aren't cutting an entire religion's worth of potential workers out of the workforce available to fix issues in case it turns out to be a bigger problem than is expected, and when we have less chance of annoying the vast army of football-loving fans of every sort?
Thanks!
Matt
participants (17)
-
Bjørn Mork -
Brian Kantor -
Christopher Morrow -
Doug Barton -
Eric Brander -
James Stahr -
Jim Popovitch -
Jimmy Hess -
Keith Medcalf -
Mark Andrews -
Mark Tinka -
Matthew Petach -
Mike Meredith -
Niels Bakker -
Owen DeLong -
Radu-Adrian Feurdean -
Stephen Satchell