In message <CAP-guGWoytsYy=2taiQxcTB3doYS+T+A1nqfi0_asjXfip3c=w@mail.gmail.com> , William Herrin writes:

On Thu, Sep 15, 2016 at 12:22 PM, Aaron C. de Bruyn <aaron@heyaaron.com> wrot e:

...

On Thu, Sep 15, 2016 at 12:31 AM, Mark Andrews <marka@isc.org> wrote:

...

QWEST isn't the only DNS provider that has broken nameservers. One shouldn't have to try and contact every DNS operator to get them to use protocol compliant servers.

Save yourself some time. Contact the DNS software vendors. ;)

I'd bet he already has. This looks like a name-and-shame to me, and probably deserved.

-Bill

Aaron, How am I supposed to know which DNS vendor to contact? DNS server fingerprinting is not a exact science. After that I then still need to work out how to contact every operator of a broken server and get them to contact the DNS vendor to get a fix. And by the way the SOA RNAME is often a blackhole or it bounces or it is syntactically invalid. The best way to get this fixed would be for nameservers to be checked for protocol compliance, by the parent zone operators or their proxies regularly. That the child zone operator be given a short (< 3 months) to fix it then all zones with that server get removed from the parent zone until the server is fixed (apply the final step in the complaints proceedures from RFC 1033) which forces the owner of the zone to fix the server or to move to someone who follows the protocol. The servers for new delegations be checked immediately and the delegation not proceed unless the delegated servers are protocol compliant. Everybody seems to think they know how to write a DNS server. The problem is that most people don't test anything other than simple queries and that includes many of the DNS vendors. Think about all the load balancer vendors that don't handle anything but a A query or only handle A and AAAA queries don't handle DNSKEY queries. There really is no excuse to not handle non-meta qtypes properly (no error not data or name error depending upon whether the name exists or not). My bet is the DNS vendor has issued a update already and that it hasn't been applied. If not Qwest can inform them that their product is broken. Fixing this should be about 10 minutes for the DNS vendor then QA. If you (collectively) haven't already checked your servers go to https://ednscomp.isc.org and check your servers. While you are there look at some of the reports. If there are any tech reporters out there can you report on the issue of non compliance in DNS servers and that it can lead to lookups failing. This issue affects everybody. Mark

-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/> -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

In message <CAEE+rGp5BG=qyDfka2KYVfYmZdjC9EZBDD=OkZCNC7264Xt48A@mail.gmail.com> , "Aaron C. de Bruyn" writes:

On Thu, Sep 15, 2016 at 2:45 PM, Mark Andrews <marka@isc.org> wrote:

...

Aaron, How am I supposed to know which DNS vendor to contact? DNS

Sorry--I should have added a /sarcasm tag. :)

...

The best way to get this fixed would be for nameservers to be checked for protocol compliance, by the parent zone operators or their proxies regularly. That the child zone operator be given a short (< 3 months) to fix it then all zones with that server get removed from the parent zone until the server is fixed (apply the final step in the complaints proceedures from RFC 1033) which forces the owner of the zone to fix the server or to move to someone who follows the protocol. The servers for new delegations be checked immediately and the delegation not proceed unless the delegated servers are protocol compliant.

Seems a bit harsh, but I'm new to the conversation. What is being out of compliance actually hurting other than the nameserver operator and the zones they host?

So your helpdesks don't get problem reports when people can't look up domain names? Recursive DNS vendors don't get bug reports when domain names can't be looked up. We don't get fixes developed because there are too many broken servers out there. Because some servers don't answer EDNS requests this leads to false positives on servers not support EDNS when they do. This in turn leads to DNSSEC validation failures as you don't get DNSSEC answers without EDNS. IPv6 deployment was put back years because AAAA DNS lookups got wrong answers. DANE deployment is slow because DNS servers give bad answers to _<port>._tcp.<server-name>/TLSA. Then there is SPF. A fare portion of the reason why the SPF record failed, despite it being architectually cleaner than using TXT records, is that some nameservers gave bad responses to SPF queries. I could go find more examples of the cost of non DNS protocol compliance.

...

My bet is the DNS vendor has issued a update already and that it hasn't been applied. If not Qwest can inform them that their product is broken. Fixing this should be about 10 minutes for the DNS vendor then QA.

Yeah, but the business upgrade cycles are the killer. Why dedicate resources to fix it unless there's a pretty clear line-of-sight to lost profits? That's why so many of my clients refuse to upgrade away from XP. It still works for what they basically need, and it's not really impacting their profit in a way the CFO can directly see. (i.e. he doesn't see people like me who will walk out of a dental office and never come back when I see a 2-plus-year-out-of-date XP machine handling patient information.)

I'm sure the same is happening in a large bureaucracy like Qwest.

Maybe you're right with a harsher penalty. Be standards compliant or you'll get a warning, then be cut off.

...

If you (collectively) haven't already checked your servers go to https://ednscomp.isc.org and check your servers. While you are there look at some of the reports.

Tested. I'm compliant. I definitely think more comprehensive tools that are easily accessible to admins and CFOs would help.

For example, when I explain various zone-related things to CFOs, I'll use http://intodns.com/. It's sorta flashy, and contains some sorta helpful information that a CFO can sorta understand.

And a big red 'X' when someone is wrong.

Unfortunately it doesn't do DNSSEC. For that, there's another tool. ...and if you want EDNS testing, there's your tool.

A tool that tests compliance for everything and spits out errors, warnings, and recommendations might go a long ways towards getting people to solve the problem.

Just my $0.02.

Nice graphs by the way.

-A

--001a11394e2c845079053c9314bd Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

...

</div><div>For example, when I explain various zone-related things to CFOs= , I'll use <a href=3D"http://intodns.com/">http://intodns.com/</a>.=C2= =A0 It's sorta flashy, and contains some sorta helpful information that= a CFO can sorta understand.</div><div><br></div><div>And a big red 'X&= #39; when someone is wrong.</div><div><br></div><div>Unfortunately it doesn= 't do DNSSEC.=C2=A0 For that, there's another tool.</div><div>...an= d if you want EDNS testing, there's your tool.</div><div><br></div><div= A tool that tests compliance for everything and spits out errors, warnings= , and recommendations might go a long ways towards getting people to solve =

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On T= hu, Sep 15, 2016 at 2:45 PM, Mark Andrews <span dir=3D"ltr"><<a href=3D"= mailto:marka@isc.org" target=3D"_blank">marka@isc.org</a>></span> wrote:= <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">Aaron,<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0How am I supposed to know which DNS vendor to co= ntact?=C2=A0 DNS<br></blockquote><div><br></div><div>Sorry--I should have a= dded a /sarcasm tag. =C2=A0:)</div><div>=C2=A0</div><blockquote class=3D"gm= ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le= ft:1ex">The best way to get this fixed would be for nameservers to be check= ed<br> for protocol compliance, by the parent zone operators or their<br> proxies regularly.=C2=A0 That the child zone operator be given a short<br> (< 3 months) to fix it then all zones with that server get removed<br> from the parent zone until the server is fixed (apply the final<br> step in the complaints proceedures from RFC 1033) which forces the<br> owner of the zone to fix the server or to move to someone who follows<br> the protocol.=C2=A0 The servers for new delegations be checked immediately<= br> and the delegation not proceed unless the delegated servers are<br> protocol compliant.<br></blockquote><div><br></div><div>Seems a bit harsh, = but I'm new to the conversation.=C2=A0 What is being out of</div><div>c= ompliance actually hurting other than the nameserver operator and the</div>= <div>zones they host?</div><div><br></div><div>=C2=A0</div><blockquote clas= s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad= ding-left:1ex">My bet is the DNS vendor has issued a update already and tha= t it<br> hasn't been applied.=C2=A0 If not Qwest can inform them that their prod= uct<br> is broken.=C2=A0 Fixing this should be about 10 minutes for the DNS<br> vendor then QA.<br></blockquote><div><br></div><div>Yeah, but the business = upgrade cycles are the killer.</div><div>Why dedicate resources to fix it u= nless there's a pretty clear line-of-sight to lost profits?</div><div>T= hat's why so many of my clients refuse to upgrade away from XP.=C2=A0 I= t still works for what they basically need, and it's not really impacti= ng their profit in a way the CFO can directly see. =C2=A0(i.e. he doesn'= ;t see people like me who will walk out of a dental office and never come b= ack when I see a 2-plus-year-out-of-date XP machine handling patient inform= ation.)</div><div><br></div><div>I'm sure the same is happening in a la= rge bureaucracy like Qwest.</div><div><br></div><div>Maybe you're right= with a harsher penalty.=C2=A0 Be standards compliant or you'll get a w= arning, then be cut off.</div><div><br></div><div>=C2=A0</div><blockquote c= lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;= padding-left:1ex"> If you (collectively) haven't already checked your servers go to<br> <a href=3D"https://ednscomp.isc.org" rel=3D"noreferrer" target=3D"_blank">h= ttps://ednscomp.isc.org</a> and check your servers.=C2=A0 While you are<br> there look at some of the reports.<br></blockquote><div><br></div><div>Test= ed.=C2=A0 I'm compliant.=C2=A0 I definitely think more comprehensive to= ols that are easily accessible to admins and CFOs would help.</div><div><br= the problem.</div><div><br></div><div>Just my $0.02.</div><div><br></div><d= iv>Nice graphs by the way.</div><div><br></div><div>-A</div></div></div></d= iv>

--001a11394e2c845079053c9314bd-- -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

In message <9442FCB1-E039-4EDD-8A0F-F5F351BC99B4@truenet.com>, Eric Tykwinski w rites:

Ironically, I always wondered why I was told not to publish SPF records, since it did make more sense to have both, and slowly remove the TXT records later. Thanks for the heads up…

What do you think really is best practice now?

For SPF the decision was made to stay with TXT. The IAB wrote RFC 5507 - Design Choices When Expanding the DNS. As for testing there is: https://tools.ietf.org/html/draft-ietf-dnsop-no-response-issue-04 There is general consensus that the tests are correct to the level that they cover. There can always be more tests added. There is less consensus on how we get from where we are now to where we need to be. The EDNS tests tool was the starting point for this draft. Mark

Sincerely,

Eric Tykwinski TrueNet, Inc. P: 610-429-8300

...

On Sep 15, 2016, at 7:30 PM, Mark Andrews <marka@isc.org> wrote:

So your helpdesks don't get problem reports when people can't look up domain names? Recursive DNS vendors don't get bug reports when domain names can't be looked up. We don't get fixes developed because there are too many broken servers out there.

Because some servers don't answer EDNS requests this leads to false positives on servers not support EDNS when they do. This in turn leads to DNSSEC validation failures as you don't get DNSSEC answers without EDNS.

IPv6 deployment was put back years because AAAA DNS lookups got wrong answers.

DANE deployment is slow because DNS servers give bad answers to _<port>._tcp.<server-name>/TLSA.

Then there is SPF. A fare portion of the reason why the SPF record failed, despite it being architectually cleaner than using TXT records, is that some nameservers gave bad responses to SPF queries.

I could go find more examples of the cost of non DNS protocol compliance. -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

In message <CAP-guGWXqBuUUak=HgqckFqS2Pgxa0e30RQ_G=qQNQ_C-hagbA@mail.gmail.com> , William Herrin writes:

On Thu, Sep 15, 2016 at 7:30 PM, Mark Andrews <marka@isc.org> wrote:

...

Then there is SPF. A fare portion of the reason why the SPF record failed, despite it being architectually cleaner than using TXT records, is that some nameservers gave bad responses to SPF queries.

Hi Mark,

I'm going to stop you there. The SPF record type failed because resolvers can't pass requests between clients and authoritative servers unless they can parse them. New DNS record types essentially require a universal software upgrade before they work. And universal software upgrades do not happen well or in anything approaching a timely manner. The next new DNS record type will fail for the same reason. And the one after that.

Again lack of DNS compliance. Go read STD 13 then tell me that Microsoft ships a standards compliant resolver. They still don't last time I checked. Libresolv could look up any <qname,qtype,class> tuple from back when the UCB developed it. You *never* needed universal support for new types. It is a myth. You just need the authoritative servers to support the type and the client to support the type. Everything else treated it as a opaque blob. That is why compression pointers were only allowed for well known types. That is why records have a length field. What STD 13 missed was a presentation format for unknown types. There were implementations that got that wrong, including named, but we fixed that well before SPF ever became a issue. We have RFC 3597 which allows authoritative servers to load and save records they don't know the internal structure of. This was published in September 2003. All the major DNS vendors support it. This addressed the oversight in STD 13. You have lazy operators that haven't designed their web tools to support RFC 3597. Mark

TXT is the DNS record type that's extensible without a software upgrade. Like it or lump it.

Regards, Bill Herrin

-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/> -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

On Thu, 15 Sep 2016 09:22:10 -0700, "Aaron C. de Bruyn" said:

On Thu, Sep 15, 2016 at 12:31 AM, Mark Andrews <marka@isc.org> wrote:

...

QWEST isn't the only DNS provider that has broken nameservers. One shouldn't have to try and contact every DNS operator to get them to use protocol compliant servers.

Save yourself some time. Contact the DNS software vendors. ;)

Often, the vendor *is* on the ball, but the customer isn't. Remember that Windows XP didn't enable IPv6 by default, and *still* has some 10% market share.