Thanks for the update that dnssec STILL causes more real world problems than it solves. ..... That said, arin is a pro outfit. If they can screw it up, like nasa, so can you. No your threats and deploy wisely ---------- Forwarded message --------- From: John Curran <jcurran@istaff.org> Date: Fri, Jan 11, 2019 at 6:36 AM Subject: Re: ARIN NS down? To: Suresh Ramasubramanian <ops.lists@gmail.com> CC: NANOG <nanog@nanog.org> Suresh - We’re aware and working the problem. It looks to me like expired RRSIG/DNSKEY’s for the zone, so if you’re using a DNSSEC validating resolver (e.g. Google, Cloudflare, Cogent) then ARIN.NET <http://arin.net/> is unreachable. ARIN’s engineering team is working on resolution now. /John John Curran President and CEO American Registry for Internet Numbers On 11 Jan 2019, at 9:27 AM, Suresh Ramasubramanian <ops.lists@gmail.com> wrote: couldn't get address for 'ns1.arin.net': not found couldn't get address for 'ns2.arin.net': not found couldn't get address for 'u.arin.net': not found couldn't get address for 'ns3.arin.net': not found dig: couldn't get address for 'ns1.arin.net': no more srs@Sureshs-MacBook-Pro-2 19:56:18 <~> $ dig +trace +norec whois.arin.net ; <<>> DiG 9.10.6 <<>> +trace +norec whois.arin.net ;; global options: +cmd . 2230 IN NS m.root-servers.net. . 2230 IN NS b.root-servers.net. . 2230 IN NS c.root-servers.net. . 2230 IN NS d.root-servers.net. . 2230 IN NS e.root-servers.net. . 2230 IN NS f.root-servers.net. . 2230 IN NS g.root-servers.net. . 2230 IN NS h.root-servers.net. . 2230 IN NS i.root-servers.net. . 2230 IN NS j.root-servers.net. . 2230 IN NS a.root-servers.net. . 2230 IN NS k.root-servers.net. . 2230 IN NS l.root-servers.net. . 2230 IN RRSIG NS 8 0 518400 20190121050000 20190108040000 16749 . JqXTRb0qik0Iy1zDpwKRfKr1iZjTeiJRTk1GCfIWh9dFFvhN0c7Fiz6H lbNfhgQbPsacG0b/1I3rguS13H2guX7apppK2w88h+z8mzym2Bw1C1HR ZR3ocj/jHLJbMqHdQ+DFyRdw/AxCXBdhnbX46C8+unhQ03D/MzS0M0t4 vgadYi7BN4sa+iZIilwFV56n2dOfpzyO+evVbcnTLRZ6D4bjCHZLCtO8 EDziAPUbVAPZWiflb7/Y2dECe5gbOuGYYU/xv/Pal5+v9cjgMjcf8buG S+iTIL/lnus0JJSRDmkM6yzfYMBXC2ZqhOp+Ls+EfvmqFjIZzi394XCi pdKRZw== ;; Received 525 bytes from 10.0.0.1#53(10.0.0.1) in 40 ms net. 172800 IN NS g.gtld-servers.net. net. 172800 IN NS c.gtld-servers.net. net. 172800 IN NS j.gtld-servers.net. net. 172800 IN NS e.gtld-servers.net. net. 172800 IN NS h.gtld-servers.net. net. 172800 IN NS k.gtld-servers.net. net. 172800 IN NS m.gtld-servers.net. net. 172800 IN NS i.gtld-servers.net. net. 172800 IN NS f.gtld-servers.net. net. 172800 IN NS b.gtld-servers.net. net. 172800 IN NS a.gtld-servers.net. net. 172800 IN NS d.gtld-servers.net. net. 172800 IN NS l.gtld-servers.net. net. 86400 IN DS 35886 8 2 7862B27F5F516EBE19680444D4CE5E762981931842C465F00236401D 8BD973EE net. 86400 IN RRSIG DS 8 1 86400 20190124130000 20190111120000 16749 . uahpltN27UkKaFJRaAU1on+IpC2lpgZo84XEM7Pk7dQysKfSnqUkaVLY PXQf9kvgW5eOx/+BttQB2OWFLckJs8vv5ScOpz7dDhs8zR2FPLm93HTD 4F/XEKDNOQbFGSA3g4pZq3fatY7kFEkV9sFTH90WqJt0sXe64LYFcwr2 FtrJaS/yhEV4XDbsN3RLkBP58bf526LPpvonwSZsMUTDZcnXtUnc57ZI dlTHg2snNhVWu4qJfHDsEQPwOZagRXJhjlRT8Ox/7HwXvplmRfmeuhZb Vj5kdiY+3j0RTxpLRCG/SZRDIRcvdFKh9umdwQvAzuTS0xzO8OyPw9q8 8QCCYg== ;; Received 1171 bytes from 192.112.36.4#53(g.root-servers.net) in 207 ms arin.net. 172800 IN NS ns1.arin.net. arin.net. 172800 IN NS ns2.arin.net. arin.net. 172800 IN NS u.arin.net. arin.net. 172800 IN NS ns3.arin.net. arin.net. 86400 IN DS 48281 5 2 6EB0CCF325A8101A768C93D10CE084303D3714D4E92FEE53D6E683D2 22291017 arin.net. 86400 IN DS 48281 5 1 FCBF93357C8FE3247CECB2CD277F45EB955EE4CE arin.net. 86400 IN RRSIG DS 8 2 86400 20190117062448 20190110051448 6140 net. stuWyfC0PDuk2hNF/Bnz0lnypk+bA/slTa2KYznjmoLXDtq7v1obJq41 ZfloQKXuC7MnzpCQj70GU9ZESZq1/XU+u6wDmCqmEUbJ3kyrILxkVrln bTEySJWPmurpwUVzDVfvqFpXEOhWxOjDu6drZMcC3wG9EdPqBuFC6wlf FIQ= couldn't get address for 'ns1.arin.net': not found couldn't get address for 'ns2.arin.net': not found couldn't get address for 'u.arin.net': not found couldn't get address for 'ns3.arin.net': not found dig: couldn't get address for 'ns1.arin.net': no more

On Fri, Jan 11, 2019 at 8:10 AM Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:

On Fri, Jan 11, 2019 at 07:58:25AM -0800, Ca By <cb.list6@gmail.com> wrote a message of 488 lines which said:

...

No your threats and deploy wisely

Say no to the threats :-)

This is nanog, so i used the cisco no Its like , negate threats :)

It's because you see problems it causes, and do not see problems it solves ;)

...

Thanks for the update that dnssec STILL causes more real world problems than it solves. 

hmmm. has anyone set about to measure that? randy

On Fri, 11 Jan 2019, Ca By wrote:

Thanks for the update that dnssec STILL causes more real world problems than it solves.

Do you feel the same way about RPKI? -- Mikael Abrahamsson email: swmike@swm.pp.se

On 11 Jan 2019, at 10:39 AM, John Curran <jcurran@arin.net<mailto:jcurran@arin.net>> wrote: On Fri, Jan 11, 2019 at 07:57:25PM +0530, couldn't get address for 'ns1.arin.net<http://ns1.arin.net/>': not found Folks - This has been resolved - arin.net<http://arin.net/> zone is again correctly signed. Post-mortem forthcoming, Folks - The ARIN.NET<http://ARIN.NET> zone on our public signed DNS servers are populated via an internal DNS server and associated workflow. As part of system maintenance near the end of 2018, the zone file used by the master internal DNS server was updated incorrectly, resulting in an invalid zone file. Since the zone file was invalid, the zone did not reload on our internal master, and the associated workflow to DNSSEC sign and push this zone to the public servers did not execute. Our monitoring systems reported being green until the signatures expired as they presently check that the SOA's match on the internal and external nameservers. At approximately 8:30AM eastern time today (11 January 2019), ARIN operations started seeing issues within its monitoring. Initial review suggested the problem was DNSSEC-related due to expired signatures. We pulled the DS record from the zone so that DNSSEC validation would not be performed by those validating resolvers that had not already cached our DS records. Upon further investigation we determined that it was the result of human error in editing a zone file that went undetected and resulted in interruption of our routine zone publication process. The issue was fixed and signed zones where then pushed out at 10:25 AM ET. The DS record was reinstated in the parent at 10:30AM ET. As a result of this incident, we will add additional alerting to the zone loading process for any errors and perform monitoring of zone signature lifetimes, with appropriate alerting for any potential expiration of DNSSEC signatures. My apologies for this incident – while ARIN does have some fragility in our older systems (which we have been working aggressively to phase out via system refresh and replacements), it is not acceptable to have this situation with key infrastructure such as our DNS zones. We will prioritize the necessary alert and monitor changes and I will report back to the community once that has been completed. Thank you for your patience in this regard. /John John Curran President and CEO American Registry for Internet Numbers

On 11 Jan 2019, at 3:59 PM, John Curran <jcurran@arin.net<mailto:jcurran@arin.net>> wrote: ... My apologies for this incident – while ARIN does have some fragility in our older systems (which we have been working aggressively to phase out via system refresh and replacements), it is not acceptable to have this situation with key infrastructure such as our DNS zones. We will prioritize the necessary alert and monitor changes and I will report back to the community once that has been completed. Folks - I indicated that we would report back once appropriate DNSSEC monitoring is in place - this has now been completed (ref: attached announcement of same) Thanks again for your patience in this matter, /John John Curran President and CEO American Registry for Internet Numbers Begin forwarded message: From: ARIN <info@arin.net<mailto:info@arin.net>> Subject: [arin-announce] DNSSEC Monitoring Enhancements Date: 4 February 2019 at 11:32:25 AM EST To: <arin-announce@arin.net<mailto:arin-announce@arin.net>> On 31 January, ARIN deployed DNSSEC monitoring enhancements, including proactive RRSIG expiration checking, zone syntax checking, and DNSSEC validation. We are monitoring from various disparate locations across the Internet with these checks. This effort was undertaken in response to the incident that occurred on 11 January, detailed in the incident report below. Improved monitoring of DNSSEC and the arin.net<http://arin.net> zone will provide earlier alerts of any issues such as Resource Record Signature (RRSIG) expiration and any issues with DNSSEC validation. These enhancements will provide early warning of potential issues, prevent outages, and improve our ability to troubleshoot DNSSEC problems if they occur in the future. Regards, Mark Kosters Chief Technology Officer American Registry for Internet Numbers (ARIN) Incident Report: On 11 January 2019, at approximately 8:30 a.m. ET, ARIN monitoring systems alerted that some arin.net<http://arin.net> properties were unreachable. All users with validating DNS resolvers were unable to look up resources within arin.net<http://arin.net> and thus were unable to reach them. ARIN’s www.arin.net<http://www.arin.net> and ftp.arin.net<http://ftp.arin.net> sites and Whois, RPKI, and DNS services were affected for those users who use validating resolvers. ARIN’s Engineering staff determined that DNSSEC validation for the arin.net<http://arin.net> zone was failing and temporarily unpublished Delegation Signer (DS) records with our registrar so that we could investigate the problem. Upon troubleshooting, ARIN staff discovered that the removal of a resource record had created a spurious record, which caused a script to fail to reload. New versions of the zone could not be loaded, and the zone file in use expired. After determining the cause of the problem, the offending file was removed and the zone was reloaded. Delegation Signer (DS) records were republished and the zone validated, restoring service at approximately 10:30 a.m. ET. _______________________________________________ ARIN-Announce