ARIN contact needed: something bad happens with legacy IPv4 block's reverse delegations
We (at Kurchatov Insitute) still use 144.206.0.0/16, the legacy block, and seeing the breakage rooted at ARIN since this night, {{{ $ dig +trace -t soa 206.144.in-addr.arpa ; <<>> DiG 9.10.4-P6 <<>> +trace -t soa 206.144.in-addr.arpa ;; global options: +cmd . 206351 IN NS a.root-servers.net. . 206351 IN NS c.root-servers.net. . 206351 IN NS d.root-servers.net. . 206351 IN NS e.root-servers.net. . 206351 IN NS l.root-servers.net. . 206351 IN NS f.root-servers.net. . 206351 IN NS g.root-servers.net. . 206351 IN NS h.root-servers.net. . 206351 IN NS j.root-servers.net. . 206351 IN NS m.root-servers.net. . 206351 IN NS k.root-servers.net. . 206351 IN NS i.root-servers.net. . 206351 IN NS b.root-servers.net. . 514733 IN RRSIG NS 8 0 518400 20170329170000 20170316160000 61045 . OjsjoA6WCcThV3BqhAyMaXI3bU1m228Gl8nvaR074qBtem/RjaFJh1Oe r7LPI6W15jgbRGuCY7/GUNgDex4ZM43yvx2iY+2GpSk9b2/pKGbDaDIp X1Hd8418206eow1P/SgPqtT+2LzjM+lz/HKUvCyPoN4uX5/7GDExn8ir 2Q/vw81Za2nJ4Ji6oeGAmE8g6V3RJfTI0El/CP9Vl2/r0jWDZZ+wtYRA uwqvQYj+7VZc5+UJDJ3/Gne1LBnzXKnZRwnfJtZenhfZ7X20D9PiXvfJ M2s1ryqZvg8K5RqYt/r8pIiq1Udd1KWQBgdN7Q629+jNSngZQtH19R9H fM7h1w== ;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms in-addr.arpa. 172800 IN NS e.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS f.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS d.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS c.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS b.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS a.in-addr-servers.arpa. in-addr.arpa. 86400 IN DS 53696 8 2 13E5501C56B20394DA921B51412D48B7089C5EB6957A7C58553C4D4D 424F04DF in-addr.arpa. 86400 IN DS 63982 8 2 AAF4FB5D213EF25AE44679032EBE3514C487D7ABD99D7F5FEC3383D0 30733C73 in-addr.arpa. 86400 IN DS 47054 8 2 5CAFCCEC201D1933B4C9F6A9C8F51E51F3B39979058AC21B8DF1B1F2 81CBC6F2 in-addr.arpa. 86400 IN RRSIG DS 8 2 86400 20170330000000 20170316230000 33786 arpa. gqNN6MmLVxFtSG0oxdyoYVSXZlp6vY9yxpnJW89TYCHfTkv+ZmklM76O IHdTznEhLHXbyr4BaxjFCshsC3WacdH/YZYa/SZynJ9Q1N6/bogrIUTn qs61Y/YD4Sk5lI7YvUxVnPrF3lk1pY8dpoTkprTLZngoeQEsm552inqG ypU= ;; Received 733 bytes from 198.41.0.4#53(a.root-servers.net) in 64 ms 144.in-addr.arpa. 86400 IN NS r.arin.net. 144.in-addr.arpa. 86400 IN NS u.arin.net. 144.in-addr.arpa. 86400 IN NS x.arin.net. 144.in-addr.arpa. 86400 IN NS y.arin.net. 144.in-addr.arpa. 86400 IN NS z.arin.net. 144.in-addr.arpa. 86400 IN NS arin.authdns.ripe.net. 144.in-addr.arpa. 86400 IN DS 62856 5 1 484154736CF9D4DC4F1C2B807BDC06E5B1645AFF 144.in-addr.arpa. 86400 IN RRSIG DS 8 3 86400 20170328085556 20170307105911 4341 in-addr.arpa. X3nPKgjQbRef6BxK3msEXi/IcfpG1rylY2xeWwfNO6fJB5x/QT38ZCA5 XMs6CBeXb59tpQFbgjwZX+prqSpjGFtZ+phFuzsaGmuPOF0FFypMHj7F swykEFvaPY5z4d8mo9FKFJetF2gZfzYthwJWW0x2oo2H2BG0lEedVUCb 1aGs/HE= ;; Received 380 bytes from 193.0.9.1#53(f.in-addr-servers.arpa) in 48 ms 144.in-addr.arpa. 0 IN SOA z.arin.net. dns-ops.arin.net. 2017022432 1800 900 691200 10800 144.in-addr.arpa. 0 IN RRSIG SOA 5 3 86400 20170331083220 20170317073220 33345 144.in-addr.arpa. k2sZXuQQR3MyyRiB9Wd7fw45yZTbNokjQtFFNY+aCEa/85AnSyuomLlZ jZs4A0bO376/Z1v+bTHoeFqsyHr/xuWHX74r0QC/TCeP0amc+w8RqCvw Yox1TfoJxwhblGNRCgyLw52WszMYyr49EfWPfpHAKFU+G94gdilf3C6x GOM= 144.in-addr.arpa. 10800 IN NSEC 0.144.in-addr.arpa. NS SOA RRSIG NSEC DNSKEY 144.in-addr.arpa. 10800 IN RRSIG NSEC 5 3 10800 20170331083220 20170317073220 33345 144.in-addr.arpa. opvG8SC7+UjHSTtBBtekWNkhLFMIxFxOdejJ7sM7t8lyGNzM6sOABeSI ADXfo8q3p4YFBHgBU5fRCAhBdiQ/AiZC0dLMnHb4WdKEv5bDsBbq5Pt6 vabJaIv7Gizw7kBy1JZ/ZrC4Jmp4EX0HiYA+Ak+aQggD7gdI/6tFDNpl N7M= 205.144.in-addr.arpa. 10800 IN NSEC 207.144.in-addr.arpa. NS RRSIG NSEC 205.144.in-addr.arpa. 10800 IN RRSIG NSEC 5 4 10800 20170331083220 20170317073220 33345 144.in-addr.arpa. F1e5mA7C3Mx10YSaNtshzfzfER8pudDOQkUoe+jeDhHeDZeR8FM1FoqW dqrNh5X6JVNlTdWGio6evnSkxnaoJFYyeYJtc+tJ8dTd5APJxJCSOfhH VfRNy7VHs2PdElRo1rRwMw+5Zncc0u8uPdmFDv2ZG8Vv3zj5C6l7rMla 3SA= ;; Received 718 bytes from 199.212.0.63#53(z.arin.net) in 128 ms }}} Seems like the other /16 from 144.in-addr.arpa are affected too (at least). ARIN tickets are engaged, but I am seeking some live person: having the whole reverse zone being unavailable isn't the stuff that is to be handled in 5x8 or alike. If anybody knows what happened, can point me to my errors in assertions or give an e-mail/phone of 24x7-like person or service at ARIN -- will be tremendous. Thanks! -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
On Fri, Mar 17, 2017 at 12:03:58PM +0300, Eygene Ryabinkin <rea+nanog@grid.kiae.ru> wrote a message of 71 lines which said:
Seems like the other /16 from 144.in-addr.arpa are affected too (at least).
Also in 164.in-addr.arpa, it seems?
171 also seems affected. Job On Fri, 17 Mar 2017 at 10:54, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:
On Fri, Mar 17, 2017 at 12:03:58PM +0300, Eygene Ryabinkin <rea+nanog@grid.kiae.ru> wrote a message of 71 lines which said:
Seems like the other /16 from 144.in-addr.arpa are affected too (at least).
Also in 164.in-addr.arpa, it seems?
Job, good day. Fri, Mar 17, 2017 at 09:55:56AM +0000, Job Snijders wrote:
171 also seems affected.
Not the whole one, it seems: {{{ $ dig +trace -t soa 1.171.in-addr.arpa ; <<>> DiG 9.10.4-P6 <<>> +trace -t soa 1.171.in-addr.arpa ;; global options: +cmd . 202634 IN NS m.root-servers.net. . 202634 IN NS i.root-servers.net. . 202634 IN NS k.root-servers.net. . 202634 IN NS c.root-servers.net. . 202634 IN NS a.root-servers.net. . 202634 IN NS d.root-servers.net. . 202634 IN NS l.root-servers.net. . 202634 IN NS e.root-servers.net. . 202634 IN NS g.root-servers.net. . 202634 IN NS b.root-servers.net. . 202634 IN NS j.root-servers.net. . 202634 IN NS h.root-servers.net. . 202634 IN NS f.root-servers.net. . 511016 IN RRSIG NS 8 0 518400 20170329170000 20170316160000 61045 . OjsjoA6WCcThV3BqhAyMaXI3bU1m228Gl8nvaR074qBtem/RjaFJh1Oe r7LPI6W15jgbRGuCY7/GUNgDex4ZM43yvx2iY+2GpSk9b2/pKGbDaDIp X1Hd8418206eow1P/SgPqtT+2LzjM+lz/HKUvCyPoN4uX5/7GDExn8ir 2Q/vw81Za2nJ4Ji6oeGAmE8g6V3RJfTI0El/CP9Vl2/r0jWDZZ+wtYRA uwqvQYj+7VZc5+UJDJ3/Gne1LBnzXKnZRwnfJtZenhfZ7X20D9PiXvfJ M2s1ryqZvg8K5RqYt/r8pIiq1Udd1KWQBgdN7Q629+jNSngZQtH19R9H fM7h1w== ;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms in-addr.arpa. 172800 IN NS a.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS b.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS c.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS d.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS e.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS f.in-addr-servers.arpa. in-addr.arpa. 86400 IN DS 47054 8 2 5CAFCCEC201D1933B4C9F6A9C8F51E51F3B39979058AC21B8DF1B1F2 81CBC6F2 in-addr.arpa. 86400 IN DS 53696 8 2 13E5501C56B20394DA921B51412D48B7089C5EB6957A7C58553C4D4D 424F04DF in-addr.arpa. 86400 IN DS 63982 8 2 AAF4FB5D213EF25AE44679032EBE3514C487D7ABD99D7F5FEC3383D0 30733C73 in-addr.arpa. 86400 IN RRSIG DS 8 2 86400 20170330000000 20170316230000 33786 arpa. gqNN6MmLVxFtSG0oxdyoYVSXZlp6vY9yxpnJW89TYCHfTkv+ZmklM76O IHdTznEhLHXbyr4BaxjFCshsC3WacdH/YZYa/SZynJ9Q1N6/bogrIUTn qs61Y/YD4Sk5lI7YvUxVnPrF3lk1pY8dpoTkprTLZngoeQEsm552inqG ypU= ;; Received 731 bytes from 193.0.14.129#53(k.root-servers.net) in 41 ms 171.in-addr.arpa. 86400 IN NS ns1.apnic.net. 171.in-addr.arpa. 86400 IN NS ns2.lacnic.net. 171.in-addr.arpa. 86400 IN NS ns3.apnic.net. 171.in-addr.arpa. 86400 IN NS ns4.apnic.net. 171.in-addr.arpa. 86400 IN NS apnic.authdns.ripe.net. 171.in-addr.arpa. 86400 IN NS apnic1.dnsnode.net. 171.in-addr.arpa. 86400 IN NS tinnie.arin.net. 171.in-addr.arpa. 86400 IN DS 49699 5 1 0240801C96480900CC6D93130AF45CFE86EDE940 171.in-addr.arpa. 86400 IN DS 49699 5 2 6E0BA96F05B4D39C1668979731E040213BB6130DE33E86E063B5F7F7 5C465C88 171.in-addr.arpa. 86400 IN RRSIG DS 8 3 86400 20170330042932 20170309125911 4341 in-addr.arpa. RKcPZJdG1MBrwpfa1mIK7ikIU585i1Jv+UkEBHxuM3BxxAbD+ht0W04C ljboyXJfYK8ly/YTDqNUkWKZLwDHlkq/rgNwTG63sPT8iK8qya+2qUVB iTXVsD2HaV1V/KwgJjlWHRNnlwkK0YZ5Q7tevXaq4yyLT2Q2mu5dkhFC evNQOyI= ;; Received 482 bytes from 199.253.183.183#53(b.in-addr-servers.arpa) in 69 ms 171.in-addr.arpa. 0 IN SOA ns1.apnic.net. read-txt-record-of-zone-first-dns-admin.apnic.net. 5276 7200 1800 604800 172800 171.in-addr.arpa. 0 IN RRSIG SOA 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. dH+uuGq9pgMc9TEMkTwrf9NGMj5zFnbZt3kjlNSF0kPzGk3WUr5g84jv 0SN88Y26eP/Op4Vv8JewyVu2IT1POnmiNpDYELkJxrTWnPPc6CJ2wugR TrFLOq6eIsqtrCcWbatQ7XSXXj1UglqVVydlX1ExoIh9MdP+1eBneD1I 4OU= 171.in-addr.arpa. 172800 IN NSEC 10.171.in-addr.arpa. NS SOA TXT RRSIG NSEC DNSKEY 171.in-addr.arpa. 172800 IN RRSIG NSEC 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. QieHZ0W7jx3QCgLuvqtKtFoLPYkxn5R9nRO1oMbSSXkdF+S4iQQkVlX7 DyQvoL69hKA+S9idfernf6DwNTQgeWFfCdjUyOjMCa3Ag/S8ARZs7K4F N9DxB/cBCZm7KTzwAzXFe3cJ/d1Wo45Tx9jf6Drx551oqOw1gmDIL4y6 +k4= ;; Received 530 bytes from 202.12.31.140#53(ns4.apnic.net) in 347 ms }}} since it is APNIC who respond to queries about it. Any more specific reverse records that are cross ARIN and die there from 171.x.y.z? -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
Eygene - We are aware there’s an issue and working on it presently with RIPE. Expect additional updates shortly. /John John Curran President and CEO ARIN
On 17 Mar 2017, at 11:04 AM, Eygene Ryabinkin <rea+nanog@grid.kiae.ru> wrote:
Job, good day.
Fri, Mar 17, 2017 at 09:55:56AM +0000, Job Snijders wrote:
171 also seems affected.
Not the whole one, it seems: {{{ $ dig +trace -t soa 1.171.in-addr.arpa
; <<>> DiG 9.10.4-P6 <<>> +trace -t soa 1.171.in-addr.arpa ;; global options: +cmd . 202634 IN NS m.root-servers.net. . 202634 IN NS i.root-servers.net. . 202634 IN NS k.root-servers.net. . 202634 IN NS c.root-servers.net. . 202634 IN NS a.root-servers.net. . 202634 IN NS d.root-servers.net. . 202634 IN NS l.root-servers.net. . 202634 IN NS e.root-servers.net. . 202634 IN NS g.root-servers.net. . 202634 IN NS b.root-servers.net. . 202634 IN NS j.root-servers.net. . 202634 IN NS h.root-servers.net. . 202634 IN NS f.root-servers.net. . 511016 IN RRSIG NS 8 0 518400 20170329170000 20170316160000 61045 . OjsjoA6WCcThV3BqhAyMaXI3bU1m228Gl8nvaR074qBtem/RjaFJh1Oe r7LPI6W15jgbRGuCY7/GUNgDex4ZM43yvx2iY+2GpSk9b2/pKGbDaDIp X1Hd8418206eow1P/SgPqtT+2LzjM+lz/HKUvCyPoN4uX5/7GDExn8ir 2Q/vw81Za2nJ4Ji6oeGAmE8g6V3RJfTI0El/CP9Vl2/r0jWDZZ+wtYRA uwqvQYj+7VZc5+UJDJ3/Gne1LBnzXKnZRwnfJtZenhfZ7X20D9PiXvfJ M2s1ryqZvg8K5RqYt/r8pIiq1Udd1KWQBgdN7Q629+jNSngZQtH19R9H fM7h1w== ;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms
in-addr.arpa. 172800 IN NS a.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS b.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS c.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS d.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS e.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS f.in-addr-servers.arpa. in-addr.arpa. 86400 IN DS 47054 8 2 5CAFCCEC201D1933B4C9F6A9C8F51E51F3B39979058AC21B8DF1B1F2 81CBC6F2 in-addr.arpa. 86400 IN DS 53696 8 2 13E5501C56B20394DA921B51412D48B7089C5EB6957A7C58553C4D4D 424F04DF in-addr.arpa. 86400 IN DS 63982 8 2 AAF4FB5D213EF25AE44679032EBE3514C487D7ABD99D7F5FEC3383D0 30733C73 in-addr.arpa. 86400 IN RRSIG DS 8 2 86400 20170330000000 20170316230000 33786 arpa. gqNN6MmLVxFtSG0oxdyoYVSXZlp6vY9yxpnJW89TYCHfTkv+ZmklM76O IHdTznEhLHXbyr4BaxjFCshsC3WacdH/YZYa/SZynJ9Q1N6/bogrIUTn qs61Y/YD4Sk5lI7YvUxVnPrF3lk1pY8dpoTkprTLZngoeQEsm552inqG ypU= ;; Received 731 bytes from 193.0.14.129#53(k.root-servers.net) in 41 ms
171.in-addr.arpa. 86400 IN NS ns1.apnic.net. 171.in-addr.arpa. 86400 IN NS ns2.lacnic.net. 171.in-addr.arpa. 86400 IN NS ns3.apnic.net. 171.in-addr.arpa. 86400 IN NS ns4.apnic.net. 171.in-addr.arpa. 86400 IN NS apnic.authdns.ripe.net. 171.in-addr.arpa. 86400 IN NS apnic1.dnsnode.net. 171.in-addr.arpa. 86400 IN NS tinnie.arin.net. 171.in-addr.arpa. 86400 IN DS 49699 5 1 0240801C96480900CC6D93130AF45CFE86EDE940 171.in-addr.arpa. 86400 IN DS 49699 5 2 6E0BA96F05B4D39C1668979731E040213BB6130DE33E86E063B5F7F7 5C465C88 171.in-addr.arpa. 86400 IN RRSIG DS 8 3 86400 20170330042932 20170309125911 4341 in-addr.arpa. RKcPZJdG1MBrwpfa1mIK7ikIU585i1Jv+UkEBHxuM3BxxAbD+ht0W04C ljboyXJfYK8ly/YTDqNUkWKZLwDHlkq/rgNwTG63sPT8iK8qya+2qUVB iTXVsD2HaV1V/KwgJjlWHRNnlwkK0YZ5Q7tevXaq4yyLT2Q2mu5dkhFC evNQOyI= ;; Received 482 bytes from 199.253.183.183#53(b.in-addr-servers.arpa) in 69 ms
171.in-addr.arpa. 0 IN SOA ns1.apnic.net. read-txt-record-of-zone-first-dns-admin.apnic.net. 5276 7200 1800 604800 172800 171.in-addr.arpa. 0 IN RRSIG SOA 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. dH+uuGq9pgMc9TEMkTwrf9NGMj5zFnbZt3kjlNSF0kPzGk3WUr5g84jv 0SN88Y26eP/Op4Vv8JewyVu2IT1POnmiNpDYELkJxrTWnPPc6CJ2wugR TrFLOq6eIsqtrCcWbatQ7XSXXj1UglqVVydlX1ExoIh9MdP+1eBneD1I 4OU= 171.in-addr.arpa. 172800 IN NSEC 10.171.in-addr.arpa. NS SOA TXT RRSIG NSEC DNSKEY 171.in-addr.arpa. 172800 IN RRSIG NSEC 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. QieHZ0W7jx3QCgLuvqtKtFoLPYkxn5R9nRO1oMbSSXkdF+S4iQQkVlX7 DyQvoL69hKA+S9idfernf6DwNTQgeWFfCdjUyOjMCa3Ag/S8ARZs7K4F N9DxB/cBCZm7KTzwAzXFe3cJ/d1Wo45Tx9jf6Drx551oqOw1gmDIL4y6 +k4= ;; Received 530 bytes from 202.12.31.140#53(ns4.apnic.net) in 347 ms }}} since it is APNIC who respond to queries about it. Any more specific reverse records that are cross ARIN and die there from 171.x.y.z? -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute"
Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
You can find a detailed announcement from the RIPE NCC here: https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html <https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html> -Alex Band
On 17 Mar 2017, at 12:31, John Curran <jcurran@istaff.org> wrote:
Eygene -
We are aware there’s an issue and working on it presently with RIPE. Expect additional updates shortly.
/John
John Curran President and CEO ARIN
On 17 Mar 2017, at 11:04 AM, Eygene Ryabinkin <rea+nanog@grid.kiae.ru> wrote:
Job, good day.
Fri, Mar 17, 2017 at 09:55:56AM +0000, Job Snijders wrote:
171 also seems affected.
Not the whole one, it seems: {{{ $ dig +trace -t soa 1.171.in-addr.arpa
; <<>> DiG 9.10.4-P6 <<>> +trace -t soa 1.171.in-addr.arpa ;; global options: +cmd . 202634 IN NS m.root-servers.net. . 202634 IN NS i.root-servers.net. . 202634 IN NS k.root-servers.net. . 202634 IN NS c.root-servers.net. . 202634 IN NS a.root-servers.net. . 202634 IN NS d.root-servers.net. . 202634 IN NS l.root-servers.net. . 202634 IN NS e.root-servers.net. . 202634 IN NS g.root-servers.net. . 202634 IN NS b.root-servers.net. . 202634 IN NS j.root-servers.net. . 202634 IN NS h.root-servers.net. . 202634 IN NS f.root-servers.net. . 511016 IN RRSIG NS 8 0 518400 20170329170000 20170316160000 61045 . OjsjoA6WCcThV3BqhAyMaXI3bU1m228Gl8nvaR074qBtem/RjaFJh1Oe r7LPI6W15jgbRGuCY7/GUNgDex4ZM43yvx2iY+2GpSk9b2/pKGbDaDIp X1Hd8418206eow1P/SgPqtT+2LzjM+lz/HKUvCyPoN4uX5/7GDExn8ir 2Q/vw81Za2nJ4Ji6oeGAmE8g6V3RJfTI0El/CP9Vl2/r0jWDZZ+wtYRA uwqvQYj+7VZc5+UJDJ3/Gne1LBnzXKnZRwnfJtZenhfZ7X20D9PiXvfJ M2s1ryqZvg8K5RqYt/r8pIiq1Udd1KWQBgdN7Q629+jNSngZQtH19R9H fM7h1w== ;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms
in-addr.arpa. 172800 IN NS a.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS b.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS c.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS d.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS e.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS f.in-addr-servers.arpa. in-addr.arpa. 86400 IN DS 47054 8 2 5CAFCCEC201D1933B4C9F6A9C8F51E51F3B39979058AC21B8DF1B1F2 81CBC6F2 in-addr.arpa. 86400 IN DS 53696 8 2 13E5501C56B20394DA921B51412D48B7089C5EB6957A7C58553C4D4D 424F04DF in-addr.arpa. 86400 IN DS 63982 8 2 AAF4FB5D213EF25AE44679032EBE3514C487D7ABD99D7F5FEC3383D0 30733C73 in-addr.arpa. 86400 IN RRSIG DS 8 2 86400 20170330000000 20170316230000 33786 arpa. gqNN6MmLVxFtSG0oxdyoYVSXZlp6vY9yxpnJW89TYCHfTkv+ZmklM76O IHdTznEhLHXbyr4BaxjFCshsC3WacdH/YZYa/SZynJ9Q1N6/bogrIUTn qs61Y/YD4Sk5lI7YvUxVnPrF3lk1pY8dpoTkprTLZngoeQEsm552inqG ypU= ;; Received 731 bytes from 193.0.14.129#53(k.root-servers.net) in 41 ms
171.in-addr.arpa. 86400 IN NS ns1.apnic.net. 171.in-addr.arpa. 86400 IN NS ns2.lacnic.net. 171.in-addr.arpa. 86400 IN NS ns3.apnic.net. 171.in-addr.arpa. 86400 IN NS ns4.apnic.net. 171.in-addr.arpa. 86400 IN NS apnic.authdns.ripe.net. 171.in-addr.arpa. 86400 IN NS apnic1.dnsnode.net. 171.in-addr.arpa. 86400 IN NS tinnie.arin.net. 171.in-addr.arpa. 86400 IN DS 49699 5 1 0240801C96480900CC6D93130AF45CFE86EDE940 171.in-addr.arpa. 86400 IN DS 49699 5 2 6E0BA96F05B4D39C1668979731E040213BB6130DE33E86E063B5F7F7 5C465C88 171.in-addr.arpa. 86400 IN RRSIG DS 8 3 86400 20170330042932 20170309125911 4341 in-addr.arpa. RKcPZJdG1MBrwpfa1mIK7ikIU585i1Jv+UkEBHxuM3BxxAbD+ht0W04C ljboyXJfYK8ly/YTDqNUkWKZLwDHlkq/rgNwTG63sPT8iK8qya+2qUVB iTXVsD2HaV1V/KwgJjlWHRNnlwkK0YZ5Q7tevXaq4yyLT2Q2mu5dkhFC evNQOyI= ;; Received 482 bytes from 199.253.183.183#53(b.in-addr-servers.arpa) in 69 ms
171.in-addr.arpa. 0 IN SOA ns1.apnic.net. read-txt-record-of-zone-first-dns-admin.apnic.net. 5276 7200 1800 604800 172800 171.in-addr.arpa. 0 IN RRSIG SOA 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. dH+uuGq9pgMc9TEMkTwrf9NGMj5zFnbZt3kjlNSF0kPzGk3WUr5g84jv 0SN88Y26eP/Op4Vv8JewyVu2IT1POnmiNpDYELkJxrTWnPPc6CJ2wugR TrFLOq6eIsqtrCcWbatQ7XSXXj1UglqVVydlX1ExoIh9MdP+1eBneD1I 4OU= 171.in-addr.arpa. 172800 IN NSEC 10.171.in-addr.arpa. NS SOA TXT RRSIG NSEC DNSKEY 171.in-addr.arpa. 172800 IN RRSIG NSEC 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. QieHZ0W7jx3QCgLuvqtKtFoLPYkxn5R9nRO1oMbSSXkdF+S4iQQkVlX7 DyQvoL69hKA+S9idfernf6DwNTQgeWFfCdjUyOjMCa3Ag/S8ARZs7K4F N9DxB/cBCZm7KTzwAzXFe3cJ/d1Wo45Tx9jf6Drx551oqOw1gmDIL4y6 +k4= ;; Received 530 bytes from 202.12.31.140#53(ns4.apnic.net) in 347 ms }}} since it is APNIC who respond to queries about it. Any more specific reverse records that are cross ARIN and die there from 171.x.y.z? -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute"
Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
Dear all, RIPE NCC have issued a statement about the issue here: https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html Our apologies for the inconvenience caused. Kind regards, Romeo Zwart RIPE NCC On 17/03/17 12:31 , John Curran wrote:
Eygene -
We are aware there’s an issue and working on it presently with RIPE. Expect additional updates shortly.
/John
John Curran President and CEO ARIN
On 17 Mar 2017, at 11:04 AM, Eygene Ryabinkin <rea+nanog@grid.kiae.ru> wrote:
Job, good day.
Fri, Mar 17, 2017 at 09:55:56AM +0000, Job Snijders wrote:
171 also seems affected.
Not the whole one, it seems: {{{ $ dig +trace -t soa 1.171.in-addr.arpa
; <<>> DiG 9.10.4-P6 <<>> +trace -t soa 1.171.in-addr.arpa ;; global options: +cmd . 202634 IN NS m.root-servers.net. . 202634 IN NS i.root-servers.net. . 202634 IN NS k.root-servers.net. . 202634 IN NS c.root-servers.net. . 202634 IN NS a.root-servers.net. . 202634 IN NS d.root-servers.net. . 202634 IN NS l.root-servers.net. . 202634 IN NS e.root-servers.net. . 202634 IN NS g.root-servers.net. . 202634 IN NS b.root-servers.net. . 202634 IN NS j.root-servers.net. . 202634 IN NS h.root-servers.net. . 202634 IN NS f.root-servers.net. . 511016 IN RRSIG NS 8 0 518400 20170329170000 20170316160000 61045 . OjsjoA6WCcThV3BqhAyMaXI3bU1m228Gl8nvaR074qBtem/RjaFJh1Oe r7LPI6W15jgbRGuCY7/GUNgDex4ZM43yvx2iY+2GpSk9b2/pKGbDaDIp X1Hd8418206eow1P/SgPqtT+2LzjM+lz/HKUvCyPoN4uX5/7GDExn8ir 2Q/vw81Za2nJ4Ji6oeGAmE8g6V3RJfTI0El/CP9Vl2/r0jWDZZ+wtYRA uwqvQYj+7VZc5+UJDJ3/Gne1LBnzXKnZRwnfJtZenhfZ7X20D9PiXvfJ M2s1ryqZvg8K5RqYt/r8pIiq1Udd1KWQBgdN7Q629+jNSngZQtH19R9H fM7h1w== ;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms
in-addr.arpa. 172800 IN NS a.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS b.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS c.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS d.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS e.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS f.in-addr-servers.arpa. in-addr.arpa. 86400 IN DS 47054 8 2 5CAFCCEC201D1933B4C9F6A9C8F51E51F3B39979058AC21B8DF1B1F2 81CBC6F2 in-addr.arpa. 86400 IN DS 53696 8 2 13E5501C56B20394DA921B51412D48B7089C5EB6957A7C58553C4D4D 424F04DF in-addr.arpa. 86400 IN DS 63982 8 2 AAF4FB5D213EF25AE44679032EBE3514C487D7ABD99D7F5FEC3383D0 30733C73 in-addr.arpa. 86400 IN RRSIG DS 8 2 86400 20170330000000 20170316230000 33786 arpa. gqNN6MmLVxFtSG0oxdyoYVSXZlp6vY9yxpnJW89TYCHfTkv+ZmklM76O IHdTznEhLHXbyr4BaxjFCshsC3WacdH/YZYa/SZynJ9Q1N6/bogrIUTn qs61Y/YD4Sk5lI7YvUxVnPrF3lk1pY8dpoTkprTLZngoeQEsm552inqG ypU= ;; Received 731 bytes from 193.0.14.129#53(k.root-servers.net) in 41 ms
171.in-addr.arpa. 86400 IN NS ns1.apnic.net. 171.in-addr.arpa. 86400 IN NS ns2.lacnic.net. 171.in-addr.arpa. 86400 IN NS ns3.apnic.net. 171.in-addr.arpa. 86400 IN NS ns4.apnic.net. 171.in-addr.arpa. 86400 IN NS apnic.authdns.ripe.net. 171.in-addr.arpa. 86400 IN NS apnic1.dnsnode.net. 171.in-addr.arpa. 86400 IN NS tinnie.arin.net. 171.in-addr.arpa. 86400 IN DS 49699 5 1 0240801C96480900CC6D93130AF45CFE86EDE940 171.in-addr.arpa. 86400 IN DS 49699 5 2 6E0BA96F05B4D39C1668979731E040213BB6130DE33E86E063B5F7F7 5C465C88 171.in-addr.arpa. 86400 IN RRSIG DS 8 3 86400 20170330042932 20170309125911 4341 in-addr.arpa. RKcPZJdG1MBrwpfa1mIK7ikIU585i1Jv+UkEBHxuM3BxxAbD+ht0W04C ljboyXJfYK8ly/YTDqNUkWKZLwDHlkq/rgNwTG63sPT8iK8qya+2qUVB iTXVsD2HaV1V/KwgJjlWHRNnlwkK0YZ5Q7tevXaq4yyLT2Q2mu5dkhFC evNQOyI= ;; Received 482 bytes from 199.253.183.183#53(b.in-addr-servers.arpa) in 69 ms
171.in-addr.arpa. 0 IN SOA ns1.apnic.net. read-txt-record-of-zone-first-dns-admin.apnic.net. 5276 7200 1800 604800 172800 171.in-addr.arpa. 0 IN RRSIG SOA 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. dH+uuGq9pgMc9TEMkTwrf9NGMj5zFnbZt3kjlNSF0kPzGk3WUr5g84jv 0SN88Y26eP/Op4Vv8JewyVu2IT1POnmiNpDYELkJxrTWnPPc6CJ2wugR TrFLOq6eIsqtrCcWbatQ7XSXXj1UglqVVydlX1ExoIh9MdP+1eBneD1I 4OU= 171.in-addr.arpa. 172800 IN NSEC 10.171.in-addr.arpa. NS SOA TXT RRSIG NSEC DNSKEY 171.in-addr.arpa. 172800 IN RRSIG NSEC 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. QieHZ0W7jx3QCgLuvqtKtFoLPYkxn5R9nRO1oMbSSXkdF+S4iQQkVlX7 DyQvoL69hKA+S9idfernf6DwNTQgeWFfCdjUyOjMCa3Ag/S8ARZs7K4F N9DxB/cBCZm7KTzwAzXFe3cJ/d1Wo45Tx9jf6Drx551oqOw1gmDIL4y6 +k4= ;; Received 530 bytes from 202.12.31.140#53(ns4.apnic.net) in 347 ms }}} since it is APNIC who respond to queries about it. Any more specific reverse records that are cross ARIN and die there from 171.x.y.z? -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute"
Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
John, Alex, Romeo, Fri, Mar 17, 2017 at 12:31:06PM +0100, John Curran wrote:
We are aware there’s an issue and working on it presently with RIPE. Expect additional updates shortly.
Fri, Mar 17, 2017 at 12:50:48PM +0100, Alex Band wrote:
You can find a detailed announcement from the RIPE NCC here: https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html <https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html>
Fri, Mar 17, 2017 at 12:52:10PM +0100, Romeo Zwart wrote:
RIPE NCC have issued a statement about the issue here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
Our apologies for the inconvenience caused.
Thanks for your work: the issue seem to be fixed. I am trying to verify if everything works as expected, there are some neats, {{{ 206.144.in-addr.arpa. 172800 IN NS ns1.rrcki.ru. 206.144.in-addr.arpa. 172800 IN NS ns.kiae.ru. 206.144.in-addr.arpa. 172800 IN NS ns3.rrcki.ru. 206.144.in-addr.arpa. 172800 IN NS ns2.grid.kiae.ru. 206.144.in-addr.arpa. 172800 IN NS ns.grid.kiae.ru. 206.144.in-addr.arpa. 10800 IN NSEC 207.144.in-addr.arpa. NS RRSIG NSEC 206.144.in-addr.arpa. 10800 IN RRSIG NSEC 5 4 10800 20170331110430 20170317100430 33345 144.in-addr.arpa. vbFwaKdRa7Jd70aAbJ 5mC37BsTzMg3nWVI5gqQLLOqSaCZfH0XUez+Uk MbTNvepziCRzH+HgSLabuvRSo4nIUP1SjOd2WX0wySSdb/blqhfmjw3l n8laqOxy/lj8TDiIuxOdw2JhM1v5x/DH4aDnwdGFfUEOdgzCU5k8LdAT oyA= ;; Received 373 bytes from 199.180.180.63#53(r.arin.net) in 198 ms ;; Question section mismatch: got 206.144.0.0.in-addr.arpa/PTR/IN ;; Question section mismatch: got 206.144.0.0.in-addr.arpa/PTR/IN 206.144.in-addr.arpa. 3600 IN SOA ns.kiae.ru. noc.rrcki.ru. 2017020803 10800 3600 1800000 3600 ;; Received 105 bytes from 144.206.239.1#53(ns.grid.kiae.ru) in 0 ms }}} about question mismatch, though my guts feel that this is the local issue at rrcki.ru. Thanks again! And for a nicely-written summary -- too. -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
On Fri, Mar 17, 2017 at 7:52 AM, Romeo Zwart <rz+nng@zwart.com> wrote:
RIPE NCC have issued a statement about the issue here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
Our apologies for the inconvenience caused.
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to corrupted data by zeroing out the data instead of using the last known good data. That's awfully brittle for such a critical service. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Dirtside Systems ......... Web: <http://www.dirtside.com/>
William Herrin <bill@herrin.us> writes:
On Fri, Mar 17, 2017 at 7:52 AM, Romeo Zwart <rz+nng@zwart.com> wrote:
RIPE NCC have issued a statement about the issue here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
Our apologies for the inconvenience caused.
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to corrupted data by zeroing out the data instead of using the last known good data. That's awfully brittle for such a critical service.
Well, it was a nice smoke test of the "RDNS required" anti-feature. All of a sudden we couldn't even send email to ourselves, having smarthosts in one of the affected zones. Nice. Maybe time to re-evaluate the usefulness of that config... Bjørn
On Fri, Mar 17, 2017 at 05:42:11PM +0100, Bjørn Mork wrote:
William Herrin <bill@herrin.us> writes:
On Fri, Mar 17, 2017 at 7:52 AM, Romeo Zwart <rz+nng@zwart.com> wrote:
RIPE NCC have issued a statement about the issue here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
Our apologies for the inconvenience caused.
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to corrupted data by zeroing out the data instead of using the last known good data. That's awfully brittle for such a critical service.
Well, it was a nice smoke test of the "RDNS required" anti-feature. All of a sudden we couldn't even send email to ourselves, having smarthosts in one of the affected zones. Nice.
Maybe time to re-evaluate the usefulness of that config...
or proper whitelisting of your own infrastructure :-) - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Fri, 17 Mar 2017 17:42:11 +0100, Bjørn Mork said:
Well, it was a nice smoke test of the "RDNS required" anti-feature. All of a sudden we couldn't even send email to ourselves, having smarthosts in one of the affected zones. Nice.
If you don't have a chaos monkey, the Internet will provide one free of charge.
On Mar 17, 2017, at 10:28 AM, valdis.kletnieks@vt.edu wrote:
On Fri, 17 Mar 2017 17:42:11 +0100, Bjørn Mork said:
Well, it was a nice smoke test of the "RDNS required" anti-feature. All of a sudden we couldn't even send email to ourselves, having smarthosts in one of the affected zones. Nice.
If you don't have a chaos monkey, the Internet will provide one free of charge.
Or to a service partner or underlying infrastructure you rely on. Isn't complexity grand... -george Sent from my iPhone
On 3/17/17, 12:26 PM, "NANOG on behalf of William Herrin" <nanog-bounces@nanog.org on behalf of bill@herrin.us> wrote: On Fri, Mar 17, 2017 at 7:52 AM, Romeo Zwart <rz+nng@zwart.com> wrote: > RIPE NCC have issued a statement about the issue here: > > https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html > > Our apologies for the inconvenience caused. Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to corrupted data by zeroing out the data instead of using the last known good data. That's awfully brittle for such a critical service. Regards, Bill Herrin Hi Bill, The analysis was not yet complete when the notice went out from RIPE. After doing a post-mortum, there were no bugs in ARIN’s software in regards to this issue. We followed exactly what RIPE told us to do. When we noticed an issue with RIPE’s updates yesterday, we notified them as well. Two zones managed by APNIC that received delegations from RIPE were also affected by RIPE’s bug – it was more than just a RIPE/ARIN thing. The three impacted RIR’s have been working closely together to get RIPE’s DNS entries correctly added into our respective authoritative DNS systems. Thanks, Mark ARIN CTO
On Fri, Mar 17, 2017 at 1:42 PM, Mark Kosters <markk@arin.net> wrote:
On 3/17/17, 12:26 PM, "NANOG on behalf of William Herrin" < nanog-bounces@nanog.org on behalf of bill@herrin.us> wrote:
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to corrupted data by zeroing out the data instead of using the last
known good
there were no bugs in ARIN’s software in regards to this issue. We followed exactly what RIPE told us to do.
Hi Mark, That shot my eyebrow up. You misspoke here, right? There's no bug -solely because- you did what the design said to do? The design calls for some self-check information and it's not a critical design bug to zero-out the publish if the self-check fails? Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Dirtside Systems ......... Web: <http://www.dirtside.com/>
On 17 Mar 2017, at 2:08 PM, William Herrin <bill@herrin.us> wrote:
On Fri, Mar 17, 2017 at 1:42 PM, Mark Kosters <markk@arin.net> wrote:
On 3/17/17, 12:26 PM, "NANOG on behalf of William Herrin" < nanog-bounces@nanog.org on behalf of bill@herrin.us> wrote:
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to corrupted data by zeroing out the data instead of using the last known good
there were no bugs in ARIN’s software in regards to this issue. We followed exactly what RIPE told us to do.
Hi Mark,
That shot my eyebrow up. You misspoke here, right? There's no bug -solely because- you did what the design said to do? The design calls for some self-check information and it's not a critical design bug to zero-out the publish if the self-check fails?
Bill - See previous reply. The data was both correctly formatted and signed, so the agreed integrity checks passed. Thanks, /John John Curran President and CEO ARIN
On Fri, Mar 17, 2017 at 2:14 PM, John Curran <jcurran@arin.net> wrote:
See previous reply. The data was both correctly formatted and signed, so the agreed integrity checks passed.
Ah, okay. So it wasn't bad counts as originally reported but no data with counts that confirmed no data. Thanks for the clarification! Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Dirtside Systems ......... Web: <http://www.dirtside.com/>
On 17 Mar 2017, at 2:17 PM, William Herrin <bill@herrin.us> wrote:
On Fri, Mar 17, 2017 at 2:14 PM, John Curran <jcurran@arin.net> wrote:
See previous reply. The data was both correctly formatted and signed, so the agreed integrity checks passed.
Ah, okay. So it wasn't bad counts as originally reported but no data with counts that confirmed no data. Thanks for the clarification!
Bill - Glad to help (and apologies for the information coming out in pieces – we’ve opted to go with updates as we learn more rather than some for comprehensive but less timely report.) Thanks! /John John Curran President and CEO ARIN
On Fri, Mar 17, 2017 at 2:31 PM, John Curran <jcurran@arin.net> wrote:
Glad to help (and apologies for the information coming out in pieces – we’ve opted to go with updates as we learn more rather than some for comprehensive but less timely report.)
Also very much appreciated. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Dirtside Systems ......... Web: <http://www.dirtside.com/>
Dear John, Bill and all, On 17/03/17 19:31 , John Curran wrote:
On 17 Mar 2017, at 2:17 PM, William Herrin <bill@herrin.us> wrote:
On Fri, Mar 17, 2017 at 2:14 PM, John Curran <jcurran@arin.net> wrote:
See previous reply. The data was both correctly formatted and signed, so the agreed integrity checks passed.
Ah, okay. So it wasn't bad counts as originally reported but no data with counts that confirmed no data. Thanks for the clarification!
Bill -
Glad to help (and apologies for the information coming out in pieces – we’ve opted to go with updates as we learn more rather than some for comprehensive but less timely report.)
We have been slow to clarify this from the RIPE NCC end, for which I apologize. As was already mentioned by Mark and John in previous messages in this thread, the initial report from the RIPE NCC wasn't complete, which has lead to unnecessary confusion. A follow up message with additional detail was sent to the RIPE NCC DNS working group list earlier today: https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003401.html We hope that this clarifies matters sufficiently. Kind regards, Romeo Zwart RIPE NCC
On 03/17/2017 10:42 AM, Mark Kosters wrote:
On 3/17/17, 12:26 PM, "NANOG on behalf of William Herrin" <nanog-bounces@nanog.org on behalf of bill@herrin.us> wrote:
On Fri, Mar 17, 2017 at 7:52 AM, Romeo Zwart <rz+nng@zwart.com> wrote:
RIPE NCC have issued a statement about the issue here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
Our apologies for the inconvenience caused.
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to corrupted data by zeroing out the data instead of using the last known good data. That's awfully brittle for such a critical service.
Regards, Bill Herrin
Hi Bill,
The analysis was not yet complete when the notice went out from RIPE. After doing a post-mortum, there were no bugs in ARIN’s software in regards to this issue. We followed exactly what RIPE told us to do. When we noticed an issue with RIPE’s updates yesterday, we notified them as well.
My eyebrows reacted to this the same way Bill's did. It sounds like this is at least a semi-automated system. Such things should have sanity checks on the receiving side when told to remove large gobs of data, even if the instructions validate correctly. More fundamentally, according to the RIPE report they are sending you something called "zonelets" which you then process into actual DNS data. Can you say something about the relative merit of this system, vs. simply delegating the right zones to the right parties and letting the DNS do what it was intended to do? At minimum the fact that this automated system was allowed to wipe out great chunks of important data calls it into question. And sure, you can all 3 fix the bugs you found this time around, but up until these bugs were triggered you all thought the system was functioning perfectly, in spite of it ending up doing something that obviously was not intended. Doug
On 18 Mar 2017, at 9:58 PM, Doug Barton <dougb@dougbarton.us> wrote:
My eyebrows reacted to this the same way Bill's did. It sounds like this is at least a semi-automated system. Such things should have sanity checks on the receiving side when told to remove large gobs of data, even if the instructions validate correctly.
More fundamentally, according to the RIPE report they are sending you something called "zonelets" which you then process into actual DNS data. Can you say something about the relative merit of this system, vs. simply delegating the right zones to the right parties and letting the DNS do what it was intended to do?
At minimum the fact that this automated system was allowed to wipe out great chunks of important data calls it into question. And sure, you can all 3 fix the bugs you found this time around, but up until these bugs were triggered you all thought the system was functioning perfectly, in spite of it ending up doing something that obviously was not intended.
Doug - We could indeed decide to ignore correctly formatted and signed information if it doesn’t match some heuristics that we put in place (e.g. empty zone, zone with only 1 entry, zone that changes more than 10% in size, etc.) Some downsides with this approach is that that: 1) we’d be establishing heuristics for data that originates with a different organization and absent knowledge of their business changes, and 2) this would be mean that there could be occasions where proper data cannot be installed without manual intervention (because the changes happens to be outside of whatever heuristics have previously been put in place.) Despite the associated risk, we are happy to install such checks if RIPE requests them, but are this time are processing them as we agreed to do so – which is whenever we receive correctly formatted and properly signed requests from them. (You should inquire to RIPE for more detail regarding their future intentions in this regard.) As to why DNS-native zone operations are not utilized, the challenge is that reverse DNS zones for IPv4 and DNS operations are on octet boundaries, but IPv4 address blocks may be aligned on any bit boundary. Thus, a single IPv4 octet range may contain IPv4 address blocks that are administered by multiple RIRs, making it is necessary for one RIR to be authoritative for the entire zone and other RIRs to send information seperately on their IPv4 address blocks in that same range so that it gets included in the appropriate zone file. Excellent questions - thanks! /John John Curran President and CEO ARIN
Thanks for the response, John. Some thoughts below. On 03/18/2017 08:58 PM, John Curran wrote:
On 18 Mar 2017, at 9:58 PM, Doug Barton <dougb@dougbarton.us <mailto:dougb@dougbarton.us>> wrote:
My eyebrows reacted to this the same way Bill's did. It sounds like this is at least a semi-automated system. Such things should have sanity checks on the receiving side when told to remove large gobs of data, even if the instructions validate correctly.
More fundamentally, according to the RIPE report they are sending you something called "zonelets" which you then process into actual DNS data. Can you say something about the relative merit of this system, vs. simply delegating the right zones to the right parties and letting the DNS do what it was intended to do?
At minimum the fact that this automated system was allowed to wipe out great chunks of important data calls it into question. And sure, you can all 3 fix the bugs you found this time around, but up until these bugs were triggered you all thought the system was functioning perfectly, in spite of it ending up doing something that obviously was not intended.
Doug -
We could indeed decide to ignore correctly formatted and signed information if it doesn’t match some heuristics that we put in place (e.g. empty zone, zone with only 1 entry, zone that changes more than 10% in size, etc.)
I have used the latter type of system with good results, for what it's worth. And funny you should mention 10%, as that's what I've found to be fairly commonly at least a yellow flag, if not a big red one.
Some downsides with this approach is that that: 1) we’d be establishing heuristics for data that originates with a different organization and absent knowledge of their business changes,
If you're not already having ongoing discussions about said changes well in advance, your system is broken.
and 2) this would be mean that there could be occasions where proper data cannot be installed without manual intervention (because the changes happens to be outside of whatever heuristics have previously been put in place.)
See above. Also, not putting in place *new* changes on a scale sufficient to trip the heuristics is far superior to automatically putting in place changes that take huge chunks of address space off the network. Or am I missing something? And if you're having regular conversations with your "customers" in this scenario about upcoming major changes, tripping the alarm should only happen if someone, somewhere, made a mistake. Thus, human intervention is required regardless. But, see below.
Despite the associated risk, we are happy to install such checks if RIPE requests them, but are this time are processing them as we agreed to do so – which is whenever we receive correctly formatted and properly signed requests from them. (You should inquire to RIPE for more detail regarding their future intentions in this regard.)
Already did, thanks. :) Meanwhile, one could make a legitimate argument that even absent specific guidance from RIPE, ARIN should have a sufficient level of concern for the health of the larger Internet to consider unilateral action here. At least in the form of delaying implementation until some human coordination takes place. Personally, I don't buy, "They told us to do it!" as a legitimate response here.
As to why DNS-native zone operations are not utilized, the challenge is that reverse DNS zones for IPv4 and DNS operations are on octet boundaries, but IPv4 address blocks may be aligned on any bit boundary.
Yes, deeply familiar with that problem. Are you dealing with any address blocks smaller than a /24? If the answer is no (which it almost certainly is), what challenges are you facing that you haven't figured out how to overcome yet? (Even < /24 blocks can be dealt with, obviously, but I'd be interested to learn that there are problems with /24 and up that are too difficult to solve.) Personally, I would be happy to donate my time to help y'all sort this out, and I'm sure there are others who would also be willing to help. Doug
On 19 Mar 2017, at 12:27 AM, Doug Barton <dougb@dougbarton.us> wrote:
...
Despite the associated risk, we are happy to install such checks if RIPE requests them, but are this time are processing them as we agreed to do so – which is whenever we receive correctly formatted and properly signed requests from them. (You should inquire to RIPE for more detail regarding their future intentions in this regard.)
Already did, thanks. :) Meanwhile, one could make a legitimate argument that even absent specific guidance from RIPE, ARIN should have a sufficient level of concern for the health of the larger Internet to consider unilateral action here. At least in the form of delaying implementation until some human coordination takes place.
We’ll process RIPE’s requests in whatever manner they direct, as it is their customers that are affected by whatever decision they make in this regard. Thanks! /John John Curran President and CEO ARIN
On 03/18/2017 09:40 PM, John Curran wrote:
On 19 Mar 2017, at 12:27 AM, Doug Barton <dougb@dougbarton.us> wrote:
...
Despite the associated risk, we are happy to install such checks if RIPE requests them, but are this time are processing them as we agreed to do so – which is whenever we receive correctly formatted and properly signed requests from them. (You should inquire to RIPE for more detail regarding their future intentions in this regard.)
Already did, thanks. :) Meanwhile, one could make a legitimate argument that even absent specific guidance from RIPE, ARIN should have a sufficient level of concern for the health of the larger Internet to consider unilateral action here. At least in the form of delaying implementation until some human coordination takes place.
We’ll process RIPE’s requests in whatever manner they direct, as it is their customers that are affected by whatever decision they make in this regard.
I'll let others chime in on whether they they think that's a reasonable response. I've already said my piece. Meanwhile, my offer to help y'all fix your DNS was a sincere one. Feel free to hit me up off list. Doug
On 19 Mar 2017, at 12:50 AM, Doug Barton <dougb@dougbarton.us<mailto:dougb@dougbarton.us>> wrote: ... Meanwhile, my offer to help y'all fix your DNS was a sincere one. Feel free to hit me up off list. Doug - You’d want to make that offer to the RIPE NCC (as they experienced the issues with their DNS systems.) Let me know if you need contact info. Thanks, /John John Curran President and CEO ARIN
On 03/18/2017 10:53 PM, John Curran wrote:
On 19 Mar 2017, at 12:50 AM, Doug Barton <dougb@dougbarton.us <mailto:dougb@dougbarton.us>> wrote:
... Meanwhile, my offer to help y'all fix your DNS was a sincere one. Feel free to hit me up off list.
Doug -
You’d want to make that offer to the RIPE NCC
My offer was in response to your assertion that normal DNS techniques of delegation were not sufficient to the unique problems ARIN has to deal with in regards to the address space you manage delegations for. Subsequent to our conversation however, Shane Kerr was kind enough to explain the problem that the "zonelets" are designed to solve: https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003406.html Short version, ARIN maintains foo/8, but bar/16 within it is managed by RIPE, who wants to delegate it directly to the registered party for that block. They use a zonelet to tell ARIN how to do that. As you have indicated that ARIN will not make any changes to its existing practices without specific instructions from RIPE, I will offer my suggestions to them instead. :) best, Doug
On Sat, Mar 18, 2017 at 09:27:11PM -0700, Doug Barton wrote:
As to why DNS-native zone operations are not utilized, the challenge is that reverse DNS zones for IPv4 and DNS operations are on octet boundaries, but IPv4 address blocks may be aligned on any bit boundary.
Yes, deeply familiar with that problem. Are you dealing with any address blocks smaller than a /24? If the answer is no (which it almost certainly is), what challenges are you facing that you haven't figured out how to overcome yet? (Even < /24 blocks can be dealt with, obviously, but I'd be interested to learn that there are problems with /24 and up that are too difficult to solve.)
Hypotheically: 10.11.0.0/16 (11.10.in-addr.arpa) is managed by ARIN 10.11.16.0/20 is ARIN space 10.11.32.0/20 is RIPE space If ARIN delegated 32.11.10.in-addr.arpa through 47.11.10.in-addr.arpa to a RIPE nameserver, there's no good way for RIPE to then delegate, say, 10.11.34.0/24 (34.11.10.in-addr.arpa) to the nameserver of the entity to which RIPE has allocated 10.11.34.0. (Sure, it can be done, using the same techniques as are used for allocations of longer-than-/24, but recipients of /24 and larger reasonably expect to have the X.X.X.in-addr.arpa delegated to their nameservers.) So, instead, RIPE communicates to ARIN the proper delegations for 32.11.10.in-addr.arpa through 47.11.10.in-addr.arpa, and ARIN merges those into 11.10.in-addr.arpa. One way for RIPE to communicate those delegations to ARIN is to put then into some other zone, which ARIN could then zone-transfer. But ARIN would still need a process to merge the data from that other e with the real 11.10.in-addr.arpa zone. But that has the same risks as the current process, which apparently communicates those delegations via something other than zone-transfer. -- Brett
On Mon, Mar 20, 2017 at 3:27 PM, Brett Frankenberger <rbf+nanog@panix.com> wrote:
If ARIN delegated 32.11.10.in-addr.arpa through 47.11.10.in-addr.arpa to a RIPE nameserver, there's no good way for RIPE to then delegate, say, 10.11.34.0/24 (34.11.10.in-addr.arpa) to the nameserver of the entity to which RIPE has allocated 10.11.34.0. (Sure, it can be done, using the same techniques as are used for allocations of longer-than-/24, but recipients of /24 and larger reasonably expect to have the X.X.X.in-addr.arpa delegated to their nameservers.)
Hi Brett, The last I tried it, the servers which the glue claims are authoritative for a zone could assert that they themselves are not authoritative and offer new glue for completely different servers asserted to be authoritative. I had to fake a parent zone in Bind. This was before DNSSEC. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Dirtside Systems ......... Web: <http://www.dirtside.com/>
On 17 Mar 2017, at 12:26 PM, William Herrin <bill@herrin.us<mailto:bill@herrin.us>> wrote: On Fri, Mar 17, 2017 at 7:52 AM, Romeo Zwart <rz+nng@zwart.com<mailto:rz%2Bnng@zwart.com>> wrote:
RIPE NCC have issued a statement about the issue here:
https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html
Our apologies for the inconvenience caused.
Hmm. That sounds like an ARIN-side bug too. ARIN's code responded to corrupted data by zeroing out the data instead of using the last known good data. That's awfully brittle for such a critical service. Agreed in principle - receiving incorrect data (improperly formatted, corrupted, or not properly signed) should result in appropriate notice and no change to the running system. This is actually the case with ARIN’s systems. However, we received a properly formatted and signed zonelet file, albeit one which contained zero records. APNIC also received similar correctly formatted/signed zonelet files as a record of the RIPE bug, and the three RIRs have been working closely together to get the correct RIPE data loaded back into our authoritative DNS systems. Thanks! /John John Curran President and CEO ARIN
Folks - RIPE NCC has backed out the change at issue, and we are again processing zonelet files received from them. They’ve posted more details here - <https://www.ripe.net/ripe/mail/archives/dns-wg/2017-March/003394.html> FYI, /John John Curran President and CEO ARIN On 17 Mar 2017, at 12:31 PM, John Curran <jcurran@istaff.org<mailto:jcurran@istaff.org>> wrote: Eygene - We are aware there’s an issue and working on it presently with RIPE. Expect additional updates shortly. /John John Curran President and CEO ARIN On 17 Mar 2017, at 11:04 AM, Eygene Ryabinkin <rea+nanog@grid.kiae.ru<mailto:rea+nanog@grid.kiae.ru>> wrote: Job, good day. Fri, Mar 17, 2017 at 09:55:56AM +0000, Job Snijders wrote: 171 also seems affected. Not the whole one, it seems: {{{ $ dig +trace -t soa 1.171.in-addr.arpa ; <<>> DiG 9.10.4-P6 <<>> +trace -t soa 1.171.in-addr.arpa ;; global options: +cmd . 202634 IN NS m.root-servers.net<http://m.root-servers.net>. . 202634 IN NS i.root-servers.net<http://i.root-servers.net>. . 202634 IN NS k.root-servers.net<http://k.root-servers.net>. . 202634 IN NS c.root-servers.net<http://c.root-servers.net>. . 202634 IN NS a.root-servers.net<http://a.root-servers.net>. . 202634 IN NS d.root-servers.net<http://d.root-servers.net>. . 202634 IN NS l.root-servers.net<http://l.root-servers.net>. . 202634 IN NS e.root-servers.net<http://e.root-servers.net>. . 202634 IN NS g.root-servers.net<http://g.root-servers.net>. . 202634 IN NS b.root-servers.net<http://b.root-servers.net>. . 202634 IN NS j.root-servers.net<http://j.root-servers.net>. . 202634 IN NS h.root-servers.net<http://h.root-servers.net>. . 202634 IN NS f.root-servers.net<http://f.root-servers.net>. . 511016 IN RRSIG NS 8 0 518400 20170329170000 20170316160000 61045 . OjsjoA6WCcThV3BqhAyMaXI3bU1m228Gl8nvaR074qBtem/RjaFJh1Oe r7LPI6W15jgbRGuCY7/GUNgDex4ZM43yvx2iY+2GpSk9b2/pKGbDaDIp X1Hd8418206eow1P/SgPqtT+2LzjM+lz/HKUvCyPoN4uX5/7GDExn8ir 2Q/vw81Za2nJ4Ji6oeGAmE8g6V3RJfTI0El/CP9Vl2/r0jWDZZ+wtYRA uwqvQYj+7VZc5+UJDJ3/Gne1LBnzXKnZRwnfJtZenhfZ7X20D9PiXvfJ M2s1ryqZvg8K5RqYt/r8pIiq1Udd1KWQBgdN7Q629+jNSngZQtH19R9H fM7h1w== ;; Received 1097 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms in-addr.arpa. 172800 IN NS a.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS b.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS c.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS d.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS e.in-addr-servers.arpa. in-addr.arpa. 172800 IN NS f.in-addr-servers.arpa. in-addr.arpa. 86400 IN DS 47054 8 2 5CAFCCEC201D1933B4C9F6A9C8F51E51F3B39979058AC21B8DF1B1F2 81CBC6F2 in-addr.arpa. 86400 IN DS 53696 8 2 13E5501C56B20394DA921B51412D48B7089C5EB6957A7C58553C4D4D 424F04DF in-addr.arpa. 86400 IN DS 63982 8 2 AAF4FB5D213EF25AE44679032EBE3514C487D7ABD99D7F5FEC3383D0 30733C73 in-addr.arpa. 86400 IN RRSIG DS 8 2 86400 20170330000000 20170316230000 33786 arpa. gqNN6MmLVxFtSG0oxdyoYVSXZlp6vY9yxpnJW89TYCHfTkv+ZmklM76O IHdTznEhLHXbyr4BaxjFCshsC3WacdH/YZYa/SZynJ9Q1N6/bogrIUTn qs61Y/YD4Sk5lI7YvUxVnPrF3lk1pY8dpoTkprTLZngoeQEsm552inqG ypU= ;; Received 731 bytes from 193.0.14.129#53(k.root-servers.net<http://k.root-servers.net>) in 41 ms 171.in-addr.arpa. 86400 IN NS ns1.apnic.net<http://ns1.apnic.net>. 171.in-addr.arpa. 86400 IN NS ns2.lacnic.net<http://ns2.lacnic.net>. 171.in-addr.arpa. 86400 IN NS ns3.apnic.net<http://ns3.apnic.net>. 171.in-addr.arpa. 86400 IN NS ns4.apnic.net<http://ns4.apnic.net>. 171.in-addr.arpa. 86400 IN NS apnic.authdns.ripe.net<http://apnic.authdns.ripe.net>. 171.in-addr.arpa. 86400 IN NS apnic1.dnsnode.net<http://apnic1.dnsnode.net>. 171.in-addr.arpa. 86400 IN NS tinnie.arin.net<http://tinnie.arin.net>. 171.in-addr.arpa. 86400 IN DS 49699 5 1 0240801C96480900CC6D93130AF45CFE86EDE940 171.in-addr.arpa. 86400 IN DS 49699 5 2 6E0BA96F05B4D39C1668979731E040213BB6130DE33E86E063B5F7F7 5C465C88 171.in-addr.arpa. 86400 IN RRSIG DS 8 3 86400 20170330042932 20170309125911 4341 in-addr.arpa. RKcPZJdG1MBrwpfa1mIK7ikIU585i1Jv+UkEBHxuM3BxxAbD+ht0W04C ljboyXJfYK8ly/YTDqNUkWKZLwDHlkq/rgNwTG63sPT8iK8qya+2qUVB iTXVsD2HaV1V/KwgJjlWHRNnlwkK0YZ5Q7tevXaq4yyLT2Q2mu5dkhFC evNQOyI= ;; Received 482 bytes from 199.253.183.183#53(b.in-addr-servers.arpa) in 69 ms 171.in-addr.arpa. 0 IN SOA ns1.apnic.net<http://ns1.apnic.net>. read-txt-record-of-zone-first-dns-admin.apnic.net<http://read-txt-record-of-zone-first-dns-admin.apnic.net>. 5276 7200 1800 604800 172800 171.in-addr.arpa. 0 IN RRSIG SOA 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. dH+uuGq9pgMc9TEMkTwrf9NGMj5zFnbZt3kjlNSF0kPzGk3WUr5g84jv 0SN88Y26eP/Op4Vv8JewyVu2IT1POnmiNpDYELkJxrTWnPPc6CJ2wugR TrFLOq6eIsqtrCcWbatQ7XSXXj1UglqVVydlX1ExoIh9MdP+1eBneD1I 4OU= 171.in-addr.arpa. 172800 IN NSEC 10.171.in-addr.arpa. NS SOA TXT RRSIG NSEC DNSKEY 171.in-addr.arpa. 172800 IN RRSIG NSEC 5 3 172800 20170416100102 20170317090102 7858 171.in-addr.arpa. QieHZ0W7jx3QCgLuvqtKtFoLPYkxn5R9nRO1oMbSSXkdF+S4iQQkVlX7 DyQvoL69hKA+S9idfernf6DwNTQgeWFfCdjUyOjMCa3Ag/S8ARZs7K4F N9DxB/cBCZm7KTzwAzXFe3cJ/d1Wo45Tx9jf6Drx551oqOw1gmDIL4y6 +k4= ;; Received 530 bytes from 202.12.31.140#53(ns4.apnic.net<http://ns4.apnic.net>) in 347 ms }}} since it is APNIC who respond to queries about it. Any more specific reverse records that are cross ARIN and die there from 171.x.y.z? -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
On Fri, Mar 17, 2017 at 12:03:58PM +0300, Eygene Ryabinkin <rea+nanog@grid.kiae.ru> wrote a message of 71 lines which said:
We (at Kurchatov Insitute) still use 144.206.0.0/16, the legacy block, and seeing the breakage rooted at ARIN since this night, {{{ $ dig +trace -t soa 206.144.in-addr.arpa
According to DNSDB, it arrived yesterday, around 1630 UTC.
participants (15)
-
Alex Band -
Bjørn Mork -
Brett Frankenberger -
Doug Barton -
Eygene Ryabinkin -
George William Herbert -
Jared Mauch -
Job Snijders -
John Curran -
John Curran -
Mark Kosters -
Romeo Zwart -
Stephane Bortzmeyer -
valdis.kletnieks＠vt.edu -
William Herrin