AOL DNS - temporary resolution of problem
At about noon today NetworkTwo (formerly Autonet) noticed heavy usage of our Internet links and DNS. When we investigated we discovered what you already know ... someone pointed AOL's root server entry at us. We contacted AOL about the same time they contacted us. AOL asked us to load their primary zone file on our DNS, but it quickly became apparent that our upstream pipe and our DNS server could not handle the load. We (AOL and N2) contacted NetworkTwo's upstream provider MichNet (aka Merit of nanog@merit.edu fame). Merit loaned us their new, not yet in service, DNS server. This was loaded with both the AOL and Autonet primary zones. Merit then hijacked the 206.88.0.x network and redirected it to their server, where AOL and Autonet are currently resolving. Some of my clients are affected, but most have been pointed to other name servers. The InterNIC folks predict it will take 18 hours for the root servers to be up to date. We will monitor the situation throughout the weekend, and take apart this hack when the number of queries drops off. On behalf of NetworkTwo, I'd like to thank the on call staff at Merit and AOL, all of whom pitched in totally professional way with time and equipment to solve this problem. Thanks also to Goodnet (spelling?), a peer of AOL and MichNet, who offered equipment and bandwidth that we might have needed, but didn't. On a personal note, it's nice to find out that people can still work together in a crisis. Now if we can only get NSI to secure the domain update process ... With hopes for a calmer weekend, Dave Hares -- David L. Hares, Director of Network Engineering NetworkTwo Communications Group Phone: (313) 995-6539 175 Jackson Plaza FAX : (313) 995-6458 Ann Arbor, MI 48106 (USA) Email: dhares@networktwo.net
Wow. I thought originally that this was a hijack; good to see that it wasn't. The question that I have remaining is, "How'd this happen?" How did the primary DNS mysteriously change? On Fri, 16 Oct 1998, David Hares - AutoNet wrote:
At about noon today NetworkTwo (formerly Autonet) noticed heavy usage of our Internet links and DNS. When we investigated we discovered what you already know ... someone pointed AOL's root server entry at us. We contacted AOL about the same time they contacted us. AOL asked us to load their primary zone file on our DNS, but it quickly became apparent that our upstream pipe and our DNS server could not handle the load. We (AOL and N2) contacted NetworkTwo's upstream provider MichNet (aka Merit of nanog@merit.edu fame). Merit loaned us their new, not yet in service, DNS server. This was loaded with both the AOL and Autonet primary zones. Merit then hijacked the 206.88.0.x network and redirected it to their server, where AOL and Autonet are currently resolving. Some of my clients are affected, but most have been pointed to other name servers.
The InterNIC folks predict it will take 18 hours for the root servers to be up to date. We will monitor the situation throughout the weekend, and take apart this hack when the number of queries drops off.
On behalf of NetworkTwo, I'd like to thank the on call staff at Merit and AOL, all of whom pitched in totally professional way with time and equipment to solve this problem. Thanks also to Goodnet (spelling?), a peer of AOL and MichNet, who offered equipment and bandwidth that we might have needed, but didn't.
On a personal note, it's nice to find out that people can still work together in a crisis. Now if we can only get NSI to secure the domain update process ...
With hopes for a calmer weekend,
Dave Hares
-- David L. Hares, Director of Network Engineering NetworkTwo Communications Group Phone: (313) 995-6539 175 Jackson Plaza FAX : (313) 995-6458 Ann Arbor, MI 48106 (USA) Email: dhares@networktwo.net
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ISPF, The Forum for ISPs by ISPs. October 26-28, 1998, Atlanta, GA. Three days of clues, news, and views from the industry's best and brightest. http://www.ispf.com/ for information and registration. Atheism is a non-prophet organization. I route, therefore I am. Alex Rubenstein, alex@nac.net, KC2BUO, ISP/C Charter Member Father of the Network and Head Bottle-Washer Net Access Corporation, 9 Mt. Pleasant Tpk., Denville, NJ 07834 Don't choose a spineless ISP; we have more backbone! http://www.nac.net -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
On Fri, 16 Oct 1998 alex@nac.net wrote:
Wow. I thought originally that this was a hijack; good to see that it wasn't.
It was a hijak, but not by the admins at AutoNet (or NetworkTwo). Take a look at the follow URL, the third paragraph down: http://www.news.com/News/Item/0,4,27655,00.html?st.ne.fd.gif.d AOL was just using the MAIL-FROM auth. By setting this who ever was listed as the Technical or administrative contact could alter the domain. Internic just checks to see if the from address is a valid one and if so the ACK is not required (I can tell you about this from an experience we had). Therefore even a crude forgery can change the domain servers if the auth is MAIL-FROM. The strange thing is that the contacts listed for AOL (i.e. the previous contacts if they were changed) received the piece of email that the change was going through and did nothing about it until it was too late. When this happened to us we jumped right on things and noone was the wiser on the internet (although I guess AutoNet couldn't handle the DNS traffic which is generated for AOL's web servers so that would be a problem, even if things were caught). bye, ken emery
The question that I have remaining is, "How'd this happen?"
How did the primary DNS mysteriously change?
On Fri, 16 Oct 1998, David Hares - AutoNet wrote:
At about noon today NetworkTwo (formerly Autonet) noticed heavy usage of our Internet links and DNS. When we investigated we discovered what you already know ... someone pointed AOL's root server entry at us. We contacted AOL about the same time they contacted us. AOL asked us to load their primary zone file on our DNS, but it quickly became apparent that our upstream pipe and our DNS server could not handle the load. We (AOL and N2) contacted NetworkTwo's upstream provider MichNet (aka Merit of nanog@merit.edu fame). Merit loaned us their new, not yet in service, DNS server. This was loaded with both the AOL and Autonet primary zones. Merit then hijacked the 206.88.0.x network and redirected it to their server, where AOL and Autonet are currently resolving. Some of my clients are affected, but most have been pointed to other name servers.
The InterNIC folks predict it will take 18 hours for the root servers to be up to date. We will monitor the situation throughout the weekend, and take apart this hack when the number of queries drops off.
On behalf of NetworkTwo, I'd like to thank the on call staff at Merit and AOL, all of whom pitched in totally professional way with time and equipment to solve this problem. Thanks also to Goodnet (spelling?), a peer of AOL and MichNet, who offered equipment and bandwidth that we might have needed, but didn't.
On a personal note, it's nice to find out that people can still work together in a crisis. Now if we can only get NSI to secure the domain update process ...
With hopes for a calmer weekend,
Dave Hares
-- David L. Hares, Director of Network Engineering NetworkTwo Communications Group Phone: (313) 995-6539 175 Jackson Plaza FAX : (313) 995-6458 Ann Arbor, MI 48106 (USA) Email: dhares@networktwo.net
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ISPF, The Forum for ISPs by ISPs. October 26-28, 1998, Atlanta, GA. Three days of clues, news, and views from the industry's best and brightest. http://www.ispf.com/ for information and registration.
Atheism is a non-prophet organization. I route, therefore I am. Alex Rubenstein, alex@nac.net, KC2BUO, ISP/C Charter Member Father of the Network and Head Bottle-Washer Net Access Corporation, 9 Mt. Pleasant Tpk., Denville, NJ 07834 Don't choose a spineless ISP; we have more backbone! http://www.nac.net -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
participants (3)
-
alex＠nac.net -
David Hares - AutoNet -
ken emery