- NANOG - lists.nanog.org

The weak link? DNS
by Sean Donelan 27 Mar '03

27 Mar '03

Watching the Iraqi Ururklink and Al Jazeera over the weekend what struck me is how many different ways network administrators can mess up. Although malicious actors have been trying (and succeeding) to exploit vulnerabilities, the worst problems seem to be self-inflicted. Administrators had used firewalls and locked down their web sites, sometimes so well they couldn't handle the traffic load. But the real weak link was their DNS servers. For example, Al Jazeera had time-to-live set of their domain records set to 15 minutes, making them even more vulnerable to increasing the load on their systems. Of course, Al Jazeera had other problems too. What even stranger about the Iraqi state provider Uruklink.net is the DNS servers are now self-identifying with earlier (with known bugs) versions of BIND. Last week the Uruklink name server 62.145.94.1 was running 8.2.2-P5, but now is running 8.1.2. Although the web site for www.uruklink.net is up, DNS lookups for www.uruklink.net return various other IP addresses (not in 62.145.94.0/24). Including some addresses running web sites claiming the site is "owned." In reality, the site isn't owned, you are being redirected to a unrelated web site.

11 15

Both Iraqi state provider Uruklink.net name servers offline
by Sean Donelan 27 Mar '03

27 Mar '03

Despite very old recommendations, the Iraqi state provider Uruklink.net kept all of its name servers on the same subnet. Although this is recognized as a poor design, many domain name server operators worldwide do the same thing. nic1.baghdadlink.net. 2D IN A 62.145.94.1 nic2.baghdadlink.net. 2D IN A 62.145.94.2 The nic2 (62.145.94.2) has been offline for over a week. Yesterday the remaining name server nic1 (62.145.94.1) was running an old version of bind (8.1.2). It was returning obviously bogus answers to queries. In the last 24 hours, the name server application on nic1 (62.145.94.1) went offline. The server is online (responds to pings), but neither tcp or udp port 53 responds. The name server application may have crashed, been trashed, or shutdown by the system administrator.

2 1

Classified
by Len Rose 27 Mar '03

27 Mar '03

Please refrain from discussing anything! http://news.com.com/2100-1028-994216.html sigh.

1 0

summary (Re: how to get people to upgrade?)
by Paul Vixie 27 Mar '03

27 Mar '03

in addition to many public comments (cc'd to nanog or just sent there), i received a number of private replies. here's a representative sample: > problem is if the default is off you will probably not catch the > clueless folk that you want to target, better would be default on and > the clueful world can choose to ignore it... > > btw, my impression of the vulns in earlier binds was that everyone who > mattered had upgraded and perhaps only corporates were left on old > version, of course you prolly know more than me on that. pv: (the roots and most tld servers always update, and the vendors always have patches, but by no means everyone who matters always upgrades.) > I'd certainly consider using a service offered by ISC on this basis for any > servers for which I have any responsibility. pv: (four other people said the same; thanks for your enthusiasm! one added:) > ... However, since the problem applies to multiple software packages, > perhaps it is worth the effort developing a unified standard for this > kind of purpose (RFC, something similar) and passing it on to package > maintainers, distribution maintainers, etc? Creating a universal > "versioning system" would solve a lot of problems with keeping software > updated. i think that's true, but beyond the scope of ISC's resources, and also likely to take a very long time to settle. interestingly, since it's a protocol, the ietf should care, but since it's about software, the ietf probably won't care. > In my experience people are perfectly capable of shutting their eyes for > the need to upgrade. Harrassment by e-mail won't change much there. pv: (this is interesting, since it represents the prevalent view here that just because people want the functionality that some software provides, does not imply any commitment to keeping it from being used as an entry point for an attack nor launch point for attacks on others. this should have been obvious but i'd never thought about it that way before today.) > 1. I have another idea though, during setup of the server ask for email > address of list administrator, but keep that on the server itself. > 2. Setup some dns server that provides dns record if there are necessary > updates (here is one example in reverse dns notation...: > 1.2.9.bind.updates.isc.org and set it to particular ip or > particular MX or whatever to show if update is needed). > Possibly have this special update dns recorb be HINFO to url where more > information on update is available. > 3. When bind starts if the record in #2 exists and shows that update is > necessary, have bind server email to address entered in 1 and with > abscense of that have it email to postmaster(a)computername.com and if HINFO > exists, it can take that and enter this as custome email. pv: (alas, in most bind installations, there is no script that's run to install the software or generate a config, and where this is done, it's platform/vendor specific, i.e., not written by ISC even if we had one. as to #2, there's a lot more value in MIM'ing the response to a query than there is in MIM'ing the reporting of a version:hostname:email tuple, and i don't thing we should create new attack vectors. as to #3, not every bind server host has the ability to send e-mail at all.) > This idea could be taken further, with a service offering that vendors > and/or users support. Instead of having the server only answer for BIND, it > could answer for any number of products. Vendors would have to supply the > details on the latest software versions and patches, hopefully funding the > listings (in the case of software sold for dollars, open source would have > to be supported in another way, such as by user subscription). pv: (i love this idea, and if somebody else leads, ISC would try to follow.) > I think the key to getting people to upgrade may be one of: > > 1) provide an automated audit tool which can look at an existing > installation and flag known issues, and do pre-installation checks > for required supporting packages which may also need to be > installed -- sort of an automated mentoring system. Given the problems > with misconfigured software in some remote parts of the world, it > would probably be worth building that code in a way that facilitates > internationalization from the get go... > > 2) have the application periodically check for updated versions, > and nag/nudge admins when appropriate (or possibly even auto-update) > > Yes, I know, neither of these should be necessary, and yes, I know > that automation of installation tasks poses its own set of problems > but we live in the time of Windows Update and "system admins" > who aren't what they used to be, I'm afraid. pv: (i suspect that something like this, as an outboard tool capable of running on a NOC machine, is the right short term approach. it'll need the ability to scan for BIND instances in the local address space, keep track of their version numbers, and download (via https:// i guess) a file from some central repository indicating version availability and safety, and to send mail at various junctures. i wish i had this for NTP as well, now that i think of it. maybe this should be a NOCOL thing.)

2 1

RE: how to get people to upgrade? (Re: The weak link? DNS)
by Kuhtz, Christian 26 Mar '03

26 Mar '03

> On 26 Mar 2003, Jeffrey C. Ollie wrote: > > What I would like to see is somewhat of the idea in > > reverse. The ISC would host a zone that would contain TXT records with > > security/bug advisories for every version: > > I really like this solution. It seems clean and unobjectionable, while > getting the job done. What's the job, though? The way I see it, the issue isn't that there aren't enough notifications of BIND vulnerabilities. Administrator inertia is the root cause. I don't see how an automatism such as the one described changes human behavior. And unless you change that inertia, no amount of notification, databases, registries, yada yada yada will make any difference. Thanks, Christian ***** "The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers."

7 7

RE: Odd DNS Traffic
by McBurnett, Jim 26 Mar '03

26 Mar '03

Michael, Do you have a packet sniff of the traffic? Possibly a sniff of at least 10000 packets? HMMM.. I have seen some increase at our Corp DNS, but not that much... drop me a note offlist with the sniff.. I would like to look at this.. Jim > -----Original Message----- > From: Support Team [mailto:support@snworks.com] > Sent: Wednesday, March 26, 2003 4:01 PM > To: nanog(a)merit.edu > Subject: Odd DNS Traffic > > > > First I would like to note I am new to the list and group. > It's nice to > be here. > > Second, since Monday, March 24th at approx 1am we have been suffering > from "odd" DNS traffic to our two primary DNS servers. The > odd traffic > has increased our bandwidth utilization by about 20 Mbps, which is > obviously putting a hurting on our network and our DNS servers. > > I know this must also be affecting other networks, and if anything the > root servers. If anyone has any suggestions, etc, they would be much > appreciated. > > Thank you, > Michael Mannella > Support Team > Synergy Networks, Inc. > > Here are the symptoms: > ============================================ > > The odd traffic started with the root servers, namely > (a-m).gtld-servers.net . Most of the traffic is still coming > from them, > but other servers have also started sending us this odd traffic. > > We have 3 dns servers, only two are being affected, they are > our Primary > and Secondary servers that are listed with Network Solutions. > The third > server (that is not being affected) is not listed with NetSol > and has no > DNS records setup in it. It is strictly being used for lookups. > > The odd traffic is listed as a "DNS Spoof attempt" on our firewall. > > The odd traffic looks like this: > > Rcv 192.48.79.30 0cbb R Q [0084 A NOERROR] > (8)ÎÒµÄµç»°(3)COM(0) > UDP response info at 01ADC8BC > Socket = 380 > Remote addr 192.48.79.30, port 53 > Time Query=147367, Queued=0, Expire=0 > Buf length = 0x0200 (512) > Msg length = 0x010e (270) > Message: > XID 0x0cbb > Flags 0x8400 > QR 1 (response) > OPCODE 0 (QUERY) > AA 1 > TC 0 > RD 0 > RA 0 > Z 0 > RCODE 0 (NOERROR) > QCOUNT 0x1 > ACOUNT 0x1 > NSCOUNT 0xd > ARCOUNT 0x0 > Offset = 0x000c, RR count = 0 > Name "(8)ÎÒµÄµç»°(3)COM(0)" > QTYPE A (1) > QCLASS 1 > ANSWER SECTION: > Offset = 0x001e, RR count = 0 > Name "[C00C](8)ÎÒµÄµç»°(3)COM(0)" > TYPE A (1) > CLASS 1 > TTL 300 > DLEN 4 > DATA 198.41.1.35 > AUTHORITY SECTION: > Offset = 0x002e, RR count = 0 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 20 > DATA (1)g(12)gtld-servers(3)net(0) > Offset = 0x004e, RR count = 1 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)h[C03C](12)gtld-servers(3)net(0) > Offset = 0x005e, RR count = 2 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)d[C03C](12)gtld-servers(3)net(0) > Offset = 0x006e, RR count = 3 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)j[C03C](12)gtld-servers(3)net(0) > Offset = 0x007e, RR count = 4 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)i[C03C](12)gtld-servers(3)net(0) > Offset = 0x008e, RR count = 5 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)l[C03C](12)gtld-servers(3)net(0) > Offset = 0x009e, RR count = 6 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)b[C03C](12)gtld-servers(3)net(0) > Offset = 0x00ae, RR count = 7 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)e[C03C](12)gtld-servers(3)net(0) > Offset = 0x00be, RR count = 8 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)a[C03C](12)gtld-servers(3)net(0) > Offset = 0x00ce, RR count = 9 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)k[C03C](12)gtld-servers(3)net(0) > Offset = 0x00de, RR count = 10 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)f[C03C](12)gtld-servers(3)net(0) > Offset = 0x00ee, RR count = 11 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)c[C03C](12)gtld-servers(3)net(0) > Offset = 0x00fe, RR count = 12 > Name "[C015](3)COM(0)" > TYPE NS (2) > CLASS 1 > TTL 172800 > DLEN 4 > DATA (1)m[C03C](12)gtld-servers(3)net(0) > ADDITIONAL SECTION: > > The DNS server encountered an invalid domain name in a packet from > 192.48.79.30. The packet is > rejected. > >

1 0

suggestion for IBX in Washington DC
by K. Scott Bethke 26 Mar '03

26 Mar '03

I need a recommendation for an IBX/colo environment located in the city of Washington DC itself.. I know Tyson/McLean is Paradise but looking for a good solution actually IN the DC portion of lata 236 Our main goals are Transit availability from good providers (Worldcom is a must) and good peering options -Scotty

2 1

Odd DNS Traffic
by Support Team 26 Mar '03

26 Mar '03

First I would like to note I am new to the list and group. It's nice to be here. Second, since Monday, March 24th at approx 1am we have been suffering from "odd" DNS traffic to our two primary DNS servers. The odd traffic has increased our bandwidth utilization by about 20 Mbps, which is obviously putting a hurting on our network and our DNS servers. I know this must also be affecting other networks, and if anything the root servers. If anyone has any suggestions, etc, they would be much appreciated. Thank you, Michael Mannella Support Team Synergy Networks, Inc. Here are the symptoms: ============================================ The odd traffic started with the root servers, namely (a-m).gtld-servers.net . Most of the traffic is still coming from them, but other servers have also started sending us this odd traffic. We have 3 dns servers, only two are being affected, they are our Primary and Secondary servers that are listed with Network Solutions. The third server (that is not being affected) is not listed with NetSol and has no DNS records setup in it. It is strictly being used for lookups. The odd traffic is listed as a "DNS Spoof attempt" on our firewall. The odd traffic looks like this: Rcv 192.48.79.30 0cbb R Q [0084 A NOERROR] (8)ÎÒµÄµç»°(3)COM(0) UDP response info at 01ADC8BC Socket = 380 Remote addr 192.48.79.30, port 53 Time Query=147367, Queued=0, Expire=0 Buf length = 0x0200 (512) Msg length = 0x010e (270) Message: XID 0x0cbb Flags 0x8400 QR 1 (response) OPCODE 0 (QUERY) AA 1 TC 0 RD 0 RA 0 Z 0 RCODE 0 (NOERROR) QCOUNT 0x1 ACOUNT 0x1 NSCOUNT 0xd ARCOUNT 0x0 Offset = 0x000c, RR count = 0 Name "(8)ÎÒµÄµç»°(3)COM(0)" QTYPE A (1) QCLASS 1 ANSWER SECTION: Offset = 0x001e, RR count = 0 Name "[C00C](8)ÎÒµÄµç»°(3)COM(0)" TYPE A (1) CLASS 1 TTL 300 DLEN 4 DATA 198.41.1.35 AUTHORITY SECTION: Offset = 0x002e, RR count = 0 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 20 DATA (1)g(12)gtld-servers(3)net(0) Offset = 0x004e, RR count = 1 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)h[C03C](12)gtld-servers(3)net(0) Offset = 0x005e, RR count = 2 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)d[C03C](12)gtld-servers(3)net(0) Offset = 0x006e, RR count = 3 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)j[C03C](12)gtld-servers(3)net(0) Offset = 0x007e, RR count = 4 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)i[C03C](12)gtld-servers(3)net(0) Offset = 0x008e, RR count = 5 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)l[C03C](12)gtld-servers(3)net(0) Offset = 0x009e, RR count = 6 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)b[C03C](12)gtld-servers(3)net(0) Offset = 0x00ae, RR count = 7 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)e[C03C](12)gtld-servers(3)net(0) Offset = 0x00be, RR count = 8 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)a[C03C](12)gtld-servers(3)net(0) Offset = 0x00ce, RR count = 9 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)k[C03C](12)gtld-servers(3)net(0) Offset = 0x00de, RR count = 10 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)f[C03C](12)gtld-servers(3)net(0) Offset = 0x00ee, RR count = 11 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)c[C03C](12)gtld-servers(3)net(0) Offset = 0x00fe, RR count = 12 Name "[C015](3)COM(0)" TYPE NS (2) CLASS 1 TTL 172800 DLEN 4 DATA (1)m[C03C](12)gtld-servers(3)net(0) ADDITIONAL SECTION: The DNS server encountered an invalid domain name in a packet from 192.48.79.30. The packet is rejected.

1 0

how to get people to upgrade? (Re: The weak link? DNS)
by Pedro R Marques 26 Mar '03

26 Mar '03

> what i really want to talk > about is: how to get people to upgrade their software when defects > are found. > > sending out announcements through CERT and the bind-announce m/l > isn't working. Paul, I seems to me that you are assuming that the problem is not enought information gets to system admins... that may be the case in some instances, but it is my belief that the majority of the cases have to do with the fact that systems are not administered. i.e. they where setup once and there are assumed to be running without need for maintenance. imho, this is a very reasonable expectation... unfortunatly most software is not really up to what people expect in this regard. If you want to address this issue my suggestion would be to make BIND automatically update itself... and option that needs to default to ON and that can be disabled in managed systems where admins are expected to read CERT and act upon it. This is sort of the direction other systems are taking... microsoft, which is quite competent on the consumer market, tends to have this automated update tools that can be turned off but are a pain to do so. The result is imperfect but imho better than what would be without such tools. I'm afraid that BIND is currently a consumer tool also and that you should not expect an administrator to be present, even if someone was around to initialy setup the system. Pedro.

1 0

Re: how to get people to upgrade? (Re: The weak link? DNS)
by Michael.Dillon＠radianz.com 26 Mar '03

26 Mar '03

> The ISC would host a zone that would contain TXT records with > security/bug advisories for every version: I have a better idea. ISC could set up a web page that would contain security/bug advisories for every version. In order to make it easier for people to find this web page, it could be listed in various directories such as CERT. And it could also be put into search engines so that someone could go to Google, type in "BIND bug site:isc.org" and click "I'm Feeling Lucky". > yadda yadda yadda... Indeed! Let's face it folks, DNS is not the tool of choice for publishing anything other than the mapping between hostnames and IP addresses. For some things the web is better. For others LDAP is better. And for the problem that Paul mentioned at the beginning there is only one solution; the press. The fact is that Paul wants to catch the attention of people who aren't paying attention right now. He has stated that email notices don't work and we can assume that the BIND security web page, and CERT's web pages also don't work. No, there is only one thing Paul can do right now. a. Collect some realistic numbers as to the number of DNS servers that are vulnerable to the various exploits. Ask people who do network scanning to provide some statistics on what they see and scale them up according to the number of hosts in the host-count database. For example, assume that someone has scanned N hosts and discovers that N/10 are running BIND and that 40% of those are vulnerable. Also assume that the hostcount shows 20 million hosts in the world. Infer from this that there are 2 million BIND servers and that there are 800,000 vulnerable ones. But do the calculations with some real data, not my example figures. b. Write a press release with the following headline: N Thousand Internet Servers Suffer Same Fate as Iraqi Server Or maybe write some variation on this but keep it scary and keep Iraq in the headline. The body of the press release should explain the attacks that the Iraqi server was suffering from, then point out how many other servers are also vulnerable and then point out how terrorists could use these vulnerabilities against us. If you can, name some specific organizations where you know the vulnerability exists. Pick a large organization or two that has many nameservers and the vulnerability exists in some obscure corner of the organization, not on their main nameservers. They really can't sue over this because you haven't damaged them by disclosing enough detail for an attacker to use. c. release to both the national press and the computer/network trade press. That's how you get people's attention and that's also how the clueful technical people get the authority and funding to go in and fix the vulnerable boxes. As long as we continue to play games and pretend that Internet operations is still the old boys club that it once was, we will continue to suffer from these nagging issues. People on the NANOG mailing list do not run the Internet anymore. NANOG's market share of network operations people has been steadily shrinking as the Internet has grown. This may still be the moist clueful gathering place, but there are an awful lot of people out there today designing, building and operating networks, who have never heard of NANOG. There is no universal forum anymore. The Internet isn't special anymore. --Michael Dillon

3 2