Watching the Iraqi Ururklink and Al Jazeera over the weekend what struck me is how many different ways network administrators can mess up. Although malicious actors have been trying (and succeeding) to exploit vulnerabilities, the worst problems seem to be self-inflicted. Administrators had used firewalls and locked down their web sites, sometimes so well they couldn't handle the traffic load. But the real weak link was their DNS servers. For example, Al Jazeera had time-to-live set of their domain records set to 15 minutes, making them even more vulnerable to increasing the load on their systems. Of course, Al Jazeera had other problems too. What even stranger about the Iraqi state provider Uruklink.net is the DNS servers are now self-identifying with earlier (with known bugs) versions of BIND. Last week the Uruklink name server 62.145.94.1 was running 8.2.2-P5, but now is running 8.1.2. Although the web site for www.uruklink.net is up, DNS lookups for www.uruklink.net return various other IP addresses (not in 62.145.94.0/24). Including some addresses running web sites claiming the site is "owned." In reality, the site isn't owned, you are being redirected to a unrelated web site.
sean@donelan.com (Sean Donelan) writes:
What even stranger about the Iraqi state provider Uruklink.net is the DNS servers are now self-identifying with earlier (with known bugs) versions of BIND. Last week the Uruklink name server 62.145.94.1 was running 8.2.2-P5, but now is running 8.1.2. ...
at http://www.isc.org/products/BIND/bind-security.html we see: Name: "BIND: Remote Execution of Code" [Added 11.12.2002] Versions affected: BIND 4.9.5 to 4.9.10 BIND 8.1, 8.2 to 8.2.6, 8.3.0 to 8.3.3 Severity: SERIOUS Exploitable: Remotely Type: Possibility to execute arbitrary code. Description: When constructing a response containing SIG records a incorrect space allows a write buffer overflow. It is then possible to execute code with the privileges of named. the list goes on. i'm sure several folks will use this as an opportunity to hawk their own alternative non-BIND DNS solution, i wish you well except plz change the Subject: header on your reply since what i really want to talk about is: how to get people to upgrade their software when defects are found. sending out announcements through CERT and the bind-announce m/l isn't working. so here's a proposal. we (speaking for ISC here) could add a config option (default to OFF) to make bind send some kind of registration packet at boot time, containing an e-mail address for a technical contact for that server, and perhaps its hostname as well. the destination would be configurable, and the format would be open, and we would include in the distribution a tool capable of catching these. any campus/WAN admin who wanted to run their own "BIND registration system" could do so. anyone who wanted to simply config their server to send registration data to ISC could do so. for data received at ISC, we'd (a) keep it completely private other than public statistics, (b) clean it of obvious trash (some people will sent registration data for president@whitehouse.gov just for fun; we know that), and (c) use the contact information only in the event that a security defect discovered in that version. remember, the default would be OFF. given such a feature, whose default was OFF, would anyone here who uses BIND stop using it out of protest? if so plz answer publically (on nanog). given such a feature, would anyone here create their own registration system so they had their own database of local BIND instances on their campus/WAN, or would anyone here config their servers to send registration data to ISC? if so plz answer privately (i'll summarize to the list.) -- Paul Vixie
Personaly I'v not looked favorably at given my email to various lists, although its probably way too late and everyone by now has it... 1. I have another idea though, during setup of the server ask for email address of list administrator, but keep that on the server itself. 2. Setup some dns server that provides dns record if there are necessary updates (here is one example in reverse dns notation...: 1.2.9.bind.updates.isc.org and set it to particular ip or particular MX or whatever to show if update is needed). Possibly have this special update dns recorb be HINFO to url where more information on update is available. 3. When bind starts if the record in #2 exists and shows that update is necessary, have bind server email to address entered in 1 and with abscense of that have it email to postmaster@computername.com and if HINFO exists, it can take that and enter this as custome email. The above also makes it unnecessary for ISC to maintain that huge email database or email everybody (and probably get bunch of people angry at you for spamming...). Anyway just a thought... On 26 Mar 2003, Paul Vixie wrote:
sean@donelan.com (Sean Donelan) writes:
What even stranger about the Iraqi state provider Uruklink.net is the DNS servers are now self-identifying with earlier (with known bugs) versions of BIND. Last week the Uruklink name server 62.145.94.1 was running 8.2.2-P5, but now is running 8.1.2. ...
at http://www.isc.org/products/BIND/bind-security.html we see:
Name: "BIND: Remote Execution of Code" [Added 11.12.2002] Versions affected: BIND 4.9.5 to 4.9.10 BIND 8.1, 8.2 to 8.2.6, 8.3.0 to 8.3.3 Severity: SERIOUS Exploitable: Remotely Type: Possibility to execute arbitrary code. Description:
When constructing a response containing SIG records a incorrect space allows a write buffer overflow. It is then possible to execute code with the privileges of named.
the list goes on. i'm sure several folks will use this as an opportunity to hawk their own alternative non-BIND DNS solution, i wish you well except plz change the Subject: header on your reply since what i really want to talk about is: how to get people to upgrade their software when defects are found.
sending out announcements through CERT and the bind-announce m/l isn't working.
so here's a proposal. we (speaking for ISC here) could add a config option (default to OFF) to make bind send some kind of registration packet at boot time, containing an e-mail address for a technical contact for that server, and perhaps its hostname as well. the destination would be configurable, and the format would be open, and we would include in the distribution a tool capable of catching these. any campus/WAN admin who wanted to run their own "BIND registration system" could do so. anyone who wanted to simply config their server to send registration data to ISC could do so. for data received at ISC, we'd (a) keep it completely private other than public statistics, (b) clean it of obvious trash (some people will sent registration data for president@whitehouse.gov just for fun; we know that), and (c) use the contact information only in the event that a security defect discovered in that version. remember, the default would be OFF.
given such a feature, whose default was OFF, would anyone here who uses BIND stop using it out of protest? if so plz answer publically (on nanog).
given such a feature, would anyone here create their own registration system so they had their own database of local BIND instances on their campus/WAN, or would anyone here config their servers to send registration data to ISC? if so plz answer privately (i'll summarize to the list.)
Thinking about it again, this would have additional advantage of collecting statistics at where bind is being used (you get ips of the servers) and what versions they are running. So even if they did not update the software, you can still find out where they are by ip address and if situation is very very serious, some kind of proactive contact option would still be available. On Wed, 26 Mar 2003 william@elan.net wrote:
Personaly I'v not looked favorably at given my email to various lists, although its probably way too late and everyone by now has it...
1. I have another idea though, during setup of the server ask for email address of list administrator, but keep that on the server itself. 2. Setup some dns server that provides dns record if there are necessary updates (here is one example in reverse dns notation...: 1.2.9.bind.updates.isc.org and set it to particular ip or particular MX or whatever to show if update is needed). Possibly have this special update dns recorb be HINFO to url where more information on update is available. 3. When bind starts if the record in #2 exists and shows that update is necessary, have bind server email to address entered in 1 and with abscense of that have it email to postmaster@computername.com and if HINFO exists, it can take that and enter this as custome email.
The above also makes it unnecessary for ISC to maintain that huge email database or email everybody (and probably get bunch of people angry at you for spamming...). Anyway just a thought...
On 26 Mar 2003, Paul Vixie wrote:
sean@donelan.com (Sean Donelan) writes:
What even stranger about the Iraqi state provider Uruklink.net is the DNS servers are now self-identifying with earlier (with known bugs) versions of BIND. Last week the Uruklink name server 62.145.94.1 was running 8.2.2-P5, but now is running 8.1.2. ...
at http://www.isc.org/products/BIND/bind-security.html we see:
Name: "BIND: Remote Execution of Code" [Added 11.12.2002] Versions affected: BIND 4.9.5 to 4.9.10 BIND 8.1, 8.2 to 8.2.6, 8.3.0 to 8.3.3 Severity: SERIOUS Exploitable: Remotely Type: Possibility to execute arbitrary code. Description:
When constructing a response containing SIG records a incorrect space allows a write buffer overflow. It is then possible to execute code with the privileges of named.
the list goes on. i'm sure several folks will use this as an opportunity to hawk their own alternative non-BIND DNS solution, i wish you well except plz change the Subject: header on your reply since what i really want to talk about is: how to get people to upgrade their software when defects are found.
sending out announcements through CERT and the bind-announce m/l isn't working.
so here's a proposal. we (speaking for ISC here) could add a config option (default to OFF) to make bind send some kind of registration packet at boot time, containing an e-mail address for a technical contact for that server, and perhaps its hostname as well. the destination would be configurable, and the format would be open, and we would include in the distribution a tool capable of catching these. any campus/WAN admin who wanted to run their own "BIND registration system" could do so. anyone who wanted to simply config their server to send registration data to ISC could do so. for data received at ISC, we'd (a) keep it completely private other than public statistics, (b) clean it of obvious trash (some people will sent registration data for president@whitehouse.gov just for fun; we know that), and (c) use the contact information only in the event that a security defect discovered in that version. remember, the default would be OFF.
given such a feature, whose default was OFF, would anyone here who uses BIND stop using it out of protest? if so plz answer publically (on nanog).
given such a feature, would anyone here create their own registration system so they had their own database of local BIND instances on their campus/WAN, or would anyone here config their servers to send registration data to ISC? if so plz answer privately (i'll summarize to the list.)
On Wed, 2003-03-26 at 09:24, Paul Vixie wrote:
so here's a proposal. we (speaking for ISC here) could add a config option (default to OFF) to make bind send some kind of registration packet at boot time, containing an e-mail address for a technical contact for that server, and perhaps its hostname as well.
[...]
given such a feature, whose default was OFF, would anyone here who uses BIND stop using it out of protest? if so plz answer publically (on nanog).
I would not use such a feature, and I suspect that most people who would use such a feature would not have a clue that it was there or how to turn it on. What I would like to see is somewhat of the idea in reverse. The ISC would host a zone that would contain TXT records with security/bug advisories for every version: $ORIGIN . security-notice.bind IN SOA ns.isc.org. postmaster.isc.org. 1 7200 3600 604800 3600 $ORIGIN security-notice.bind. 8.3.3 IN TXT "Name: BIND: Multiple Denial of Service [yadda yadda yadda...]" 4.9.10 IN TXT "Name: LIBRESOLV: buffer overrun [yadda yadda yadda...]" yadda yadda yadda... Ideally the zone would be DNSSEC signed as well. Then, by default, BIND would query the zone periodically (perhaps every 24 hours or so) for it's version. If any records are found it would log a message and/or send email to root@localhost, which would be repeated periodically (I'd log a message every time that a check was performed, but I'd only email once a week). There would be config options so that the clueful admin could customize/disable this behavior to his or her liking. This way no one would be collecting a central database of email addresses, but everyone would get notified of security advisories in a timely manner. Jeff
On 26 Mar 2003, Jeffrey C. Ollie wrote: > What I would like to see is somewhat of the idea in > reverse. The ISC would host a zone that would contain TXT records with > security/bug advisories for every version: I really like this solution. It seems clean and unobjectionable, while getting the job done. -Bill
* vixie@vix.com (Paul Vixie) [Wed 26 Mar 2003, 16:24 CET]:
so here's a proposal. we (speaking for ISC here) could add a config option (default to OFF) to make bind send some kind of registration packet at boot time, containing an e-mail address for a technical contact for that server, and perhaps its hostname as well. the destination would be configurable, and the format would be open, and we would include in the distribution a tool capable of catching these. any campus/WAN admin who wanted to run their own "BIND registration system" could do so. anyone who wanted to simply config their server to send registration data to ISC could do so. for data received at ISC, we'd (a) keep it completely private other than public statistics, (b) clean it of obvious trash (some people will sent registration data for president@whitehouse.gov just for fun; we know that), and (c) use the contact information only in the event that a security defect discovered in that version. remember, the default would be OFF.
given such a feature, whose default was OFF, would anyone here who uses BIND stop using it out of protest? if so plz answer publically (on nanog).
So how much would this differ from `make install' running this shell script? --- cat <<EOF Congratulations! You have just successfully installed BIND ${VERSION}. If you want to be kept up-to-date on future announcements on BIND (for example, security vulnerabilities found in your version), say Y here. This is highly recommended! EOF read answer case "$answer" in y|Y|[Yy][Ee][Ss]) echo subscribe | mail bind-announce-request@isc.org ;; *) echo "You can still see security updates at" echo "http://www.isc.org/products/bind/" echo "Please check there regularly!" esac --- -- Niels. -- subvertise me
i see that a lot of folks are responding publically. sorry to spawn a thread. niels=nanog@bakker.net (Niels Bakker) writes:
So how much would this differ from `make install' running this shell script?
most bind installations are prefab -- the come with the operating system and there's no "make install" done after the host has a name. christian.kuhtz@BellSouth.com ("Kuhtz, Christian") writes:
Administrator inertia is the root cause. I don't see how an automatism such as the one described changes human behavior. And unless you change that inertia, no amount of notification, databases, registries, yada yada yada will make any difference.
this argues for time bombs, where the software will stop working after it detects some condition (too much time has passed, or an advertisement for newer software is seen, or a vulnerability notice is seen). this would be wildly unpopular, contrary to the open source philosophy, and never adopted. roque@sbcglobal.net (Pedro R Marques) writes:
If you want to address this issue my suggestion would be to make BIND automatically update itself... and option that needs to default to ON and that can be disabled in managed systems where admins are expected to read CERT and act upon it.
this solution implies a trust relationship between a server operator and the software provider which in fact never exists in reality. even my microsoft sysadmin friends carefully eyeball any "software update" patch before they'll put it on production iron. then there's the local customization issue -- and the binary problem, since many name server hosts do not have compilers. again this would be contrary to the open source philosophy. *** i don't want to have this be bimodal (run binaries from someone you're required to trust, or else run source and be out of date most of the time) since neither mode is interesting or useful. i do agree that other open source packages (openssl for example, or apache) would benefit from a good answer to the "how to get folks to upgrade" question. however, i'm not sure a single answer will fit all packages. having the server check for updates and issue local mail is appealing, but i'm more concerned about MIM when fetching update information than i am with simply registering package version numbers, hosts, and e-mail addresses. -- Paul Vixie
PV> Date: 26 Mar 2003 21:22:59 +0000 PV> From: Paul Vixie PV> having the server check for updates and issue local mail is This assumes the configured notification email address remains valid, and the recipient doesn't sort them into a "I'll get around to it" folder. PV> appealing, but i'm more concerned about MIM when fetching PV> update information than i am with simply registering package PV> version numbers, hosts, and e-mail addresses. Distribute BIND with public key. Updates are encrypted or signed with its counterpart. Eddy -- Brotsman & Dreger, Inc. - EverQuick Internet Division Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 (785) 865-5885 Lawrence and [inter]national Phone: +1 (316) 794-8922 Wichita ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
On Wed, 26 Mar 2003, E.B. Dreger wrote:
PV> From: Paul Vixie PV> appealing, but i'm more concerned about MIM when fetching PV> update information than i am with simply registering package PV> version numbers, hosts, and e-mail addresses.
Distribute BIND with public key. Updates are encrypted or signed with its counterpart.
But don't distributors already provide this service? Several Linux distributions (at least Redhat and Debian) and Unix companies (Sun at least) already provide [semi-]automatic updates of packages like bind. Just look at the vendor list in the average CERT notice. Someone who downloads, compiles and installs bind directly from the ISC is already indicating that they want to go beyond the safe vendor supplied version thats good-enough for 99% of people. I'm also worried about any concept of trying to "force" people to upgrade, even with bind I use some features (namely an external named-xfer program) of bind v8 that arn't available in bind v9 . For the servers which I need this on I run bind 8.3.3 (Vendor backported with the 8.3.4 fixes) of copy the named-xfer program over to the bind 9 box. -- Simon Lyall. | Newsmaster | Work: simon.lyall@ihug.co.nz Senior Network/System Admin | Postmaster | Home: simon@darkmere.gen.nz Ihug Ltd, Auckland, NZ | Asst Doorman | Web: http://www.darkmere.gen.nz
SL> Date: Thu, 27 Mar 2003 09:55:08 +1200 (NZST) SL> From: Simon Lyall SL> I'm also worried about any concept of trying to "force" SL> people to upgrade, even with bind I use some features (namely SL> an external named-xfer program) of bind v8 that arn't SL> available in bind v9 . For the servers which I need this on I I'm curious... why not use "dig @wherever zone.example axfr" instead? SL> run bind 8.3.3 (Vendor backported with the 8.3.4 fixes) of SL> copy the named-xfer program over to the bind 9 box. Agreed re the hazards of being heavy-handed. However, I'm sure there would be those who disabled the automated checkup... the question is, would they be the crowd for whom the approach in question was intended? Eddy -- Brotsman & Dreger, Inc. - EverQuick Internet Division Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 (785) 865-5885 Lawrence and [inter]national Phone: +1 (316) 794-8922 Wichita ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
Perhaps nameservers could periodically poll dig @?.root-servers.net 2.2.9.is-vuln.bind. txt chaos or something similar; I suggest using roots because DNS queries to them are far less likely to be filtered. If corresponding RR is valid (see below), shut down BIND, thus forcing admins to look into the problem. Harsh? Yes. Sadly, "it runs, so it must be correct" is far more common an attitude than "it must be correct before it's allowed to run". Worried about spoofing? Distribute the public key with BIND, and let the TXT response be encoded. Of course, the clueless generally don't compile from source. I wonder how long it would be before some vendor circumvented such controls so their userbase wouldn't be hassled with such silliness as forced critical upgrades. Eddy -- Brotsman & Dreger, Inc. - EverQuick Internet Division Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 (785) 865-5885 Lawrence and [inter]national Phone: +1 (316) 794-8922 Wichita ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
so here's a proposal. we (speaking for ISC here) could add a config option (default to OFF) to make bind send some kind of registration packet at boot time, containing an e-mail address for a technical contact for that server, and perhaps its hostname as well. the destination would be configurable, and the format would be open, and we would include in the distribution a tool capable of catching these. any campus/WAN admin who wanted to run their own "BIND registration system" could do so. anyone who wanted to simply config their server to send registration data to ISC could do so. for data received at ISC, we'd (a) keep it completely private other than public statistics, (b) clean it of obvious trash (some people will sent registration data for president@whitehouse.gov just for fun; we know that), and (c) use the contact information only in the event that a security defect discovered in that version. remember, the default would be OFF.
Isn't the problem with this that in order to get the code out, people need to upgrade and you therefor risk ending up with only notifying the people that upgrade anyway? - kurtis -
In a message written on Wed, Mar 26, 2003 at 04:09:06AM -0500, Sean Donelan wrote:
For example, Al Jazeera had time-to-live set of their domain records set to 15 minutes, making them even more vulnerable to increasing the load on their systems. Of course, Al Jazeera had other problems too.
This is very much a double edged sword. If they decided to add more servers, or Akamize, or set up traditional mirrors a long TTL would have prevented a large number of people from using them as ISP's continued to serve up cached entries. Imagine not being able to get new servers properly loaded for a week if you followed more traditional TTL guidelines. For web content, I would recomend a TTL approximately equal to the longest you would expect a single user to view your web site. Worst case, if every user had to ask your DNS servers directly, would then be one query for every web visitor. Given that there will be caching at larger ISP's and the like it will actually be even less. If you can scale your web infrastructure to serve the pages to that number of users, scaling DNS to answer one query for each one of those users should be a trivial exercise. In addition to making it easier to change the service on the fly and have those changes take effect, it also makes it easier on smaller companies that cache content. How many small ISP's or corporate caching servers keep entries around for a week or more when one person spent 10 minutes at one site? The shorter TTL allows these boxes to get rid of the junk entries much faster. Depending on your web content I'd recommend 15 minute to 1 hour TTL values. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
I can not go into details, but suffice it to say DNS was just a symptom of other events, not the problem itself. DNS TTL on the global load balancing system was at 5 seconds and DNS load never rose above trivial. ----- Original Message ----- From: "Sean Donelan" <sean@donelan.com> To: <nanog@merit.edu> Sent: Wednesday, March 26, 2003 4:09 AM Subject: The weak link? DNS
Watching the Iraqi Ururklink and Al Jazeera over the weekend what struck me is how many different ways network administrators can mess up. Although malicious actors have been trying (and succeeding) to exploit vulnerabilities, the worst problems seem to be self-inflicted.
Administrators had used firewalls and locked down their web sites, sometimes so well they couldn't handle the traffic load.
But the real weak link was their DNS servers.
For example, Al Jazeera had time-to-live set of their domain records set to 15 minutes, making them even more vulnerable to increasing the load on their systems. Of course, Al Jazeera had other problems too.
What even stranger about the Iraqi state provider Uruklink.net is the DNS servers are now self-identifying with earlier (with known bugs) versions of BIND. Last week the Uruklink name server 62.145.94.1 was running 8.2.2-P5, but now is running 8.1.2. Although the web site for www.uruklink.net is up, DNS lookups for www.uruklink.net return various other IP addresses (not in 62.145.94.0/24). Including some addresses running web sites claiming the site is "owned." In reality, the site isn't owned, you are being redirected to a unrelated web site.
On Wed, 26 Mar 2003, Matt Buford wrote:
I can not go into details, but suffice it to say DNS was just a symptom of other events, not the problem itself. DNS TTL on the global load balancing system was at 5 seconds and DNS load never rose above trivial.
Al Jazeera's main problem looks like its web site or upstream decided to drop them as a customer.
participants (11)
-
Bill Woodcock -
E.B. Dreger -
Jeffrey C. Ollie -
Kurt Erik Lindqvist -
Leo Bicknell -
Matt Buford -
Niels Bakker -
Paul Vixie -
Sean Donelan -
Simon Lyall -
william＠elan.net