Computer systems blamed for feeble hurricane response?
This is the first I've heard of this... Via The Inquirer: [snip] REPORTERS at the Wall Street Journal said they have seen documents which show that a swift response by the US federal government to Hurricane Katrina was hampered because FEMA computer servers crashed. Michael Brown, FEMA's head, resigned yesterday after being recalled by the Department of Homeland Security to Washington DC. Attempts by agencies to spur the Federal Emergency Management Agency into urgent action were met with bouncing emails, the Journal said. It quoted a Department of Health official as saying every email it had sent to FEMA staff bounced. "They need a better internet provider during disasters," the Journal quoted her or him as saying. A number of US agencies made desperate calls to the Department of Homeland Security and to Congresswomen and men, the article claimed. [Subscription required.] The newspaper did not say which computer systems FEMA uses. [snip] http://www.theinquirer.net/?article=26125 - ferg -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg@netzero.net or fergdawg@sbcglobal.net ferg's tech blog: http://fergdawg.blogspot.com/
on Tue, Sep 13, 2005 at 01:13:19PM +0000, Fergie (Paul Ferguson) quoth:
Attempts by agencies to spur the Federal Emergency Management Agency into urgent action were met with bouncing emails, the Journal said.
It quoted a Department of Health official as saying every email it had sent to FEMA staff bounced. "They need a better internet provider during disasters," the Journal quoted her or him as saying.
Does anyone know what their mail infrastructure looks like? From what I can see, they don't even have an MX record for fema.gov... -- hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com antispam news, solutions for sendmail, exim, postfix: http://enemieslist.com/
At 09:31 AM 13/09/2005, Steven Champeon wrote:
Does anyone know what their mail infrastructure looks like? From what I can see, they don't even have an MX record for fema.gov...
No MX record, and the A record for fema.gov does not accept smtp traffic. # telnet fema.gov smtp Trying 205.128.1.44... telnet: connect to address 205.128.1.44: Operation timed out telnet: Unable to connect to remote host # Then again, it might be that they use different email addresses ? @dhs.gov ?
set type=soa fema.gov Server: ns.fema.gov Address: 166.112.200.142
fema.gov origin = ns.fema.gov mail addr = root.ns2.fema.gov serial = 2005090901 refresh = 10800 (3H) retry = 3600 (1H) expire = 604800 (1W) minimum ttl = 1800 (30M) fema.gov nameserver = ns.fema.gov fema.gov nameserver = ns2.fema.gov ns.fema.gov internet address = 166.112.200.142 ns2.fema.gov internet address = 162.83.67.144 Looks Solaris'ish # telnet ns2.fema.gov smtp Trying 162.83.67.144... Connected to ns2.fema.gov. Escape character is '^]'. 220 ns2.fema.gov ESMTP Sendmail 8.11.7p1+Sun/8.11.7; Tue, 13 Sep 2005 09:49:36 -0400 (EDT) ---Mike
on Tue, Sep 13, 2005 at 09:54:42AM -0400, Mike Tancsa wrote:
At 09:31 AM 13/09/2005, Steven Champeon wrote:
Does anyone know what their mail infrastructure looks like? From what I can see, they don't even have an MX record for fema.gov...
No MX record, and the A record for fema.gov does not accept smtp traffic.
# telnet fema.gov smtp Trying 205.128.1.44... telnet: connect to address 205.128.1.44: Operation timed out telnet: Unable to connect to remote host # Then again, it might be that they use different email addresses ? @dhs.gov ?
Their "contact us" page on fema.gov lists several @fema.gov addresses, so I doubt it.
fema.gov nameserver = ns.fema.gov fema.gov nameserver = ns2.fema.gov ns.fema.gov internet address = 166.112.200.142 ns2.fema.gov internet address = 162.83.67.144
Looks Solaris'ish
# telnet ns2.fema.gov smtp Trying 162.83.67.144... Connected to ns2.fema.gov. Escape character is '^]'. 220 ns2.fema.gov ESMTP Sendmail 8.11.7p1+Sun/8.11.7; Tue, 13 Sep 2005 09:49:36 -0400 (EDT)
Well, how is any automated system supposed to find it? Sheesh. Apparently, that host accepts mail to postmaster; we'll see if it is actually delivered/read/responded to. -- hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com antispam news, solutions for sendmail, exim, postfix: http://enemieslist.com/
At 10:29 AM 13/09/2005, Steven Champeon wrote:
on Tue, Sep 13, 2005 at 09:54:42AM -0400, Mike Tancsa wrote:
Looks Solaris'ish
# telnet ns2.fema.gov smtp Trying 162.83.67.144... Connected to ns2.fema.gov. Escape character is '^]'. 220 ns2.fema.gov ESMTP Sendmail 8.11.7p1+Sun/8.11.7; Tue, 13 Sep 2005 09:49:36 -0400 (EDT)
Well, how is any automated system supposed to find it? Sheesh.
Apparently, that host accepts mail to postmaster; we'll see if it is actually delivered/read/responded to.
SOA said root.ns2.fema.gov. It might be someone actually read's roots mail ? I will cc that addr so if its read, they can see the thread at http://www.merit.edu/mail.archives/nanog/msg11505.html and perhaps comment. ---Mike
-- hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com antispam news, solutions for sendmail, exim, postfix: http://enemieslist.com/
On Tue, 13 Sep 2005, Fergie (Paul Ferguson) wrote:
It quoted a Department of Health official as saying every email it had sent to FEMA staff bounced. "They need a better internet provider during disasters," the Journal quoted her or him as saying.
A number of US agencies made desperate calls to the Department of Homeland Security and to Congresswomen and men, the article claimed.
The newspaper did not say which computer systems FEMA uses.
$ dig mx fema.gov ;; ANSWER SECTION: fima.org. 3600 IN MX 0 smtp.secureserver.net. fima.org. 3600 IN MX 10 mailstore1.secureserver.net. ;; AUTHORITY SECTION: fima.org. 3600 IN NS PARK5.secureserver.net. fima.org. 3600 IN NS PARK6.secureserver.net. [This is Godaddy and their datacenter is obviously in Arizona] $ dig fima.org [snip] $ ;; ANSWER SECTION: fema.gov. 1800 IN A 205.128.1.44 ;; AUTHORITY SECTION: fema.gov. 1800 IN NS ns.fema.gov. fema.gov. 1800 IN NS ns2.fema.gov. $ whois -h completewhois.com 205.128.1.44 [snip] Level 3 Communications, Inc. LVLT-ORG-205-128 (NET-205-128-0-0-1) 205.128.0.0 - 205.131.255.255 Federal Emergency Management Agency FEDEMERGENCY-1-18 (NET-205-128-1-0-1) 205.128.1.0 - 205.128.1.127 Note: They also have 192.206.40.0/24 (not routed), 205.142.100.0/22 (not routed), 64.119.224.0/20 (not in bgp) and 166.112.0.0/16 (announced by 2828 - XO). While its possible that L3 or XO could have been down with one of their southern links, I really dont think it would effect their Washington, DC customers. -- William Leibzon Elan Networks william@elan.net
In message <Pine.LNX.4.62.0509130618010.16184@sokol.elan.net>, "william(at)elan .net" writes:
On Tue, 13 Sep 2005, Fergie (Paul Ferguson) wrote:
It quoted a Department of Health official as saying every email it had sent to FEMA staff bounced. "They need a better internet provider during disasters," the Journal quoted her or him as saying.
A number of US agencies made desperate calls to the Department of Homeland Security and to Congresswomen and men, the article claimed.
The newspaper did not say which computer systems FEMA uses.
$ dig mx fema.gov ;; ANSWER SECTION: fima.org. 3600 IN MX 0 smtp.secureserver.net. fima.org. 3600 IN MX 10 mailstore1.secureserver.net
That's interesting -- I'm not getting that response. --Steven M. Bellovin, http://www.cs.columbia.edu/~smb
On Tue, Sep 13, 2005 at 10:08:59AM -0400, Steven M. Bellovin wrote:
In message <Pine.LNX.4.62.0509130618010.16184@sokol.elan.net>, "william(at)elan .net" writes:
;; ANSWER SECTION: fima.org. 3600 IN MX 0 smtp.secureserver.net. fima.org. 3600 IN MX 10 mailstore1.secureserver.net That's interesting -- I'm not getting that response.
Second that. Just glanced at the fema website - their contact us section lists a mixture of @dhs.gov as well as @fema.gov addresses. John
Steven M. Bellovin wrote:
In message <Pine.LNX.4.62.0509130618010.16184@sokol.elan.net>, "william(at)elan .net" writes:
not say which computer systems FEMA uses.
$ dig mx fema.gov ;; ANSWER SECTION: fima.org. 3600 IN MX 0 smtp.secureserver.net. fima.org. 3600 IN MX 10 mailstore1.secureserver.net
That's interesting -- I'm not getting that response.
Sure you will. If you dig fima.org and not fema.gov as it appears above. Fema.gov doesn't have any mx. Thanks, Christian
On 13/09/05, Steven M. Bellovin <smb@cs.columbia.edu> wrote:
$ dig mx fema.gov ;; ANSWER SECTION: fima.org. 3600 IN MX 0 smtp.secureserver.net. fima.org. 3600 IN MX 10 mailstore1.secureserver.net
That's interesting -- I'm not getting that response.
Er, who is fIma.org and were you looking for fEma.org instead? -- Suresh Ramasubramanian (ops.lists@gmail.com)
The newspaper did not say which computer systems FEMA uses.
$ dig mx fema.gov ;; ANSWER SECTION: fima.org. 3600 IN MX 0 smtp.secureserver.net. fima.org. 3600 IN MX 10 mailstore1.secureserver.net
That's interesting -- I'm not getting that response.
Sorry about that, as you could probably get from dig, I did it on fima.gov instead ... correct one is: ----------------------------------------------------------------- ;; QUESTION SECTION: ;fema.gov. IN MX ;; AUTHORITY SECTION: fema.gov. 1642 IN SOA ns.fema.gov. root.ns2.fema.gov. 2005090901 10800 3600 604800 1800 ----------------------------------------------------------------- Which indeed means they have no MX servers listed and that MAY be a problem for some mail servers (though normally mail servers are supposed to send email based on A record then). Obviously not having MX record is not considered to be good email service setup in this century and it also means if they receive too many messages and their mail server can not handle all the connections, the mail will bounce (since there is no secondary mail server to go to). -- William Leibzon Elan Networks william@elan.net
On Tuesday 13 September 2005 09:23, william(at)elan.net wrote:
Which indeed means they have no MX servers listed and that MAY be a problem for some mail servers (though normally mail servers are supposed to send email based on A record then).
Obviously not having MX record is not considered to be good email service setup in this century and it also means if they receive too many messages and their mail server can not handle all the connections, the mail will bounce (since there is no secondary mail server to go to).
Actually it is worse than that. fema.gov has an IP (205.128.1.44) which does not respond for mail so most MTA will try the IP first, meaning that most mail will fail even is ns.fema.gov or ns2.fema.gov do answer for mail. -- Larry Smith SysAd ECSIS.NET sysad@ecsis.net
william(at)elan.net wrote:
Which indeed means they have no MX servers listed and that MAY be a problem for some mail servers (though normally mail servers are supposed to send email based on A record then).
Uh, which mainstream mail server out there is ignorant enough not to send to A record?
On Tue, 13 Sep 2005, Christian Kuhtz wrote:
william(at)elan.net wrote:
Which indeed means they have no MX servers listed and that MAY be a problem for some mail servers (though normally mail servers are supposed to send email based on A record then).
Uh, which mainstream mail server out there is ignorant enough not to send to A record?
I came around windows mail server that ddnt (not exchange, some small one that I don't remember now). There are also unix php scripts that don't work properly with it. Also earlier versions of postfix did not properly retry delivery if the domain had no MX and connection to they server did not work. Other mail server may also have various types of "unusual" behavior when they see no MX. Also some servers like exim have option not to send email if there is no MX record (or rather turn off default behavior of falling back to A record if MX is not there). So having no MX server is really not such a good idea nowdays... Obviously FEMA's problems are a lot worth since ip address 205.128.1.44 is behind firewall and does not accept port 25 connections. -- William Leibzon Elan Networks william@elan.net
On Tue, 13 Sep 2005 10:39:21 EDT, Christian Kuhtz said:
william(at)elan.net wrote:
Which indeed means they have no MX servers listed and that MAY be a problem for some mail servers (though normally mail servers are supposed to send email based on A record then).
Uh, which mainstream mail server out there is ignorant enough not to send to A record?
There's no MX record for fema.gov. The *single* A record doesn't answer on port 25. And there's no mail server I know of that's on enough crack that it thinks trying the 2 NS entries is acceptable....
Valdis.Kletnieks@vt.edu wrote:
On Tue, 13 Sep 2005 10:39:21 EDT, Christian Kuhtz said:
william(at)elan.net wrote:
Which indeed means they have no MX servers listed and that MAY be a problem for some mail servers (though normally mail servers are supposed to send email based on A record then).
Uh, which mainstream mail server out there is ignorant enough not to send to A record?
There's no MX record for fema.gov. The *single* A record doesn't answer on port 25. And there's no mail server I know of that's on enough crack that it thinks trying the 2 NS entries is acceptable....
That wasn't the question, I'm well aware of the situation. But thanks for playing ;-)
$ dig mx fema.gov ;; ANSWER SECTION: fima.org. 3600 IN MX 0 smtp.secureserver.net. fima.org. 3600 IN MX 10 mailstore1.secureserver.net
That's interesting -- I'm not getting that response.
from tokyo roam.psg.com:/usr/home/randy> dig mx fema.gov. ; <<>> DiG 9.3.1 <<>> mx fema.gov. ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9180 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;fema.gov. IN MX ;; AUTHORITY SECTION: fema.gov. 1797 IN SOA ns.fema.gov. root.ns2.fema.gov. 2005090901 10800 3600 604800 1800 ;; Query time: 0 msec ;; SERVER: 202.232.15.98#53(202.232.15.98) ;; WHEN: Wed Sep 14 10:23:20 2005 ;; MSG SIZE rcvd: 74 and roam.psg.com:/usr/home/randy> doc -p -w fema.gov Doc-2.2.3: doc -p -w fema.gov Doc-2.2.3: Starting test of fema.gov. parent is gov. Doc-2.2.3: Test date - Wed Sep 14 10:23:48 JST 2005 ERROR: NS list from fema.gov. authoritative servers does not === match NS list from parent (gov.) servers ERROR: nse.algx.net. claims to be authoritative, but does not appear in NS list from authoritative servers ERROR: nsf.algx.net. claims to be authoritative, but does not appear in NS list from authoritative servers Summary: ERRORS found for fema.gov. (count: 3) Done testing fema.gov. Wed Sep 14 10:23:52 JST 200 5
On 9/13/05, Fergie (Paul Ferguson) <fergdawg@netzero.net> wrote:
Attempts by agencies to spur the Federal Emergency Management Agency into urgent action were met with bouncing emails, the Journal said.
while the lot of you can debate proper DNS records and what OS their mail server might be running, does anyone else find it highly odd and worrisome that they're sending emails to alert FEMA of a crisis, instead of, I don't know - phone calls? if I'm a federal agency and I require FEMA's resources immediately, I'm going to pick up the phone and call them; not fire off an email marked "urgent". aaron.glenn
does anyone else find it highly odd and worrisome that they're sending emails to alert FEMA of a crisis, instead of, I don't know - phone calls? if I'm a federal agency and I require FEMA's resources immediately, I'm going to pick up the phone and call them; not fire off an email marked "urgent".
Imagine the following email: I have just received a phone call from one of my constituents who was visiting friends in New Orleans. She is trapped along with 50 other people on the second floor of the Baptist Church at the corner of ABC Street and XYZ Avenue approximately a mile west of the Superdome. The nearest building with any part of it above water is the Odeon Theatre 3 blocks north of the church. They have had no water to drink for over 24 hours and they fear that some of the children and elderly are literally dying of thirst. Is there a fax number I can send this information to? What part of the above email message is it preferably to communicate by telephone instead of email? Many people choose the medium of communication based on whether or not they want a record of the information communicated. If they want the content kept secret, they use the phone. If they want the content recorded, they use email. In my hypothetical example, a politican from another state is trying to help a constituent. Naturally, being a politician, they prefer to have a record of the content. Also, the sender of the email recognizes that some of the information is important to have in written form, such as the address, distance, direction, number of blocks. Things like that can get wrongly transcribed on a voice call. This is a life or death situation so it can be argued that it is TOO IMPORTANT to risk a voice call. If only FEMA's email infrastructure was geared for emergencies... Or their web page. Or the web page of the American Red Cross. Fact is that a lot of organizations got caught with their pants down because they were not prepared to respond to an emergency and they were not prepared to use modern communications methods. Anyone who was searching for friends and relatives during the aftermath knows how chaotic it was to find information about the whereabouts of the refugees. This is a real wake-up call for all kinds of organizations, not just FEMA and not just government agencies. Could your diesel gensets cope with an extended running period like the one that DirectNIC has experienced? Do you have enough bottled water in your data center to keep critical staff ALIVE in the case of an extended emergency like this? Anyone who runs any type of critical infrastructure will have dozens of lessons to learn after analyzing the outcome of the New Orleans disaster, even moreso than the 911 commission or the Columbia accident inquiry. --Michael Dillon
On Sep 13, 2005, at 1:13 PM, Fergie (Paul Ferguson) wrote:
Attempts by agencies to spur the Federal Emergency Management Agency into urgent action were met with bouncing emails, the Journal said.
http://www.fema.gov/staff/extended.jsp Lists an "IT Services Division" that has ~250 possible points of contact. Surely one of them has some clue... :-/ I think this sort of problem shows the endemic disease currently in place at FEMA. It's not just an "IT gaffe" or firewall mistake. It's a failure much more serious, sadly. -David
participants (14)
-
Aaron Glenn -
Christian Kuhtz -
David Ulevitch -
Fergie (Paul Ferguson) -
John Kinsella -
Larry Smith -
Michael.Dillon＠btradianz.com -
Mike Tancsa -
Randy Bush -
Steven Champeon -
Steven M. Bellovin -
Suresh Ramasubramanian -
Valdis.Kletnieks＠vt.edu -
william(at)elan.net