- NANOG - lists.nanog.org

att.net email issues?
by Jim Popovitch 27 Jan '03

27 Jan '03

Is anyone seeing problems delivering email to att.net? I see about 50% rejected, 50% accepted. The 50% accepted never make it to the endusers. Been like this for the last 36 hours. Anyone else? -Jim P.

8 11

Banc of America Article
by Alex Rubenstein 27 Jan '03

27 Jan '03

http://biz.yahoo.com/rb/030125/tech_virus_boa_1.html Let's make the assumption that the outage of ATM's that BoA suffered was caused by last nights 'SQL Slammer' virus. The following things can then be assumed: a) BoA's network has Microsoft SQL Servers on them. b) BoA has not applied SP3 (available for a week) or the patch for this particular problem (SQL Slammer) (available for many months). c) somehow, this attack spawned on the public internet made it's way to BoA's SQL servers, bypassing firewalls (did they have firewalls?). Another article states, "Bank of America Corp., one of the nation's largest banks, said many customers could not withdraw money from its 13,000 ATM machines because of technical problems caused by the attack. A spokeswoman, Lisa Gagnon, said the bank restored service to nearly all ATMs by late Saturday afternoon and that customers' money and personal information had not been at risk." Does anyone else, based upon the assumptions above, believe this statement to be patently incorrect (specifically, the part about 'personal information had not been at risk.') ? I find these statement made by BoA, based upon assumptions which are probably correct, to be very scary. Comments? -- Alex Rubenstein, AR97, K2AHR, alex(a)nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --

10 11

Re: management interface accessability (was Re: Worm / UDP1434)
by Steven M. Bellovin 27 Jan '03

27 Jan '03

In message <20030126172907.GA31694(a)f00f.org>, Chris Wedgwood writes: > >On Sun, Jan 26, 2003 at 01:37:16AM +0000, Paul Vixie wrote: > >> ... If you are relying on their ACL's to protect your telnet and >> snmp access, but are otherwise allowing their management interfaces >> to hear traffic from the whole Internet, then you should turn in >> your badge and go back to bagging groceries or whatever it is you >> used to do. > >Some would argue this should apply to those exposing MSSQL to the >outside world such that it could even receive malicious port 1434 >packets... Therein lies the rub. I'm curious -- every medium or large company I'm aware of had Code Red on the inside of the firewalls. What happened this time? Did it get inside? If so, has anyone analyzed how? --Steve Bellovin, http://www.research.att.com/~smb (me) http://www.wilyhacker.com (2nd edition of "Firewalls" book)

9 14

RE: Banc of America Article
by Temkin, David 27 Jan '03

27 Jan '03

I think you're leaving out a very viable possibility in your summary... What if BoA took a proactive approach and shut down their SQL environment (even though none of us known conclusively if they're a SQL or Oracle shop) to verify that it was in fact clean and not compromised. When you're talking about access to billions of dollars, it's not worth taking a chance. They might have actually followed proper security protocol and verified their systems were clean before re-activating them. Just a thought. -Dave > -----Original Message----- > From: Alex Rubenstein [SMTP:alex@nac.net] > Sent: Sunday, January 26, 2003 10:59 AM > To: Ray Burkholder > Cc: nanog(a)nanog.org > Subject: RE: Banc of America Article > > > > Let me summarize, then ask a question: > > a) BoA uses the public internet for ATM transactions. The public internet > was so dead, that every one of thier ATM machines was dead for many hours, > even many hours longer than the public internet was dead. > > b) BoA uses it's own network for it's on ATM transactions. Somewhere on > the a public to private connection, a firewall wasn't doing it's job, or > there wasn't a firewall. Things were broken for a while, until they were > able to fix all thier SQL servers. > > I guess my point is, if it were a), not every ATM would be dead all the > time, and things would have been fixed in only a little while. Not many > internet 'backbones' (at least ones BoA would have used for this > application) were down as long as BoA's ATM's were. > > On the other hand, I think it's more likely that BoA had unprotected SQL > servers, and they got it. It took a long while for BoA IT people to make > it out of bed saturday morning to fix the problem. > > I still clearly say that I don't know what happened, and I did make > assumptions (as I said in the original mail) -- but I'd still place my > money on b). > > > > On Sun, 26 Jan 2003, Ray Burkholder wrote: > > > Actually, I think too many assumptions were made. > > > > Let's simplify. > > > > We know UUNet traffic capabilities were reduced significantly. Uunet > > has many big customers. Other big carriers had similar affects on their > > networks, probably particularly at peering points. > > > > We know many companies use public or private VPN services from major > > carriers such as these, and that both VPN types may use public internet > > carriers. > > > > I think therefore that the only true conclusion we could say is that if > > BoA's traffic was not prioritized, it therefore suffered collateral > > damage primarily due to traffic not being able to get through between > > ATM's and the central processing center. > > > > -- Alex Rubenstein, AR97, K2AHR, alex(a)nac.net, latency, Al Reuben -- > -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net -- > > > IMPORTANT:The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

2 1

RE: Banc of America Article
by Ray Burkholder 27 Jan '03

27 Jan '03

Actually, I think too many assumptions were made. Let's simplify. We know UUNet traffic capabilities were reduced significantly. Uunet has many big customers. Other big carriers had similar affects on their networks, probably particularly at peering points. We know many companies use public or private VPN services from major carriers such as these, and that both VPN types may use public internet carriers. I think therefore that the only true conclusion we could say is that if BoA's traffic was not prioritized, it therefore suffered collateral damage primarily due to traffic not being able to get through between ATM's and the central processing center. Ray Burkholder > -----Original Message----- > From: Alex Rubenstein [mailto:alex@nac.net] > Sent: January 25, 2003 18:45 > To: nanog(a)nanog.org > Subject: Banc of America Article > > > > > http://biz.yahoo.com/rb/030125/tech_virus_boa_1.html > > Let's make the assumption that the outage of ATM's that BoA > suffered was > caused by last nights 'SQL Slammer' virus. > > The following things can then be assumed: > > a) BoA's network has Microsoft SQL Servers on them. > > b) BoA has not applied SP3 (available for a week) or the > patch for this > particular problem (SQL Slammer) (available for many months). > > c) somehow, this attack spawned on the public internet made > it's way to > BoA's SQL servers, bypassing firewalls (did they have firewalls?). > > Another article states, "Bank of America Corp., one of the nation's > largest banks, said many customers could not withdraw money from its > 13,000 ATM machines because of technical problems caused by > the attack. A > spokeswoman, Lisa Gagnon, said the bank restored service to nearly all > ATMs by late Saturday afternoon and that customers' money and personal > information had not been at risk." > > Does anyone else, based upon the assumptions above, believe > this statement > to be patently incorrect (specifically, the part about 'personal > information had not been at risk.') ? > > I find these statement made by BoA, based upon assumptions which are > probably correct, to be very scary. > > Comments? > > > -- Alex Rubenstein, AR97, K2AHR, alex(a)nac.net, latency, Al Reuben -- > -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net -- > > >

5 5

Root servers
by Peter Salus 27 Jan '03

27 Jan '03

vixie wrote: >I did DNS queries of all of the root servers several times throughout the >event. Other than G, I was eventually able to get responses from all of >them. I'd like to report that I did at least one DNS query on Sunday morning from three different machines around the country. My results were identical to Paul's. Only g didn't respond. Peter ----------------------------------------------------------- Peter H. Salus Chief Knowledge Officer, Matrix NetSystems Ste. 300 5001 Plaza on the Lake Austin, TX 78746 +1 512 697-0613 -----------------------------------------------------------

1 0

Possible source of worm..
by Len Rose 27 Jan '03

27 Jan '03

According to the South China Morning Post: http://www.scmp.com "Roy Ko Wai-tak of the Hong Kong Computer Response Team says his team is tracing the source of the Sapphire worm, which is believed to have come from Asia, maybe even Hong Kong." - Ricky Chang I can't get access to the news article, seemingly you have to have premium access.

2 1

Re: Tracing where it started
by George Bakos 27 Jan '03

27 Jan '03

Graphs of our observances are available at: http://people.ists.dartmouth.edu/~gbakos/sapphire Here's the earliest port 1434 probe that I find. Localtimes are EST. Pay no attention to the port 123 business; I like to include ntp with my dumps to facilitate correlation: [root@bunta hpot]# tcpslice 1041153985 1041154648 ../tcpdump.1041060689 | tcpdump -ttttnr - port 1434 or port 123 or port 53 12/29/2002 09:26:25.248240 140.162.8.25.123 > 64.222.84.217.123: v4 server strat 2 poll 10 prec -16 (DF) [tos 0x10] 12/29/2002 09:37:23.203055 216.150.155.11.53 > 64.222.84.217.1434: [|domain] And the dump: 12/29/2002 09:37:23.203055 216.150.155.11.53 > 64.222.84.217.1434: [|domain] 4500 0021 c8ef 0000 7b11 6d83 d896 9b0b 40de 54d9 0035 059a 000d eeab 0200 0000 00 I ran through packet logs from several networks starting Dec 1. This is the earliest I can find. As indicated above, there was certainlny no prior dns request. Just for poops & snickers, let's have a peek at 216.150.155.11, shall we? NetRange: 216.150.150.0 - 216.150.157.255 CIDR: 216.150.150.0/23, 216.150.152.0/22, 216.150.156.0/23 NetName: EASYCGI-150-157 NetHandle: NET-216-150-150-0-1 Parent: NET-216-150-128-0-1 NetType: Reassigned NameServer: NS1.EASY-CGI.COM NameServer: NS2.EASY-CGI.COM Comment: RegDate: 2002-06-19 Updated: 2002-08-08 [gbakos@lt1 gbakos]$ nc 216.150.155.11 80 GET / HTTP/1.0 HTTP/1.0 404 Not Found Server: Microsoft-IIS/5.0 Date: Mon, 27 Jan 2003 03:38:32 GMT Content-Type: text/html Content-Length: 111 Age: 440 X-Cache: HIT from bunta.alpinista.dyndns.org Connection: close <html><head><title>Site Not Found</title></head> <body>No web site is configured at this address.</body></html> Why doesn't this surprise me? Anyone want to run this guy down and apply the "sucker rod" section of syslogd(8) ? On Sun, 26 Jan 2003 09:11:11 -0800 John Sage <jsage(a)finchhaven.com> wrote: > Tom et al: > > On Sat, Jan 25, 2003 at 09:59:42PM -0500, tom glaab wrote: > > Johannes Ullrich wrote: > > > > >wow... excellent catch. here is some data I have: > > > > > > > Hmmm... > > > > I first see 67.8.33.179 on 20 January: > > <snippage> > > > But found my first (and only, prior to 20 Jan) hits on udp/1434 much > > earlier: > > > > Dec 29 06:32:36 chopper kernel: Packet log: input DENY eth0 PROTO=17 > > 12.10.144.249:53 x.y.z.83:1434 L=33 S=0x00 I=38557 F=0x0000 T=108 (#1303) > > Dec 29 06:32:36 chopper kernel: Packet log: input DENY eth0 PROTO=17 > > 12.10.144.249:53 x.y.z.84:1434 L=33 S=0x00 I=63999 F=0x0000 T=108 (#1303) > > Dec 29 06:32:36 chopper kernel: Packet log: input DENY eth0 PROTO=17 > > 12.10.144.249:53 x.y.z.85:1434 L=33 S=0x00 I=12853 F=0x0000 T=108 (#1303) > > Dec 29 06:32:36 chopper kernel: Packet log: input DENY eth0 PROTO=17 > > 12.10.144.249:53 x.y.z.86:1434 L=33 S=0x00 I=61180 F=0x0000 T=108 (#1303) > > I'm betting the dlen=33 is this: > > Generated by ACID v0.9.6b21 on Sun January 26, 2003 08:53:59 > ------------------------------------------------------------------------------ > #(458 - 93) [2002-10-16 13:16:44] UDP inbound to 1434 MS SQL monitor > IPv4: 217.226.25.204 -> 12.82.130.126 > hlen=5 TOS=0 dlen=33 ID=1541 flags=0 offset=0 TTL=115 chksum=48968 > UDP: port=53 -> dport: 1434 len=13 > Payload: length = 5 > > 000 : 02 00 00 00 00 ..... > ------------------------------------------------------------------------------ > #(524 - 103) [2002-11-20 01:03:39] UDP inbound to 1434 MS SQL monitor > IPv4: 80.128.175.135 -> 12.82.141.35 > hlen=5 TOS=0 dlen=33 ID=57947 flags=0 offset=0 TTL=116 chksum=51955 > UDP: port=53 -> dport: 1434 len=13 > Payload: length = 5 > > 000 : 02 00 00 00 00 ..... > ------------------------------------------------------------------------------ > > This is all I've got with src port = 53 AND dst port = 1434 > > > - John > -- > Has the preparation > of your heart been ready? > Almost, calm down. > > PGP key: http://www.finchhaven.com/pages/gpg_pubkey.html > Fingerprint: C493 9F26 05A9 6497 9800 4EF6 5FC8 F23D 35A4 F705 -- George Bakos Institute for Security Technology Studies Dartmouth College gbakos(a)ists.dartmouth.edu voice 603-646-0665 fax 603-646-0666 Key fingerprint = D646 8F91 F795 27EC FF8B 8C95 B102 9EB2 081E CB85

1 0

official ISC statement concerning nonimpact on f-root from sql worm
by Paul Vixie 27 Jan '03

27 Jan '03

someone wrote me privately and asked: > Hi, Paul - Some of the articles on the 1434 worm talk about > 5 or more root nameservers being down. a lot of people are pretty confused about the data they have available; sometimes it's because they have the wrong data, and sometimes it's because they don't know how to interpret it properly. > Do you know if this was just sheer volume, ... all of the root name servers i monitor (which is, well, i guess, all of them) looked like they were "up" the whole time of the ms-sql flash worm. latency and jitter were a bit wider than normal, but there was no packet loss at all. (note: icmp is a poor test; some roots won't answer "ping".) looking at rob thomas's graphs tends to confirm that this was generally true for most people who monitor the roots, no matter what their point of "view". in the f-root case, our traffic volume went DOWN by a little bit during the ms-sql flashworm, which corresponded to a number of bgp session resets from our peers. my assumption so far is that a lot of the normal root service traffic bound for "f-root" didn't get here due to OPN (which is "other people's networks" -- the bane of all reliability goals and programmes.) > ... or were some of the routers providing connectivity to them > actually allowing in UDP 1434? > > I'd expect that, given the variety of different attacks on them, > at least most of them would block anything except DNS requests > except from approved locations, and preferably do so at the ISP routers > instead of whatever access lines they're on.... at f-root we don't generate "icmp port unreachable" when non-53 udp is heard -- it's a difference between "deny" and "reject" in freebsd's ipfw and ip6fw subsystems. (most of the f-root hosts are now better able to drop this traffic than their upstream routers are, due to hardware ages.) i'd like to know who it is that couldn't reach 5 of the 13 servers, so if anyone on nanog@ has heard that, please tell me what you heard and who you heard it from so i can get to the bottom of it.

3 2

UDP/1434 --- request for raw data and/or logs
by Chris Wedgwood 27 Jan '03

27 Jan '03

Hi, I wanted to create some nice pretty graphs of the amount of the recent UDP/1434 activity from my logs, I figured with 26k events and only a tiny trickle still coming this should be easy... Sadly it is not, the number of events I have logged very quickly rises from 0 to about 24 packets per (30-second) interval in less than a minute or so. Once at around 24 packets/interval it holds fairly steady gently declining... Might someone else out there have better data to work with? I ideally want data from *before* the event started, not once it was underway... ... replies of the list to cw(a)f00f.org please! Thanks!, --cw

1 0