August 2001 - NANOG - lists.nanog.org

RE: trapdoor.merit.edu and other impatient Postfix mailers everywhere (fwd)
by Larry Sheldon 02 Aug '01

02 Aug '01

> I think MAPS managed to DOS themselves. Since they are refusing queries > BIND can't cache the response and so it just keeps trying with every new > e-mail. The traffic hitting MAPS servers has to be impressive. If I were in a bad mood I'd see some poetry in that. Instead, I'll say that I think we have removed their stuff from our sendmail.cf files. But there are no doubt others in our address space that I don't know about. So. Somebody send me the IP addresses of their servers, I'll spend a couple of ACL slots and block-and-log accesses to their servers to protect them and to identify folk here that need to be told. Yes, I know I could look them up. After dinner I might. -- -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- . . - L. F. (Larry) Sheldon, Jr. - . Unix Systems and Network Administration . - Creighton University Computer Center-Old Gym - . 2500 California Plaza . - Omaha, Nebraska, U.S.A. 68178 Two identifying characteristics - . lsheldon(a)creighton.edu of System Administrators: . - 402 280-2254 (work) Infallibility, and the ability to - . 402 681-4726 (cellular) learn from their mistakes. . - 402 332-4622 (residence) - . http://www.creighton.edu/~lsheldon Adapted from Stephen Pinker . -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-

1 0

trapdoor.merit.edu and other impatient Postfix mailers everywhere
by kai＠pac-rim.net 02 Aug '01

02 Aug '01

If you are receiving larger numbers of reports about failing mail transport from your customers these days, here is why. Apparently, the nameservers for the blackholes.mail-abuse.org and dialups.mail-abuse.org zones are overloaded (I get <10% ICMP_echo's back, and virtually never any DNS answers), causing a more than 60-second connection-established -> 220 Hello banner answering delay on typical Unix-based Sendmail (and other SMTP) servers with default DNS resolver timeouts. This by itself should not pose a problem given that RFC 1123 5.3.2 stipulates a 5-minute timeout for the banner, but it appears that a SIGNIFICANT number of mailers out there are losing their patience after only about 60s: the postfix mailer running the outgoing NANOG-L mail (trapdoor.merit.edu) is one of them (guess how I found out, heh) Upon closer examination, I find thousands of these mailers, all suddenly appearing with lots of Null-connections in my sendmail logs: Aug 2 12:08:49 sonet sendmail[26668]: NOQUEUE: Null connection \ from postfix(a)trapdoor.merit.edu [198.108.1.26] While I am not sure that the non-responsiveness of the mail-abuse.org DNS servers for their "subscription-only" query zones is intentional (how do you shed traffic coming from 1000's of sites that you no longer wish to serve?), I am just amazed at the wide proliferation of blazingly RFC 1123-violating implementations/configurations of mail servers around: why for once, do none of these servers get MORE patient (after deciding for the first time that 2MB for 60s of their precious server RAM oughta be enough of their resources wasted for a delivery attempt for a particular mail to a particular host) with a host that is not answering 'fast enough' for them? Have people forgotten the robustness principle and no longer feel responsible in any shape or form ? (see RFC 1123, 1.2.1) : A vendor who develops computer communication software for the Internet protocol suite (or any other protocol suite!) and then fails to maintain and update that software for changing specifications is going to leave a trail of unhappy customers. The Internet is a large communication network, and the users are in constant contact through it. Experience has shown that knowledge of deficiencies in vendor software propagates quickly through the Internet technical community. E.g.: this is not MAPS' fault. This is large site's (Hello Yahoo!) SMTP MTAs (and excuse me for not having researched the shipped default timeouts for Postfix here, I am not blaming Postfix or any particular SMTP MTA here, as administrators tend to have their hands too deeply in the config files) screwing it up for all of us: They are mistakenly thinking that, (while violating RFC 1123 that was designed with interoperability and stability in mind, not max. profit margins) defining an arbitrarily small number of resources for their flawed business model does not have consequences beyond their own service. MAPS's changing server arrangements just happen to be the coincidential "contributing failure" here, but the true cause is apparently marketing, financial and other blithering idiots at the wheel at Yahoo, Verizon, Flonetwork (and 1000's of other dot-coms) making resource decisions over the heads, and beyond any sane reason, of responsible technical personnel that knows better than them what will fly and what won't, and why. And remember: when mail breaks, your phones don't stop ringing. bye,Kai -- "Just say No" to Spam Kai Schlichting New York, Palo Alto, You name it Sophisticated Technical Peon Kai's SpamShield <tm> is FREE! http://www.SpamShield.org | | LeasedLines-FrameRelay-IPLs-ISDN-PPP-Cisco-Consulting-VoiceFax-Data-Muxes WorldWideWebAnything-Intranets-NetAdmin-UnixAdmin-Security-ReallyHardMath

6 5

Re: MicroSoft amplification?
by Tony Rall 02 Aug '01

02 Aug '01

I see it too, on that address and on the second of the 3 addresses mapped to www.microsoft.com (the third address doesn't respond at all). Most likely this is due to a not very smart load distribution system. I suspect each of these addresses really front-ends about 10 web servers. The load distributor doesn't know what to do with ICMP packets so it sends them to all of the servers (and they all respond, in the case of ICMP echo). This probably makes PMTUD work a lot better, but it sucks for ICMP Echo. (I wonder if all Akamai setups are so affected.) Tony Rall So with all the noise about Code Red, and in the process of trying to recover from various attacks, I happened to try to ping www.microsoft.com. It's probably par for the course that this happens: Wed Aug 1 14:05:29 bross@ogre:~ $ ping www.microsoft.com PING www.microsoft.akadns.net (207.46.197.100): 56 data bytes 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=37.5 ms 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=41.2 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=42.8 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=43.9 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=45.0 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=46.1 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=47.3 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=48.4 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=49.5 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=1 ttl=45 time=57.6 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=39.8 ms 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=41.4 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=42.7 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=43.3 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=44.4 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=45.5 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=46.8 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=47.9 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=49.0 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=2 ttl=45 time=51.6 ms (DUP!) 64 bytes from 207.46.197.100: icmp_seq=3 ttl=45 time=39.6 ms I find it interesting and almost amusing that MicroSoft's own web server can be used for amplification attacks. -- Brandon Ross 404-522-5400 EVP Engineering, NetRail http://www.netrail.net AIM: BrandonNR ICQ: 2269442 Read RFC 2644!

4 3

RE: trapdoor.merit.edu and other impatient Postfix mailers everyw here
by Roeland Meyer 02 Aug '01

02 Aug '01

> From: kai(a)pac-rim.net [mailto:kai@pac-rim.net] > Sent: Thursday, August 02, 2001 10:23 AM Hello Kai, Having many of the dot-coms, that you blast so thouroughly here, as clients ... > E.g.: this is not MAPS' fault. This is large site's (Hello Yahoo!) > SMTP MTAs (and excuse me for not having researched the shipped default > timeouts for Postfix here, I am not blaming Postfix or any particular > SMTP MTA here, as administrators tend to have their hands too deeply > in the config files) screwing it up for all of us: > > They are mistakenly thinking that, (while violating RFC 1123 that > was designed with interoperability and stability in mind, not max. > profit margins) defining an arbitrarily small number of resources > for their flawed business model does not have consequences beyond > their own service. Read first part "administrators ... hands too deeply ...", then read second part "abitrarily small ...", then realize that most of their bosses don't even know enough to tell them "what" to do, let alone "how". Rather, they trust then to know what they're doing as they give them the goals. As an example; I had one client complain that their office LAN was always short on disk space. They couldn't understand it since they had just installed 250 GB of RAID. It turns out that they had a BOFH over there that was doling out DASD, 100 MB at a time (Quotas), for a 25-man office. The BOFH was running Gnutella and warez on the "unused" space. This is a page right out of the BOFH book. The BOFH is gone, the quota system is disabled, and they hired a Linux kid as SysAdmin. They now have sufficient space. The kid sometimes has to figure things out, but he isn't going on any power-trips anytime soon. ... an adequate trade-off. > MAPS's changing server arrangements just happen to be the > coincidential "contributing failure" here, but the true cause is > apparently marketing, financial and other blithering idiots at > the wheel at Yahoo, Verizon, Flonetwork (and 1000's of other > dot-coms) making resource decisions over the heads, and beyond > any sane reason, of responsible technical personnel that knows > better than them what will fly and what won't, and why. Often, it is the SysAdmins, not the business folk that make these decisions. Often without the business folk fully realizing what's going on.

1 0

Code Red host list
by John Kristoff 02 Aug '01

02 Aug '01

The following is a list of about 18,400 host IPs that are thought to be or were infected by the second round of the recent Code Red worm. Again, I hope this is helpful. I unfortunately missed some data and we continue to see new IPs continue to come in, but hopefully this is a good start. As usual, more complete monitoring would probably have turned up significantly more. http://condor.depaul.edu/~jkristof/codered-hosts.txt This is being sent to multiple people/lists in order to ensure that it reaches all interested parties. Apologies to those that may receive this note more than once. John

1 0

Re: Code Red growth stats
by Sean Donelan 02 Aug '01

02 Aug '01

On Wed, 01 August 2001, k claffy wrote: > also by AS and per country as of 20:00 GMT > http://worm-security-survey.caida.org/AS_summary.txt So much for China's claim they didn't have any infected computers. It looks like the worm doesn't discriminated based on political system.

2 1

Re: Code Red growth stats
by Steven M. Bellovin 02 Aug '01

02 Aug '01

In message <20010801190627.A7553(a)caida.org>, k claffy writes: > >hmm, not sure about that, smb. > >albeit crippled caida monitor (we're working on it), >it does seem to have reversed slope again: >http://www.caida.org/analysis/security/code-red/aug1-live-hosts.gif Fascinating; thanks. SANS hasn't updated their plots lately, so I can't compare. Anyone else with any data to post? (On the other hand -- any chance that the dip recorded at CAIDA is due to the measurement problems?) If it has indeed turned up again, I'm at a loss to explain it. While I'm sure there are some IIS servers on home machines, I doubt there are that many. But I don't have another explanation to offer. --Steve Bellovin, http://www.research.att.com/~smb

7 6

Re: Code Red growth stats
by Larry Sheldon 02 Aug '01

02 Aug '01

I did a poor job of formating this message. Somebody else said this (Idon't remember who at the moment): > Indeed. I've seen 1215 probes since the start of August, and a rough glance > shows something like 30% or more are dialups, cable modems and DSL lines. > Better than 50% appear to be addresses without INADDR. I said this: > Here too I think a sociologist might be useful (by the way I am not a > sociologist, I disagree with some sociology, but I have lunch with > one or more nearly every week). > > What is likely to be the population characteristic most common among > xDSL connected folk? > > I think it is small [home|business|shop] relatively unsophisticated, > poorly staffed and more likely to have all the bells and whistles on. > > And maybe the most frequently occuring thing in the population. I am sorry to have stolen somebody else's words--I did not mean to do that.

1 0

Re: Code Red growth stats
by Larry Sheldon 02 Aug '01

02 Aug '01

Indeed. I've seen 1215 probes since the start of August, and a rough glance shows something like 30% or more are dialups, cable modems and DSL lines. Better than 50% appear to be addresses without INADDR. Here too I think a sociologist might be useful (by the way I am not a sociologist, I disagree with some sociology, but I have lunch with one or more nearly every week). What is likely to be the population characteristic most common among xDSL connected folk? I think it is small [home|business|shop] relatively unsophisticated, poorly staffed and more likely to have all the bells and whistles on. And maybe the most frequently occuring thing in the population.

1 0

Re: MicroSoft amplification?
by Tony Rall 02 Aug '01

02 Aug '01

On Thursday, 2001/08/02 at 00:49 AST, Valdis.Kletnieks(a)vt.edu wrote: > On Wed, 01 Aug 2001 16:26:55 PDT, Tony Rall <trall(a)almaden.ibm.com> said: > > echo). This probably makes PMTUD work a lot better, but it sucks for ICMP > > Or totally horques it up entirely if the actual data path used has a > different PMTU. No way this will work if 9 paths are clean and one > requires a frag. ;) > > I won't discuss what to do if you get back 10 FRAG NEEDED packets, with > differing frag sizes ;) That's not the way it works. You've got a load-balancing-system (LBS) front-ending (using a single IP address) a cluster 10 web servers. A client on the other side of the LBS initiates an 80/tcp connection. The LBS directs it to (let's say) server 6. Once data starts flowing on the connection, nothing interesting happens until server 6 sends a large packet to the client with (as on all of its packets) the Don't-frag flag on. That packet reaches a link with a smaller mtu. The router on that link returns (to the server complex) and ICMP unreachable, fragmentation needed packet (type 3, code 4). That ICMP reaches the LBS; it has to decide what to do with it. Some LBSs will just discard any ICMP packets addressed to the cluster. The one used by MS instead forwards it to all the back-end servers. The servers that don't have a session with the client may just discard the ICMP packet (or they may simply update info in their routing table (I know that Aix does that)). The server that does have a session with the client will repackage his data packet (per the newly learned mtu) and send the smaller packet. The path between the chosen server to the client is no more ambiguous than any other PMTUD situation. Which is to say that, yes, the path could change from packet to packet, but that isn't brought on by the presence of the LBS, it's just a shortcoming of the PMTUD mechanism. In fact outbound traffic from the clustered servers often doesn't even go through the LBS. (Note that if the client is also using PMTUD and happens to send a large enough packet to trigger it, the only ICMP unreachable sent would be towards the client. Even if the link mtu causing the unreachable is on the server side of the LBS, there will be only one unreachable sent - no ambiguity at all.) (Also note that it isn't necessary for an LBS to forward all ICMPs to make PMTUD work. It just has to forward the unreachable, fragmentation needed packets. And it doesn't have to forward those to all the back-end servers. There is enough info in the unreachable message to determine which connection this ICMP message relates to - the 80/tcp connection between the client and server 6. So the LBS could know that this ICMP only has to be forwarded to server 6. I don't know of any LBS that is smart enough to handle it this way.) Tony Rall

2 1