- Test - lists.nanog.org

Statistical Games Providers Play (RE: availability and resiliency)
by Sean Donelan 30 Sep '00

30 Sep '00

Whenever providers start throwing numbers around, you've lost the battle. I always suggest people should talk to their insurance agents, not their technical people. Insurance agents are very good at understanding risk, and how much to spend mitigating that risk. Sometimes it is cheaper to buy a new computer every few years than trying to build the perfect protection around it. The used car dealer is almost never the best source of information about car insurance. Don't expect any better from a sales person at a provider. You are going to end up with expensive undercoat and fabric protection package. As far as I know, the system with the highest publicly stated reliability and availability is FEDWIRE. Fedwire exceeds everything I've seen at NORAD, NASA, or any service provider (carrier, internet, web hosting, etc). Fedwire has five-way redundancy of some systems. It also has the full faith and credit of the US Treasury backing up its service guarantee. A software error in 1985 resulted in a $23 billion (with a "B") accounting imbalance. If I take Fedwire as the upper limit, I need to ask what about providers whose claims exceed those delivered by Fedwire? They aren't lying, but you need to understand the numbers. And if any provider does think their system exceeds Fedwire, I would love a tour of your facility. Due to their history as a regulated monopoly, telephone companies have developed interesting ways to calculated reliability. For example, some telephone companies ignore events which exceed the design parameters of the network. Or in other words, they don't include Mother's Day in their calculations. Some telephone companies also don't include disruptions due to Acts Of God or Force Majure in their reported numbers. I chuckle whenever I hear someone say "carrier-grade." Availability statistics are much like flood and storm statistics. A once in 100 year flood does not mean it will flood only once in any 100 year period. You can have back to back floods. And you can have back to back computer failures. Nor does it limit the length of an outage. You could have a 43 minute failure in Year 1, and no failures in Years 2-5. Or an 86 minute failure in Year 1 and no failures in Years 2-10. Or even a 86 minute failure in Year 1, and a 86 minute failure in Year 2, and no failures in Years 3-20. Remember in statistics when you calculated the series to infinity. If you are still around in Year Infinity, then you can discuss X 9's of availability. Asking a provider how many 9's of reliability they provide, or the MTBF of their systems is really a red herring. What you really want to know, and what you should ask is When a failure does occur (and it will): how will you respond? how will you keep me informed? what do I need to do? and after you understand those answers how often would I expect this? No matter how many 9's you have, there is always a .1, 01, .001, .0001, etc chance. Murphy is exceedingly good at his job. Ok, if you are still reading, and you still want to build a system as reliable as Fedwire, lets talk. Fedwire has shown it can be done, however expect to pay as much as Fedwire. On the other hand, if you are willing to settle for just a little less, the price drops dramatically. Its a lot cheaper to build a system to meet a certain level of design risk, and buying insurance to cover the excess. It may double the price to add another "9" of reliability, but only 10% to cover the risk with insurance. I am not a lawyer, banker, insurance agent, doctor, or indian chief. You should always consult a licensed professional for advice.

2 1

Re: Disabling QAZ (was Re: Port 139 scans)
by Ben Browning 29 Sep '00

29 Sep '00

At 05:02 PM 9/29/00 -0400, Dana Hudes wrote: >I am willing to scrap together a script to shutdown >the virus on an infected machine and put it in a CGI web page. Well, that solves the problem until the reboot. After that, the registry key opens that puppy right back up. The trick is to gut it COMPLETELY. This virus supposedly supports three commands : upload, run and quit. I can't get upload to work, and I lost the manpage(ha, ha). It is possible to upload a file (perhaps compiled c?) that rips out the registry entry and renames the appropriate files on reboot. In fact, one could (legality aside) write up the program to use QAZ as the delivery mechanism for its own death. There's something poetic about that... I have a copy of the worm zipped here- if you'd like it drop me a private email. >I'm not sure about volume but initially I think I can host >it. In the event my 1Mbit connection is overwhelmed I'll > need another place.... >What stops me at the moment is that I have no authorization > to test against any infected machine. >I need a target. I'd offer mine, but I have it isolated. >I'm willing to also try for making the connection to the >share and removing the infection but I'm not sure I can >get it in time. >At least a shutdown page would do something. Half measures merely delay the inevitable- I believe it is best to expunge it right off the bat and never have to deal with the recurrences. >I will start writing my code and await direct e-mail >with authorization and a target IP address to test against. >Note that I have plenty of potential test targets in my >Samba logs :-( but no legal authority to connect to those machines. My current thought is to simply put up a .reg and .bat file up on the web, with instructions on how to use it. Run the .reg to kill the registry key, and run the .bat file to rename the files after the reboot. Of course, it may be easier to simply have a standard email explaining the virus and the removal procedure (my current solution, if anyone wants a copy of the email drop me a line). I will stick with this approach unless the script fully removes (as opposed to temporarily disabling) the virus. Another interesting note- the virus will not allow your computer to reboot if someone is connected to the telnet port. On a side note, if anyone knows a good logfile parsing perl script that pulls out all the IP addresses in a log, I'd love a copy. I have one, but it is very clunky and I daresay a better perl coder than I has tackled this issue. I only ask because this worm has increased the number of other peoples(variously formatted) logfiles in my inbox by about 900%. :) --- Ben Browning <benb(a)oz.net> oz.net Network Operations Tel (206) 443-8000 Fax (206) 443-0500 http://www.oz.net/

1 0

RE: availability and resiliency
by Roeland M.J. Meyer 29 Sep '00

29 Sep '00

> >Hosts meeting three nines, or better, typically have redundant power > >supplies and integrated UPS, bootable RAID for the OS, > redundant NICs, > >and SMP CPU configurations. > > um...is an smp cpu configuration really going to help your uptime? or > are there operating systems or hardware out there that can say to > themselves "hmph! cpu 2 seems not to be working correctly...i'd > better spin it down." That is a natural function these days. Fail-safe to the "off" state. Most SMP OS's can recognise when one, out of an SMP CPU set, goes down. > just for fun a few years back i decided to check if the sun e4000 we > had had hot-swappable cpus (i figured it didn't, but why not try it?) > and i pulled one of the boards. it didn't like it too much. None of what I claimed is required to be hot-swappable to get 99.9%. Enough online hot-spares will keep the system up long enough so that you can replace the entire box. 99.9% allows over 8 hours per year outage. You should be able to swap out a host in less than 0.5 hours.

1 0

RE: Disabling QAZ (was Re: Port 139 scans)
by Roeland M.J. Meyer 29 Sep '00

29 Sep '00

Just like they probably don't know that they're infected, they probably won't know that they've been disinfected. At least the first time. > -----Original Message----- > From: Dana Hudes [mailto:dhudes@hudes.org] > Sent: Friday, September 29, 2000 2:03 PM > To: Dan Hollis; nanog(a)merit.edu > Subject: Re: Disabling QAZ (was Re: Port 139 scans) > > > > I am willing to scrap together a script to shutdown the virus > on an infected machine and put it in a CGI web page. > I'm not sure about volume but initially I think I can host > it. In the event my 1Mbit connection is overwhelmed I'll need > another place.... > What stops me at the moment is that I have no authorization > to test against any infected machine. > I need a target. > I'm willing to also try for making the connection to the > share and removing the infection but I'm not sure I can get > it in time. > At least a shutdown page would do something. > I will start writing my code and await direct e-mail with > authorization and a target IP address to test against. > Note that I have plenty of potential test targets in my Samba > logs :-( but no legal authority to connect to those machines. > > ----- Original Message ----- > From: "Dan Hollis" <goemon(a)sasami.anime.net> > To: <nanog(a)merit.edu> > Sent: Friday, September 29, 2000 4:42 PM > Subject: Re: Disabling QAZ (was Re: Port 139 scans) > > > > > > On Fri, 29 Sep 2000, John Fraizer wrote: > > > On Fri, 29 Sep 2000, Dan Hollis wrote: > > > > It would be cool if someone would make a tool that > would auto-disinfect > > > > users... > > > Yep. The problem with that is that current laws on the > books (in the US > > > at least) make this an illegal solution. If memory > serves me correctly, > > > the one I'm thinking about is worded something like: > > > "...any person who without authorization, accesses, > modifies, deletes or > > > destroys..." > > > > A web page that users themselves must click "OK, disinfect > me"? Seems > > authorization enough to me... > > > > > The penalties are pretty stiff too. The best of > intentions don't negate > > > the fact that it's illegal. > > > > When the user initiates the disinfection themselves? > > > > -Dan > > > >

3 2

RE: Disabling QAZ (was Re: Port 139 scans)
by Roeland M.J. Meyer 29 Sep '00

29 Sep '00

I never argued that. > -----Original Message----- > From: Dana Hudes [mailto:dhudes@hudes.org] > Sent: Friday, September 29, 2000 2:43 PM > To: Roeland M.J. Meyer > Cc: Dan Hollis; nanog(a)merit.edu > Subject: RE: Disabling QAZ (was Re: Port 139 scans) > > > that does not give me authority to connect to stranger's PCs > > On Fri, 29 Sep 2000, Roeland M.J. Meyer wrote: > > > Just like they probably don't know that they're infected, > they probably > > won't know that they've been disinfected. At least the first time. > > > > > -----Original Message----- > > > From: Dana Hudes [mailto:dhudes@hudes.org] > > > Sent: Friday, September 29, 2000 2:03 PM > > > To: Dan Hollis; nanog(a)merit.edu > > > Subject: Re: Disabling QAZ (was Re: Port 139 scans) > > > > > > > > > > > > I am willing to scrap together a script to shutdown the virus > > > on an infected machine and put it in a CGI web page. > > > I'm not sure about volume but initially I think I can host > > > it. In the event my 1Mbit connection is overwhelmed I'll need > > > another place.... > > > What stops me at the moment is that I have no authorization > > > to test against any infected machine. > > > I need a target. > > > I'm willing to also try for making the connection to the > > > share and removing the infection but I'm not sure I can get > > > it in time. > > > At least a shutdown page would do something. > > > I will start writing my code and await direct e-mail with > > > authorization and a target IP address to test against. > > > Note that I have plenty of potential test targets in my Samba > > > logs :-( but no legal authority to connect to those machines. > > > > > > ----- Original Message ----- > > > From: "Dan Hollis" <goemon(a)sasami.anime.net> > > > To: <nanog(a)merit.edu> > > > Sent: Friday, September 29, 2000 4:42 PM > > > Subject: Re: Disabling QAZ (was Re: Port 139 scans) > > > > > > > > > > > > > > On Fri, 29 Sep 2000, John Fraizer wrote: > > > > > On Fri, 29 Sep 2000, Dan Hollis wrote: > > > > > > It would be cool if someone would make a tool that > > > would auto-disinfect > > > > > > users... > > > > > Yep. The problem with that is that current laws on the > > > books (in the US > > > > > at least) make this an illegal solution. If memory > > > serves me correctly, > > > > > the one I'm thinking about is worded something like: > > > > > "...any person who without authorization, accesses, > > > modifies, deletes or > > > > > destroys..." > > > > > > > > A web page that users themselves must click "OK, disinfect > > > me"? Seems > > > > authorization enough to me... > > > > > > > > > The penalties are pretty stiff too. The best of > > > intentions don't negate > > > > > the fact that it's illegal. > > > > > > > > When the user initiates the disinfection themselves? > > > > > > > > -Dan > > > > > > > > > > > > >

1 0

RE: Disabling QAZ (was Re: Port 139 scans)
by Carter, Gregory 29 Sep '00

29 Sep '00

-----BEGIN PGP SIGNED MESSAGE----- Greetings, The windows 2000 resource kit comes with a small tool that will add/remove/list shares on a machine. It is called share.vbs and works on other 2k boxes however I have not yet tried it on windows 9x platforms. It might be worth looking into however obviously the legalities of such actions still remain unfortunately. Cheers, Greg +(gcarter(a)infoDNS.com)-------------------------------------------------+ | infoDNS http://www.infodns.com/ | | Senior Network Administrator bits/keyID 1024/7DF9C285 | | Register your domain name today for only $15/yr. | +--------[ DC 50 57 59 C3 76 46 E8 EB 75 A8 94 FE 96 9E D3 ]----------+ - -----Original Message----- From: John Fraizer [mailto:nanog@EnterZone.Net] Sent: Friday, September 29, 2000 1:30 PM To: Dan Hollis Cc: Mike Lewinski; nanog(a)merit.edu Subject: Re: Disabling QAZ (was Re: Port 139 scans) On Fri, 29 Sep 2000, Dan Hollis wrote: > > On Fri, 29 Sep 2000, Mike Lewinski wrote: > > "exit" will close the connection but not the QAZ server, while "quit" does > > appear to shut it down. You can also "run x". Once QAZ has been shutdown, > > it's also possible to connect to the share and manually delete the infected > > notepad.exe, although I haven't yet figured out if there's a way to unshare > > someone's drives remotely via command line (if I did this, I wouldn't be > > able to get back in to clean the infection). > > It would be cool if someone would make a tool that would auto-disinfect > users... > > -Dan > > Yep. The problem with that is that current laws on the books (in the US at least) make this an illegal solution. If memory serves me correctly, the one I'm thinking about is worded something like: "...any person who without authorization, accesses, modifies, deletes or destroys..." The penalties are pretty stiff too. The best of intentions don't negate the fact that it's illegal. - --- John Fraizer EnterZone, Inc -----BEGIN PGP SIGNATURE----- Version: PGPfreeware 6.5.8 for non-commercial use <http://www.pgp.com> iQCVAwUBOdT7PG2piax9+cKFAQFVcQQAnfq88CRumG9MBsxGJOYqLX7mdCGoFozj yIPTPohfwSu56Q0ekNecwaHNFKLqfwBZY6LkWZg12Mt5k+ozaPNBv5F4w2sRCEh7 2n1VIdrXqNfczTPpPSmJWFwsY4O7xFWV791oUYhLWTdML13WpErOoLArz1wqwAPa 5AuN1Lw/cNc= =EEXT -----END PGP SIGNATURE-----

1 0

RE: availability and resiliency
by Roeland M.J. Meyer 29 Sep '00

29 Sep '00

> From: Lionel Lauer [mailto:longword@newsguy.com] > Sent: Friday, September 29, 2000 10:54 AM > > On Fri, 29 Sep 2000 17:39:04 +0100, Andrew Bangs <andrewb(a)demon.net> > wrote: > > > > >On Thu, Sep 28, 2000 at 02:39:40PM -0600, Irwin Lazar wrote: > >> Hi all, > >> Does anyone know if a template exists for what it takes to > provide 5 9's of > >> availability, 4 9's etc.for Internet data centers? > Specifically I'm looking > >> for something that would say "if you want 5 9's of > availability, here's what > >> you need to do", and so on. > > > >For 5 9s you need: > > > >1) Lots of money > >2) Lots of clue > >3) Lots of luck > >4) Lots of balls > > > >You can do 4 9s with any 3 of the above. > > Too true. > > But you forgot to include 'halfway-clued management' - > without that you > haven't got a hope in hell of even getting three 9's. ;) Covered under 2 (above). You need clued mgmnt even to ask for it, let alone "pay" for it. Non-clued management gets ingress filtered at the initial cost estimate layer, for cost-estimate layers that are properly operational.

1 0

Re: Disabling QAZ (was Re: Port 139 scans)
by Dan Hollis 29 Sep '00

29 Sep '00

On Fri, 29 Sep 2000, Mike Lewinski wrote: > the e-mail or not. I believe that this SMTP isn't actually responsible for > _any_ legitimate mail, a check on MX records for yeah.net shows that it's > not listed there. Perhaps the attackers have modified the MTA itself now to > hide their tracks, making it look like that address has been disabled (the > virus doesn't know this, and will keep trying to send at every reboot, btw). How about asking the tier1's to null0 route that chinese MTA? We are blocking 139/tcp and 7597/tcp on our borders. -Dan

1 0

The Cidr Report
by Tony Bates 29 Sep '00

29 Sep '00

This is an auto-generated mail on Fri Sep 29 12:00:00 PDT 2000 It is not checked before it leaves my workstation. However, hopefully you will find this report interesting and will take the time to look through this to see if you can improve the amount of aggregation you perform. The report is split into sections: 0) General Status List the route table history for the last week, list any possibly bogus routes seen and give some status on ASes. 1) Gains by aggregating at the origin AS level This lists the "Top 30" players who if they decided to aggregate their announced classful prefixes at the origin AS level could make a significant difference in the reduction of the current size of the Internet routing table. This calculation does not take into account the inclusion of holes when forming an aggregate so it is possible even larger reduction should be possible. 2) Weekly Delta A summary of the last weeks changes in terms of withdrawn and added routes. Please note that this is only a snapshot but does give some indication of ASes participating in CIDR. Clearly, it is generally a good thing to see a large amont of withdrawls. 3) Interesting aggregates Interesting here means not an aggregate made as a set of classful routes. Thanks to xara.net for giving me access to their routing tables once a day. Please send any comments about this report directly to me. Check http://www.employees.org/~tbates/cidr-report.html for a daily update of this report. ------------------------------------------------------------------------------ CIDR REPORT for 29Sep00 0) General Status Table History ------------- Date Prefixes 220900 87996 230900 87972 240900 88049 250900 88476 260900 88933 270900 88898 280900 88696 290900 88909 Check http://www.employees.org/~tbates/cidr.plot.html for a plot of the table history. Possible Bogus Routes --------------------- AS Summary ---------- Number of ASes in routing system: 8613 Number of ASes announcing only one prefix: 4912 (2732 cidr, 2180 classful) Largest number of cidr routes: 895 announced by AS701 Largest number of classful routes: 1087 announced by AS701 1) Gains by aggregating at the origin AS level --- 29Sep00 --- ASnum NetsNow NetsCIDR NetGain % Gain Description AS271 431 153 278 64.5% BCnet Backbone AS1785 623 377 246 39.5% AppliedTheory Communications AS1221 614 453 161 26.2% TELSTRA-AS AS816 581 437 144 24.8% UUNET Canada (ASN-UUNETCA-AS4) AS5696 421 285 136 32.3% WinStar AS AS9269 177 47 130 73.4% Hong Kong CTI AS2609 128 8 120 93.8% EUnet-TN AS7657 337 220 117 34.7% The Internet Group Limited AS7046 390 277 113 29.0% COQUI-NET PRTC Internet AS9304 124 17 107 86.3% Hutchcity AS7545 158 55 103 65.2% TPG Internet Pty Ltd AS4755 214 116 98 45.8% Videsh Sanchar Nigam Ltd. India AS705 354 258 96 27.1% ALTERNET-AS AS6429 210 114 96 45.7% FirstCom Internet AS9706 103 9 94 91.3% Pusan Metropolitan City Office of AS7496 125 34 91 72.8% Power Up AS8013 488 402 86 17.6% PSINET-CA AS174 538 452 86 16.0% Performance Systems International AS4293 338 255 83 24.6% Internal ASN for C&W AS6595 138 57 81 58.7% DODDSEUR AS13999 90 9 81 90.0% UNKNOWN AS3908 255 176 79 31.0% Supernet, Inc. AS1727 166 92 74 44.6% MRMS-WEST AS577 248 175 73 29.4% Bell Backbone AS701 1087 1018 69 6.3% Alternet AS7018 579 512 67 11.6% AT&T WorldNet Service Backbone AS724 227 161 66 29.1% DLA-ASNBLOCK-AS AS8895 102 37 65 63.7% Saudi Arabia AS AS5106 99 36 63 63.6% AADS-COLUMBUS AS3749 119 57 62 52.1% TECNET For the rest of the previous weeks gain information please see http://www.employees.org:80/~tbates/cidr-report.html 2) Weekly Delta Please see http://www.employees.org:80/~tbates/cidr-report.html for this part of the report 3) Interesting aggregates Please see http://www.employees.org:80/~tbates/cidr-report.html for this part of the report

1 0

availability and resiliency
by Irwin Lazar 29 Sep '00

29 Sep '00

Hi all, Does anyone know if a template exists for what it takes to provide 5 9's of availability, 4 9's etc.for Internet data centers? Specifically I'm looking for something that would say "if you want 5 9's of availability, here's what you need to do", and so on. Ideally, the template would define issues such as divergent fiber, redundant power, hot spares, etc. Thanks in advance, Irwin

3 2