- NANOG - lists.nanog.org

RE: availability and resiliency
by Roeland M.J. Meyer 30 Sep '00

30 Sep '00

It refers to the percent uptime of a host or site. 99% is two "9"'s (2x9?) and 99.9% is three nines. Getting a single host to meet more than three nines (99.9%) can be a challenge ( <8.76 hours outage, per year), but can be more easily met with multiple hosts in a site. Four nines (99.99% uptime, <0.88 hours annual downtime) is extremely difficult for a single host, less difficult for internal data centers, and (given lots of $$$) a bit easier for a internet site (using multiply redundant hosts). Five nines (99.999%, <5.26 annual minutes down) is almost impossible for a single affordable host to meet. This is where we enter the world of High-Availability (H-A) systems. These are usually high transaction flow critical systems and are found in large corps, telcos, and reliable internet sites. At this time, only governments are willing to part with the required cash to build systems meeting six nines (99.9999%, <0.53 minutes annual downtime), or better (NASA, NORAD, US Space Command, etc). Usually, this is done using multiple site redundancy. Hosts meeting three nines, or better, typically have redundant power supplies and integrated UPS, bootable RAID for the OS, redundant NICs, and SMP CPU configurations. -----Original Message----- From: Leo Nelson [mailto:lnelson@axient.com] Sent: Friday, September 29, 2000 10:08 AM To: 'nop(a)alt.net'; Andrew Bangs Cc: nanog(a)merit.edu Subject: RE: availability and resiliency Pardon my ignorance, but what the heck does a "9" refer too? Is it a UPS, rack, floor space, circuit... Thanks in advance -leo -----Original Message----- From: Lionel Lauer [ mailto:longword@newsguy.com] Sent: Friday, September 29, 2000 10:54 AM To: Andrew Bangs Cc: nanog(a)merit.edu Subject: Re: availability and resiliency On Fri, 29 Sep 2000 17:39:04 +0100, Andrew Bangs <andrewb(a)demon.net> wrote: > >On Thu, Sep 28, 2000 at 02:39:40PM -0600, Irwin Lazar wrote: >> Hi all, >> Does anyone know if a template exists for what it takes to provide 5 9's of >> availability, 4 9's etc.for Internet data centers? Specifically I'm looking >> for something that would say "if you want 5 9's of availability, here's what >> you need to do", and so on. > >For 5 9s you need: > >1) Lots of money >2) Lots of clue >3) Lots of luck >4) Lots of balls > >You can do 4 9s with any 3 of the above. Too true. But you forgot to include 'halfway-clued management' - without that you haven't got a hope in hell of even getting three 9's. ;) -- W . | ,. w , "Some people are alive only because \|/ \|/ it is illegal to kill them." Perna condita delenda est ---^----^---------------------------------------------------------------

5 4

Inventor of the Internet Technology / Father of Modern Day Networking
by Ehud Gavron 30 Sep '00

30 Sep '00

My apologies for this non-op note. I was hoping to tap the wisdom of the collective. http://www.lk.cs.ucla.edu/ http://www.lk.cs.ucla.edu/LK/Inet/keys.html Is there a person who invented the Internet, and is this person him? Cheers, Ehud

14 20

Re: Disabling QAZ (was Re: Port 139 scans)
by Dan Hollis 30 Sep '00

30 Sep '00

On Fri, 29 Sep 2000, John Fraizer wrote: > On Fri, 29 Sep 2000, Dan Hollis wrote: > > It would be cool if someone would make a tool that would auto-disinfect > > users... > Yep. The problem with that is that current laws on the books (in the US > at least) make this an illegal solution. If memory serves me correctly, > the one I'm thinking about is worded something like: > "...any person who without authorization, accesses, modifies, deletes or > destroys..." A web page that users themselves must click "OK, disinfect me"? Seems authorization enough to me... > The penalties are pretty stiff too. The best of intentions don't negate > the fact that it's illegal. When the user initiates the disinfection themselves? -Dan

6 5

Statistical Games Providers Play (RE: availability and resiliency)
by Sean Donelan 30 Sep '00

30 Sep '00

Whenever providers start throwing numbers around, you've lost the battle. I always suggest people should talk to their insurance agents, not their technical people. Insurance agents are very good at understanding risk, and how much to spend mitigating that risk. Sometimes it is cheaper to buy a new computer every few years than trying to build the perfect protection around it. The used car dealer is almost never the best source of information about car insurance. Don't expect any better from a sales person at a provider. You are going to end up with expensive undercoat and fabric protection package. As far as I know, the system with the highest publicly stated reliability and availability is FEDWIRE. Fedwire exceeds everything I've seen at NORAD, NASA, or any service provider (carrier, internet, web hosting, etc). Fedwire has five-way redundancy of some systems. It also has the full faith and credit of the US Treasury backing up its service guarantee. A software error in 1985 resulted in a $23 billion (with a "B") accounting imbalance. If I take Fedwire as the upper limit, I need to ask what about providers whose claims exceed those delivered by Fedwire? They aren't lying, but you need to understand the numbers. And if any provider does think their system exceeds Fedwire, I would love a tour of your facility. Due to their history as a regulated monopoly, telephone companies have developed interesting ways to calculated reliability. For example, some telephone companies ignore events which exceed the design parameters of the network. Or in other words, they don't include Mother's Day in their calculations. Some telephone companies also don't include disruptions due to Acts Of God or Force Majure in their reported numbers. I chuckle whenever I hear someone say "carrier-grade." Availability statistics are much like flood and storm statistics. A once in 100 year flood does not mean it will flood only once in any 100 year period. You can have back to back floods. And you can have back to back computer failures. Nor does it limit the length of an outage. You could have a 43 minute failure in Year 1, and no failures in Years 2-5. Or an 86 minute failure in Year 1 and no failures in Years 2-10. Or even a 86 minute failure in Year 1, and a 86 minute failure in Year 2, and no failures in Years 3-20. Remember in statistics when you calculated the series to infinity. If you are still around in Year Infinity, then you can discuss X 9's of availability. Asking a provider how many 9's of reliability they provide, or the MTBF of their systems is really a red herring. What you really want to know, and what you should ask is When a failure does occur (and it will): how will you respond? how will you keep me informed? what do I need to do? and after you understand those answers how often would I expect this? No matter how many 9's you have, there is always a .1, 01, .001, .0001, etc chance. Murphy is exceedingly good at his job. Ok, if you are still reading, and you still want to build a system as reliable as Fedwire, lets talk. Fedwire has shown it can be done, however expect to pay as much as Fedwire. On the other hand, if you are willing to settle for just a little less, the price drops dramatically. Its a lot cheaper to build a system to meet a certain level of design risk, and buying insurance to cover the excess. It may double the price to add another "9" of reliability, but only 10% to cover the risk with insurance. I am not a lawyer, banker, insurance agent, doctor, or indian chief. You should always consult a licensed professional for advice.

2 1

Re: Disabling QAZ (was Re: Port 139 scans)
by Ben Browning 29 Sep '00

29 Sep '00

At 05:02 PM 9/29/00 -0400, Dana Hudes wrote: >I am willing to scrap together a script to shutdown >the virus on an infected machine and put it in a CGI web page. Well, that solves the problem until the reboot. After that, the registry key opens that puppy right back up. The trick is to gut it COMPLETELY. This virus supposedly supports three commands : upload, run and quit. I can't get upload to work, and I lost the manpage(ha, ha). It is possible to upload a file (perhaps compiled c?) that rips out the registry entry and renames the appropriate files on reboot. In fact, one could (legality aside) write up the program to use QAZ as the delivery mechanism for its own death. There's something poetic about that... I have a copy of the worm zipped here- if you'd like it drop me a private email. >I'm not sure about volume but initially I think I can host >it. In the event my 1Mbit connection is overwhelmed I'll > need another place.... >What stops me at the moment is that I have no authorization > to test against any infected machine. >I need a target. I'd offer mine, but I have it isolated. >I'm willing to also try for making the connection to the >share and removing the infection but I'm not sure I can >get it in time. >At least a shutdown page would do something. Half measures merely delay the inevitable- I believe it is best to expunge it right off the bat and never have to deal with the recurrences. >I will start writing my code and await direct e-mail >with authorization and a target IP address to test against. >Note that I have plenty of potential test targets in my >Samba logs :-( but no legal authority to connect to those machines. My current thought is to simply put up a .reg and .bat file up on the web, with instructions on how to use it. Run the .reg to kill the registry key, and run the .bat file to rename the files after the reboot. Of course, it may be easier to simply have a standard email explaining the virus and the removal procedure (my current solution, if anyone wants a copy of the email drop me a line). I will stick with this approach unless the script fully removes (as opposed to temporarily disabling) the virus. Another interesting note- the virus will not allow your computer to reboot if someone is connected to the telnet port. On a side note, if anyone knows a good logfile parsing perl script that pulls out all the IP addresses in a log, I'd love a copy. I have one, but it is very clunky and I daresay a better perl coder than I has tackled this issue. I only ask because this worm has increased the number of other peoples(variously formatted) logfiles in my inbox by about 900%. :) --- Ben Browning <benb(a)oz.net> oz.net Network Operations Tel (206) 443-8000 Fax (206) 443-0500 http://www.oz.net/

1 0

RE: availability and resiliency
by Roeland M.J. Meyer 29 Sep '00

29 Sep '00

> >Hosts meeting three nines, or better, typically have redundant power > >supplies and integrated UPS, bootable RAID for the OS, > redundant NICs, > >and SMP CPU configurations. > > um...is an smp cpu configuration really going to help your uptime? or > are there operating systems or hardware out there that can say to > themselves "hmph! cpu 2 seems not to be working correctly...i'd > better spin it down." That is a natural function these days. Fail-safe to the "off" state. Most SMP OS's can recognise when one, out of an SMP CPU set, goes down. > just for fun a few years back i decided to check if the sun e4000 we > had had hot-swappable cpus (i figured it didn't, but why not try it?) > and i pulled one of the boards. it didn't like it too much. None of what I claimed is required to be hot-swappable to get 99.9%. Enough online hot-spares will keep the system up long enough so that you can replace the entire box. 99.9% allows over 8 hours per year outage. You should be able to swap out a host in less than 0.5 hours.

1 0

RE: Disabling QAZ (was Re: Port 139 scans)
by Roeland M.J. Meyer 29 Sep '00

29 Sep '00

Just like they probably don't know that they're infected, they probably won't know that they've been disinfected. At least the first time. > -----Original Message----- > From: Dana Hudes [mailto:dhudes@hudes.org] > Sent: Friday, September 29, 2000 2:03 PM > To: Dan Hollis; nanog(a)merit.edu > Subject: Re: Disabling QAZ (was Re: Port 139 scans) > > > > I am willing to scrap together a script to shutdown the virus > on an infected machine and put it in a CGI web page. > I'm not sure about volume but initially I think I can host > it. In the event my 1Mbit connection is overwhelmed I'll need > another place.... > What stops me at the moment is that I have no authorization > to test against any infected machine. > I need a target. > I'm willing to also try for making the connection to the > share and removing the infection but I'm not sure I can get > it in time. > At least a shutdown page would do something. > I will start writing my code and await direct e-mail with > authorization and a target IP address to test against. > Note that I have plenty of potential test targets in my Samba > logs :-( but no legal authority to connect to those machines. > > ----- Original Message ----- > From: "Dan Hollis" <goemon(a)sasami.anime.net> > To: <nanog(a)merit.edu> > Sent: Friday, September 29, 2000 4:42 PM > Subject: Re: Disabling QAZ (was Re: Port 139 scans) > > > > > > On Fri, 29 Sep 2000, John Fraizer wrote: > > > On Fri, 29 Sep 2000, Dan Hollis wrote: > > > > It would be cool if someone would make a tool that > would auto-disinfect > > > > users... > > > Yep. The problem with that is that current laws on the > books (in the US > > > at least) make this an illegal solution. If memory > serves me correctly, > > > the one I'm thinking about is worded something like: > > > "...any person who without authorization, accesses, > modifies, deletes or > > > destroys..." > > > > A web page that users themselves must click "OK, disinfect > me"? Seems > > authorization enough to me... > > > > > The penalties are pretty stiff too. The best of > intentions don't negate > > > the fact that it's illegal. > > > > When the user initiates the disinfection themselves? > > > > -Dan > > > >

3 2

RE: Disabling QAZ (was Re: Port 139 scans)
by Roeland M.J. Meyer 29 Sep '00

29 Sep '00

I never argued that. > -----Original Message----- > From: Dana Hudes [mailto:dhudes@hudes.org] > Sent: Friday, September 29, 2000 2:43 PM > To: Roeland M.J. Meyer > Cc: Dan Hollis; nanog(a)merit.edu > Subject: RE: Disabling QAZ (was Re: Port 139 scans) > > > that does not give me authority to connect to stranger's PCs > > On Fri, 29 Sep 2000, Roeland M.J. Meyer wrote: > > > Just like they probably don't know that they're infected, > they probably > > won't know that they've been disinfected. At least the first time. > > > > > -----Original Message----- > > > From: Dana Hudes [mailto:dhudes@hudes.org] > > > Sent: Friday, September 29, 2000 2:03 PM > > > To: Dan Hollis; nanog(a)merit.edu > > > Subject: Re: Disabling QAZ (was Re: Port 139 scans) > > > > > > > > > > > > I am willing to scrap together a script to shutdown the virus > > > on an infected machine and put it in a CGI web page. > > > I'm not sure about volume but initially I think I can host > > > it. In the event my 1Mbit connection is overwhelmed I'll need > > > another place.... > > > What stops me at the moment is that I have no authorization > > > to test against any infected machine. > > > I need a target. > > > I'm willing to also try for making the connection to the > > > share and removing the infection but I'm not sure I can get > > > it in time. > > > At least a shutdown page would do something. > > > I will start writing my code and await direct e-mail with > > > authorization and a target IP address to test against. > > > Note that I have plenty of potential test targets in my Samba > > > logs :-( but no legal authority to connect to those machines. > > > > > > ----- Original Message ----- > > > From: "Dan Hollis" <goemon(a)sasami.anime.net> > > > To: <nanog(a)merit.edu> > > > Sent: Friday, September 29, 2000 4:42 PM > > > Subject: Re: Disabling QAZ (was Re: Port 139 scans) > > > > > > > > > > > > > > On Fri, 29 Sep 2000, John Fraizer wrote: > > > > > On Fri, 29 Sep 2000, Dan Hollis wrote: > > > > > > It would be cool if someone would make a tool that > > > would auto-disinfect > > > > > > users... > > > > > Yep. The problem with that is that current laws on the > > > books (in the US > > > > > at least) make this an illegal solution. If memory > > > serves me correctly, > > > > > the one I'm thinking about is worded something like: > > > > > "...any person who without authorization, accesses, > > > modifies, deletes or > > > > > destroys..." > > > > > > > > A web page that users themselves must click "OK, disinfect > > > me"? Seems > > > > authorization enough to me... > > > > > > > > > The penalties are pretty stiff too. The best of > > > intentions don't negate > > > > > the fact that it's illegal. > > > > > > > > When the user initiates the disinfection themselves? > > > > > > > > -Dan > > > > > > > > > > > > >

1 0

RE: Disabling QAZ (was Re: Port 139 scans)
by Carter, Gregory 29 Sep '00

29 Sep '00

-----BEGIN PGP SIGNED MESSAGE----- Greetings, The windows 2000 resource kit comes with a small tool that will add/remove/list shares on a machine. It is called share.vbs and works on other 2k boxes however I have not yet tried it on windows 9x platforms. It might be worth looking into however obviously the legalities of such actions still remain unfortunately. Cheers, Greg +(gcarter(a)infoDNS.com)-------------------------------------------------+ | infoDNS http://www.infodns.com/ | | Senior Network Administrator bits/keyID 1024/7DF9C285 | | Register your domain name today for only $15/yr. | +--------[ DC 50 57 59 C3 76 46 E8 EB 75 A8 94 FE 96 9E D3 ]----------+ - -----Original Message----- From: John Fraizer [mailto:nanog@EnterZone.Net] Sent: Friday, September 29, 2000 1:30 PM To: Dan Hollis Cc: Mike Lewinski; nanog(a)merit.edu Subject: Re: Disabling QAZ (was Re: Port 139 scans) On Fri, 29 Sep 2000, Dan Hollis wrote: > > On Fri, 29 Sep 2000, Mike Lewinski wrote: > > "exit" will close the connection but not the QAZ server, while "quit" does > > appear to shut it down. You can also "run x". Once QAZ has been shutdown, > > it's also possible to connect to the share and manually delete the infected > > notepad.exe, although I haven't yet figured out if there's a way to unshare > > someone's drives remotely via command line (if I did this, I wouldn't be > > able to get back in to clean the infection). > > It would be cool if someone would make a tool that would auto-disinfect > users... > > -Dan > > Yep. The problem with that is that current laws on the books (in the US at least) make this an illegal solution. If memory serves me correctly, the one I'm thinking about is worded something like: "...any person who without authorization, accesses, modifies, deletes or destroys..." The penalties are pretty stiff too. The best of intentions don't negate the fact that it's illegal. - --- John Fraizer EnterZone, Inc -----BEGIN PGP SIGNATURE----- Version: PGPfreeware 6.5.8 for non-commercial use <http://www.pgp.com> iQCVAwUBOdT7PG2piax9+cKFAQFVcQQAnfq88CRumG9MBsxGJOYqLX7mdCGoFozj yIPTPohfwSu56Q0ekNecwaHNFKLqfwBZY6LkWZg12Mt5k+ozaPNBv5F4w2sRCEh7 2n1VIdrXqNfczTPpPSmJWFwsY4O7xFWV791oUYhLWTdML13WpErOoLArz1wqwAPa 5AuN1Lw/cNc= =EEXT -----END PGP SIGNATURE-----

1 0

RE: availability and resiliency
by Roeland M.J. Meyer 29 Sep '00

29 Sep '00

> From: Lionel Lauer [mailto:longword@newsguy.com] > Sent: Friday, September 29, 2000 10:54 AM > > On Fri, 29 Sep 2000 17:39:04 +0100, Andrew Bangs <andrewb(a)demon.net> > wrote: > > > > >On Thu, Sep 28, 2000 at 02:39:40PM -0600, Irwin Lazar wrote: > >> Hi all, > >> Does anyone know if a template exists for what it takes to > provide 5 9's of > >> availability, 4 9's etc.for Internet data centers? > Specifically I'm looking > >> for something that would say "if you want 5 9's of > availability, here's what > >> you need to do", and so on. > > > >For 5 9s you need: > > > >1) Lots of money > >2) Lots of clue > >3) Lots of luck > >4) Lots of balls > > > >You can do 4 9s with any 3 of the above. > > Too true. > > But you forgot to include 'halfway-clued management' - > without that you > haven't got a hope in hell of even getting three 9's. ;) Covered under 2 (above). You need clued mgmnt even to ask for it, let alone "pay" for it. Non-clued management gets ingress filtered at the initial cost estimate layer, for cost-estimate layers that are properly operational.

1 0