- NANOG - lists.nanog.org

FWD: Explanation for the recent major downtime
by jc dill 15 Sep '05

15 Sep '05

My personal website is hosted with DreamHost. They sent this out to their customers today. Of interest to NANOG is the bit about the N+1 redundant genset system having 2 generators quickly fail, and in doing so having the UPS fail and the entire data center go dark. Something to consider in your data center design... jc DreamHost Announcement Team wrote: > On Monday, September 12 the greater Los Angeles area experienced a major power outage > affecting large sections of the city, including our main data center. The power outage began > shortly before 1pm PST and continued until about 4:30pm PST. Our data center is equipped > with a redundant backup power system with both battery UPS systems and diesel generators, > but the backup failed and our entire data center was powered down. > > We have previously covered much of this information on our official weblog (http:// > blog.dreamhost.com/) but many of you have not seen that information so we will summarize > the events here. > > When the grid power to our building was cut, the UPS system kicked in and kept everything in > the building up and running. The five generators also fired up and began providing power. > The building needs four generators to operate at full power so the system is designed to > tolerate a single failure. Unfortunately, two of the five generators failed within minutes of each > other. We receive our power from the building housing our data center and they also manage > the redundant power system. We do not know the exact reason for the generator failures at > this time. We have received some vague explanations that we have not found to be satisfactory. > Regardless, the remaining three generators were not sufficient to meet the building's power > needs and that caused the emergency electrical systems to transfer into a “load shedding > mode” and the building’s UPS system to turn itself off, thus preventing permanent UPS and > related equipment damage. That shut everything down, including emergency lighting, and the > building was evacuated. > > About 15 minutes later, one of the generators was started up to power emergency lighting and > a couple of our senior technicians made their way into the (still evacuated) building and down > to our data center to assess the damage. Since the backup power had failed, our own data > center power remained off until the main grid power came back. We then proceeded to slowly > power up our equipment. Servers (and all computers) consume significantly more power when > booting up than when up and running so there is some risk of overloading the power circuits if > too many of them are flipped on at once. Keeping that in mind, we powered everything on as > quickly as possible. At that time the majority of our services were fully back up and running > but some services were still down and we began the process of systematically verifying all > services and making any necessary repairs and adjustments. Whenever a large number of > servers suddenly loses power a certain small percentage of them will not come back up properly > and when you have several hundred servers it takes awhile to verify all of them. > > Once our own access to our servers was restored our staff continued working into the night to > restore as much service as possible and to respond to as many of your support cases as > possible. Some of our staff continued working all the way through the night and we were able > to restore almost everything that first night. > > Tuesday (September 13) started off early with all of us addressing the residual issues. At > around noon that day one of our core routers experienced an internal failure stemming from > damage previously sustained during the power outage. Our routers handle all of the Internet > traffic coming in and out of our network and they are set up in a redundant way to minimize > network disruption when a failure does occur. In this case, the main cpu of the router (called > the 'supervisor') died and the secondary one took over. Everything continued working almost as > it should have, but there is a remaining router issue that we are still working with Cisco support > on. That issue is responsible for the slower than normal performance of our network and it will > be resolved absolutely as soon as possible. > > During this outage, our off-network Emergency Status Page (http://status.dreamhost.com/) > proved to be an invaluable resource for disseminating information among our customers. That > status page remained up throughout the power outage and was updated regularly as we > received new information. Unfortunately, not everyone knows about it and we will be working > to improve that situation in the coming days. Those bloggers among you that did know to > check the status page were extra helpful in passing along the information to other > dreamhosters who were still in the dark. Thank you to everyone who helped out with that! > > This announcement will be followed by another explaining what went wrong with our processes > and what we plan to do to address them. That will come in the next few days. > > We will be continuing to provide more detailed information on our official weblog found here: > http://blog.dreamhost.com/ > > Also, everyone who has not bookmarked our Emergency Status Page should do so now. That > page is found here: > http://status.dreamhost.com/ > > We will be improving on the basic page we have there to provide as useful of an avenue of > information as possible. > > If you have any additional questions about this outage, please let us know. We will be happy to > address all of your questions or concerns. > > > The Un-Happy DreamHost Powerless Team > > > > > > > > >

2 1

Re: 12/8 problems? (fwd)
by Manish Karir 15 Sep '05

15 Sep '05

---------- Forwarded message ---------- Date: Wed, 14 Sep 2005 16:39:54 -0400 (EDT) From: Manish Karir <mkarir(a)merit.edu> To: nanog(a)merit.edu Subject: Re: 12/8 problems? I'm sorry I'm a bit late on this thread but wanted to point out that you can view some of the relevant information on this, as seen from routeviews at: http://bgpinspect.merit.edu - At the bottom half of the page, select routeviews peer, e.g. sprint - select "Prefix-Exact" as query type - Enter prefix into the Query box: "12.0.0.0/8" (without the quotes) - Select start date as Aug 25, select end date as Sept 14 - Click submit-query. Alternately you can just get the pdf files of the results page from the above queries from the following 2 links: http://bgpinspect.merit.edu/reports/glx-12-8.pdf http://bgpinspect.merit.edu/reports/sprint-12-8.pdf After a few more cleanups and changes, we will be re-announcing the bgp-inspect project hopefully before the next nanog mtg. as a generally available service. thanks manish ---- Date: Mon, 12 Sep 2005 18:02:29 -0400 From: Richard A Steenbergen <ras(a)e-gerbil.net> Subject: Re: 12/8 problems? On Sat, Sep 10, 2005 at 06:15:38AM -0700, Eric Louie wrote: > > FYI, happened again this morning for (at least) 12/8 > duration approx 30 minutes > starting at 5:45 AM PDT. Notice that AT&T is no longer taking chances, and is announcing 2 /9s. - -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ----------

1 0

The return of the wildcard domain...
by Jim McBurnett 15 Sep '05

15 Sep '05

All, I saw this evening that CentralNic had added *.uk.com to point to itself. I thought this was of operational interest considering the effect Verisign sitefinder had. (Sorry Martin, no offence intended) So I guess the question to us all is: how will this affect our networks as a whole? Or will it at all? Or is it already? I tried this with the almost funny domain www.idunno.uk.com It does a redirect to http://www.uk.com/results/idunno?src=idunno And came back with a "The domain name "idunno.uk.com" is available for registration. Would you like to buy it" So will ICANN act on this? CentralNIC also runs the uk.com, us.com, eu.com and de.com domains. For a "news" reference: http://www.theregister.co.uk/2005/09/14/centralnic_wildcard/ Anyway, I thought I would start a new topic.. Hopefully this does not add too much to the overall S:N...... Later, Jim McBurnett

5 4

re: commonly blocked ports (but not on backbones)
by brett watson 14 Sep '05

14 Sep '05

> seems to me this is the wrong question... a default security > "posture" (network or system, isp or enterprise or any type of > entity) should be: "if it's not explicitly allowed, it's denied." apologies, i see the original poster was talking about a *backbone*... my mind was on campus/edge/customer networks. this policy, of course, does not apply to backbones (unless you want an avalanche of customer calls). -b

1 0

CAT5 surge/lightning strike protection recommendations?
by R.P. Aditya 14 Sep '05

14 Sep '05

I have a bunch of cat5 buried about 1 ft below the surface connecting multiple buildings on a campus (short runs) and lightning strikes nearby have caused surges along one or more of the cables and burnt out switch ports. I would like to protect the switch ports -- there seem to be lots of products on the market. Anyone have recommendations (tested/practical is best :-)? The APC Protectnet PNET1 and PRM24 seem quite nice and not too expensive -- if they work....pros? cons? Thanks, Adi

13 17

Re: commonly blocked ISP ports
by Fergie (Paul Ferguson) 14 Sep '05

14 Sep '05

A couple of decent barometers: http://www.dshield.org/topports.php and: http://www.mynetwatchman.com/default.asp - ferg -- Luke Parrish <lukep(a)centurytel.net> wrote: Not quite looking for tips to manage my network and ACL's or if should or should not be blocking, more looking for actual ports that other ISP's are blocking and why. For example: port 5 worm 2.5 port 67 virus 8.2 At 03:12 PM 9/14/2005, Valdis.Kletnieks(a)vt.edu wrote: >On Wed, 14 Sep 2005 14:42:56 CDT, Luke Parrish said: > > We have a list, some reactive and some proactive, however we need to > remove > > ports that are no longer a threat and add new ones as they are published. > >All ports that are open are threats, at least potentially. What you *should* >be doing is: > >a) When you block a new port due to a current exploit, log the fact. >b) Work with customers/users to make sure they're patched, and that new >machines >are patched before they go live. >c) When probing for the port stops (which it never does), or some sufficient >number of downstream boxes are patched and safe, remove the block. > >Either that, or block the world, and open ports on request. > >Remember - *you* are the only one on this list who really knows if a given >port is a threat anymore.... > >(And that's totally skipping all the noise about corporate firewalls >versus ISP >firewalls and different expectations regarding security/transparency...) Luke Parrish Centurytel Internet Operations 318-330-6661 -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(a)netzero.net or fergdawg(a)sbcglobal.net ferg's tech blog: http://fergdawg.blogspot.com/

1 0

Re: commonly blocked ISP ports
by Scott Weeks 14 Sep '05

14 Sep '05

----- Original Message Follows ----- From: Luke Parrish <lukep(a)centurytel.net> To: nanog(a)merit.edu Subject: commonly blocked ISP ports Date: Wed, 14 Sep 2005 14:42:56 -0500 > Everyone, > > Does anyone have a reference point for commonly blocked > ports? > > We have a list, some reactive and some proactive, however > we need to remove ports that are no longer a threat and > add new ones as they are published. See the NANOG archives. There is a LOT of discussion regarding this issue there. Please everyone, let's don't do this again. It has not been long enough since the last one and my "D" key is beginning to wear out after the last couple of days... scott

1 0

cogent and level3
by Joseph Nuara 14 Sep '05

14 Sep '05

Does anyone know what the story is with Cogent and L3? I noticed that my Cogent site (IN NY) is using a path to one of my providers (IN NJ) via asia as opposed to the local and preferred L3 peer. After several days I was finally told that L3 and Cogent are working through some peering negotiations and cogent is moving traffic off their L3 peer in anticipation of a depeering (I guess they are trying to avoid the whole France Telecom thing that happend last time). Does anyone have a better clue as to what is going on and where the negotiations stand?

2 1

Load Balancing between multiple BGP peer connections
by Joe Shen 14 Sep '05

14 Sep '05

Hi, How could load on multiple BGP peer links be balanced automatically? The situation we are facing: ---------------------------| | Service provider | | | --R1--------------------R2--- |\ | | \ E-BGP | | \ ----------------\ | | \| ----R3--------------------R4-| | Our Network | | (OSPF) | ----------------------------- The three links between our network and Service provider network are all 1Gbps. Now, we noticed that load on link R1-R3, R1-R4 is about 50% (in/out), but load on R2-R4 are about 90% ( in/out). How could we balance those load on the three links automatically? or must we tune the route mannually? Is there any technical guide on this? thanks in advance. Joe Send instant messages to your online friends http://asia.messenger.yahoo.com

3 3

Re: CAT5 surge/lightning strike protection recommendations?
by Frank Coluccio 14 Sep '05

14 Sep '05

re: ""what did your electricians say ..." Back to lurking in a moment, but I should note that fire and safety code issues are so shrouded in matters of locality and jurisdiction, at times, and bound up in industry and governmental standards that are so esotric in nature, that the typical electrical worker, and I dare say even most licensed electricians and IT consultants, are totally unaware of the specifics, or they are given to gross misconceptions and half-truths. This is so, not so much due to the complex nature of the matter at hand, but due to a lack of regular exposure to it. Nufsed. Frank A. Coluccio DTI Consulting Inc. 212-587-8150 Office 347-526-6788 Mobile On Wed Sep 14 10:58 , 'Hannigan, Martin' sent: I asked him "what did your electricians say" and the entire ground conversation went south. Good post. Thanks. -- Martin Hannigan (c) 617-388-2663 VeriSign, Inc. (w) 703-948-7018 Network Engineer IV Operations & Infrastructure hannigan(a)verisign.com > -----Original Message----- > From: owner-nanog(a)merit.edu [owner-nanog(a)merit.edu','','','')">owner-nanog(a)merit.edu]On Behalf Of > Frank Coluccio > Sent: Wednesday, September 14, 2005 10:52 AM > To: nanog(a)merit.edu > Subject: Re: CAT5 surge/lightning strike protection recommendations? > > > > There's not much left to interpretation and preferences here, aside > > from the choice of medium to be used. I should also add that > some of the > > advice that has been posted in this thread, as well-meaning > and thoughtful as it > > has been, has been downright dangerous to follow. > > > > If one is going to run copper cable between buildings, or outdoors, in > > general, in lieu of the better alternatives of fiber or wireless, then > > there are local and national electrical, fire and safety > codes in effect > > that spell out what you must do, enforceable under the penalty of law. > > And while certain of those precautions have been spelled out > upstream in > > piecemeal fashion, none thus far has been entirely accurate > or complete. > > > > Yes, when running copper between buildings, lightning > arresters/circuit > > - i.e., protection - are a must, but they must be placed > within a couple > > of feet of the building point of entry, or POE. Think about > it. Does it > > make great sense to protect a cable from surges deep within > the interior > > of a building if the cable traverses vast distances on premises > > unprotected between the point of entry and the terminal point. > > Therefore, the stipulation of performing grounding, bonding and surge > > protections at the point where the cable enters the building (in > > potentially at addition points, elsewhere, when required). > > > > Also, if the copper cable is "armored" with a corrugated steal jacket, > > as many outside plant cables are, then the shielding (the armor) must > > also be "bonded" to earth ground at the POE, as well. So the issue > > becomes one not only of grounding, but bonding, as well. And while I'm > > on that subject, be aware that many FIBER OPTIC cables designed for > > inter-building/outside plant use are also armored and must be > treated in > > the same manner. > > > > BICSI (Building Industries Consulting Systems International) > > www.bicsi.org does a good job of rolling up all of the relevant > > standards, as do a number of other sources. For some good coverage of > > safety, grounding and bonding principles and techniques see the > > following Structured Cabling Supplement reference by Panduit > (taken from > > the Cisco CCNA Networking Academy Program) > > > > http://www.tecmiami.com/cisco/extra/CCNA1_CS_1_en.pdf > > > > Frank A. Coluccio > > DTI Consulting Inc. > > 212-587-8150 Office > > 347-526-6788 Mobile > > >

2 1