July 2007 - Test - lists.nanog.org

Re: San Francisco Power Outage
by George William Herbert 25 Jul '07

25 Jul '07

Michael Dillon writes: >> And the stories that the power guy I'm working with tells >> about foreign facilities, particularly in middle east war >> zones, are really scary... > >> We fundamentally do not have the facilities problem >> completely nailed down to the point that things will never >> drop. Level 4 >> datacenters can, and will, fail. Nothing you can do including >> just doing 48V DC for everything are truly foolproof solutions. > >A single level 4 datacenter is a Single Point of Failure! > >Two of those middle-eastern style facilities is... ? >Has anyone actually kept track of all these data center failures over >the years and done some statistical analysis on it? Maybe two half-baked >data centers is better than one over the long run? > >Remember that one 10-12 years ago in (Palo Alto, Mountainview?) where a >lady in a car caused a backhoe driver to move out of the way which >resulted in him cutting a gas line which resulted in the fire department >evacuating the data center, cutting off electricity in the area, and >forbidding the diesel generators to be switched on? Santa Clara. I was working right outside the evacuation radius. Which exchange point was in the building? PB-NAP? CIX? I remember we had a net-dark event associated, but not which one. It was a bad day... The lesson, as you point out, is that geographical redundancy is sometimes necessary. This is as true for providers as for datacenter end-users... -george william herbert gherbert(a)retro.com

1 0

Re: Why do we use facilities with EPO's?
by Warren Kumari 25 Jul '07

25 Jul '07

On Jul 25, 2007, at 3:35 PM, Patrick W. Gilmore wrote: > > On Jul 25, 2007, at 2:03 PM, Tuc at T-B-O-H.NET wrote: > >>> If they can be avoided, why do we put up with them? Do we really >>> want our colo in downtown San Francisco bad enough to take the risk >>> of having a single point of failure? How can we, as engineers, ask >>> questions about how many generators, how much fuel, and yet take >>> for granted that there is one button on the wall that makes it all >>> turn off? Is it simply that having colo in the middle of the city >>> is so convenient that it overrides the increased cost and the >>> reduced >>> redundancy that are necessitated by that location? >>> >> You forgot the default "Single Point of Failure" in anything.. >> >> HUMANS. > > The earth is a SPoF. Let's put DCs on the moon. > > Besides, safety always overrides convenience. And I don't think > that is a bad trade off. Me neither... Having multiple redundant sites (and a well designed network between them) is almost always going to be better than a single, wildly redundant site. No matter how much redundancy you build into a single site, you cannot (realistically) engineer away things like floods, etc. Planning your redundancy and testing it though is very important... Random anecdote (from a friend, I don't know if it true or not): Back in the day (before cheap international circuits), a very large financial in New York needed connectivity to some branches in Europe, so they bought some capacity on a satellite transponder and built their own ground-station (not cheap) fairly close to NY. They then realized that the needed a redundant ground station in case the first one failed or something similar, so the built a second ground- station, just outside Jersey City.... One of the satellite connectivity failure modes is... rain fade..... W > > -- > TTFN, > patrick > > -- "Does Emacs have the Buddha nature? Why not? It has bloody well everything else!"

1 0

Re: 365 Main - an operators' nightmare?
by Jim Popovitch 25 Jul '07

25 Jul '07

On Tue, 2007-07-24 at 19:26 -0700, Rusty Hodge wrote: > > Think that's good? It gets better.... > > > > http://valleywag.com/tech/breaking/angry-mob-gathers-outside-sf- > > datacenter-282053.php > > That article states that only Colo 4 was affected. > > I'm in Colo 7 and it was affected as well. > > You're not seriously believing the disgruntled employee story are you? No. ;-) But it is otherwise believable. I've seen people hit big-red-buttons in disbelief before, doing so in anger seems very plausible. -Jim p.

4 3

[Nanog] test
by Larry J. Blunk 25 Jul '07

25 Jul '07

testing

1 0

ASN Name of the week
by Marshall Eubanks 25 Jul '07

25 Jul '07

This just appeared in my BGP ASN Number Name Handle Location Organization 40543 1-800-GOT-JUNK [ABI19-ARIN] {Vancouver, BC, CA} 1-800-GOT-JUNK I guess somebody thinks that whois has advertising potential. Regards Marshall

7 9

Problems getting Cisco router and Motorola Nextlevel system to work together
by Brian Raaen 25 Jul '07

25 Jul '07

I am having some difficulties involving using a Cisco 7200 router to terminate ATM sessions from a motorola nextlevel IPTV system. The router is currently configured to use IRB which is a hybrid process. The problems is that the IRB process is overloaded and is dropping traffic faster than it can process it. I opened a case with Cisco TAC, and they recommended using RBE instead of IRB. While I have been trying to plan migrating the system to RBE I discovered that Motorola uses a concept called "dynamic pvc's" to assign the pvc's to the CPE devices (a IPTV unit that has a data port). The device uses two PVC's one for data and one for IPTV. The system dynamically assigns the PVCs when the CPE devices connects. This looks like it would not work with RBE, since the pvc can change before the dhcp lease expires. Having this router dropping traffic, has been causing severe problems for end users and is causing an ongoing system outage. I am currently trying to work with both Motorola and Cisco, however both vendors are blaming the problem on the other vendor. I am not sure what to do. Motorola says their system only works with IRB and Cisco says the router will not function with this size network using IRB. Has anyone else arrived at a working solution using a Cisco 7200 router to terminate a Motorola nextlevel system support approximately 2000-3000 end users. I would be extremely gratefull if anyone who has worked with this type of system could help shed some light on this problem. Thank you in advance. -- Brian Raaen Network Engineer braaen (at) zcorum (dot) com

3 4

more on SF outage
by Fred Heutte 25 Jul '07

25 Jul '07

The story is evolving, as you'd expect. An excerpt from the latest update at the Chron is below. Nothing at this hour has appeared on 365main.com. But go there and take a look and decide if *your* colo has better stuff. http://www.365main.com/365_main_tour_2.html And now that we know this isn't enough . . . your move. fh PS I am not a customer of 365 Main or any colo for that matter. ---------------------- http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2007/07/24/BAG9NR67253.DTL… The datacenter 365 Main -- which hosts a number of Internet sites, including Craigslist, Yelp and RedEnvelope -- also lost power, causing frustration among many Web surfers. Several of the sites posted messages to explain to their users that they were having difficulties, while others said they were undergoing scheduled maintenance or upgrades. After about 45 mintues, power to the datacenter was restored by back-up generators that remained in service until PG&E could confirm that its supply was stable. There was no immediate response from the facility as to why the generators were not turned on right after the blackout. Some of the sites, including Craigslist, remained down even after power was restored, as administrators ensured that data in the server hadn't been damaged, among other checks.

2 1

Fw: Re: San Francisco Power Outage
by Fred Heutte 25 Jul '07

25 Jul '07

Don't be so fast to point the finger. Generally speaking, blame is obvious from the initial news reports but tends to diminish with retrospective fact-based assessment. For example: it's "obvious" that serious net sites need multihoming. But what if your multihomed bits go through the same pipe (or worse, through the same fiber)? Who do you blame when you find out? Worse, in terms of blame: who can you go to beforehand who actually knows where that can happen? I well remember this slide from Sean Donelan's talk at NANOG23: --------------------------- http://www.nanog.org/mtg-0110/ppt/donelan_files/v3_document.htm What Didn't Work - Diversity and Avoidance * Equipment in the World Trade Center primarily served tenants in complex (shared fate) * SONET ring through WTC tower 1 and alternate path through WTC tower 2 * Damage to 140 West Street central office and surrounding underground infrastructure * Backup circuit routed through same facility * Advanced data circuits (ISDN/DSL) concentrated in a few central offices --------------------------- The real answer, found elsewhere in Sean's talk, is that the design of the net has always encouraged redundancy as an engineering principle. Stress situations is where that pays off, even though it can't solve every possible eventuality (and as has already been noted, redundant equipment also fails as well as creating more complex failure modes). The net had problems on 9/11, especially around the WTC, but Sean's slides document remarkable resiliency even in that area. The power went off at a key spot in the San Francisco infrastructure today. But as far as I know, even though it was mentioned in the Chron article, Craigslist stayed online because they have a distributed and redundant system (which is not to say, impervious to all failure modes). Some shortcomings are obvious, but all I am saying is, before rushing to cast blame, it's a good idea to try and collect some facts. fh ------------------- http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2007/07/24/BAG9NR67253.DTL… Power restored in San Francisco Marisa Lagos and Matthew B. Stannard, Chronicle Staff Writers Tuesday, July 24, 2007 (07-24) 16:57 PDT SAN FRANCISCO -- Between 30,000 and 50,000 Pacific Gas and Electric Co. customers in San Francisco and the northern Peninsula lost power for several hours this afternoon after what witnesses described as an explosion under a manhole cover on Mission Street, the utility said. Brian Swanson, a spokesman for the utility, said power failures were reported throughout wide swaths of the east side of San Francisco, including downtown and at PG&E's own office on Beale Street near the Ferry Building. The outage first occurred at about 1:50 p.m., and electricity flickered on and off at least five times before power was restored at about 4 p.m. PG&E officials said the source of the power outage was an underground failure. Standing at a manhole in a plaza at 560 Mission St. in San Francisco, where witnesses reported hearing an explosion, Swanson said it could have been the source of the outage, but officials were still investigating. The incident recalled an August 2005 explosion in an underground vault at Post and Kearny streets that critically injured a woman who was walking by. At the time, PG&E blamed high levels of moisture in the attached high-voltage chambers and said it was checking the safety of about 1,000 other high-voltage chambers. Swanson said today's incident -- in which no one was injured -- was caused by some sort of fault in the line. "It is completely unrelated to what happened two years ago," he said. Witnesses said they heard an explosion at about 1:50 p.m., then saw flames coming from the manhole. Actor Torino Von Jones, 32, said he was filming a Fruit of the Loom commercial down the block at the time. "We were standing over there waiting for the camera cue when we heard a big explosion," he said. "Flames came up taller than I am, and I'm 6-foot-2." "Naturally, when you hear an explosion, you think the worst," Von Jones said. Nevertheless, he hurried back to work. "We're Fruit of the Loom -- we've got to make this commercial." The outage briefly affected some Muni buses and trains, but all were back to normal by 3 p.m., a spokeswoman said. Workers at several downtown and South of Market offices were reportedly sent home for the day following the outage. Additionally, the datacenter 365 Main -- which hosts Web sites including Craigslist and Yelp -- lost power. ------ mail forwarded, original message follows ------ To: nanog(a)merit.edu From: sethm(a)rollernet.us <Seth Mattinen> Subject: Re: San Francisco Power Outage Date: Tue, 24 Jul 2007 15:54:08 -0700 Jonathan Lassoff wrote: > > Just a heads up to anyone on list that PG&E has just sustained a large > outage in San Francisco that has caused a few hiccups (both network, > electrical, infrastructural, etc.) around the city. > > I've confirmed that both customers in 365 Main and parts of telecom 1 > have both sustained brief blackouts. No word yet form 200 Paul. > > Anyone in the area that could use a hand with anything, I'll probably > be wrapping up fixes for my stuff soon, and would be glad to help > however I can. > I have a question: does anyone seriously accept "oh, power trouble" as a reason your servers went offline? Where's the generators? UPS? Testing said combination of UPS and generators? What if it was important? I honestly find it hard to believe anyone runs a facility like that and people actually *pay* for it. If you do accept this is a good reason for failure, why? ~Seth

1 0

365 Main - an operators' nightmare?
by joe mcguckin 25 Jul '07

25 Jul '07

I don't know if this is true, but it's more exciting reading than blaming it on a 'power outage'... http://valleywag.com/tech/breakdowns/a-drunk-employee-kills-all-of- the-websites-you-care-about-282021.php Joe McGuckin ViaNet Communications joe(a)via.net 650-207-0372 cell 650-213-1302 office 650-969-2124 fax

3 2

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking )
by Paul Ferguson 25 Jul '07

25 Jul '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -- Christopher Morrow <christopher.morrow(a)verizonbusiness.com> wrote: >I'd love to see CPE dsl/cable-modem providers integrate with a 'service' >that lists out 'bad' things. it'd be nice if the user could even tailor >that list (just C&C or C&C + child-porn or C&C older not than X >days/hours/minutes) ... I think it might even help, and be vendor >>agnostic (from a provide and hardware) perspective. Ironically, that is exactly part of a product announcement that we (Trend Micro) are making on 30 July. Since this topic arose, I saw Trend mentioned as a possible product "culprit" in this scenario, but it isn't. Yet. :-) The particular service to be announced on Monday (BIS, or Botnet Identification Service), is nothing more than a BGP feed of _known_ and _vetted_ botnet C&Cs as /32s, intended to be a black-hole feed. Interested folks should either e-mail me off-list, or just wait for the official announcement on 30 July. Cheers, - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.6.2 (Build 2014) wj8DBQFGplq5q1pz9mNUZTMRAnFzAKCicaHuvoTwJk92hPOOu2E/ofjhegCcCrMc XCA4rpUCimConxtKV/Qrsfs= =N2f1 -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/

3 2