July 2007 - Test - lists.nanog.org

TWTC issue with Foundry routers?
by David Hubbard 26 Jul '07

26 Jul '07

Anyone know of any changes that were made with TWTC (AS 4323) last night that may have affected those running Foundry routers? We peer with a number of providers and last night our TWTC connection went down with: Jul 25 15:57:22:N:BGP Peer 1.2.3.49 DOWN (Attribute Flags Error) Jul 25 15:57:14:N:BGP Peer 1.2.3.49 UP (ESTABLISHED) If I debug updates on that session I get: (Lines added for readability) Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE 142.166.102.0/24 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE w/attr: Origin=IGP AS_PATH=AS_SEQ(2) 4323 8881 8881 8881 30915 NextHop=1.2.3.49 COMMUNITY=4323:51 4323:501 4323:1003 4323:2001 4323:2503 4323:34510 4323:50000 65101:1003 65102:4 65103:1 65104:301 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE 193.27.220.0/23 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE w/attr: Origin=IGP AS_PATH=AS_SEQ(2) 4323 2828 19092 14188 14188 14188 14188 14188 NextHop=1.2.3.49 COMMUNITY=4323:51 4323:501 4323:1015 4323:2503 4323:36410 4323:50000 65101:1015 65102:4 65103:1 65104:301 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE 64.13.0.0/22 Jul 25 15:57:21 BGP: 1.2.3.49 rcv invalid COMMUNITY attribute flag d0 Jul 25 15:57:21 BGP: 1.2.3.49 rcv UPDATE w/attr: Origin=IGP AS_PATH=AS_SEQ(2) 4323 12956 3352 NextHop=1.2.3.49 ATOMIC_AGGREGATE AGGREGATOR AS=3352 Speaker=81.46.63.133 The router is a Foundry NetIron 400 running their 7.8 code. We have two of these talking to Level 3, TWTC, Cogent, Uunet and AT&T and only the TWTC had an issue. They sent me a default route instead of full routes and the session came up and was stable; go back to full routes and error. They admitted to me this afternoon that three other customers are having the same issue. That's when we started wondering if they changed something that the Foundry code doesn't like. Interesting though is that they claim to not be sending me communities while the output above indicates they are. Any ideas; be nice to get the link back up. :-) Thanks, David

3 2

Re: San Francisco Power Outage
by George William Herbert 26 Jul '07

26 Jul '07

Seth wrote: >Jonathan Lassoff wrote: >> >> Just a heads up to anyone on list that PG&E has just sustained a large >> outage in San Francisco that has caused a few hiccups (both network, >> electrical, infrastructural, etc.) around the city. >> >> I've confirmed that both customers in 365 Main and parts of telecom 1 >> have both sustained brief blackouts. No word yet form 200 Paul. >> >> Anyone in the area that could use a hand with anything, I'll probably >> be wrapping up fixes for my stuff soon, and would be glad to help >> however I can. > >I have a question: does anyone seriously accept "oh, power trouble" as a >reason your servers went offline? Where's the generators? UPS? Testing >said combination of UPS and generators? What if it was important? I >honestly find it hard to believe anyone runs a facility like that and >people actually *pay* for it. > >If you do accept this is a good reason for failure, why? Unfortunate real-world lesson: there is a functional difference between pushing the UPS test cutover button, and some of the stuff that can happen out on the power lines (including rapid voltage swings, harmonics, etc). I know 365 Main has the equipment and tests it, I've been standing outside when the generators spool up. I've had generator firmware upgrades generate reporting info on the serial uplink that flipped the UPSes into permanent error state until the Liebert guys got off the plane with the replacement mainboard. I've had grid voltage fluctuations that toasted VSDs in chillers. I watched a building's electrical service go "pop" when a transformer blew and ran 10kv into the 220 mains for a fraction of a second as it arced. I was at home but called in after a 5 MW generator popped under a sufficiently badly harmonic UPS and PDU load of only about 2.4 MW. I had a client who forgot to wire the A/C into the UPS, and nearly melted a whole server room. And the stories that the power guy I'm working with tells about foreign facilities, particularly in middle east war zones, are really scary... We fundamentally do not have the facilities problem completely nailed down to the point that things will never drop. Level 4 datacenters can, and will, fail. Nothing you can do including just doing 48V DC for everything are truly foolproof solutions. -george williiam herbert gherbert(a)retro.com

4 3

Reliance / Flag telecom buys Yipes - $300m
by Suresh Ramasubramanian 26 Jul '07

26 Jul '07

http://www.thehindubusinessline.com/2007/07/18/stories/2007071850650400.htm -- Suresh Ramasubramanian (ops.lists(a)gmail.com)

1 0

Re: iPhone and Network Disruptions ...
by Prof. Robert Mathews (OSIA) 26 Jul '07

26 Jul '07

Hank, Warren, & Fellow Nanogers: Looking at this issue with an *'interoperability lens,'* I remain puzzled by a personal observation that at least in the publicized case of Duke University's Wi-Fi net being effected, the "ARP storms" did not negatively impact network operations UNTIL the presence of iPhones on campus. The nagging point in my mind therefore, is: why have other Wi-Fi devices (laptops, HPCs/PDAs, Smartphones etc.,) NOT caused the 'type' of ARP flooding, which was made visible in Duke's Wi-Fi environment? Why did this issue become MOST prominent with the introduction of Apple's iPhone on campus? In that sense, my *original question* regarding iPhone's 'unique' operational circumstance(s) will have/need to be considered. Initial analysis tells me that we may not be far into that aspect but, we might need to... Again, I wish to thank you for the responses. All my best, Robert. --

7 6

Cogent issues in Chicago?
by Brandon Galbraith 26 Jul '07

26 Jul '07

Anyone experiencing issues with their Cogent connectivity right now? -brandon

1 0

San Francisco Power Outage
by Jonathan Lassoff 25 Jul '07

25 Jul '07

Just a heads up to anyone on list that PG&E has just sustained a large outage in San Francisco that has caused a few hiccups (both network, electrical, infrastructural, etc.) around the city. I've confirmed that both customers in 365 Main and parts of telecom 1 have both sustained brief blackouts. No word yet form 200 Paul. Anyone in the area that could use a hand with anything, I'll probably be wrapping up fixes for my stuff soon, and would be glad to help however I can. Cheers, jonathan -- Jonathan Lassoff echo thejof | sed 's/^/jof(a)/;s/$/.com/' http://thejof.com 415-215-2464 GPG: 0xC8579EE5

20 26

Call for Volunteers for NANOG Mailing List Committee
by Joe Abley 25 Jul '07

25 Jul '07

People of NANOG, Members of the Mailing List Committee come from a variety of technical and organisational backgrounds, but share a desire to help the NANOG list function as an effective resource for network operators. The Steering Committee, in consultation with the current Mailing List Committee, are interested in hearing from people who are interested in joining the team, volunteering their time and energy to help make the NANOG list better. We are looking for people who care about the NANOG list, who are habitual e-mail addicts, and who have a thorough understanding of NANOG culture. If this sounds like you, we would encourage you to put your name forward. To volunteer, send a note to steering(a)nanog.org containing a brief summary of how you think the NANOG list could be better, and how you think you could help make a difference. We'll consider every response we get before 4 August 2007. Thanks for your time, Joe Abley, for the NANOG SC Alex Pilosov, for the NANOG MLC

1 0

Re: Why do we use facilities with EPO's?
by George William Herbert 25 Jul '07

25 Jul '07

>If they can be avoided, why do we put up with them? Major electrical fire, or arcing in the datacenter. Flood in the datacenter. Accidental water sprinkler discharge in the datacenter. Equipment fire that the FM-200 didn't put out, and you want not to have the sprinklers go off if you can avoid it. Equipment fire in a room without FM-200 in the first place. Equipment fire with sprinkler discharge, and you'd really rather just dry out the equipment than have to replace it all. Electrical worker who's electrocuted himself and going to die if you can't make the power go away. There are a very few exceptions, but for our practical purposes, people really ought to simply go to multiple site redundancy rather than thinking about bending major safety assumptions in how we operate individual buildings. You may have a few less outages, but you may also kill someone. What the Navy does on ships, for critical ship safety and combat systems; what the FAA does for their radar facilities and air traffic control facilities; what Telcos do, these are different operating regimes, and there are associated higher risk acceptance with the different equipment setups and safety procedures. -george william herbert gherbert(a)retro.com

2 1

An Internet IPv6 Transition Plan
by John Curran 25 Jul '07

25 Jul '07

Folks - There's quite a few IPv6 transition technologies, each with its own camp of supporters based on particular world view of the hardest & easiest system elements to change. One of the challenges this poses is that it's very easy to get caught up in the various transition approaches and miss the high-level view of what needs to be accomplished. In an effort to communicate one possible transition plan in a technology agnostic manner, I've written an Internet draft which highlights the expectations that organizations could face over the next few years: http://www.ietf.org/internet-drafts/draft-jcurran-v6transitionplan-00.txt I'd be interested in hearing any and all feedback from the NANOG community on this draft; feel free to send such privately if you'd prefer a degree of anonymity, or have the urge to use language inappropriate in public... ;-) Thanks! /John

7 11

Re: Why do we use facilities with EPO's?
by George William Herbert 25 Jul '07

25 Jul '07

>I've always wondered who died or was injured and caused the EPO to >come in to existence. There have been lots of "EPO caused downtime" >stories, but does anyone on the NANOG list even have one single >"Thank God for the EPO" story? I'll feel better about the general >state of the world if I know that the EPO actually has a real valid >use that has been ACTUALLY PROVEN IN PRACTICE rather than just in >someone's mind. > > >-Jerry <----Is so anti EPO, he has no remote EPO buttons, and even >has the irrational fear about the jumper on the "EPO terminal strip" >inside his UPSes coming undone. A friend of mine is a volunteer firefighter (and ex-ISP CEO, used to be on the list). I'm not sure that he'd voluntarily enter a burning datacenter to put it out if there wasn't an EPO. Firefighters won't use water on live electrical fires. If your response plan to a fire in the datacenter is "the building burns down and hope nobody's inside it still" then you're set. I've seen someone electrocuted and frozen on the wire in a non-datacenter setting; we flipped the building breaker, which was further than an EPO would have been. It wasn't through his chest, and was only 110 V, so it probably didn't make a difference that it took a minute to turn it off instead of 10 seconds. There were no severe burns, etc. I've seen equipment catch on fire in a datacenter. If I hadn't been able to cut off the power locally, the EPO was the last line of defense... People I know have hit the EPO when sprinklers discharged in the datacenter. If you're lucky these won't happen to you. But that's not why safety rules are put in place. Unluck happens. -george william herbert gherbert(a)retro.com

1 0