October 2012 - Test - lists.nanog.org

NJ impact
by Alex Rubenstein 07 Nov '12

07 Nov '12

I had to summarize this recently for a news article I was interviewed for, so I figured I forward: -- Of our three datacenters, this is what we saw: Parsippany 1 (OCT) - The worst we saw here was several sub-second power hits. UPS's held without problem, and we did not transfer to generator at all yet. Parsippany 2 (WBR) - Transferred to generator at about 7:55 PM EST Monday as a precautionary measure due to ongoing utility power hits. However, shortly after transfer, utility voltage went to 0 on all phases; around 10p power returned, but abnormally high (seeing about 550 volts on 480 volt bus). We retransferred last night as utility voltage settled down. Cedar Knolls 1 (MMU) - Briefly transferred to generator around 7:10, then back to utility. We then force transferred to generator around 8pm and stayed until this morning. Returned to utility and all systems are normal. We have accommodated many customers (and non-customers) by allowing them to use our conference rooms and some offices, mainly because they lost power at their buildings, lost access to their buildings, or generators ran out of fuel. We literally have card tables setup in hallways and common areas where people sit and use Wifi to conduct business. There has been no service interruption of any sort to any of our datacenter customers, as of this writing. We lost one span of fiber optics between MMU and 165 Halsey, Newark (NWR), network healed in less than 1 second. It was imperceptible. That was early though, Monday afternoon. This has since restored today. We have sales people onsite who are actively taking sales calls, and there is a measurable amount of customers moving equipment into our facilities as I write this. Many are moving equipment from offices with no power and no ETR in sight. We have electricians onsite to help accommodate. The facilities department has been manning all locations around the clock. All sites remain accessible by road. There was never a situation where you could not over the last 48 hours. We have no equipment in 111 8th, so I cannot offer any telemetry there. However, our gear in 60 Hudson St (Tel-X 9th floor) is still online. We saw no interruption at 165 Halsey, except for some very brief utility hits. UPS's held. We are seeing hundreds, if not thousands, of DSL / T1 / T3 / Ethernet customers down - presumably mostly power loss on the CPE. We lost our connection to Covad, presumably they have gear in a building which lost power. These are only just starting to come back. We have seen substantial interruption to the cellular networks; in many cases, ATT Wireless (who NAC uses) is unable to complete a call. In certain areas, there is no signal at all. Where I live, in 07874 (Byram), many roads are impassable and most everyone does not have power. Trees are down everywhere. I heard that a Byram squad car had a tree land on it while responding to a call. Schools are closed, Halloween has been "rescheduled" according to our governor. My cable is down, cable modem is down, POTS service is out - yet I still have xDSL (loop by Verizon, IP by NAC) and I run on that successfully. I am unable to get any ATT cell service at my house (usually at least three bars). I am on generator at my house, consuming about 2 gal/hour (propane), with a little over 500 gallons left. Gasonline is becoming an issue. As I understand, having talked to our oil supplier, there is 0 refinery capacity in NJ at this time. All gasoline is being imported from PA, DE, MD, etc. Most stations don't have power. Ones that do are generally out of gas. If they have gas, they have mile long lines and are rationing. Diesel is unavailable if you had not pre-arranged for it. Widespread power outages remain. I am running on generator at my house (Generac QT048) on a 1000 gallon tank of propane, have been since Monday about 7pm. It is estimated at least 7 to 10 days before power restoration. I have yet to see one JCPL truck anywhere near where I live. That is all I have. This is the road leading to the Byram Police Dept - I took this yesterday morning: https://sphotos-a.xx.fbcdn.net/hphotos-prn1/12696_4311181250657_900557870_n… This is typically what Northern NJ looks like: https://sphotos-a.xx.fbcdn.net/hphotos-ash3/548544_4311185050752_1011136048… https://sphotos-b.xx.fbcdn.net/hphotos-ash3/61115_4317167440308_289655722_n… (not the top of the telephone pole hanging from the wires)

2 2

NSA and the exchanges
by andy lam 06 Nov '12

06 Nov '12

Anyone knows if there's a way to find out how involved NSA monitors 151 front street at Toronto? NSA allegedly monitors data centres in the US, but does it have the same influence at a building sitting in its neighbor's soil? There's something on the web like www.ixmaps.ca that tries to piece it together. but not sure how helpful the information on there really is? feedback welcome.

8 8

Network scan tool/appliance horror stories
by Pedersen, Sean 02 Nov '12

02 Nov '12

We're evaluating several tools at the moment, and one vendor wants to dynamically scan our network to pick up hosts - SNMP, port-scans, WMI, the works. I was curious if anyone had any particularly gruesome horror stories of scanning tools run amok.

14 14

IOS architecture
by Andrey Khomyakov 01 Nov '12

01 Nov '12

Does anyone know if there is an updated version of this book? Year 2000 seems so distant now... http://www.amazon.com/dp/B0051TM4RC/ref=rdr_kindle_ext_tmb --Andrey

9 8

Google burp
by Blair Trosper 31 Oct '12

31 Oct '12

I guess I'll be the one to ask...what's going on over at Google? Service interruptions and front-end errors all over the place across what appears to be all services, though Gmail seems to have bounced back up. Google's service disruption is about to bring Twitter's service to its knees as people complain and try to figure out what's going on. Blair Trosper Updraft Networks & The North Texas GigaPOP

9 10

75th Broad up/down?
by Jeff Shultz 31 Oct '12

31 Oct '12

An online magazine I work with is, we believe, hosted at 75 Broad or 111 8th. It dropped out Monday night, right after they announced that the diesel pumps had fried due to flooding at 75 Broad, came back up early Tuesday morning, and then died again this morning. IP address is in 74.63.44.0/24 block (which may be part of a larger block). Pinging 74.63.44.255 gets me replies from 208.122.44.210 only. Anyone know the current status of these facilities? I've got an e-mail in to noc@internap but haven't heard back - and if they have a facility down they're probably too busy to reply to a single web hosting customer. I have seen the report from earlier this morning indicating that 111 8th had an issue that's been fixed - but the Internap NOC line (877.843.4662) still indicates that NYM008 is having a power outage. Thanks! -- Jeff Shultz

3 2

Flood affecting US east coast communication facilities?
by Kauto Huopio 31 Oct '12

31 Oct '12

Greetings all, Any reports on damage to communications facilities on US east coast? --Kauto -- Kauto Huopio - kauto(a)huopio.fi (dayjob @ CERT-FI )

6 5

New York Crews?
by Justin Wilson 31 Oct '12

31 Oct '12

Anyone know of lists, contacts, etc. for companies looking for I.T. Folks for help with cleanup and such on the eastern seaboard? I am guessing there will be a demand for anyone from cable pullers to Engineers. I have some free time on my hands and would gladly take a cut in pay to go out and work with the cleanup. I can terminate cables, climb towers, etc. I am sure I am not the only underemployed I.T. Guy who could spend a week or two helping a data center, or other entity. Just wondering if anyone knew of any resources, groups, contacts. Thanks, Justin -- Justin Wilson <j2sw(a)mtin.net> Aol & Yahoo IM: j2sw http://www.mtin.net/blog xISP News http://www.twitter.com/j2sw Follow me on Twitter

3 2

altdb contact
by Serajul Hossain 31 Oct '12

31 Oct '12

Dear Nanog member, I would like to know if there is anyone on this mailing list who is volunteering to help manage altdb.net It seems like no one is actually responding to db-admin(a)altdb.net We need to update information on our new address block that was originally owned by a different company. Regards, Serajul

1 0

Twitter Issue
by Rashed Alwarrag 30 Oct '12

30 Oct '12

Hi All Was there is any global issue in Twitter.com here is saudi arabia we were not able to access twitter.com from 3:30 to 06:30 GMT any idea ? *Rashed Alwarrag *

2 1