- Test - lists.nanog.org

[Nanog] test
by Larry J. Blunk 25 Jul '07

25 Jul '07

testing

1 0

ASN Name of the week
by Marshall Eubanks 25 Jul '07

25 Jul '07

This just appeared in my BGP ASN Number Name Handle Location Organization 40543 1-800-GOT-JUNK [ABI19-ARIN] {Vancouver, BC, CA} 1-800-GOT-JUNK I guess somebody thinks that whois has advertising potential. Regards Marshall

7 9

Problems getting Cisco router and Motorola Nextlevel system to work together
by Brian Raaen 25 Jul '07

25 Jul '07

I am having some difficulties involving using a Cisco 7200 router to terminate ATM sessions from a motorola nextlevel IPTV system. The router is currently configured to use IRB which is a hybrid process. The problems is that the IRB process is overloaded and is dropping traffic faster than it can process it. I opened a case with Cisco TAC, and they recommended using RBE instead of IRB. While I have been trying to plan migrating the system to RBE I discovered that Motorola uses a concept called "dynamic pvc's" to assign the pvc's to the CPE devices (a IPTV unit that has a data port). The device uses two PVC's one for data and one for IPTV. The system dynamically assigns the PVCs when the CPE devices connects. This looks like it would not work with RBE, since the pvc can change before the dhcp lease expires. Having this router dropping traffic, has been causing severe problems for end users and is causing an ongoing system outage. I am currently trying to work with both Motorola and Cisco, however both vendors are blaming the problem on the other vendor. I am not sure what to do. Motorola says their system only works with IRB and Cisco says the router will not function with this size network using IRB. Has anyone else arrived at a working solution using a Cisco 7200 router to terminate a Motorola nextlevel system support approximately 2000-3000 end users. I would be extremely gratefull if anyone who has worked with this type of system could help shed some light on this problem. Thank you in advance. -- Brian Raaen Network Engineer braaen (at) zcorum (dot) com

3 4

more on SF outage
by Fred Heutte 25 Jul '07

25 Jul '07

The story is evolving, as you'd expect. An excerpt from the latest update at the Chron is below. Nothing at this hour has appeared on 365main.com. But go there and take a look and decide if *your* colo has better stuff. http://www.365main.com/365_main_tour_2.html And now that we know this isn't enough . . . your move. fh PS I am not a customer of 365 Main or any colo for that matter. ---------------------- http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2007/07/24/BAG9NR67253.DTL… The datacenter 365 Main -- which hosts a number of Internet sites, including Craigslist, Yelp and RedEnvelope -- also lost power, causing frustration among many Web surfers. Several of the sites posted messages to explain to their users that they were having difficulties, while others said they were undergoing scheduled maintenance or upgrades. After about 45 mintues, power to the datacenter was restored by back-up generators that remained in service until PG&E could confirm that its supply was stable. There was no immediate response from the facility as to why the generators were not turned on right after the blackout. Some of the sites, including Craigslist, remained down even after power was restored, as administrators ensured that data in the server hadn't been damaged, among other checks.

2 1

Fw: Re: San Francisco Power Outage
by Fred Heutte 25 Jul '07

25 Jul '07

Don't be so fast to point the finger. Generally speaking, blame is obvious from the initial news reports but tends to diminish with retrospective fact-based assessment. For example: it's "obvious" that serious net sites need multihoming. But what if your multihomed bits go through the same pipe (or worse, through the same fiber)? Who do you blame when you find out? Worse, in terms of blame: who can you go to beforehand who actually knows where that can happen? I well remember this slide from Sean Donelan's talk at NANOG23: --------------------------- http://www.nanog.org/mtg-0110/ppt/donelan_files/v3_document.htm What Didn't Work - Diversity and Avoidance * Equipment in the World Trade Center primarily served tenants in complex (shared fate) * SONET ring through WTC tower 1 and alternate path through WTC tower 2 * Damage to 140 West Street central office and surrounding underground infrastructure * Backup circuit routed through same facility * Advanced data circuits (ISDN/DSL) concentrated in a few central offices --------------------------- The real answer, found elsewhere in Sean's talk, is that the design of the net has always encouraged redundancy as an engineering principle. Stress situations is where that pays off, even though it can't solve every possible eventuality (and as has already been noted, redundant equipment also fails as well as creating more complex failure modes). The net had problems on 9/11, especially around the WTC, but Sean's slides document remarkable resiliency even in that area. The power went off at a key spot in the San Francisco infrastructure today. But as far as I know, even though it was mentioned in the Chron article, Craigslist stayed online because they have a distributed and redundant system (which is not to say, impervious to all failure modes). Some shortcomings are obvious, but all I am saying is, before rushing to cast blame, it's a good idea to try and collect some facts. fh ------------------- http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2007/07/24/BAG9NR67253.DTL… Power restored in San Francisco Marisa Lagos and Matthew B. Stannard, Chronicle Staff Writers Tuesday, July 24, 2007 (07-24) 16:57 PDT SAN FRANCISCO -- Between 30,000 and 50,000 Pacific Gas and Electric Co. customers in San Francisco and the northern Peninsula lost power for several hours this afternoon after what witnesses described as an explosion under a manhole cover on Mission Street, the utility said. Brian Swanson, a spokesman for the utility, said power failures were reported throughout wide swaths of the east side of San Francisco, including downtown and at PG&E's own office on Beale Street near the Ferry Building. The outage first occurred at about 1:50 p.m., and electricity flickered on and off at least five times before power was restored at about 4 p.m. PG&E officials said the source of the power outage was an underground failure. Standing at a manhole in a plaza at 560 Mission St. in San Francisco, where witnesses reported hearing an explosion, Swanson said it could have been the source of the outage, but officials were still investigating. The incident recalled an August 2005 explosion in an underground vault at Post and Kearny streets that critically injured a woman who was walking by. At the time, PG&E blamed high levels of moisture in the attached high-voltage chambers and said it was checking the safety of about 1,000 other high-voltage chambers. Swanson said today's incident -- in which no one was injured -- was caused by some sort of fault in the line. "It is completely unrelated to what happened two years ago," he said. Witnesses said they heard an explosion at about 1:50 p.m., then saw flames coming from the manhole. Actor Torino Von Jones, 32, said he was filming a Fruit of the Loom commercial down the block at the time. "We were standing over there waiting for the camera cue when we heard a big explosion," he said. "Flames came up taller than I am, and I'm 6-foot-2." "Naturally, when you hear an explosion, you think the worst," Von Jones said. Nevertheless, he hurried back to work. "We're Fruit of the Loom -- we've got to make this commercial." The outage briefly affected some Muni buses and trains, but all were back to normal by 3 p.m., a spokeswoman said. Workers at several downtown and South of Market offices were reportedly sent home for the day following the outage. Additionally, the datacenter 365 Main -- which hosts Web sites including Craigslist and Yelp -- lost power. ------ mail forwarded, original message follows ------ To: nanog(a)merit.edu From: sethm(a)rollernet.us <Seth Mattinen> Subject: Re: San Francisco Power Outage Date: Tue, 24 Jul 2007 15:54:08 -0700 Jonathan Lassoff wrote: > > Just a heads up to anyone on list that PG&E has just sustained a large > outage in San Francisco that has caused a few hiccups (both network, > electrical, infrastructural, etc.) around the city. > > I've confirmed that both customers in 365 Main and parts of telecom 1 > have both sustained brief blackouts. No word yet form 200 Paul. > > Anyone in the area that could use a hand with anything, I'll probably > be wrapping up fixes for my stuff soon, and would be glad to help > however I can. > I have a question: does anyone seriously accept "oh, power trouble" as a reason your servers went offline? Where's the generators? UPS? Testing said combination of UPS and generators? What if it was important? I honestly find it hard to believe anyone runs a facility like that and people actually *pay* for it. If you do accept this is a good reason for failure, why? ~Seth

1 0

365 Main - an operators' nightmare?
by joe mcguckin 25 Jul '07

25 Jul '07

I don't know if this is true, but it's more exciting reading than blaming it on a 'power outage'... http://valleywag.com/tech/breakdowns/a-drunk-employee-kills-all-of- the-websites-you-care-about-282021.php Joe McGuckin ViaNet Communications joe(a)via.net 650-207-0372 cell 650-213-1302 office 650-969-2124 fax

3 2

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking )
by Paul Ferguson 24 Jul '07

24 Jul '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -- Christopher Morrow <christopher.morrow(a)verizonbusiness.com> wrote: >I'd love to see CPE dsl/cable-modem providers integrate with a 'service' >that lists out 'bad' things. it'd be nice if the user could even tailor >that list (just C&C or C&C + child-porn or C&C older not than X >days/hours/minutes) ... I think it might even help, and be vendor >>agnostic (from a provide and hardware) perspective. Ironically, that is exactly part of a product announcement that we (Trend Micro) are making on 30 July. Since this topic arose, I saw Trend mentioned as a possible product "culprit" in this scenario, but it isn't. Yet. :-) The particular service to be announced on Monday (BIS, or Botnet Identification Service), is nothing more than a BGP feed of _known_ and _vetted_ botnet C&Cs as /32s, intended to be a black-hole feed. Interested folks should either e-mail me off-list, or just wait for the official announcement on 30 July. Cheers, - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.6.2 (Build 2014) wj8DBQFGplq5q1pz9mNUZTMRAnFzAKCicaHuvoTwJk92hPOOu2E/ofjhegCcCrMc XCA4rpUCimConxtKV/Qrsfs= =N2f1 -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/

3 2

"A drunk employee kills all of the websites you care about"
by Michael Greb 24 Jul '07

24 Jul '07

<http://valleywag.com/tech/breakdowns/a-drunk-employee-kills-all-of-the-webs…> or <http://tinyurl.com/25ly7m>. ------8<------- A drunk employee kills all of the websites you care about 365 Main365 Main, a datacenter on the edge of San Francisco's Financial District, is popular with Soma startups for its proximity and its state-of-the-art facilities. Or it used to be, anyway, until a power outage took down sites including Craigslist, Six Apart's TypePad and LiveJournal blogging sites, local listings site Yelp, and blog search engine Technorati. The cause? You won't believe it. A source close to the company says: Someone came in shitfaced drunk, got angry, went berserk, and fucked up a lot of stuff. There's an outage on 40 or so racks at minimum. Whoever it is, while we like how you roll in theory, in practice, we'd appreciate it if you laid off the servers running websites we actually use. We're sure 365 Main will deny that such a thing could ever happen. And, conveniently, the neighborhood is having power troubles, too. But here's a question: When you have several levels of redundant power, what could bring your customers' servers down other than something like an employee physically ripping the plugs out of the wall? Update: Technorati's Dave Sifry just sent this email: Folks, I just wanted to let you know, it looks like San Francisco is having a MAJOR power event, with outages from the Financial district all the way down to Daly City. One of our colos at 365 Main Street has experienced a power outage (never mind that they always swear up and down that this kind of event can't possibly happen, oh no, they have multiple redundant systems and they charge us up the wazoo to make sure that we'll have business continuity, so of course, this isn't really happening, oh yes) however, our other data centers are all up and running, so we hope to be back up and running as quickly as possible. I'll keep you all updated on progress, and I appreciate you bearing with us as we work our way through this... Dave ------8<-------

1 0

Email lists pertaining to NA colo sites?
by Tuc at T-B-O-H.NET 24 Jul '07

24 Jul '07

Hi, I was wondering if someone was running a single repository of lists run by either Colo sites, or others in support of colo sites, in the North America. (Like for 25 Broadway {Adam?}, or even more specifically 118th, 5th floor, room 518 and anyone that might be in that room in the next week or so. :) ) Maybe there is a general list where someone can post "Hey, I'm in 25B, 5th floor and I need a patch cable" or even more specifically "Hey, I need someone who can get to my cabinet at the specificied location and use a pen to reboot one of my servers so I can save the $43.75 and not spend $5 in gas, $6 for tolls, and $17 for parking to press a button, wait 30 seconds, and press again". Thanks, Tuc

1 0

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking )
by Paul Ferguson 24 Jul '07

24 Jul '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -- "Chris L. Morrow" <christopher.morrow(a)verizonbusiness.com> wrote: >On Tue, 24 Jul 2007, Paul Ferguson wrote: >> >> The particular service to be announced on Monday (BIS, or Botnet >> Identification Service), is nothing more than a BGP feed of _known_ >> and _vetted_ botnet C&Cs as /32s, intended to be a black-hole feed. >> >> Interested folks should either e-mail me off-list, or just wait for >> the official announcement on 30 July. >> > >note that this will take out vhost systems... unless they are vetted off >the list, which is certainly possible of course. But of course. :-) - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.6.2 (Build 2014) wj8DBQFGpm6Rq1pz9mNUZTMRAuVXAJ4jxpw0BRVlv8qmNXZQ9P++VHTjcwCg9JGy 591aYUFqou4ziRL5TQASDGg= =lQba -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/

1 0