- NANOG - lists.nanog.org

RE: Verizon outage in Southern California?
by Olsen, Jason 19 Oct '05

19 Oct '05

> From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On > Behalf Of Matthew Black > Sent: Tuesday, October 18, 2005 3:13 PM > Telephone service is beginning to be restored in the Long > Beach area but is still sporadic. Our ATM WAN link through Sprint came back up around 1345 Central time, and the two DS1s for the school's Internet service were revived about fifteen minutes ago (1507 CDT). They've been rock-solid so something must be going right out there. When I called Sprint about any information they might have for the outage the tech said that the area was down due to a Verizon DACS failure. That must have been a spectacular failure, because I'm reading that it wiped out most everything ( http://www2.presstelegram.com/news/ci_3128087 indicates four tandems hit?! ) in the area. The articles are primarily focusing on the impact to E911 services, followed with the hit to POTS lines. I have yet to see any mention of impact to data in any of 'em. Here's what intrigues me about this outage: if it wiped out E911, most of the POTS and also impacted data services (as Jay Hannigan and I can attest), how did the cell towers that are also served by the network live through it? Jason "Feren" Olsen Senior Network Engineer DeVry, Inc Em: jolsen(a)devry.com Ph: 630-645-1607 One Tower Lane INOC-DBA: 19258*526 Fx: 630-389-2929 Oakbrook Terrace, IL 60181

5 4

RE: Scalability issues in the Internet routing system
by Susan Hares 19 Oct '05

19 Oct '05

Elmer: You are not cooking for the routing table (hence loss of information). You are cooking for the forwarding entries in the line cards.. As to architectural discussion, I believe the IRTF is publishing work from On NG architecture. I'll be glad to send you pointers. 2 panels of routing experts suggested algorithms and changes. That will tell you some of the things that were thought of. Sue -----Original Message----- From: Elmar K. Bins [mailto:elmi@4ever.de] Sent: Wednesday, October 19, 2005 3:31 AM To: Susan Hares Cc: nanog(a)nanog.org Subject: Re: Scalability issues in the Internet routing system Susasn, > Using the compression ("cooking") per router can provide one level of > abstraction [reduction of prefix space] at router. So cooking down your > Large number of routes to a "minimum" set of routes can provide some > leverage against the prefix growth. By cooking down the prefixes you unfortunately lose topology information which might be a bad thing, and at the same moment disrespect the site's wish to how it would like to be routed. Another bad thing, if you think of companies/sites paying for the entire network in the long run. Apart from that, IMHO cooking down the prefixes only buys time, but does not solve the problem. More people will multihome, and with the current mechanisms and routing cloud, they have to do it by injecting prefixes. I'm not sure whether this hasn't long become an architectural question and should be moved to the (new) IETF arch list. Opinions? Yours, Elmi. PS: Btw, anyone can give me a hint on where to discuss new ideas for e.g. routing schemes (and finding out whether it's an old idea)? -- "Begehe nur nicht den Fehler, Meinung durch Sachverstand zu substituieren." (PLemken, <bu6o7e$e6v0p$2(a)ID-31.news.uni-berlin.de>) --------------------------------------------------------------[ ELMI-RIPE ]---

1 0

FW: Verizon outage in Southern California?
by Hannigan, Martin 18 Oct '05

18 Oct '05

> > > >> From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On > >> Behalf Of Matthew Black > >> Sent: Tuesday, October 18, 2005 3:13 PM > > > I'm not completely familiar with the telco jargon. > Does Tandem mean the same as a local central office, where > POTS lines terminate at the switch? Long Beach has a population > of 470,000. The C/Os I know of are: A tandem office is a CO primarily used as an aggregated switch point between local CO's. Think interconnection of local CO's or long haul tandems. > Alamitos at 7th Street and Termino, ZIP 90814 > > Clark near Clark Ave and Pacific Coast Highway, ZIP 90804 > > LongBeach at 6th Street and Elm Ave, ZIP 90802 > > Lakewood at Clark Ave and Connant St, ZIP 90808 > > LNBHCAXG at 3440 California Ave, ZIP 90807 (for my home) That's the building CLLI, the switch is LNBCHAXGDS0. This one is a 5ESS and serves 12 exchanges. 562-290 562-424 562-426 562-427 562-490 562-492 562-595 562-933 562-981 562-988 562-989 562-997 I see 7 5ESS and 1 Nortel SLC DMS 10, possibly a remote to a campus or something, in Long Beach. 507 E LEW is holding the most switching gear is likely a tandem. Um, I think this is the tandem code, PNTCMIMN50T, and it's servicing about 20 areas. > I have no idea whether cell service was truly affected. The > announcements we sent to our campus suggested people use their > cell phones for 911 service which would be serviced by the > CA Highway Patrol (Erik Estrada, etc.) or a campus telephone > which is serviced by our local campus police (sworn state police). > I was completely unaware of the outage until someone else > mentioned it in my office. If you know of an NPA-NXX of a cell phone that was impacted, send it privately and I'll tell you what CO it terminates in.

3 2

RE: Verizon outage in Southern California?
by Hannigan, Martin 18 Oct '05

18 Oct '05

> ----- Original Message ----- > From: "Hannigan, Martin" <hannigan(a)verisign.com> > To: "Matthew Black" <black(a)csulb.edu>; "NANOG" <nanog(a)nanog.org> > Sent: Tuesday, October 18, 2005 4:35 PM > Subject: FW: Verizon outage in Southern California? > > >507 E LEW is holding the most switching gear is likely > >a tandem. Um, I think this is the tandem code, PNTCMIMN50T, > >and it's servicing about 20 areas. > > Uhh, think you might have the wrong CLLI code. PNTCMIMN50T is > in Pontiac, Michigan and yes, it is a tandem. Yep, Majdi caught it and mailed me earlier. There's a data anomaly between 507 LEW and 507 Mill. I'm looking into it now, but I doubt I'll repost it to NANOG. I think we've defined this outage as much as posslible and most "get it". There's a lot of telco knowledge in SoCal. Kudos! -M<

1 0

RE: Scalability issues in the Internet routing system
by Susan Hares 18 Oct '05

18 Oct '05

(cookinjg = cooking) -- ;-).. Sue (but you knew that.) -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Susan Hares Sent: Tuesday, October 18, 2005 5:49 PM To: Tony Li; Andre Oppermann Cc: nanog(a)nanog.org Subject: RE: Scalability issues in the Internet routing system Andre: Hence my earlier point on #2 - the prefixes in the routing hit one part of Moore's law. The FIB hits another. Using the compression ("cooking") per router can provide one level of abstraction [reduction of prefix space] at router. So cooking down your Large number of routes to a "minimum" set of routes can provide some leverage against the prefix growth. Tony point still stands. The "cookinjg" way to deal with prefix growth by using a compression algorithm for FIB insertion. Moore's law hits the security filters, the route filters, and lots more - that may or may-not be able to be "cooked". Sue Hares -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Tony Li Sent: Tuesday, October 18, 2005 4:46 PM To: Andre Oppermann Cc: nanog(a)nanog.org Subject: Re: Scalability issues in the Internet routing system Andre, > capacity = prefix * path * churnfactor / second > > capacity = prefixes * packets / second > I think it is safe, even with projected AS and IP uptake, to assume > Moore's law can cope with this. > > This one is much harder to cope with as the number of prefixes and > the link speeds are rising. Thus the problem is multiplicative to > quadratic. You'll note that the number of prefixes is key to both of your equations. If the number of prefixes exceeds Moore's law, then it will be very difficult to get either of your equations to remain under Moore's law on the left hand side. That's the whole point of the discussion. Tony

1 0

RE: Verizon outage in Southern California?
by Hannigan, Martin 18 Oct '05

18 Oct '05

> > > From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On > > Behalf Of Matthew Black > > Sent: Tuesday, October 18, 2005 3:13 PM > > > Telephone service is beginning to be restored in the Long > > Beach area but is still sporadic. > > Our ATM WAN link through Sprint came back up around 1345 Central time, > and the two DS1s for the school's Internet service were revived about > fifteen minutes ago (1507 CDT). They've been rock-solid so something > must be going right out there. > > When I called Sprint about any information they might have for the > outage the tech said that the area was down due to a Verizon DACS > failure. That must have been a spectacular failure, because > I'm reading > that it wiped out most everything ( > http://www2.presstelegram.com/news/ci_3128087 indicates four tandems > hit?! ) in the area. The articles are primarily focusing on > the impact > to E911 services, followed with the hit to POTS lines. I have yet to > see any mention of impact to data in any of 'em. Here's what > intrigues > me about this outage: if it wiped out E911, most of the POTS and also > impacted data services (as Jay Hannigan and I can attest), how did the > cell towers that are also served by the network live through it? The dependancy between all of those would be a DACS so that seems to make sense. I'm guessing the impacted circuits were DS3 or below, with Verizon providing resale of the Z ends. I'm not sure of the relation to E911 though. Could be, but it sounds odd since E911 has redundancies to tandems IIRC. My guess is water on a DACS bay or complete power loss in the CO (rarer than water on a DACS). -M<

2 1

RE: And Now for Something Completely Different (was Re: IPv6 news)
by Church, Chuck 18 Oct '05

18 Oct '05

Nanog, I've been thinking a bunch about this IPv6 multihoming issue. It seems that the method of hierarchical summarization will keep the global tables small for all single-homed end user blocks. But the multihomed ones will be the problem. The possible solution I've been thinking about is 'adjacency blocks', for lack of a better term. In theory, the first customer to home to two different ISPs causes a new large address block to be advertised upstream by these two ISPs. Further customers homing to these two ISPs get an allocation out of this same block. The two ISPs will only announce the large block. Of course there are issues involving failure and scalability... Failure would involve an ISP losing contact with end customer, but still announcing the aggregate upstream. This can be solved by requiring that two ISPs must have a direct peering agreement, before they can accept dual-homed customers. Or a possible method (maybe using communities?) where ISP B will announce the customer's actual block (the small one) to it's upstreams, if notified by ISP A that it's not reachable by them. When ISP A resumes contact with end customer, ISP B retracts the smaller prefix. Scalability is an obvious issue, as the possible number of these 'adjacency blocks' would be N * (N-1), where N is the number of ISPs in the world. Obviously pretty large. But I feel the number of ISPs that people would actually dual home to (due to reputation, regional existence, etc) is a few orders of magnitude smaller. ~100,000 prefixes (each can be an ASN, I suppose) should cover all needs, doing some simple math. The downside is that end customers are going to lose the ability to prefer traffic from one ISP versus another for inbound traffic. That alone might be a show-stopper, not sure how important it is. Since IPv6 is a whole new ballgame, maybe it's ok to change the rules... Looking for any thoughts about it. I'm sure there's things I haven't considered, but the people I've bounced it off of haven't seen any obvious problems. Flame-retardant clothes on, just in case though. Chuck >Every multi-homer will be needing their own ASN, so that's what clutters >up your routing tables. It's economy there. Btw, a lot of ASNs advertise >one network only. People surely think multihoming is important to them >(and I cannot blame them for that). >Hierarchical routing is one possible solution, with a lot of drawbacks >and problems. Forget about geographic hierarchies; there's always people >who do not peer. Visibility radius limitation is another (I cannot believe >the idea is new, I only don't know what it's called).

3 2

RE: Verizon outage in Southern California?
by Olsen, Jason 18 Oct '05

18 Oct '05

> We lost connectivity to a number of customers in the Los > Angeles and Long beach area and the local AM radio news > stations are talking about some major telephone issues > regarding Verizon. > > Anyone have more information? It seems to have started > around 02:30 local time this morning. We lost connectivity (WAN/Internet/POTS) to our Long Beach site at around 2:27 AM PDT today. Several news agencies are reporting it on the web (hooray news.google.com) citing "mechanical glitches" or bad weather. I've also heard rumors of a power outage, though I find those highly suspect. Choose your villan, I guess. http://www.mercurynews.com/mld/mercurynews/news/local/states/california/ northern_california/12933014.htm http://www.nbc4.tv/news/5115425/detail.html http://abclocal.go.com/kabc/story?section=local&id=3546999 Jason "Feren" Olsen Senior Network Engineer DeVry, Inc Em: jolsen(a)devry.com Ph: 630-645-1607 One Tower Lane INOC-DBA: 19258*526 Fx: 630-389-2929 Oakbrook Terrace, IL 60181

3 4

RE: Scalability issues in the Internet routing system - #2
by Susan Hares 18 Oct '05

18 Oct '05

Andre: #2 FIBs doesn't have to be impacted by Moore's law. In certain times of routing technology people "cooked" routing tables in order to solve #2. The key there is to run through the proposed FIB table Generated by a routing table and cook it for FIB insertion. You can often summarize (from the viewpoint of a particular router) all the routes into a manageable set of FIB entries. The issue is how much fluctuation. 10% (actually 8-14% but 10% is average) of the Internet (see route-views, etc). FIB hardware can allow you to split the prefix load and the next-hops. So, you load prefixes that fluctuate and only break the prefix-next-hop when they fluctuate. .. bottom line.. I don't think we will be limited by Moore's law for #2.. If we utilize existing technology existing in Hw and software methodology, we can change the game on #2. Sue Hares PS - My answers tend to be brief - glad to expand in private or public. Your choice. -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Andre Oppermann Sent: Tuesday, October 18, 2005 11:31 AM To: nanog(a)nanog.org Subject: Scalability issues in the Internet routing system I guess it's time to have a look at the actual scalability issues we face in the Internet routing system. Maybe the area of action becomes a bit more clear with such an assessment. In the current Internet routing system we face two distinctive scalability issues: 1. The number of prefixes*paths in the routing table and interdomain routing system (BGP) This problem scales with the number of prefixes and available paths to a particlar router/network in addition to constant churn in the reachablility state. The required capacity for a routers control plane is: capacity = prefix * path * churnfactor / second I think it is safe, even with projected AS and IP uptake, to assume Moore's law can cope with this. 2. The number of longest match prefixes in the forwarding table This problem scales with the number of prefixes and the number of packets per second the router has to process under full or expected load. The required capacity for a routers forwarding plane is: capacity = prefixes * packets / second This one is much harder to cope with as the number of prefixes and the link speeds are rising. Thus the problem is multiplicative to quadratic. Here I think Moore's law doesn't cope with the increase in projected growth in longest prefix match prefixes and link speed. Doing longest prefix matches in hardware is relatively complex. Even more so for the additional bits in IPv6. Doing perfect matches in hardware is much easier though... -- Andre

2 1

RE: Verizon outage in Southern California?
by Hannigan, Martin 18 Oct '05

18 Oct '05

> > > > We lost connectivity to a number of customers in the Los Angeles and > Long beach area and the local AM radio news stations are talking about > some major telephone issues regarding Verizon. > > Anyone have more information? It seems to have started around 02:30 > local time this morning. There was a post here earlier regarding a major outage. Did you lose POTS or circuit level connectivity to customers or both? (I see circuits live in LA) -M<

2 1