March 1998 - Test - lists.nanog.org

Provider communications, is it time yet?
by Sean Donelan 11 Mar '98

11 Mar '98

Has your management received enough shocks to system to make talking about provider communications worthwhile yet? Or are they still in a state of denial? Or have the shocks not been big enough to be noticed? The last couple of months have seen many of the same types of outages over and over again, yet provider communications during these outages seems minimal, or even non-existent. Having an outage lists is a nice thing for your sales people to say, but if no information is given during an outage, what was the point of setting up the list? Hint: in the absence of pro-active notifications, more and more people are going to start their own active monitoring, i.e. your most critical machines are going to get pinged to death by more and more people checking if those critical machines are up. I wonder what Netscape's current ping rate is? Acts of God and other natural disasters seem to get modest reporting. But anything else seems to be treated like a tree falling in the forest, if I don't say anything maybe no one will notice it fell down. That works great, sometimes. But it also means we have no way to collect reliable data on how well the "Internet" is doing. Which means every once in a while an acorn falls out of a tree, and hits chicken little reporter on the head, and we end up with stories on TV how the sky is falling. Then we get called into our management's office to explain what we plan on doing about the sky falling. Overall, most of us, I hope, understand the sky is not falling. But as an industry we have no data to back that up. If one provider goes off the air for eight hours, they may have shot a big hole in their 99.9% reliability. But what does that mean for the Internet as a whole? How much customer traffic was affected? Do we even know how much customer traffic exists on the Internet as a whole? The so-called denominator problem. In other words when one provider carrying 10% or 90% of the Internet traffic has problems, does it affect the total amount of Internet traffic, or is a substantial portion of the traffic re-routed around the damaged provider? I've actually been a bit surprised just how little my total traffic volume has been effected even when major providers or exchange points go off the air for periods of time. Maybe we've achieved TYMNET/TELENET nirvana, and any useful site on the net is connected to all of the IRCs :-). Q: What's the worst thing that can happen to a backbone? A: Go down, and no one notice. Fiber cuts, operator errors, software bugs, etc are normal network events. For the Internet, the unusual thing is supposed to be when a single event has a disruptive affect on more than just that one system. With only modest pre-planning, and good communications between providers, the Internet continues flapping on its merry way through most of these events. The myth of the nuclear bomb and the Internet is somewhat true, the net will keep working, but routing is going to be sub-optimal at the node which gets nuked. The problem is as an industry we don't seem to have a good grasp of exactly what is the root cause of most the problems. Sure, you can say fiber cuts are the number one cause of Internet outages, but how do you know? Maybe its power failures, or configuration errors, or any number of other things. And my biggest question, why do single events seem to have such a widespread effect on the Internet? Is it a design error, or an assembly error? Second hint: if you only have two name servers, and they are both on the same subnet, your life is going to suck when that subnet gets nuked. Spring planting and backhoe season is just around the corner, do you have a second fiber strand? Ok, what's this got to do with provider communications? Q: Find the value of the Internet traffic denominator, so then statements like 40% of the Internet traffic make sense. A: Get all Internet providers to give KC? the inbytes/outbytes of their network cloud for anonymization and consolidation. Q: What are the top three causes of Internet outages? Nanog did the top three problems, but not the causes. A: Show me your trouble tickets.... Ok, how to normalize the data? Q: Do you want active pinging or provider pro-active notification? A: Software is Internet law. Develop an Internet (not single-point) status board for providers that don't cooperate. A software message system for providers that do cooperate, i.e. spin control. Decentralize, and provider control. Q: How good are the alternate paths in the Internet? A: Unfortunately we don't get to find out until the disaster. Q: Who reports Internet outages first? CNN or the provider? A: CNN seems to still be reporting them first. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

1 0

ip flow-export
by Alex Rubenstein 11 Mar '98

11 Mar '98

Does anyone have some neat tools to play with collected cflowd data? TIA

2 1

Re: simple question as always..
by Danny McPherson 10 Mar '98

10 Mar '98

> could someone explain to me why I can ping but traceroute > seems to have problem reaching it. The combination of a firewall (permitting ICMP echo replies/requests, but filtering your UDP traceroute probes) .. and a traceroute client that doesn't correctly handle (IMO) the response (ICMP Type 3, Code 13: Communication Administratively Prohibited). Many traceroute clients (i.e. ftp://ftp.ee.lbl.gov/traceroute.tar.Z) better indicate things of this nature: danny@ice% traceroute auth3.ipass.com traceroute: Warning: Multiple interfaces found; using 204.74.125.252 @ eb0 traceroute to ren.iPass.COM (167.216.128.232), 30 hops max, 40 byte packets 1 fe0-2.hopper.genuity.net (207.240.3.254) 0.883 ms 0.728 ms 2.731 ms 2 fe11-1.border1.phx1.genuity.net (207.240.103.189) 1.894 ms 13.040 ms 1.377 ms 3 fe-6-0.core1.phx1.genuity.net (207.240.1.1) 1.500 ms 69.045 ms 3.208 ms 4 core1.sjc1.genuity.net (207.240.0.9) 20.429 ms 18.507 ms 18.123 ms 5 207.240.24.178 (207.240.24.178) 20.624 ms 19.139 ms 17.908 ms 6 sanjose1-nbr1.bbnplanet.net (4.0.3.189) 19.179 ms 20.695 ms 20.093 ms 7 su-bfr.bbnplanet.net (4.0.1.1) 20.322 ms 23.264 ms 26.579 ms 8 paloalto-cr8.bbnplanet.net (4.0.2.210) 26.441 ms 41.758 ms 50.069 ms 9 digitisland.bbnplanet.net (131.119.35.82) 84.241 ms 77.681 ms 84.923 ms 10 digitisland.bbnplanet.net (131.119.35.82) 129.785 ms !X * 152.450 ms !X -danny > i.e. > ping -s auth3.ipass.com. > 64 bytes from holmes.ipass.com (204.198.128.102): icmp_seq=0. time=352. ms > > yet traceroute > > ... > 6 s2/2-decker-hi.digisle.net (167.216.156.57) 147 ms 149 ms 148 ms > 7 s2/2-decker-hi.digisle.net (167.216.156.57) 205 ms * 147 ms > 8 * s2/2-decker-hi.digisle.net (167.216.156.57) 152 ms * > 9 s2/2-decker-hi.digisle.net (167.216.156.57) 148 ms * 159 ms > 10 * s2/2-decker-hi.digisle.net (167.216.156.57) 148 ms * > 11 s2/2-decker-hi.digisle.net (167.216.156.57) 177 ms * 161 ms > ... > > thnx in adv.

2 1

Update: Sprint Outage
by Bownes, Robert M. (EXCH) 10 Mar '98

10 Mar '98

Interesting that my last notice took ~2 hours to get to NANOG and back to me... Anyway, an update. Looks like mostly voice traffic involved. Site (for those who didn't recognize it, like myself) of outage is in Florida. Unless I see some data traffic affected or demand for updates, I won't update this incident anymore. Bob >---------- >Sent: Tuesday, March 10, 1998 2:07 PM >Subject: RE: Sprint Outage > >Sprint has started repair and many areas can call out 800 >numbers again. The outage is affecting any 800 numbers >having to route through the Sprint Orlando DMS 250 and the >calling areas between Jacksonville to Atlanta. More to come. > >

1 0

Sprint Outage
by Bownes, Robert M. (EXCH) 10 Mar '98

10 Mar '98

forwarded from our NOC... Sprint has a major fiber outage affecting 89 DS-3s. The outage is between Marrianna and Sneed. Repair personnel are expected to be pon trouble location by 12:15 eastern. No ETR at this time. 800 customers not being able to complete calls that are riding the Sprint network. More to come.

1 0

Re: BGP Confederation config problem...
by Richard Parker 10 Mar '98

10 Mar '98

Andy McConnell wrote: >Avi Freedman suggested using a +1 metric when leaving each member-AS. >But it doesn't seem to help - perhaps I didn't do it right. >In fact, it doesn't look like the metrics are adjusted more than 1. >... >Setting the MED is the crux of the problem - I can set it "statically", >but the idea is to make the whole confederation dynamic. I want the >confederation to work like a true "internet". (Yes, I'm asking for the >world...) Andy, Perhaps a solution, albeit an ugly solution, to the problem is to use a large route-map that matches and increments the MED for the expected range of MED values. Each router would have a route-map to increment the metric on updates that it sends. On the Cisco this configuration might look something like the following: neighbor a.b.c.d route-map INC-MED out ! route-map INC-MED permit 1 match metric 0 set metric 1 route-map INC-MED permit 2 match metric 1 set metric 2 (etc.) If dorian's remark is true... >This won't work because Cisco's bgp implementation skips over MED >from neighboring sub ASes inside a confederation in its decision >process. ...then instead use a route map to increment the local preference for each route received. You would still probably want to set the MED from the local preference before you export routes so that the external neighbors (in other ASes) of the confederation members choose to send data into the confederation at the point closest to the ultimate destination. -Richard

2 1

Re: More smurf fun
by Jeffrey Haas 10 Mar '98

10 Mar '98

Dave Rand was known to have said: > Here's the latest group of smurf-able networks. [...] > Some of the phone calls were interesting. Many of the NOC staff did not > know what a smurf attack was (so I explained it to them). Many did not want > to set no ip directed-broadcast because they didn't know what it would do. [...] > 148.59.0.0 A quick note: If you find yourself having problems from this netblock, please contact either jmh(a)msen.com (me) or noc(a)msen.com. We currently cannot disable directed broadcast due to a NASTY bug in the packet forwarding code of the Proteon routers we use on our backbone. We are working with Proteon to solve this issue. (I'm also trying to figure out if the issue is worth posting to Bugtraq.) In the meanwhile, we have implemented blackhole routing for the subnets that seemed to be at issue. > Dave Rand -- Jeffrey Haas "He that breaks a thing to find out what it is has elezar(a)pfrc.org left the paths of wisdom." (Or works for Fermilab...)

2 1

More smurf fun
by dlr＠bungi.com 10 Mar '98

10 Mar '98

Here's the latest group of smurf-able networks. Note some cool, high-connectivity networks are on here, like Bay Networks. At least one was still active on mae-west, if you can believe it. I've filed trouble tickets with each of the network's NSP's that are on here. Operators, please check to see if a network you advertise is on this list. If it is, please set no ip directed broadcast on the lan interfaces (or get your customers to do it, as appropriate). Some of the phone calls were interesting. Many of the NOC staff did not know what a smurf attack was (so I explained it to them). Many did not want to set no ip directed-broadcast because they didn't know what it would do. Most were helpful, however, called back in a timely fashion, and had their interfaces configured quickly. 128.139.0.0 132.166.0.0 132.169.0.0 132.202.0.0 132.226.0.0 132.229.0.0 132.248.0.0 132.74.0.0 132.77.0.0 132.79.0.0 132.80.0.0 132.87.0.0 134.120.0.0 134.121.0.0 134.177.0.0 134.205.0.0 134.39.0.0 134.57.0.0 134.59.0.0 134.74.0.0 134.82.0.0 134.87.0.0 136.205.0.0 138.210.0.0 138.231.0.0 138.28.0.0 138.73.0.0 139.181.0.0 139.223.0.0 139.51.0.0 139.70.0.0 139.91.0.0 140.185.0.0 148.132.0.0 148.144.0.0 148.59.0.0 149.35.0.0 192.93.124.0 198.32.136.0 198.118.128.0 199.209.166.0 -- Dave Rand dlr(a)bungi.com http://www.bungi.com

2 1

Re: BGP Confederation config problem...
by Andy McConnell 10 Mar '98

10 Mar '98

At each member AS, either IGRP or OSPF. They do not interact. By design... two of the AS's are across the Pacific from the other two :-) -Andy -- Andy McConnell アンディマッコネル Network Architect, NTT Multimedia Communications Laboratories Slogan of 105.9, the classic rock radio station in Chicago: "Of all the radio stations in Chicago ... we're one of them." On Fri, 6 Mar 1998, Blake Willis wrote: blakew> Andy, blakew> blakew> What IGP are you running on this network? blakew> blakew> -Blake Willis blakew> CAIS Engineering blakew> blakew> --------------------------------------------------------------------------- blakew> Blake Willis 703-448-4470x483 blakew> Network Engineer, New Customers blakew(a)cais.net blakew> CAIS Internet, a CGX Communications Company blakew> --------------------------------------------------------------------------- blakew> blakew> On Fri, 6 Mar 1998, Andy McConnell wrote: blakew> blakew> > I have a problem I cannot solve. If anyone is running confederations like blakew> > mine, I would appreciate some help :-) blakew> > blakew> > We have a 4-memberAS confederation, each with two IBGP peers. The blakew> > arrangement looks sort of like an octagon. the "r" is AS3 is an internal blakew> > hop, not a BGP peer. blakew> > blakew> > _______ ______ blakew> > | R-----R---r | blakew> > | / | | | | blakew> > |AS2 R | |AS3 R | blakew> > ----|-- ----|- blakew> > | | blakew> > ____|__ ____|_ blakew> > |AS4 R | |AS1 R | blakew> > | \ | | / | blakew> > | R-------R | blakew> > ------- ------ blakew> > blakew> > The problem is this: How do you get BGP to choose the shortest "AS PATH", blakew> > since internal AS paths are ignored in selecting BGP routes? Right now, blakew> > to top router in AS4 will always choose a route through (2 3 1) instead of blakew> > (1), because it prefers "external" routes (even external confederation blakew> > routes) over internal routes. blakew> > blakew> > So, when given a choice, the router on the distant side of the AS will blakew> > ALWAYS prefer the three-AS-hop path, because it is external. Is there a blakew> > way around this?!? blakew> > blakew> > Avi Freedman suggested using a +1 metric when leaving each member-AS. blakew> > (Thanks Avi!) But it doesn't seem to help - perhaps I didn't do it right. blakew> > In fact, it doesn't look like the metrics are adjusted more than 1. for blakew> > example, from AS2, the lower router sees every route outside of AS2 as blakew> > having only metric 1! The top router in AS3 does not add 1 to the metrics blakew> > it readvertises to AS2. blakew> > blakew> > I've been stewing over this problem for some time... I believe there is blakew> > some clue that I've missed. Any help would be greatly appreciated! blakew> > blakew> > -Andy blakew> > blakew> > blakew> > -- blakew> > Andy McConnell アンディマッコネル blakew> > Network Architect, NTT Multimedia Communications Laboratories blakew> > blakew> > Writing about music is like dancing about architecture. blakew> > blakew>

5 6

simple question as always..
by mhirse＠UU.NET 09 Mar '98

09 Mar '98

They seem to using access-list on their gateway routers. That would explain your problem. Mohamed Dear nanog, could someone explain to me why I can ping but traceroute seems to have problem reaching it. i.e. ping -s auth3.ipass.com. 64 bytes from holmes.ipass.com (204.198.128.102): icmp_seq=0. time=352. ms yet traceroute ... 6 s2/2-decker-hi.digisle.net (167.216.156.57) 147 ms 149 ms 148 ms 7 s2/2-decker-hi.digisle.net (167.216.156.57) 205 ms * 147 ms 8 * s2/2-decker-hi.digisle.net (167.216.156.57) 152 ms * 9 s2/2-decker-hi.digisle.net (167.216.156.57) 148 ms * 159 ms 10 * s2/2-decker-hi.digisle.net (167.216.156.57) 148 ms * 11 s2/2-decker-hi.digisle.net (167.216.156.57) 177 ms * 161 ms ... thnx in adv. tatsuya = = = = = = BEEOCB 03-3239-0607 fax 03-3239-2609 business network telecom http://www.giganet.net

2 1