- Test - lists.nanog.org

Re: Plethora of UUnet outages and instabilities
by Alex Rubenstein 02 Jul '97

02 Jul '97

All I know is they suck moose nipples, and I disconnected from them last week. They have been having problems in/around cr1.nyc1 for MONTHS now, and they are doing nothing (that I can see) to fix it. Thier first-level support is horrendous. One guy once said to me, "Routers have processors?!". Another hung up when I told him to put "CUSTOMER IS IRATE" on the trouble ticket. There are some GOOD guys over there, but them seem to be disappearing? (are you there, Mr. Hannan?) HELLO, GENUITY (anyone who is looking for a new provider, Genuity is DAMN good. and a DAMN good NOC. Sales guys are a little whacky, tho). At 02:12 PM 6/30/97 -0700, Josh Beck wrote: >Has everyone else been seeing the almost daily UUnet outages recently, >does anyone know what the true causes of these have been? --- "Don't go with a spineless ISP; we have more backbone." Alex Rubenstein -- alex(a)nac.net -- KC2BUO -- www.nac.net net @ccess corporation, 201-983-0725 -- 201-983-0725

15 21

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 01 Jul '97

01 Jul '97

This assumes that you consider web server location and web server performance to NOT be a part of overall network performance. Our view steps back a bit from that. The majority of traffic would appear to be webcentric. From an end users perspective, what does a web site on a specific network look like and how does that compare to a web site on another network? There are ENDLESS variables contributing to that including intercity links, hub architecture, host hardware, host software, peering, connectivity points with other networks, transit agreements, type of routers, ATM switching (or not). All contribute. We think most people notice Internet performance (or lack thereof) while viewing world wide web pages. If we measure such page transits, the results are indicative of the accumulation of ALL of those factors. The web sites chosen were on the network under study, operated by the network, and under their control. They have total control over the hardware and software used, how it is connected to their network, just as they have control over all other aspects of their networks. Does web server performance affect the results? I would hope so. Can we break it down into what is purely web server hardware performance, what is web server software performance, what is NIC card on the web server, what is the impact of the first router the web server is connected to, what is the impact of hub design and the interface between IP routing and ATM switching, what part is the impact of interconnections with other networks, what part is peering, what part is just goofy router games? Uh,,, NO we can't. I would posit that it is only the network engineers at the heart of this that would or should care. I don't know at this point what portion of the equation can be levied on web servers and I don't think anyone else can either. I have held for several years that the performance breakdown is in the "last inch" of the Internet between the drive controller and the disk surface. But in working with Keynote, I generated the broad theory that if that view held true, then by massively averaging measurements across time and geography, we should flatline the Internet. In other words, all results should factor to zero relatively. They didn't. They didn't to a shocking degree. And at this point I am under the broad assumption that server performance doesn't account for all of it, perhaps little of it. But I could be widely wrong on the entire initial assumption. In any event, the networks have total control and responsibility for their own web servers, much as they do for their own network if you define that as something separate from their networks. We measured web page downloads from an end user perspective, and those are the results in aggregate. If it leads to a flurry of web server upgrades, and that moves the numbers, we'll know more than we did. If it leads to a flurry of web server upgrades, and it FAILS to move the numbers, that will tell us something as well. Our broad theory is that nothing is going to improve as long as anything you do doesn't count and is not detectable by anyone anywhere. If a particular network can move their results in any fashion, that is an improvement in the end user experience, however achieved. Warmest Regards; Jack Rickard =================================================================== Jack Rickard Boardwatch Magazine Editor/Publisher 8500 West Bowles Ave., Ste. 210 jack.rickard(a)boardwatch.com Littleton, CO 80123 www.boardwatch.com Voice: (303)973-6038 =================================================================== ---------- > From: Craig A. Huegen <c-huegen(a)quadrunner.com> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: Peter Cole <Peter.Cole(a)telescan.com>; nanog(a)merit.edu; marketing(a)keynote.com > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 2:08 PM > > On Fri, 27 Jun 1997, Jack Rickard wrote: > > ==>Not an entirely whacky concept actually. The way hot potato routing works, > ==>this could actually be a "purer" test I suspect of the network internally > ==>and a purer test of connectivity of any network to all others cum Internet. > > The article says that you're measuring backbone provider performance, yet > you're including: > > * carrier's web server location > * carrier's web server performance > > ==>Keynote does a "Top 40" study of 40 popular web sites and I believe they > ==>make the results available on their web site. It is interesting to observe > ==>performance variations of the network as a whole over time. Other than > ==>that, we don't have much interest in it. It is indicative of no specific > ==>network, but of the Internet in general. > > /cah

7 6

Another UUNET Explanation
by Chad Skidmore 01 Jul '97

01 Jul '97

Here is a message that we received from UUNET today. It seems that they are full of various explanations. July 01, 1997 Dear UUNET Customer, Please accept our apologies if you have experienced any service interruptions over the past 24 hours. The cause of these problems has been traced to a software defect within our Cascade Frame Relay network. UUNET is working closely with Cascade to fully understand the cause and effects of these problems and is in the process of identifying a permanent fix. This may include rapidly deploying new software into the network. We will provide periodic updates throughout the day as further information becomes available. Routing instabilities may continue at a diminished capacity over the next 24-48 hours as switches receive updates. Thank you for your patience during this period. Again, we apologize for any inconvenience which these outages may have caused you or your customers. Sincerely, UUNET Technologies, Inc. ------------------------------------------ Chad Skidmore VP Operations & Engineering Data Source, L.L.C. http://www.dsource.com cskidmore(a)dsource.com

1 0

A good one from the UUnet NOC
by Josh Beck 01 Jul '97

01 Jul '97

Here is UUnet's latest response: apparently they've suffered massive congestion and an impacted network in about 10 minutes! That's a good sales team! Apparently they got their OC-12 network installed ahead of schedule as well, as we are finally back to normal: Type escape sequence to abort. Tracing the route to drawbridge.ascend.com (198.4.92.1) 1 520.Hssi8-0.GW1.SDG1.ALTER.NET (137.39.132.5) 4 msec 575.Hssi3-0.GW1.SDG1.ALTER.NET (157.130.224.17) 20 msec 520.Hssi8-0.GW1.SDG1.ALTER.NET (137.39.132.5) 8 msec 2 114.Hssi4-0.CR1.LAX1.Alter.Net (137.39.31.41) 16 msec 8 msec 8 msec 3 111.Hssi8-0.CR1.SCL1.Alter.Net (137.39.30.213) 28 msec 104 msec 20 msec 4 311.atm1-0.gw1.scl1.alter.net (137.39.13.149) 80 msec 60 msec 24 msec 5 external-gw.ascend.com (198.4.92.2) [AS 701] 48 msec 28 msec 40 msec 6 drawbridge.ascend.com (198.4.92.1) [AS 701] 36 msec 52 msec 32 msec Forwarded message follows: ---------- Forwarded message ---------- Date: Tue, 1 Jul 1997 13:47:05 -0400 (EDT) From: UUNET Customer Liaison <help(a)UU.NET> To: XXXXXXXXXXXXXXX Subject: Re: Customer #UXXXXX (UUXXXXXX) Please do not take the (UUxxxxxx) information out of the subject header when replying to this problem. We use this number to track specific customer issues. Thank you. The latency you are experiencing in the Bay area is being addressed in the next few months with an OC12 network. This will provide increased bandwidth and alleviate some congestion. Thanks, Steve Williams UUNET Network Support > > > We are seeing high latency on the west coast. Your NOC page states the > problems are only on the east coast : > > Type escape sequence to abort. > Tracing the route to drawbridge.ascend.com (198.4.92.1) > > 1 520.Hssi8-0.GW1.SDG1.ALTER.NET (137.39.132.5) 4 msec > 575.Hssi3-0.GW1.SDG1.ALTER.NET (157.130.224.17) 4 msec > 520.Hssi8-0.GW1.SDG1.ALTER.NET (137.39.132.5) 4 msec > 2 114.Hssi3-0.CR2.PHX1.Alter.Net (137.39.69.57) 28 msec 44 msec 32 msec > 3 122.Hssi8-0.CR2.SCL1.Alter.Net (137.39.68.130) 256 msec 272 msec 256 > msec > 4 412.atm11-0.gw1.scl1.alter.net (137.39.13.165) 272 msec 276 msec 288 > msec > 5 external-gw.ascend.com (198.4.92.2) [AS 701] 280 msec 296 msec 284 > msec > 6 drawbridge.ascend.com (198.4.92.1) [AS 701] 256 msec 248 msec 268 msec > Josh Beck jbeck(a)connectnet.com ------------------------------------------------------------------------ CONNECTNet INS, Inc. Phone: (619)450-0254 Fax: (619)450-3216 6370 Lusk Blvd., Suite F-208 San Diego, CA 92121 ------------------------------------------------------------------------

3 2

Re: Legal Question
by Carl Oppedahl 01 Jul '97

01 Jul '97

At 08:33 AM 07/01/97 -0700, Scott Reubelt wrote: >May not be for this alias but I need to ask. > >I have an ISP that may be guilty of receiving copies of everyone's >emails associated with the ISP. What can be done legally? What can be done is consulting competent counsel who are knowledgeable about Internet issues, in particular to study the ECPA (Electronic Communications Privacy Act) as it may relate to your situation. --- Carl Oppedahl, Oppedahl & Larson, patent law firm http://www.patents.com/ has hundreds of pages of answers to frequently asked questions on patent, copyright, and trademark law

3 2

RE: Plethora of UUnet outages and instabilities
by Peter Cole 01 Jul '97

01 Jul '97

I am just a multihomed customer with Two T1s running BGP4. I had no trouble with my Ts from Sprint and MCI until about two months ago. I am running a 7206 with 64meg of RAM on 11.1(12). Our customer support started getting complaints about customers not being able to reach us(AS7201) from MSN dialup in Houston. The network 153.36.0.0/14 was in my BGP tables but not my IP route table. I called Cisco and after a little work they concluded that the reason the routes were being removed was more specific routes 153.36.50.0 & 153.36.51.0 inside the CIDR were removing the 153.36.0.0/14. I didn't understand the logic but a simple distribute list ignoring announcements for the two class Cs seemed to solve the problem. Later I sent a note to Sprint, MCI, MSN, & UUNET. Spint and MCI responded quickly stating that Cisco was mistaken. Cisco wants to look into it further. I have herd nothing from MSN or UUNET. I have to update the software on the router for other reasons. I thought I would wait to see if the problem is still present after my upgrade. I have no route Damping setup and can think of no other explanation for this bizarre behavior. My point is that maybe there are other reasons that so many ASs are having problems with UUNET routes. Peter Cole of Telescan, Inc. (281)588-9155 Better computing through lack of sleep. > ---------- > From: Josh Beck[SMTP:jbeck@connectnet.com] > Sent: Monday, June 30, 1997 4:12 PM > To: nanog(a)merit.edu > Subject: Plethora of UUnet outages and instabilities > > Has everyone else been seeing the almost daily UUnet outages recently, > does anyone know what the true causes of these have been? > > Josh Beck > jbeck(a)connectnet.com > ---------------------------------------------------------------------- > -- > CONNECTNet INS, Inc. Phone: (619)450-0254 Fax: > (619)450-3216 > 6370 Lusk Blvd., Suite F-208 San Diego, CA > 92121 > ---------------------------------------------------------------------- > -- >

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by smd＠clock.org 01 Jul '97

01 Jul '97

randy(a)psg.com (Randy Bush) writes: > > Just out of curiosity, why do many (not all) of the large backbone providers > > establish their face to the web (their corporate webserver) on slow, > > badly positioned machines? > > Because it is done by marketing, not engineering? In my opinion, this alone should reduce the Value Number of any given provider. Personally, flaws or not, I welcome Boardwatch's attempts to come up with a widely-published metric for the Internet. This likely will lead other publications into similar investigations, some of which may well bring the writers and editors of various periodicals into contact with the folks at CAIDA. Also, flaws or not, Boardwatch did do something fantastically clever, and that's examining things on an end-to-end basis, rather than obsessing about details of what's going on between the endpoints. People concerned about the abysmal end-to-end throughput of even modern TCP across much of the present Internet should be rejoicing and helping other journalists develop better and more scientific approaches to categorizing expected versus observed end-to-end performance. The combination of work aimed at measuring the internals of one's network and work that measures the "(un)happiness" of certain classes of applications should be of enormous value to engineers willing to admit that they don't know all the things that affect network performance observed by end-users. The reality, however, is that most American ISPs seem to engage in knee-jerk denial or aggressive posturing whenever there is a suggestion that their network is anything but perfect. I certainly have been in the middle of that kind of thing, so I can hardly claim innocence. Such reactions are pure marketing: we can't admit that maybe there is some way we could improve things or some set of things our network doesn't do well because that would hurt our product image. People who react with marketing-think and who build with marketing-think in the first place deserve to be torn apart by analyses based in marketing-think. While the article in question isn't in my hands yet, based on Mr Rickard's and others' comments on the NANOG list, I think it's safe to guess that this is precisely what the Boardwatch study does. Sean.

3 2

Re: Plethora of UUnet outages and instabilities
by Hank Nussbacher 01 Jul '97

01 Jul '97

At 02:12 PM 6/30/97 -0700, Josh Beck wrote: >Has everyone else been seeing the almost daily UUnet outages recently, >does anyone know what the true causes of these have been? Perhaps this has something to do with it: Glitch slows UUNET to crawl June 23, 1997 Network World If the Internet felt slower than usual last Monday, you were probably a victim of the brownout on UUNET Technologies' network that slowed services in the Northeast. No one is disputing that there was a brownout. But UUNET and Cisco Systems, Inc. - which sold the Internet service provider its routers - are at odds over how it happened. The problem, both sides agree, was a router memory failure. After that, the stories diverge. According to UUNET, the ISP's hubs in Washington, D.C., and Newark, N.J., dropped packets because of a software glitch on its Cisco routers. The hubs are like points-of-presence where UUNET users' traffic is routed to and from the Internet. The routers suffered what the ISP is calling a ``memory leakage.'' Typically, after a router uses a block of memory, it puts the block back into a pool of usable memory. But that is not what happened on UUNET's routers last week. Instead of the memory being usable, it was fragmented, said Alan Taffel, vice president of marketing at UUNET. Large clusters of data that came through the routers could not be handled by the fragmented memory blocks. Once the routers ran out of memory, they automatically reset. UUNET notified Cisco, which sent over a software patch. The software Band-Aid freed up more memory in the routers cache so it would not overload, said Bob Michelet, director of corporate relations at Cisco. The patch is a temporary measure meant to be used until UUNET ups the memory on its routers, he said. UUNET started upgrading the memory on the routers, which Cisco contends was the real culprit. Cisco recommends that its ISP customers use 128M bytes of total memory on their route switch processor boards, Michelet said. UUNET uses 64M-byte memory boards on most, if not all, of its 7,500 routers. UUNET said it never received a recommendation from Cisco concerning memory for its 7,500 routers. ``But clearly, once this event occurred, we discussed the memory issue with Cisco, and we agreed the right course of action would be to upgrade the routing memory to 128M bytesÙ,'' said Jim McManus, vice president of systems engineering at UUNET. Almost every large to midsize ISP uses some Cisco equipment, so should users be concerned? That depends on who answers the question. Cisco claimed UUNET was the only ISP using 64M-byte route switch processor boards. One ISP agrees. Genuity, Inc., Bechtel Enterprises' ISP subsidiary, said it would not run a router with less than 128M bytes of memory. Genuity uses Cisco 7513 routers with fully loaded memory because the routers have to store the routing table for the entire Internet, said Rodney Joffe, chief technology officer at Genuity. Copyright 1997, Network WorldÙ > >Josh Beck jbeck(a)connectnet.com >------------------------------------------------------------------------ >CONNECTNet INS, Inc. Phone: (619)450-0254 Fax: (619)450-3216 >6370 Lusk Blvd., Suite F-208 San Diego, CA 92121 >------------------------------------------------------------------------ > >

3 2

[Fwd: Re: Legal Question]
by Scott Reubelt 01 Jul '97

01 Jul '97

-- Scott F. Reubelt sreubelt(a)whistle.com (http://www.whistle.com)

1 0

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by smd＠clock.org 01 Jul '97

01 Jul '97

lhoward(a)UU.NET writes: > 1. Although a variety of backbones is used, the study does not say > which ones. Also, even though the study does point out the assymetrical > routing of a web transavtion (hot-potato), it doesn't point out that the > traffic being measured is a brief web request (which is dumped to the > web server's backbone ASAP) answered by a long response (10KB in this > case, dumped to the querier's backbone). Ironically, the only person I can remember actively (and publically) questioning the wisdom of hot potato routing for several applications is Curtis Villamizar. While Tony Bates and I had what seemed to be good reasons for doing closest-exit routing (among other routing policies) at the time, and there are strong economic incentives for maintaining closest-exit routing among peers, there are certainly drawbacks. Considering that the Keynote/Boardwatch study purports to help content-providers decide where to locate their content, poor performance attributable to closest-exit routing should have engineers considering the wisdom of that particular routing policy with respect to that particular market segment, depending on the focus of their businesses. The nice thing about the Keynote test is that it might get some of the cleverer people around here thinking of the Internet in a more abstract way than they might be used to. When I think in a more end-to-end fashion about comment #1 above, it seems that a provider who is rated poorly deserves the rating, even though the performance problem may be attributable to a peer's network's congested lines, broken equipment, or whatever. In effect, by taking an end-to-end approach, this test also rates networks' routing policies. If the routing policy drives traffic through a competitor's suboptimal path, and this degrades end-to-end performance, then the provider with that routing policy will receive a poorer rating than otherwise. One of the things that should be thought about is that there may be ways to improve end-to-end goodput by changing one's own network based on what one can infer from tests similar to the Keynote ones (or indeed similar to Vern Paxson's tests), as much as what one can infer from what one's own equipment is doing or seeing. > All of the previous notwithstanding, I would be interested in a better > version of this study. Even more interesting would be to track how > providers do over the course of several studies--who responds well to > backbone congestion? Personally, I would take this comment as a sign that the study and its publication is astoundingly successful. Finally, I would rather see people respond to congestion in a cleverer fashion, i.e. by deploying or pressing for the deployment of better WWW transport technology. There are some people with very clever ideas about caching and lazy proxies, and some clever ideas about changing HTTP in dramatic ways, that deserve some attention, as these ideas are more likely to help people and information meet more quickly in the long run than any amount of tweaking the underlying transport fabric ever will. Sean.

1 0