- NANOG - lists.nanog.org

Re: Policy Based Routing advice
by Andrey Khomyakov 12 Aug '10

12 Aug '10

I bit more explanation: 172.25/16 is a hop away and the packets with that source IP will enter on Gi2/6 and need to exit Gi2/14. So it goes like that: 172.25/16 is vlan25 on the student router Gi0/1 has ip address 192.168.250.2 on the student router default route is towards 192.168.250.1 on the student router > On Thu, Aug 12, 2010 at 11:54 AM, Andrey Khomyakov < > khomyakov.andrey(a)gmail.com> wrote: > >> Hey all. I'm trying to setup a routing policy on a cat4503-E with Sup6-E >> and >> for some reason I can't see it taking effect. I'm definitely sourcing >> packets from 172.25.0.0/16 (the test machine had 172.25.24.25 address). >> For >> some reason the packets still go out towards the default gateway instead >> of >> what's specified in the route-map. The switch is running >> cat4500e-ENTSERVICESK9-M), Version 12.2(52)SG, RELEASE SOFTWARE (fc1) >> According to stats on the ACL and the route-map it's just not being hit >> for >> some reason. Applying the ACL directly to the interface (as an >> access-group) >> shows that the ACL is correct and I see hits, however, via the route map >> it's not being hit. I don't know what those "2 matches" are, but there >> definitely should be a lot more than 2. And in addition, I see the packets >> arriving on the firewall that is the "default gateway". >> >> Does anyone have any tips on why this might now work? >> >> >> ip access-list standard acl_Students >> permit 172.25.0.0 0.0.255.255 >> >> route-map Students-Route-Map permit 10 >> match ip address acl_Students >> set ip next-hop 192.168.168.22 >> >> interface GigabitEthernet2/6 >> no switchport >> ip address 192.168.250.1 255.255.255.252 >> ip pim dense-mode >> ip policy route-map Students-Route-Map >> >> interface GigabitEthernet2/14 >> no switchport >> ip address 192.168.168.21 255.255.255.252 >> no ip redirects >> no ip mroute-cache >> flowcontrol send desired >> >> cat4503#sh access-lists acl_Students >> Standard IP access list acl_Students >> 10 permit 172.25.0.0, wildcard bits 0.0.255.255 (2 matches) >> >> >> cat4503#sh route-map >> route-map Students-Route-Map, permit, sequence 10 >> Match clauses: >> ip address (access-lists): acl_Students >> Set clauses: >> ip next-hop 192.168.168.22 >> Policy routing matches: 2 packets, 180 bytes >> >> cat4503#sh ip route 0.0.0.0 >> Routing entry for 0.0.0.0/0, supernet >> Known via "static", distance 1, metric 0, candidate default path >> Redistributing via eigrp 179 >> Advertised by eigrp 179 >> Routing Descriptor Blocks: >> * 192.168.168.10 >> Route metric is 0, traffic share count is 1 >> >> -- >> Andrey Khomyakov >> [khomyakov.andrey(a)gmail.com] >> > > -- Andrey Khomyakov [khomyakov.andrey(a)gmail.com]

4 6

Springnet Underground
by Justin Shore 12 Aug '10

12 Aug '10

Does anyone have any experience with the Springnet Underground in Springfield, MO? In case people don't know it's a working limestone mine. In the areas that have already been mined close to the entrance, they've sold or rented out space between the rock pillars that hold up the mine roof. The city of Springfield put in a data center and started selling services out of it that complimented their city-wide fiber plant. They've been suggested as a site for hosting a cabinet full of gear. I'm wondering what the connectivity options are for that facility and so far haven't been able to get a straight answer from anyone. For the most part it looks like the SprintNet folks want to sell you their own upstream which won't cut it for our needs. AT&T serves the area of course but I will not have them as my last mile (or any mile for that matter). Does L3 or any other large carrier have facilities there? Does anyone have any experience with the facility in general? Thanks Justin

1 0

Chunghwa Telecom Tech Support Reg.
by Natarajan Balasubramanian 12 Aug '10

12 Aug '10

Hi All, Does anyone have the contact number for "Chunghwa Telecom's" Business Users - Technical Support number ? -Nat

4 5

Re: IPv6 Server Load Balancing - DSR
by Leland Vandervort 12 Aug '10

12 Aug '10

Well, Frankly our "culture" is very much open source, so if we can find something along those lines, then it would be preferred. (Hence looking at OpenSolaris and ILB). -- having said that, we do have both F5 and Foundry kit here, but it's all pre-IPv6 so doesn't have the support built in. Not really looking to replace what is in existence already for IPv4 with something new to do both, so really that reinforces the open-source avenue really. I think the biggest problem is really the DSR aspect for IPv6, since the OS/ILB solution works perfectly in NAT mode, and DSR works perfectly with IPv4 on this solution. So either I'm missing something critical on the "real" server configuration, or ILB's implementation of DSR for IPv6 doesn't really work. The "virtual" IP is bound to loopback on the real servers, exactly the same was as for IPv4. So other than something quirky going on with ND, or simply ILB not correctly rewriting the L2 frame, or there's something else more sinister afoot that I'm unable to put my finger on. Back to the drawing board... :) Thanks, Leland On 12 Aug 2010, at 19:23, William Cooper wrote: > I know there have been quite a few responses for both h/w and s/w > solutions, it's not clear > which your preference is of the two. I know there are various h/w > vendors that offer a s/w > solution (mostly in conjunction with some form of virtualization > environment), such as A10. > > I've been testing A10 for a while now, and they seem very keen on > developing parity between > v4 and v6 feature sets / performance. > > DSR is more or less a L2 trick that plays on some inherent weaknesses > and constraints > that are present with v4 local address resolution (don't mean to > preach to the chior); I think > most responses here have touched on the primary challenges of DSR with > v6. I'll be exploring > DSR with dual stack v4/6 in the near future, I'll let you know how > that turns out. > > Hmm... not sure how this helped. > > Regards, > > -Tony > > On Thu, Aug 12, 2010 at 12:40 PM, Leland Vandervort > <leland(a)taranta.discpro.org> wrote: >> Hi Owen, >> >> The DSR address is indeed on a loopback in our case. >> >> lo Link encap:Local Loopback >> inet6 addr: ::1/128 Scope:Host >> inet6 addr: xxxx:xxxx:x:xxxx::xx/128 Scope:Global >> >> >> >> The mystery continues... >> >> >> Leland >> >> >> On 12 Aug 2010, at 18:28, Owen DeLong wrote: >> >>> >>> On Aug 12, 2010, at 6:19 AM, Xavier Beaudouin wrote: >>> >>>> Hi Leland, >>>> >>>> Le 12 août 2010 à 15:11, Leland Vandervort a écrit : >>>> >>>>> OpenSolaris ILB is open solution ;) >>>>> >>>>> but yea, that's what we've started looking at -- hence LVM / HAProxy as well.. (though LVM is IPv4 only, and HAProxy is NAT only for IPv6) >>>>> >>>>> does relayd support UDP as well as TCP or is it layer7 only like HAProxy ? >>>> >>>> It does everything... :) L2 -> L7... >>>> >>>>> In the case of ILB, I'm not convinced that it's a problem with the LB itself, but rather the idiosyncrasies of ND in IPv6 that is causing the problem.. but I may be wrong... at any rate, something's amiss ... >>>> >>>> Maybe on some setup you should desactivate ND... >>>> >>>> Xavier >>> >>> If you're putting the DSR address on an interface other than loopback, you probably need to turn of DAD on the interface with the DSR address otherwise DAD >>> will shut down that address on the interface when it sees other servers with the same address. Sometimes it will shut down all but one, sometimes it will >>> shut down all. >>> >>> >>> Owen >> >> >>

1 0

IPv6 Server Load Balancing - DSR
by Leland Vandervort 12 Aug '10

12 Aug '10

Dear Colleagues, I've been scratching my head over this for the past couple of months and have come up with blanks, and several weeks of scouring various resources on the net have not yielded anything more fruitful. I'm looking at server load balancing for IPv6 and specifically need DSR (direct server return). Additionally, I need to support both TCP and UDP. I have evaluated a number of different load balancing solutions purporting to support IPv6 with varying results (and costs)... a few examples: F5 : according to marketing blurb supposedly supports IPv6 in NAT and DSR mode, both UDP and TCP. Their documentation, however, has no mention of IPv6 capability. Other disadvantage = cost... Brocade/Foundry: Similar situation to F5 Zeus: IPv6 in NAT only, and even more expensive than F5. Exceliance Aloha: IPv6 in NAT only, and ONLY in TCP (no UDP) A few others also tested... including LVM/HAProxy (same situation as Exceliance Aloha), and others... Finally in the end, only OpenSolaris ILB seems to put all the checks in the right boxes for my requirements. But there is still a problem. 1. IPv4 TCP and UDP work fine in NAT, Half-NAT, and DSR 2. IPv6 I've managed to get working, complete with healthchecks, in TCP and UDP in NAT only although the documentation stipulates that DSR is also possible (but not HalfNAT for the moment). The problem with #2: Using the same server farm behind, but in dual-stack, and configuring ILB for TCP and UDP services using NAT, everything is fine. If I configure it for DSR, immediately it fails (both with and without healthchecks). Although from the ILB host itself, I can certainly do a manual heathcheck.. (e.g. telnet <server_real_ipv6_addr> 80 and do GET / or HEAD / with no problems. Using ARP poisoning from the shell I can also perform the healthcheck on the real server via telnet using the virtual ip. The servers are configured normally for DSR.. with the virtual IP attached to a local dummy or loopback interface, and with IPv4 DSR works fine. Nevertheless, I've been unable to get DSR working with ILB -- and have found absolutely nothing around the net with working examples of IPv6 SLB with DSR. NAT mode works fine, but the real server loses visibility of the end user's IP as the requests come from the internal IP of the ILB host, and with a system that uses client IP address as part of the various criteria for session tracking, it creates a few problems... I am suspecting that the issue may be related to ND, as the behaviour is similar to the old story with doing DSR on real-servers using older linux distributions that do not by default disable proxy-ARP replies by the server for IP addresses on dummy or loopback interfaces, and of course the proxy ARP causes confusion to the load balancer and breaks the whole thing. But the real servers are recent Debian distributions, and both ipv4 ARP and ipv6 ND is disabled on the dummy interfaces, as is proxy ARP. Would anyone happen to have any useful pointers, tips, or other on how to resolve the issue? Many thanks in advance. Leland

7 10

Policy Based Routing advice
by Andrey Khomyakov 12 Aug '10

12 Aug '10

Hey all. I'm trying to setup a routing policy on a cat4503-E with Sup6-E and for some reason I can't see it taking effect. I'm definitely sourcing packets from 172.25.0.0/16 (the test machine had 172.25.24.25 address). For some reason the packets still go out towards the default gateway instead of what's specified in the route-map. The switch is running cat4500e-ENTSERVICESK9-M), Version 12.2(52)SG, RELEASE SOFTWARE (fc1) According to stats on the ACL and the route-map it's just not being hit for some reason. Applying the ACL directly to the interface (as an access-group) shows that the ACL is correct and I see hits, however, via the route map it's not being hit. I don't know what those "2 matches" are, but there definitely should be a lot more than 2. And in addition, I see the packets arriving on the firewall that is the "default gateway". Does anyone have any tips on why this might now work? ip access-list standard acl_Students permit 172.25.0.0 0.0.255.255 route-map Students-Route-Map permit 10 match ip address acl_Students set ip next-hop 192.168.168.22 interface GigabitEthernet2/6 no switchport ip address 192.168.250.1 255.255.255.252 ip pim dense-mode ip policy route-map Students-Route-Map interface GigabitEthernet2/14 no switchport ip address 192.168.168.21 255.255.255.252 no ip redirects no ip mroute-cache flowcontrol send desired cat4503#sh access-lists acl_Students Standard IP access list acl_Students 10 permit 172.25.0.0, wildcard bits 0.0.255.255 (2 matches) cat4503#sh route-map route-map Students-Route-Map, permit, sequence 10 Match clauses: ip address (access-lists): acl_Students Set clauses: ip next-hop 192.168.168.22 Policy routing matches: 2 packets, 180 bytes cat4503#sh ip route 0.0.0.0 Routing entry for 0.0.0.0/0, supernet Known via "static", distance 1, metric 0, candidate default path Redistributing via eigrp 179 Advertised by eigrp 179 Routing Descriptor Blocks: * 192.168.168.10 Route metric is 0, traffic share count is 1 -- Andrey Khomyakov [khomyakov.andrey(a)gmail.com]

1 0

net-neutrality
by Sven Olaf Kamphuis 12 Aug '10

12 Aug '10

Hi, considering the fact that several organisations have been severely undermining net-neutrality over the past few months, which they seem to see as less important than their copyright bullshit, we have decided to set an example: Should the following networks, to which list more will be added over the coming month, desire to exchange traffic with AS34109, they can obtain a traffic relay contract at sales(a)cb3rob.net, the costs of which amount to 10000 euros per month, excl. 19% VAT, if not, well, then it's simply no more internets for them... sorry peeps. 193.108.8.0/21#GEMA-NET 195.109.249.64/29#SONYMUSIC 195.143.92.160/27#SBMG1-NETS 212.123.224.240/29#Net-WEGENER-MEDIA-BV 212.123.227.64/29#BumaStemra2 212.136.193.216/29#BUMA 212.78.179.240/28#BUMA-STEMRA 213.208.242.160/29#NL-COLT-BUMA-STEMRA 217.148.80.112/28#NL-NXS-CUST-1004613 85.236.46.0/24#IX-UNIVERSAL-NET -- Greetings, Sven Olaf Kamphuis, CB3ROB Ltd. & Co. KG ========================================================================= Address: Koloniestrasse 34 VAT Tax ID: DE267268209 D-13359 Registration: HRA 42834 B BERLIN Phone: +31/(0)87-8747479 Germany GSM: +49/(0)152-26410799 RIPE: CBSK1-RIPE e-Mail: sven(a)cb3rob.net ========================================================================= <penpen> C3P0, der elektrische Westerwelle ========================================================================= Confidential: Please be advised that the information contained in this email message, including all attached documents or files, is privileged and confidential and is intended only for the use of the individual or individuals addressed. Any other use, dissemination, distribution or copying of this communication is strictly prohibited.

10 17

RE: off-topic: summary on Internet traffic growth History
by John Lee 12 Aug '10

12 Aug '10

Andrew, Earlier this week I had a meeting with the ex-Director of the Network Operations Center for MFS-Datanet/MCI whose tenure was through 1999. From 1994 to 1998 they were re-architeching the Frame Relay and ATM networks to handle the growth in traffic including these new facilities called peering points of MAE-East and MAE-West. From roughly 1990 to then end of 1996 they saw traffic on their switches grow at 50-70% growth every 6 months. By the last half of 1996 there was a head of line blocking problem on the DEC FDDI switches that was "regularly" bringing down the Internet. The architecture had lower traffic circuits were going through concentrators while higher traffic circuits were directly attached to ports on the switchs. MFS-Datanet was not going to take the hit for the interruptions to the Internet and was going to inform the trade press there was a problem with DEC FDDI switches so Digital "gave" six switches for the re-architecture of the MAEs to solve the problem. Once this problem was solved the first quarter of 1997 saw a 70% jump in traffic that quarter alone. This "historical event" would in my memory be the genesis of the 100% traffic growth in 100 days legend. (So it was only 70% in 90 days which for the marketing folks does not cut it so 100% in 100 days sounds much better?? :) ) MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network. Personal Note: from 1983 to 90 I worked for Hayes the modem folks and became the Godfather to Ascend communications with Jeanette, Rob, Jay and Steve whose team produced the TNT line of modem/ISDN to Ethernet central site concentrators (in the early ninties) that drove a large portion of the user traffic to the Internet at the time, generating the "bubble". John (ISDN) Lee ________________________________________ From: Andrew Odlyzko [odlyzko(a)umn.edu] Sent: Wednesday, August 11, 2010 12:55 PM To: nanog(a)nanog.org Subject: off-topic: summary on Internet traffic growth myths Since several members of this list requested it, here is a summary of the responses to my request for information about Internet growth during the telecom bubble, in particular the perceptions of the O'Dell/Sidgmore/WorldCom/UUNet "Internet doubling every 100 days" myth. First of all, many thanks to all those who responded, on and off-list. This involved extensive correspondence and some long phone conversations, and helped fill out the picture of those very confusing times (and also made it even clearer than before that there were many different perspectives on what was happening). The entire message is rather long, but it is written in sections, to make it easy to get the gist quickly and neglect the rest. Andrew ------------------------------------------------------------------- 1. Short summary: People who got into the game late, or had been working at small ISPs or other enterprises, were generally willing to give serious credence to the "Internet doubling every 100 days" tale. The old-timers, especially those who worked for large ISPs or other large corporate establishment or research networks, were convinced by the late 1990s that this tale was false, but did not talk about it publicly, even inside the NANOG community. ------------------------------------------------------------------- 2. Longer version: The range of views was very wide, and hard to give justice to in full. But there seemed to be two distinct groups, and the consensus views (which obviously exclude quite a few people) appear to have been: 2A: Those who entered the field in the late 1990s, especially if they worked for small ISPs or other small enterprises, tended to regard the claim seriously. (But it should be remarked that hardly anybody devoted too much effort or thought to the claim, they were too busy putting out fires in their own backyards to worry about global issues.) They remembered periods of desperate efforts to keep up with exploding demand in their businesses. We saw just a few hours ago a post about LINX growing 5.5x in one year. Somebody else wrote about growing their business's traffic 1,000x in 2 years, or about 30x per year. People involved in such incidents often tended to think that their experience during such times might not have been untypical. 2B: Those who worked at places with large traffic, and especially those who got into the field in the early 1990s, were quite sure by the late 1990s that the UUNet fable was just that. Comments regarding everything emanating from UUNet during that period included phrases like "blowing smoke," "rolling our eyes," "taking it with a rock of salt." They had no direct knowledge of what went on inside UUNet, but from watching peering traffic, talking to salespeople about customer losses and wins, and to suppliers about deliveries of equipment, they could be pretty certain that neither the traffic nor the capacity of UUNet could be exploding at the mythical rates. They could see occasional spikes in traffic growth at some customers, or in some parts of their networks, but overall could see traffic growth settling down to a fairly regular doubling or a bit more than doubling each year. However, they did not discuss this in public, and I discuss that below, in point #4. ------------------------------------------------------------------- 3. Growth spurt in mid-1990s: The old-timers also provided very informative feedback about the global Internet traffic growth spurt in the 1995-96 time frame. It did hit suddenly and unexpectedly. (I am still trying to get confirmation on this, but I believe one of the informants said that before this period, the engineers at that person's Tier-1 ISP would routinely double the forecasts provided by the marketing team. At the peak of the bubble, the marketers would demand that the engineers plan for double the capacity that the engineers thought was going to be necessary.) Moreover, the dramatic slowdown in traffic growth that took place in 1997 was, at least in a number of cases, due substantially to a capacity crunch. Router and photonic equipment manufacturers did not have the technology needed for the traffic, and ILECs were slow in supplying access as well as backbone links. Hence the traffic growth spurt in 1996-96 was followed by a capacity growth spurt in 1997-98, which helped provide more credibility for the myth. ------------------------------------------------------------------- 4. Information viscosity: This incident provides far more information confirming the concept of "information viscosity" that I wrote about in my paper, that important and relevant information was available, but was not widely dispersed. Why did the information that Internet traffic was not doubling every 100 days get out to the public? It was not a closely guarded national security secret, after all. There seemed to be many reasons operating. In the case of Genuity (as the quote from Scott Marcus in my paper explains) it was a high-level decision, that going public would hurt the company, as it might be suspected of losing market share. But that seemed to be unusual, in that Genuity, while a major Tier-1 ISP, was small and independent, and so had people like Scott who were engineers, yet involved in policy making. At other places, other dynamics operated. At AT&T, for example, I was called in to a meeting with the management of WorldNet, the AT&T ISP unit, at the end of 2000. They had been telling their customers that Internet traffic was growing 10x per year, and some of those customers asked them about the discrepancy between that claim and the estimates of some folks from AT&T Labs - Research (Kerry Coffman and myself) that growth was just 2x per year. Now it is an interesting perspective on "information viscosity" and AT&T (and other large bureaucratic organizations) that those folks had not heard of Kerry's and my work, even though we had publicized it at the company. But in any event, at that meeting, I did succeed in convincing them that Internet traffic was growing only 2x per year (using evidence in my papers with Kerry, as well as extensive additional data from within AT&T itself), and they agreed they would not propagate the myth among customers. But the interesting thing was that many of the attendees (who included quite a few engineering types, not just the management) seemed disappointed. That really surprised me. After all, AT&T's Internet traffic was growing just about 4x per year, which meant our market share was doubling, instead of being cut in half. But it seems that many of them had really bought into the Internet dream. And, furthermore, the story that we were losing market share was a good one to pry additional resources from the corporation. Several of the NANOG old-timers said that they felt constrained from speaking about the falsity of the myth of astronomical growth rates by group solidarity. After all, it was a small, select group, and bad-mouthing one of their own in front of outsiders or even NANOG newbies did not seem the right thing to do. Then there was the additional factor that one person discussed very explicitly, and that I infer also applied in other cases. The myth was useful. Back in the mid-1990s, when it was not a myth, it did spur equipment suppliers and ILECs to greater effort. And afterwards, it was handy in internal fights over power and resources. If the Internet was exploding, one should not worry about closely examining expansion plans. In some ways, the persistence of false perceptions about Internet traffic growth mirrors that of utilization rates of data networks. When I wrote the paper "Data networks are lightly utilized, and will stay that way," back in 1998, http://www.dtc.umn.edu/~odlyzko/doc/network.utilization.pdf the more clueful data network engineers all knew that utilization rates were generally low, and usually had a good understanding as to why. But top management, as well as the research community, were almost uniformly convinced that congestion was the rule. It took me a while to understand the dynamics of the situation, in which network engineers and managers found it easier to say that they were experiencing 80% utilization and 20% packet losses and needed to upgrade, without having to explain to CEOs, CIOs, and especially to clueless CFOs, that this referred just to the peak hour on one crucial corporate WAN link, and yet justified a big new investment. Few people were lying, but many had incentives to maintain delusions among the top levels of the hierarchy. These demonstrations of information viscosity of course undermine the foundations of much of modern economics, especially of the efficient markets hypothesis. But that last myth is even more durable than "Internet traffic doubling every 100 days," especially since it does not lead directly to lots of dark fiber lying around unused, and lots of companies bankrupt. Information viscosity in facts and ideas in economics is very high, so we should not expect any changes on that scene. ------------------------------------------------------------------- 5. As a reminder, the paper that led to this discussion is "Bubbles, gullibility, and other challenges for economics, psychology, sociology, and information sciences," http://www.dtc.umn.edu/~odlyzko/doc/mania03.pdf The page with source materials from the bubbles times, http://www.dtc.umn.edu/~odlyzko/isources/ now has, in addition to a transcript of the O'Dell lecture at Stanford in May 2000, a copy of the Sidgmore paper from the Vortex98 Conference of May 1998.

10 12

Re: off-topic: summary on Internet traffic growth History
by Deepak Jain 11 Aug '10

11 Aug '10

On my BB. I'm waiting for someone to correct this thread by saying MFS bought UUNET for ~2bill and WCOM absorbed MFS. That is all. ----- Original Message ----- From: Jeffrey S. Young <young(a)jsyoung.net> To: John Lee <john(a)internetassociatesllc.com> Cc: nanog(a)nanog.org <nanog(a)nanog.org>; Andrew Odlyzko <odlyzko(a)umn.edu> Sent: Wed Aug 11 19:26:29 2010 Subject: Re: off-topic: summary on Internet traffic growth History Worldcom bought MFS. Worldcom bought MCI. Worldcom bought UUnet. In your statement s/MCI/Worldcom/g I don't know if UUnet was part of Worldcom when MO first made statements about backbone growth, but I do know that internetMCI was still part of MCI and therefore, MCI was not a part of Worldcom. May seem like splitting hairs to some, but it is important to a few of us to point out that we never worked under Ebbers. Not that we had a choice :-). Growth of the NAPs during this period is a poor indicator of growth. Because of the glitch you mention in carrying capacity the tier 1's all but abandoned the NAPs for peering between themselves and from that point forward (mid '97) preferred direct peering arrangements. jy On 12/08/2010, at 4:13 AM, John Lee <john(a)internetassociatesllc.com> wrote: > Andrew, > > Earlier this week I had a meeting with the ex-Director of the Network Operations Center for MFS-Datanet/MCI whose tenure was through 1999. From 1994 to 1998 they were re-architeching the Frame Relay and ATM networks to handle the growth in traffic including these new facilities called peering points of MAE-East and MAE-West. From roughly 1990 to then end of 1996 they saw traffic on their switches grow at 50-70% growth every 6 months. By the last half of 1996 there was a head of line blocking problem on the DEC FDDI switches that was "regularly" bringing down the Internet. The architecture had lower traffic circuits were going through concentrators while higher traffic circuits were directly attached to ports on the switchs. > > > > MFS-Datanet was not going to take the hit for the interruptions to the Internet and was going to inform the trade press there was a problem with DEC FDDI switches so Digital "gave" six switches for the re-architecture of the MAEs to solve the problem. Once this problem was solved the first quarter of 1997 saw a 70% jump in traffic that quarter alone. This "historical event" would in my memory be the genesis of the 100% traffic growth in 100 days legend. (So it was only 70% in 90 days which for the marketing folks does not cut it so 100% in 100 days sounds much better?? :) ) > > > > MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network. > > > > Personal Note: from 1983 to 90 I worked for Hayes the modem folks and became the Godfather to Ascend communications with Jeanette, Rob, Jay and Steve whose team produced the TNT line of modem/ISDN to Ethernet central site concentrators (in the early ninties) that drove a large portion of the user traffic to the Internet at the time, generating the "bubble". > > > > John (ISDN) Lee > ________________________________________ > From: Andrew Odlyzko [odlyzko(a)umn.edu] > Sent: Wednesday, August 11, 2010 12:55 PM > To: nanog(a)nanog.org > Subject: off-topic: summary on Internet traffic growth myths > > Since several members of this list requested it, here is a summary > of the responses to my request for information about Internet growth > during the telecom bubble, in particular the perceptions of the > O'Dell/Sidgmore/WorldCom/UUNet "Internet doubling every 100 days" > myth. > > First of all, many thanks to all those who responded, on and off-list. > This involved extensive correspondence and some long phone conversations, > and helped fill out the picture of those very confusing times (and > also made it even clearer than before that there were many different > perspectives on what was happening). > > The entire message is rather long, but it is written in sections, > to make it easy to get the gist quickly and neglect the rest. > > Andrew > > > ------------------------------------------------------------------- > > 1. Short summary: People who got into the game late, or had been > working at small ISPs or other enterprises, were generally willing > to give serious credence to the "Internet doubling every 100 days" > tale. The old-timers, especially those who worked for large ISPs > or other large corporate establishment or research networks, were > convinced by the late 1990s that this tale was false, but did not > talk about it publicly, even inside the NANOG community. > > ------------------------------------------------------------------- > > 2. Longer version: The range of views was very wide, and hard to > give justice to in full. But there seemed to be two distinct > groups, and the consensus views (which obviously exclude quite > a few people) appear to have been: > > 2A: Those who entered the field in the late 1990s, especially > if they worked for small ISPs or other small enterprises, tended > to regard the claim seriously. (But it should be remarked that > hardly anybody devoted too much effort or thought to the claim, > they were too busy putting out fires in their own backyards to > worry about global issues.) They remembered periods of desperate > efforts to keep up with exploding demand in their businesses. > We saw just a few hours ago a post about LINX growing 5.5x in > one year. Somebody else wrote about growing their business's > traffic 1,000x in 2 years, or about 30x per year. People involved > in such incidents often tended to think that their experience > during such times might not have been untypical. > > 2B: Those who worked at places with large traffic, and especially > those who got into the field in the early 1990s, were quite sure > by the late 1990s that the UUNet fable was just that. Comments > regarding everything emanating from UUNet during that period > included phrases like "blowing smoke," "rolling our eyes," "taking > it with a rock of salt." They had no direct knowledge of what > went on inside UUNet, but from watching peering traffic, talking > to salespeople about customer losses and wins, and to suppliers > about deliveries of equipment, they could be pretty certain that > neither the traffic nor the capacity of UUNet could be exploding > at the mythical rates. They could see occasional spikes in > traffic growth at some customers, or in some parts of their > networks, but overall could see traffic growth settling down > to a fairly regular doubling or a bit more than doubling each > year. > > However, they did not discuss this in public, and I discuss that > below, in point #4. > > ------------------------------------------------------------------- > > 3. Growth spurt in mid-1990s: The old-timers also provided very > informative feedback about the global Internet traffic growth > spurt in the 1995-96 time frame. It did hit suddenly and > unexpectedly. (I am still trying to get confirmation on this, > but I believe one of the informants said that before this > period, the engineers at that person's Tier-1 ISP would routinely > double the forecasts provided by the marketing team. At the peak > of the bubble, the marketers would demand that the engineers plan > for double the capacity that the engineers thought was going > to be necessary.) Moreover, the dramatic slowdown in traffic > growth that took place in 1997 was, at least in a number of > cases, due substantially to a capacity crunch. Router and > photonic equipment manufacturers did not have the technology > needed for the traffic, and ILECs were slow in supplying > access as well as backbone links. Hence the traffic growth > spurt in 1996-96 was followed by a capacity growth spurt > in 1997-98, which helped provide more credibility for the > myth. > > ------------------------------------------------------------------- > > 4. Information viscosity: This incident provides far more > information confirming the concept of "information viscosity" > that I wrote about in my paper, that important and relevant > information was available, but was not widely dispersed. > > Why did the information that Internet traffic was not doubling > every 100 days get out to the public? It was not a closely > guarded national security secret, after all. > > There seemed to be many reasons operating. In the case of > Genuity (as the quote from Scott Marcus in my paper explains) > it was a high-level decision, that going public would hurt > the company, as it might be suspected of losing market share. > But that seemed to be unusual, in that Genuity, while a > major Tier-1 ISP, was small and independent, and so had > people like Scott who were engineers, yet involved in > policy making. At other places, other dynamics operated. > > At AT&T, for example, I was called in to a meeting with > the management of WorldNet, the AT&T ISP unit, at the > end of 2000. They had been telling their customers that > Internet traffic was growing 10x per year, and some of > those customers asked them about the discrepancy between > that claim and the estimates of some folks from AT&T > Labs - Research (Kerry Coffman and myself) that growth > was just 2x per year. Now it is an interesting perspective > on "information viscosity" and AT&T (and other large > bureaucratic organizations) that those folks had not > heard of Kerry's and my work, even though we had publicized > it at the company. But in any event, at that meeting, > I did succeed in convincing them that Internet traffic > was growing only 2x per year (using evidence in my papers > with Kerry, as well as extensive additional data from > within AT&T itself), and they agreed they would not > propagate the myth among customers. But the interesting > thing was that many of the attendees (who included quite > a few engineering types, not just the management) seemed > disappointed. That really surprised me. After all, > AT&T's Internet traffic was growing just about 4x per > year, which meant our market share was doubling, instead > of being cut in half. But it seems that many of them > had really bought into the Internet dream. And, furthermore, > the story that we were losing market share was a good one > to pry additional resources from the corporation. > > Several of the NANOG old-timers said that they felt constrained > from speaking about the falsity of the myth of astronomical > growth rates by group solidarity. After all, it was a > small, select group, and bad-mouthing one of their own > in front of outsiders or even NANOG newbies did not seem > the right thing to do. Then there was the additional > factor that one person discussed very explicitly, and > that I infer also applied in other cases. The myth was > useful. Back in the mid-1990s, when it was not a myth, > it did spur equipment suppliers and ILECs to greater > effort. And afterwards, it was handy in internal fights > over power and resources. If the Internet was exploding, > one should not worry about closely examining expansion > plans. > > In some ways, the persistence of false perceptions about > Internet traffic growth mirrors that of utilization rates > of data networks. When I wrote the paper "Data networks > are lightly utilized, and will stay that way," back in 1998, > > http://www.dtc.umn.edu/~odlyzko/doc/network.utilization.pdf > > the more clueful data network engineers all knew that > utilization rates were generally low, and usually had > a good understanding as to why. But top management, as > well as the research community, were almost uniformly > convinced that congestion was the rule. It took me a > while to understand the dynamics of the situation, in > which network engineers and managers found it easier > to say that they were experiencing 80% utilization and > 20% packet losses and needed to upgrade, without having > to explain to CEOs, CIOs, and especially to clueless > CFOs, that this referred just to the peak hour on one crucial > corporate WAN link, and yet justified a big new investment. > Few people were lying, but many had incentives to maintain > delusions among the top levels of the hierarchy. > > These demonstrations of information viscosity of course > undermine the foundations of much of modern economics, > especially of the efficient markets hypothesis. But that > last myth is even more durable than "Internet traffic > doubling every 100 days," especially since it does not > lead directly to lots of dark fiber lying around unused, > and lots of companies bankrupt. Information viscosity > in facts and ideas in economics is very high, so we > should not expect any changes on that scene. > > > ------------------------------------------------------------------- > > 5. As a reminder, the paper that led to this discussion is > "Bubbles, gullibility, and other challenges for economics, > psychology, sociology, and information sciences," > > http://www.dtc.umn.edu/~odlyzko/doc/mania03.pdf > > The page with source materials from the bubbles times, > > http://www.dtc.umn.edu/~odlyzko/isources/ > > now has, in addition to a transcript of the O'Dell lecture > at Stanford in May 2000, a copy of the Sidgmore paper from > the Vortex98 Conference of May 1998. >

1 0

MPLS providers between Portland OR and NYC
by Brent Jones 11 Aug '10

11 Aug '10

Does anyone have any recommendations or experience to share about any L2 providers in the Portland OR, and NYC markets? Specifically, providers that can do 1Gbit between those cities, with reasonable SLAs. Comments off-list would be greatly appreciated! -- Brent Jones brent(a)servuhome.net

1 0