RE: off-topic: summary on Internet traffic growth History
Andrew, Earlier this week I had a meeting with the ex-Director of the Network Operations Center for MFS-Datanet/MCI whose tenure was through 1999. From 1994 to 1998 they were re-architeching the Frame Relay and ATM networks to handle the growth in traffic including these new facilities called peering points of MAE-East and MAE-West. From roughly 1990 to then end of 1996 they saw traffic on their switches grow at 50-70% growth every 6 months. By the last half of 1996 there was a head of line blocking problem on the DEC FDDI switches that was "regularly" bringing down the Internet. The architecture had lower traffic circuits were going through concentrators while higher traffic circuits were directly attached to ports on the switchs. MFS-Datanet was not going to take the hit for the interruptions to the Internet and was going to inform the trade press there was a problem with DEC FDDI switches so Digital "gave" six switches for the re-architecture of the MAEs to solve the problem. Once this problem was solved the first quarter of 1997 saw a 70% jump in traffic that quarter alone. This "historical event" would in my memory be the genesis of the 100% traffic growth in 100 days legend. (So it was only 70% in 90 days which for the marketing folks does not cut it so 100% in 100 days sounds much better?? :) ) MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network. Personal Note: from 1983 to 90 I worked for Hayes the modem folks and became the Godfather to Ascend communications with Jeanette, Rob, Jay and Steve whose team produced the TNT line of modem/ISDN to Ethernet central site concentrators (in the early ninties) that drove a large portion of the user traffic to the Internet at the time, generating the "bubble". John (ISDN) Lee ________________________________________ From: Andrew Odlyzko [odlyzko@umn.edu] Sent: Wednesday, August 11, 2010 12:55 PM To: nanog@nanog.org Subject: off-topic: summary on Internet traffic growth myths Since several members of this list requested it, here is a summary of the responses to my request for information about Internet growth during the telecom bubble, in particular the perceptions of the O'Dell/Sidgmore/WorldCom/UUNet "Internet doubling every 100 days" myth. First of all, many thanks to all those who responded, on and off-list. This involved extensive correspondence and some long phone conversations, and helped fill out the picture of those very confusing times (and also made it even clearer than before that there were many different perspectives on what was happening). The entire message is rather long, but it is written in sections, to make it easy to get the gist quickly and neglect the rest. Andrew ------------------------------------------------------------------- 1. Short summary: People who got into the game late, or had been working at small ISPs or other enterprises, were generally willing to give serious credence to the "Internet doubling every 100 days" tale. The old-timers, especially those who worked for large ISPs or other large corporate establishment or research networks, were convinced by the late 1990s that this tale was false, but did not talk about it publicly, even inside the NANOG community. ------------------------------------------------------------------- 2. Longer version: The range of views was very wide, and hard to give justice to in full. But there seemed to be two distinct groups, and the consensus views (which obviously exclude quite a few people) appear to have been: 2A: Those who entered the field in the late 1990s, especially if they worked for small ISPs or other small enterprises, tended to regard the claim seriously. (But it should be remarked that hardly anybody devoted too much effort or thought to the claim, they were too busy putting out fires in their own backyards to worry about global issues.) They remembered periods of desperate efforts to keep up with exploding demand in their businesses. We saw just a few hours ago a post about LINX growing 5.5x in one year. Somebody else wrote about growing their business's traffic 1,000x in 2 years, or about 30x per year. People involved in such incidents often tended to think that their experience during such times might not have been untypical. 2B: Those who worked at places with large traffic, and especially those who got into the field in the early 1990s, were quite sure by the late 1990s that the UUNet fable was just that. Comments regarding everything emanating from UUNet during that period included phrases like "blowing smoke," "rolling our eyes," "taking it with a rock of salt." They had no direct knowledge of what went on inside UUNet, but from watching peering traffic, talking to salespeople about customer losses and wins, and to suppliers about deliveries of equipment, they could be pretty certain that neither the traffic nor the capacity of UUNet could be exploding at the mythical rates. They could see occasional spikes in traffic growth at some customers, or in some parts of their networks, but overall could see traffic growth settling down to a fairly regular doubling or a bit more than doubling each year. However, they did not discuss this in public, and I discuss that below, in point #4. ------------------------------------------------------------------- 3. Growth spurt in mid-1990s: The old-timers also provided very informative feedback about the global Internet traffic growth spurt in the 1995-96 time frame. It did hit suddenly and unexpectedly. (I am still trying to get confirmation on this, but I believe one of the informants said that before this period, the engineers at that person's Tier-1 ISP would routinely double the forecasts provided by the marketing team. At the peak of the bubble, the marketers would demand that the engineers plan for double the capacity that the engineers thought was going to be necessary.) Moreover, the dramatic slowdown in traffic growth that took place in 1997 was, at least in a number of cases, due substantially to a capacity crunch. Router and photonic equipment manufacturers did not have the technology needed for the traffic, and ILECs were slow in supplying access as well as backbone links. Hence the traffic growth spurt in 1996-96 was followed by a capacity growth spurt in 1997-98, which helped provide more credibility for the myth. ------------------------------------------------------------------- 4. Information viscosity: This incident provides far more information confirming the concept of "information viscosity" that I wrote about in my paper, that important and relevant information was available, but was not widely dispersed. Why did the information that Internet traffic was not doubling every 100 days get out to the public? It was not a closely guarded national security secret, after all. There seemed to be many reasons operating. In the case of Genuity (as the quote from Scott Marcus in my paper explains) it was a high-level decision, that going public would hurt the company, as it might be suspected of losing market share. But that seemed to be unusual, in that Genuity, while a major Tier-1 ISP, was small and independent, and so had people like Scott who were engineers, yet involved in policy making. At other places, other dynamics operated. At AT&T, for example, I was called in to a meeting with the management of WorldNet, the AT&T ISP unit, at the end of 2000. They had been telling their customers that Internet traffic was growing 10x per year, and some of those customers asked them about the discrepancy between that claim and the estimates of some folks from AT&T Labs - Research (Kerry Coffman and myself) that growth was just 2x per year. Now it is an interesting perspective on "information viscosity" and AT&T (and other large bureaucratic organizations) that those folks had not heard of Kerry's and my work, even though we had publicized it at the company. But in any event, at that meeting, I did succeed in convincing them that Internet traffic was growing only 2x per year (using evidence in my papers with Kerry, as well as extensive additional data from within AT&T itself), and they agreed they would not propagate the myth among customers. But the interesting thing was that many of the attendees (who included quite a few engineering types, not just the management) seemed disappointed. That really surprised me. After all, AT&T's Internet traffic was growing just about 4x per year, which meant our market share was doubling, instead of being cut in half. But it seems that many of them had really bought into the Internet dream. And, furthermore, the story that we were losing market share was a good one to pry additional resources from the corporation. Several of the NANOG old-timers said that they felt constrained from speaking about the falsity of the myth of astronomical growth rates by group solidarity. After all, it was a small, select group, and bad-mouthing one of their own in front of outsiders or even NANOG newbies did not seem the right thing to do. Then there was the additional factor that one person discussed very explicitly, and that I infer also applied in other cases. The myth was useful. Back in the mid-1990s, when it was not a myth, it did spur equipment suppliers and ILECs to greater effort. And afterwards, it was handy in internal fights over power and resources. If the Internet was exploding, one should not worry about closely examining expansion plans. In some ways, the persistence of false perceptions about Internet traffic growth mirrors that of utilization rates of data networks. When I wrote the paper "Data networks are lightly utilized, and will stay that way," back in 1998, http://www.dtc.umn.edu/~odlyzko/doc/network.utilization.pdf the more clueful data network engineers all knew that utilization rates were generally low, and usually had a good understanding as to why. But top management, as well as the research community, were almost uniformly convinced that congestion was the rule. It took me a while to understand the dynamics of the situation, in which network engineers and managers found it easier to say that they were experiencing 80% utilization and 20% packet losses and needed to upgrade, without having to explain to CEOs, CIOs, and especially to clueless CFOs, that this referred just to the peak hour on one crucial corporate WAN link, and yet justified a big new investment. Few people were lying, but many had incentives to maintain delusions among the top levels of the hierarchy. These demonstrations of information viscosity of course undermine the foundations of much of modern economics, especially of the efficient markets hypothesis. But that last myth is even more durable than "Internet traffic doubling every 100 days," especially since it does not lead directly to lots of dark fiber lying around unused, and lots of companies bankrupt. Information viscosity in facts and ideas in economics is very high, so we should not expect any changes on that scene. ------------------------------------------------------------------- 5. As a reminder, the paper that led to this discussion is "Bubbles, gullibility, and other challenges for economics, psychology, sociology, and information sciences," http://www.dtc.umn.edu/~odlyzko/doc/mania03.pdf The page with source materials from the bubbles times, http://www.dtc.umn.edu/~odlyzko/isources/ now has, in addition to a transcript of the O'Dell lecture at Stanford in May 2000, a copy of the Sidgmore paper from the Vortex98 Conference of May 1998.
On Aug 11, 2010, at 1:13 PM, John Lee wrote:
MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.
Although not directly involved in the MCI Internet operations, I read all the announcements that came across the email when I worked at MCI from early 1993 to late 1998. My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-). While this was going on MCI was building and running what was called the BIPP (Basic IP Platform) internally. That product was at least reasonably successful, enough so that some gummint powers that be required divestiture of the BIPP from the company that would come out of the proposed acquisition of MCI by Worldcom. The regulators felt that Worldcom would have too large a share of the North American Internet traffic. The BIPP went with BT IIRC, and I think finally landed in Global Crossing's assets. --Chris
On 8/11/2010 3:10 PM, Chris Boyd wrote:
On Aug 11, 2010, at 1:13 PM, John Lee wrote:
MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.
Although not directly involved in the MCI Internet operations, I read all the announcements that came across the email when I worked at MCI from early 1993 to late 1998.
My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-). While this was going on MCI was building and running what was called the BIPP (Basic IP Platform) internally. That product was at least reasonably successful, enough so that some gummint powers that be required divestiture of the BIPP from the company that would come out of the proposed acquisition of MCI by Worldcom. The regulators felt that Worldcom would have too large a share of the North American Internet traffic. The BIPP went with BT IIRC, and I think finally landed in Global Crossing's assets.
--Chris
Correct order of (in)digestion UUNet > MFS > Worldcom >< MCI > Verizon. There were other multi-way acquisitions in-between as well (CNS, ANS, etc.) -Randy.
On Aug 11, 2010, at 3:28 PM, Randy Whitney wrote:
On 8/11/2010 3:10 PM, Chris Boyd wrote:
On Aug 11, 2010, at 1:13 PM, John Lee wrote:
MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.
Although not directly involved in the MCI Internet operations, I read all the announcements that came across the email when I worked at MCI from early 1993 to late 1998.
My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-). While this was going on MCI was building and running what was called the BIPP (Basic IP Platform) internally. That product was at least reasonably successful, enough so that some gummint powers that be required divestiture of the BIPP from the company that would come out of the proposed acquisition of MCI by Worldcom. The regulators felt that Worldcom would have too large a share of the North American Internet traffic. The BIPP went with BT IIRC, and I think finally landed in Global Crossing's assets.
What happened to VBNS in all of this ? Marshall
--Chris
Correct order of (in)digestion UUNet > MFS > Worldcom >< MCI > Verizon.
There were other multi-way acquisitions in-between as well (CNS, ANS, etc.)
-Randy.
VBNS is part of VzB. On 8/11/10, Marshall Eubanks <tme@americafree.tv> wrote:
On Aug 11, 2010, at 3:28 PM, Randy Whitney wrote:
On 8/11/2010 3:10 PM, Chris Boyd wrote:
On Aug 11, 2010, at 1:13 PM, John Lee wrote:
MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.
Although not directly involved in the MCI Internet operations, I read all the announcements that came across the email when I worked at MCI from early 1993 to late 1998.
My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-). While this was going on MCI was building and running what was called the BIPP (Basic IP Platform) internally. That product was at least reasonably successful, enough so that some gummint powers that be required divestiture of the BIPP from the company that would come out of the proposed acquisition of MCI by Worldcom. The regulators felt that Worldcom would have too large a share of the North American Internet traffic. The BIPP went with BT IIRC, and I think finally landed in Global Crossing's assets.
What happened to VBNS in all of this ?
Marshall
--Chris
Correct order of (in)digestion UUNet > MFS > Worldcom >< MCI > Verizon.
There were other multi-way acquisitions in-between as well (CNS, ANS, etc.)
-Randy.
-- Sent from my mobile device William McCall, CCIE #25044
Chris Boyd <cboyd@gizmopartners.com> writes:
My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-).
MFS acquired UUnet Technologies on 12 August 1996. On 31 December 1996, MFS "merged with and into WorldCom, Inc" although it sure looked like an acquisition to those of us who were watching from the sidelines. -r
On 11 Aug 10, at 2:10 PM, Chris Boyd wrote:
My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-). While this was going on MCI was building and running what was called the BIPP (Basic IP Platform) internally. That product was at least reasonably successful, enough so that some gummint powers that be required divestiture of the BIPP from the company that would come out of the proposed acquisition of MCI by Worldcom. The regulators felt that Worldcom would have too large a share of the North American Internet traffic. The BIPP went with BT IIRC, and I think finally landed in Global Crossing's assets.
Actually, Cable & Wireless acquired the BIPP after regulators forced Worldcom to divest one of their networks. C&W developed a new network architecture as an evolution of BIPP called "N3", based on MPLS as an ATM replacement for TE. (Perhaps somebody that worked at C&W back then can comment on N3; I can't recall what it stood for.) After a few years, C&W reorganized their American operations into a separate entity, which subsequently went bankrupt. Savvis (my current employer) bought the assets out of bankruptcy court. We then upgraded the N3 network to support better QoS, higher capacity, etc, and call it the "ATN" (Application Transport Network). The current Savvis core network, AS3561, is thus the evolved offspring of the MCI Internet Services / Internet-MCI network. Of course, before all of this, MCI built the network as a commercial Internet platform in parallel to their ARPA network. That's before my time, unfortunately, so I don't know many details. For instance I'm uncertain how the ASN has changed over the years. Anybody with more history and/or corrections would be appreciated. Cheers, -Benson
N3 = new network nodes, BIPP wasn't that great a name either. The ASN was always 3561. jy On 12/08/2010, at 8:20 AM, Benson Schliesser <bensons@queuefull.net> wrote:
On 11 Aug 10, at 2:10 PM, Chris Boyd wrote:
My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-). While this was going on MCI was building and running what was called the BIPP (Basic IP Platform) internally. That product was at least reasonably successful, enough so that some gummint powers that be required divestiture of the BIPP from the company that would come out of the proposed acquisition of MCI by Worldcom. The regulators felt that Worldcom would have too large a share of the North American Internet traffic. The BIPP went with BT IIRC, and I think finally landed in Global Crossing's assets.
Actually, Cable & Wireless acquired the BIPP after regulators forced Worldcom to divest one of their networks. C&W developed a new network architecture as an evolution of BIPP called "N3", based on MPLS as an ATM replacement for TE. (Perhaps somebody that worked at C&W back then can comment on N3; I can't recall what it stood for.) After a few years, C&W reorganized their American operations into a separate entity, which subsequently went bankrupt. Savvis (my current employer) bought the assets out of bankruptcy court. We then upgraded the N3 network to support better QoS, higher capacity, etc, and call it the "ATN" (Application Transport Network). The current Savvis core network, AS3561, is thus the evolved offspring of the MCI Internet Services / Internet-MCI network.
Of course, before all of this, MCI built the network as a commercial Internet platform in parallel to their ARPA network. That's before my time, unfortunately, so I don't know many details. For instance I'm uncertain how the ASN has changed over the years. Anybody with more history and/or corrections would be appreciated.
Cheers, -Benson
BIPP was sold to C&W where it continued to use MCI transmission and facilities. In November 2000, C&W had rebuilt it on their own facilities (just a bit larger). Quite soon after the completion of the new network in 2000, C&W marketing was forecasting the need for a network that was ten times the size of their current backbone (the new network was four times the size of the original iMCI). C&W was chapter 7 within 12 months. BTW: C&W sued Worldcom and won a $250M settlement on the basis that MCI had hidden the iMCI sales and marketing team in the sale. The assets of C&W were sold to Savvis. jy On 12/08/2010, at 5:10 AM, Chris Boyd <cboyd@gizmopartners.com> wrote:
On Aug 11, 2010, at 1:13 PM, John Lee wrote:
MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.
Although not directly involved in the MCI Internet operations, I read all the announcements that came across the email when I worked at MCI from early 1993 to late 1998.
My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-). While this was going on MCI was building and running what was called the BIPP (Basic IP Platform) internally. That product was at least reasonably successful, enough so that some gummint powers that be required divestiture of the BIPP from the company that would come out of the proposed acquisition of MCI by Worldcom. The regulators felt that Worldcom would have too large a share of the North American Internet traffic. The BIPP went with BT IIRC, and I think finally landed in Global Crossing's assets.
--Chris
CIP went with BT (Concert) I still clearly remember the very long concall when we separated it from it BIPP connections. :) -jim On Wed, Aug 11, 2010 at 4:10 PM, Chris Boyd <cboyd@gizmopartners.com> wrote:
On Aug 11, 2010, at 1:13 PM, John Lee wrote:
MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.
Although not directly involved in the MCI Internet operations, I read all the announcements that came across the email when I worked at MCI from early 1993 to late 1998.
My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-). While this was going on MCI was building and running what was called the BIPP (Basic IP Platform) internally. That product was at least reasonably successful, enough so that some gummint powers that be required divestiture of the BIPP from the company that would come out of the proposed acquisition of MCI by Worldcom. The regulators felt that Worldcom would have too large a share of the North American Internet traffic. The BIPP went with BT IIRC, and I think finally landed in Global Crossing's assets.
--Chris
MCI and BT had a long courtship. BT left MCI standing at the altar after neighborhoodMCI (a consumer last mile play) announced $400M in losses, twice. WorldCom swooped in after that. jy On 12/08/2010, at 12:12 PM, jim deleskie <deleskie@gmail.com> wrote:
CIP went with BT (Concert) I still clearly remember the very long concall when we separated it from it BIPP connections. :)
-jim
On Wed, Aug 11, 2010 at 4:10 PM, Chris Boyd <cboyd@gizmopartners.com> wrote:
On Aug 11, 2010, at 1:13 PM, John Lee wrote:
MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.
Although not directly involved in the MCI Internet operations, I read all the announcements that came across the email when I worked at MCI from early 1993 to late 1998.
My recollection is that Worldcom bought out MFS. UUnet was a later acquisition by the Worldcom monster (no, no biases here :-). While this was going on MCI was building and running what was called the BIPP (Basic IP Platform) internally. That product was at least reasonably successful, enough so that some gummint powers that be required divestiture of the BIPP from the company that would come out of the proposed acquisition of MCI by Worldcom. The regulators felt that Worldcom would have too large a share of the North American Internet traffic. The BIPP went with BT IIRC, and I think finally landed in Global Crossing's assets.
--Chris
Worldcom bought MFS. Worldcom bought MCI. Worldcom bought UUnet. In your statement s/MCI/Worldcom/g I don't know if UUnet was part of Worldcom when MO first made statements about backbone growth, but I do know that internetMCI was still part of MCI and therefore, MCI was not a part of Worldcom. May seem like splitting hairs to some, but it is important to a few of us to point out that we never worked under Ebbers. Not that we had a choice :-). Growth of the NAPs during this period is a poor indicator of growth. Because of the glitch you mention in carrying capacity the tier 1's all but abandoned the NAPs for peering between themselves and from that point forward (mid '97) preferred direct peering arrangements. jy On 12/08/2010, at 4:13 AM, John Lee <john@internetassociatesllc.com> wrote:
Andrew,
Earlier this week I had a meeting with the ex-Director of the Network Operations Center for MFS-Datanet/MCI whose tenure was through 1999. From 1994 to 1998 they were re-architeching the Frame Relay and ATM networks to handle the growth in traffic including these new facilities called peering points of MAE-East and MAE-West. From roughly 1990 to then end of 1996 they saw traffic on their switches grow at 50-70% growth every 6 months. By the last half of 1996 there was a head of line blocking problem on the DEC FDDI switches that was "regularly" bringing down the Internet. The architecture had lower traffic circuits were going through concentrators while higher traffic circuits were directly attached to ports on the switchs.
MFS-Datanet was not going to take the hit for the interruptions to the Internet and was going to inform the trade press there was a problem with DEC FDDI switches so Digital "gave" six switches for the re-architecture of the MAEs to solve the problem. Once this problem was solved the first quarter of 1997 saw a 70% jump in traffic that quarter alone. This "historical event" would in my memory be the genesis of the 100% traffic growth in 100 days legend. (So it was only 70% in 90 days which for the marketing folks does not cut it so 100% in 100 days sounds much better?? :) )
MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.
Personal Note: from 1983 to 90 I worked for Hayes the modem folks and became the Godfather to Ascend communications with Jeanette, Rob, Jay and Steve whose team produced the TNT line of modem/ISDN to Ethernet central site concentrators (in the early ninties) that drove a large portion of the user traffic to the Internet at the time, generating the "bubble".
John (ISDN) Lee ________________________________________ From: Andrew Odlyzko [odlyzko@umn.edu] Sent: Wednesday, August 11, 2010 12:55 PM To: nanog@nanog.org Subject: off-topic: summary on Internet traffic growth myths
Since several members of this list requested it, here is a summary of the responses to my request for information about Internet growth during the telecom bubble, in particular the perceptions of the O'Dell/Sidgmore/WorldCom/UUNet "Internet doubling every 100 days" myth.
First of all, many thanks to all those who responded, on and off-list. This involved extensive correspondence and some long phone conversations, and helped fill out the picture of those very confusing times (and also made it even clearer than before that there were many different perspectives on what was happening).
The entire message is rather long, but it is written in sections, to make it easy to get the gist quickly and neglect the rest.
Andrew
-------------------------------------------------------------------
1. Short summary: People who got into the game late, or had been working at small ISPs or other enterprises, were generally willing to give serious credence to the "Internet doubling every 100 days" tale. The old-timers, especially those who worked for large ISPs or other large corporate establishment or research networks, were convinced by the late 1990s that this tale was false, but did not talk about it publicly, even inside the NANOG community.
-------------------------------------------------------------------
2. Longer version: The range of views was very wide, and hard to give justice to in full. But there seemed to be two distinct groups, and the consensus views (which obviously exclude quite a few people) appear to have been:
2A: Those who entered the field in the late 1990s, especially if they worked for small ISPs or other small enterprises, tended to regard the claim seriously. (But it should be remarked that hardly anybody devoted too much effort or thought to the claim, they were too busy putting out fires in their own backyards to worry about global issues.) They remembered periods of desperate efforts to keep up with exploding demand in their businesses. We saw just a few hours ago a post about LINX growing 5.5x in one year. Somebody else wrote about growing their business's traffic 1,000x in 2 years, or about 30x per year. People involved in such incidents often tended to think that their experience during such times might not have been untypical.
2B: Those who worked at places with large traffic, and especially those who got into the field in the early 1990s, were quite sure by the late 1990s that the UUNet fable was just that. Comments regarding everything emanating from UUNet during that period included phrases like "blowing smoke," "rolling our eyes," "taking it with a rock of salt." They had no direct knowledge of what went on inside UUNet, but from watching peering traffic, talking to salespeople about customer losses and wins, and to suppliers about deliveries of equipment, they could be pretty certain that neither the traffic nor the capacity of UUNet could be exploding at the mythical rates. They could see occasional spikes in traffic growth at some customers, or in some parts of their networks, but overall could see traffic growth settling down to a fairly regular doubling or a bit more than doubling each year.
However, they did not discuss this in public, and I discuss that below, in point #4.
-------------------------------------------------------------------
3. Growth spurt in mid-1990s: The old-timers also provided very informative feedback about the global Internet traffic growth spurt in the 1995-96 time frame. It did hit suddenly and unexpectedly. (I am still trying to get confirmation on this, but I believe one of the informants said that before this period, the engineers at that person's Tier-1 ISP would routinely double the forecasts provided by the marketing team. At the peak of the bubble, the marketers would demand that the engineers plan for double the capacity that the engineers thought was going to be necessary.) Moreover, the dramatic slowdown in traffic growth that took place in 1997 was, at least in a number of cases, due substantially to a capacity crunch. Router and photonic equipment manufacturers did not have the technology needed for the traffic, and ILECs were slow in supplying access as well as backbone links. Hence the traffic growth spurt in 1996-96 was followed by a capacity growth spurt in 1997-98, which helped provide more credibility for the myth.
-------------------------------------------------------------------
4. Information viscosity: This incident provides far more information confirming the concept of "information viscosity" that I wrote about in my paper, that important and relevant information was available, but was not widely dispersed.
Why did the information that Internet traffic was not doubling every 100 days get out to the public? It was not a closely guarded national security secret, after all.
There seemed to be many reasons operating. In the case of Genuity (as the quote from Scott Marcus in my paper explains) it was a high-level decision, that going public would hurt the company, as it might be suspected of losing market share. But that seemed to be unusual, in that Genuity, while a major Tier-1 ISP, was small and independent, and so had people like Scott who were engineers, yet involved in policy making. At other places, other dynamics operated.
At AT&T, for example, I was called in to a meeting with the management of WorldNet, the AT&T ISP unit, at the end of 2000. They had been telling their customers that Internet traffic was growing 10x per year, and some of those customers asked them about the discrepancy between that claim and the estimates of some folks from AT&T Labs - Research (Kerry Coffman and myself) that growth was just 2x per year. Now it is an interesting perspective on "information viscosity" and AT&T (and other large bureaucratic organizations) that those folks had not heard of Kerry's and my work, even though we had publicized it at the company. But in any event, at that meeting, I did succeed in convincing them that Internet traffic was growing only 2x per year (using evidence in my papers with Kerry, as well as extensive additional data from within AT&T itself), and they agreed they would not propagate the myth among customers. But the interesting thing was that many of the attendees (who included quite a few engineering types, not just the management) seemed disappointed. That really surprised me. After all, AT&T's Internet traffic was growing just about 4x per year, which meant our market share was doubling, instead of being cut in half. But it seems that many of them had really bought into the Internet dream. And, furthermore, the story that we were losing market share was a good one to pry additional resources from the corporation.
Several of the NANOG old-timers said that they felt constrained from speaking about the falsity of the myth of astronomical growth rates by group solidarity. After all, it was a small, select group, and bad-mouthing one of their own in front of outsiders or even NANOG newbies did not seem the right thing to do. Then there was the additional factor that one person discussed very explicitly, and that I infer also applied in other cases. The myth was useful. Back in the mid-1990s, when it was not a myth, it did spur equipment suppliers and ILECs to greater effort. And afterwards, it was handy in internal fights over power and resources. If the Internet was exploding, one should not worry about closely examining expansion plans.
In some ways, the persistence of false perceptions about Internet traffic growth mirrors that of utilization rates of data networks. When I wrote the paper "Data networks are lightly utilized, and will stay that way," back in 1998,
http://www.dtc.umn.edu/~odlyzko/doc/network.utilization.pdf
the more clueful data network engineers all knew that utilization rates were generally low, and usually had a good understanding as to why. But top management, as well as the research community, were almost uniformly convinced that congestion was the rule. It took me a while to understand the dynamics of the situation, in which network engineers and managers found it easier to say that they were experiencing 80% utilization and 20% packet losses and needed to upgrade, without having to explain to CEOs, CIOs, and especially to clueless CFOs, that this referred just to the peak hour on one crucial corporate WAN link, and yet justified a big new investment. Few people were lying, but many had incentives to maintain delusions among the top levels of the hierarchy.
These demonstrations of information viscosity of course undermine the foundations of much of modern economics, especially of the efficient markets hypothesis. But that last myth is even more durable than "Internet traffic doubling every 100 days," especially since it does not lead directly to lots of dark fiber lying around unused, and lots of companies bankrupt. Information viscosity in facts and ideas in economics is very high, so we should not expect any changes on that scene.
-------------------------------------------------------------------
5. As a reminder, the paper that led to this discussion is "Bubbles, gullibility, and other challenges for economics, psychology, sociology, and information sciences,"
http://www.dtc.umn.edu/~odlyzko/doc/mania03.pdf
The page with source materials from the bubbles times,
http://www.dtc.umn.edu/~odlyzko/isources/
now has, in addition to a transcript of the O'Dell lecture at Stanford in May 2000, a copy of the Sidgmore paper from the Vortex98 Conference of May 1998.
I think for most of us iMCI'ers its a very big diffrence that iMCI != MCIWorldcom -jim Sent from my BlackBerry device on the Rogers Wireless Network -----Original Message----- From: "Jeffrey S. Young" <young@jsyoung.net> Date: Thu, 12 Aug 2010 09:26:29 To: John Lee<john@internetassociatesllc.com> Cc: nanog@nanog.org<nanog@nanog.org>; Andrew Odlyzko<odlyzko@umn.edu> Subject: Re: off-topic: summary on Internet traffic growth History Worldcom bought MFS. Worldcom bought MCI. Worldcom bought UUnet. In your statement s/MCI/Worldcom/g I don't know if UUnet was part of Worldcom when MO first made statements about backbone growth, but I do know that internetMCI was still part of MCI and therefore, MCI was not a part of Worldcom. May seem like splitting hairs to some, but it is important to a few of us to point out that we never worked under Ebbers. Not that we had a choice :-). Growth of the NAPs during this period is a poor indicator of growth. Because of the glitch you mention in carrying capacity the tier 1's all but abandoned the NAPs for peering between themselves and from that point forward (mid '97) preferred direct peering arrangements. jy On 12/08/2010, at 4:13 AM, John Lee <john@internetassociatesllc.com> wrote:
Andrew,
Earlier this week I had a meeting with the ex-Director of the Network Operations Center for MFS-Datanet/MCI whose tenure was through 1999. From 1994 to 1998 they were re-architeching the Frame Relay and ATM networks to handle the growth in traffic including these new facilities called peering points of MAE-East and MAE-West. From roughly 1990 to then end of 1996 they saw traffic on their switches grow at 50-70% growth every 6 months. By the last half of 1996 there was a head of line blocking problem on the DEC FDDI switches that was "regularly" bringing down the Internet. The architecture had lower traffic circuits were going through concentrators while higher traffic circuits were directly attached to ports on the switchs.
MFS-Datanet was not going to take the hit for the interruptions to the Internet and was going to inform the trade press there was a problem with DEC FDDI switches so Digital "gave" six switches for the re-architecture of the MAEs to solve the problem. Once this problem was solved the first quarter of 1997 saw a 70% jump in traffic that quarter alone. This "historical event" would in my memory be the genesis of the 100% traffic growth in 100 days legend. (So it was only 70% in 90 days which for the marketing folks does not cut it so 100% in 100 days sounds much better?? :) )
MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network.
Personal Note: from 1983 to 90 I worked for Hayes the modem folks and became the Godfather to Ascend communications with Jeanette, Rob, Jay and Steve whose team produced the TNT line of modem/ISDN to Ethernet central site concentrators (in the early ninties) that drove a large portion of the user traffic to the Internet at the time, generating the "bubble".
John (ISDN) Lee ________________________________________ From: Andrew Odlyzko [odlyzko@umn.edu] Sent: Wednesday, August 11, 2010 12:55 PM To: nanog@nanog.org Subject: off-topic: summary on Internet traffic growth myths
Since several members of this list requested it, here is a summary of the responses to my request for information about Internet growth during the telecom bubble, in particular the perceptions of the O'Dell/Sidgmore/WorldCom/UUNet "Internet doubling every 100 days" myth.
First of all, many thanks to all those who responded, on and off-list. This involved extensive correspondence and some long phone conversations, and helped fill out the picture of those very confusing times (and also made it even clearer than before that there were many different perspectives on what was happening).
The entire message is rather long, but it is written in sections, to make it easy to get the gist quickly and neglect the rest.
Andrew
-------------------------------------------------------------------
1. Short summary: People who got into the game late, or had been working at small ISPs or other enterprises, were generally willing to give serious credence to the "Internet doubling every 100 days" tale. The old-timers, especially those who worked for large ISPs or other large corporate establishment or research networks, were convinced by the late 1990s that this tale was false, but did not talk about it publicly, even inside the NANOG community.
-------------------------------------------------------------------
2. Longer version: The range of views was very wide, and hard to give justice to in full. But there seemed to be two distinct groups, and the consensus views (which obviously exclude quite a few people) appear to have been:
2A: Those who entered the field in the late 1990s, especially if they worked for small ISPs or other small enterprises, tended to regard the claim seriously. (But it should be remarked that hardly anybody devoted too much effort or thought to the claim, they were too busy putting out fires in their own backyards to worry about global issues.) They remembered periods of desperate efforts to keep up with exploding demand in their businesses. We saw just a few hours ago a post about LINX growing 5.5x in one year. Somebody else wrote about growing their business's traffic 1,000x in 2 years, or about 30x per year. People involved in such incidents often tended to think that their experience during such times might not have been untypical.
2B: Those who worked at places with large traffic, and especially those who got into the field in the early 1990s, were quite sure by the late 1990s that the UUNet fable was just that. Comments regarding everything emanating from UUNet during that period included phrases like "blowing smoke," "rolling our eyes," "taking it with a rock of salt." They had no direct knowledge of what went on inside UUNet, but from watching peering traffic, talking to salespeople about customer losses and wins, and to suppliers about deliveries of equipment, they could be pretty certain that neither the traffic nor the capacity of UUNet could be exploding at the mythical rates. They could see occasional spikes in traffic growth at some customers, or in some parts of their networks, but overall could see traffic growth settling down to a fairly regular doubling or a bit more than doubling each year.
However, they did not discuss this in public, and I discuss that below, in point #4.
-------------------------------------------------------------------
3. Growth spurt in mid-1990s: The old-timers also provided very informative feedback about the global Internet traffic growth spurt in the 1995-96 time frame. It did hit suddenly and unexpectedly. (I am still trying to get confirmation on this, but I believe one of the informants said that before this period, the engineers at that person's Tier-1 ISP would routinely double the forecasts provided by the marketing team. At the peak of the bubble, the marketers would demand that the engineers plan for double the capacity that the engineers thought was going to be necessary.) Moreover, the dramatic slowdown in traffic growth that took place in 1997 was, at least in a number of cases, due substantially to a capacity crunch. Router and photonic equipment manufacturers did not have the technology needed for the traffic, and ILECs were slow in supplying access as well as backbone links. Hence the traffic growth spurt in 1996-96 was followed by a capacity growth spurt in 1997-98, which helped provide more credibility for the myth.
-------------------------------------------------------------------
4. Information viscosity: This incident provides far more information confirming the concept of "information viscosity" that I wrote about in my paper, that important and relevant information was available, but was not widely dispersed.
Why did the information that Internet traffic was not doubling every 100 days get out to the public? It was not a closely guarded national security secret, after all.
There seemed to be many reasons operating. In the case of Genuity (as the quote from Scott Marcus in my paper explains) it was a high-level decision, that going public would hurt the company, as it might be suspected of losing market share. But that seemed to be unusual, in that Genuity, while a major Tier-1 ISP, was small and independent, and so had people like Scott who were engineers, yet involved in policy making. At other places, other dynamics operated.
At AT&T, for example, I was called in to a meeting with the management of WorldNet, the AT&T ISP unit, at the end of 2000. They had been telling their customers that Internet traffic was growing 10x per year, and some of those customers asked them about the discrepancy between that claim and the estimates of some folks from AT&T Labs - Research (Kerry Coffman and myself) that growth was just 2x per year. Now it is an interesting perspective on "information viscosity" and AT&T (and other large bureaucratic organizations) that those folks had not heard of Kerry's and my work, even though we had publicized it at the company. But in any event, at that meeting, I did succeed in convincing them that Internet traffic was growing only 2x per year (using evidence in my papers with Kerry, as well as extensive additional data from within AT&T itself), and they agreed they would not propagate the myth among customers. But the interesting thing was that many of the attendees (who included quite a few engineering types, not just the management) seemed disappointed. That really surprised me. After all, AT&T's Internet traffic was growing just about 4x per year, which meant our market share was doubling, instead of being cut in half. But it seems that many of them had really bought into the Internet dream. And, furthermore, the story that we were losing market share was a good one to pry additional resources from the corporation.
Several of the NANOG old-timers said that they felt constrained from speaking about the falsity of the myth of astronomical growth rates by group solidarity. After all, it was a small, select group, and bad-mouthing one of their own in front of outsiders or even NANOG newbies did not seem the right thing to do. Then there was the additional factor that one person discussed very explicitly, and that I infer also applied in other cases. The myth was useful. Back in the mid-1990s, when it was not a myth, it did spur equipment suppliers and ILECs to greater effort. And afterwards, it was handy in internal fights over power and resources. If the Internet was exploding, one should not worry about closely examining expansion plans.
In some ways, the persistence of false perceptions about Internet traffic growth mirrors that of utilization rates of data networks. When I wrote the paper "Data networks are lightly utilized, and will stay that way," back in 1998,
http://www.dtc.umn.edu/~odlyzko/doc/network.utilization.pdf
the more clueful data network engineers all knew that utilization rates were generally low, and usually had a good understanding as to why. But top management, as well as the research community, were almost uniformly convinced that congestion was the rule. It took me a while to understand the dynamics of the situation, in which network engineers and managers found it easier to say that they were experiencing 80% utilization and 20% packet losses and needed to upgrade, without having to explain to CEOs, CIOs, and especially to clueless CFOs, that this referred just to the peak hour on one crucial corporate WAN link, and yet justified a big new investment. Few people were lying, but many had incentives to maintain delusions among the top levels of the hierarchy.
These demonstrations of information viscosity of course undermine the foundations of much of modern economics, especially of the efficient markets hypothesis. But that last myth is even more durable than "Internet traffic doubling every 100 days," especially since it does not lead directly to lots of dark fiber lying around unused, and lots of companies bankrupt. Information viscosity in facts and ideas in economics is very high, so we should not expect any changes on that scene.
-------------------------------------------------------------------
5. As a reminder, the paper that led to this discussion is "Bubbles, gullibility, and other challenges for economics, psychology, sociology, and information sciences,"
http://www.dtc.umn.edu/~odlyzko/doc/mania03.pdf
The page with source materials from the bubbles times,
http://www.dtc.umn.edu/~odlyzko/isources/
now has, in addition to a transcript of the O'Dell lecture at Stanford in May 2000, a copy of the Sidgmore paper from the Vortex98 Conference of May 1998.
participants (10)
-
Benson Schliesser -
Chris Boyd -
deleskie＠gmail.com -
Jeffrey S. Young -
jim deleskie
-
John Lee -
Marshall Eubanks -
Randy Whitney -
Robert E. Seastrom -
William McCall