Andrew, Earlier this week I had a meeting with the ex-Director of the Network Operations Center for MFS-Datanet/MCI whose tenure was through 1999. From 1994 to 1998 they were re-architeching the Frame Relay and ATM networks to handle the growth in traffic including these new facilities called peering points of MAE-East and MAE-West. From roughly 1990 to then end of 1996 they saw traffic on their switches grow at 50-70% growth every 6 months. By the last half of 1996 there was a head of line blocking problem on the DEC FDDI switches that was "regularly" bringing down the Internet. The architecture had lower traffic circuits were going through concentrators while higher traffic circuits were directly attached to ports on the switchs. MFS-Datanet was not going to take the hit for the interruptions to the Internet and was going to inform the trade press there was a problem with DEC FDDI switches so Digital "gave" six switches for the re-architecture of the MAEs to solve the problem. Once this problem was solved the first quarter of 1997 saw a 70% jump in traffic that quarter alone. This "historical event" would in my memory be the genesis of the 100% traffic growth in 100 days legend. (So it was only 70% in 90 days which for the marketing folks does not cut it so 100% in 100 days sounds much better?? :) ) MCI bought MFS-Datanet because MCI had the customers and MFS-Datanet had all of the fiber running to key locations at the time and could drastically cut MCI's costs. UUNET "merged" with MCI and their traffic was put on this same network. MCI went belly up and Verizon bought the network. Personal Note: from 1983 to 90 I worked for Hayes the modem folks and became the Godfather to Ascend communications with Jeanette, Rob, Jay and Steve whose team produced the TNT line of modem/ISDN to Ethernet central site concentrators (in the early ninties) that drove a large portion of the user traffic to the Internet at the time, generating the "bubble". John (ISDN) Lee ________________________________________ From: Andrew Odlyzko [odlyzko@umn.edu] Sent: Wednesday, August 11, 2010 12:55 PM To: nanog@nanog.org Subject: off-topic: summary on Internet traffic growth myths Since several members of this list requested it, here is a summary of the responses to my request for information about Internet growth during the telecom bubble, in particular the perceptions of the O'Dell/Sidgmore/WorldCom/UUNet "Internet doubling every 100 days" myth. First of all, many thanks to all those who responded, on and off-list. This involved extensive correspondence and some long phone conversations, and helped fill out the picture of those very confusing times (and also made it even clearer than before that there were many different perspectives on what was happening). The entire message is rather long, but it is written in sections, to make it easy to get the gist quickly and neglect the rest. Andrew ------------------------------------------------------------------- 1. Short summary: People who got into the game late, or had been working at small ISPs or other enterprises, were generally willing to give serious credence to the "Internet doubling every 100 days" tale. The old-timers, especially those who worked for large ISPs or other large corporate establishment or research networks, were convinced by the late 1990s that this tale was false, but did not talk about it publicly, even inside the NANOG community. ------------------------------------------------------------------- 2. Longer version: The range of views was very wide, and hard to give justice to in full. But there seemed to be two distinct groups, and the consensus views (which obviously exclude quite a few people) appear to have been: 2A: Those who entered the field in the late 1990s, especially if they worked for small ISPs or other small enterprises, tended to regard the claim seriously. (But it should be remarked that hardly anybody devoted too much effort or thought to the claim, they were too busy putting out fires in their own backyards to worry about global issues.) They remembered periods of desperate efforts to keep up with exploding demand in their businesses. We saw just a few hours ago a post about LINX growing 5.5x in one year. Somebody else wrote about growing their business's traffic 1,000x in 2 years, or about 30x per year. People involved in such incidents often tended to think that their experience during such times might not have been untypical. 2B: Those who worked at places with large traffic, and especially those who got into the field in the early 1990s, were quite sure by the late 1990s that the UUNet fable was just that. Comments regarding everything emanating from UUNet during that period included phrases like "blowing smoke," "rolling our eyes," "taking it with a rock of salt." They had no direct knowledge of what went on inside UUNet, but from watching peering traffic, talking to salespeople about customer losses and wins, and to suppliers about deliveries of equipment, they could be pretty certain that neither the traffic nor the capacity of UUNet could be exploding at the mythical rates. They could see occasional spikes in traffic growth at some customers, or in some parts of their networks, but overall could see traffic growth settling down to a fairly regular doubling or a bit more than doubling each year. However, they did not discuss this in public, and I discuss that below, in point #4. ------------------------------------------------------------------- 3. Growth spurt in mid-1990s: The old-timers also provided very informative feedback about the global Internet traffic growth spurt in the 1995-96 time frame. It did hit suddenly and unexpectedly. (I am still trying to get confirmation on this, but I believe one of the informants said that before this period, the engineers at that person's Tier-1 ISP would routinely double the forecasts provided by the marketing team. At the peak of the bubble, the marketers would demand that the engineers plan for double the capacity that the engineers thought was going to be necessary.) Moreover, the dramatic slowdown in traffic growth that took place in 1997 was, at least in a number of cases, due substantially to a capacity crunch. Router and photonic equipment manufacturers did not have the technology needed for the traffic, and ILECs were slow in supplying access as well as backbone links. Hence the traffic growth spurt in 1996-96 was followed by a capacity growth spurt in 1997-98, which helped provide more credibility for the myth. ------------------------------------------------------------------- 4. Information viscosity: This incident provides far more information confirming the concept of "information viscosity" that I wrote about in my paper, that important and relevant information was available, but was not widely dispersed. Why did the information that Internet traffic was not doubling every 100 days get out to the public? It was not a closely guarded national security secret, after all. There seemed to be many reasons operating. In the case of Genuity (as the quote from Scott Marcus in my paper explains) it was a high-level decision, that going public would hurt the company, as it might be suspected of losing market share. But that seemed to be unusual, in that Genuity, while a major Tier-1 ISP, was small and independent, and so had people like Scott who were engineers, yet involved in policy making. At other places, other dynamics operated. At AT&T, for example, I was called in to a meeting with the management of WorldNet, the AT&T ISP unit, at the end of 2000. They had been telling their customers that Internet traffic was growing 10x per year, and some of those customers asked them about the discrepancy between that claim and the estimates of some folks from AT&T Labs - Research (Kerry Coffman and myself) that growth was just 2x per year. Now it is an interesting perspective on "information viscosity" and AT&T (and other large bureaucratic organizations) that those folks had not heard of Kerry's and my work, even though we had publicized it at the company. But in any event, at that meeting, I did succeed in convincing them that Internet traffic was growing only 2x per year (using evidence in my papers with Kerry, as well as extensive additional data from within AT&T itself), and they agreed they would not propagate the myth among customers. But the interesting thing was that many of the attendees (who included quite a few engineering types, not just the management) seemed disappointed. That really surprised me. After all, AT&T's Internet traffic was growing just about 4x per year, which meant our market share was doubling, instead of being cut in half. But it seems that many of them had really bought into the Internet dream. And, furthermore, the story that we were losing market share was a good one to pry additional resources from the corporation. Several of the NANOG old-timers said that they felt constrained from speaking about the falsity of the myth of astronomical growth rates by group solidarity. After all, it was a small, select group, and bad-mouthing one of their own in front of outsiders or even NANOG newbies did not seem the right thing to do. Then there was the additional factor that one person discussed very explicitly, and that I infer also applied in other cases. The myth was useful. Back in the mid-1990s, when it was not a myth, it did spur equipment suppliers and ILECs to greater effort. And afterwards, it was handy in internal fights over power and resources. If the Internet was exploding, one should not worry about closely examining expansion plans. In some ways, the persistence of false perceptions about Internet traffic growth mirrors that of utilization rates of data networks. When I wrote the paper "Data networks are lightly utilized, and will stay that way," back in 1998, http://www.dtc.umn.edu/~odlyzko/doc/network.utilization.pdf the more clueful data network engineers all knew that utilization rates were generally low, and usually had a good understanding as to why. But top management, as well as the research community, were almost uniformly convinced that congestion was the rule. It took me a while to understand the dynamics of the situation, in which network engineers and managers found it easier to say that they were experiencing 80% utilization and 20% packet losses and needed to upgrade, without having to explain to CEOs, CIOs, and especially to clueless CFOs, that this referred just to the peak hour on one crucial corporate WAN link, and yet justified a big new investment. Few people were lying, but many had incentives to maintain delusions among the top levels of the hierarchy. These demonstrations of information viscosity of course undermine the foundations of much of modern economics, especially of the efficient markets hypothesis. But that last myth is even more durable than "Internet traffic doubling every 100 days," especially since it does not lead directly to lots of dark fiber lying around unused, and lots of companies bankrupt. Information viscosity in facts and ideas in economics is very high, so we should not expect any changes on that scene. ------------------------------------------------------------------- 5. As a reminder, the paper that led to this discussion is "Bubbles, gullibility, and other challenges for economics, psychology, sociology, and information sciences," http://www.dtc.umn.edu/~odlyzko/doc/mania03.pdf The page with source materials from the bubbles times, http://www.dtc.umn.edu/~odlyzko/isources/ now has, in addition to a transcript of the O'Dell lecture at Stanford in May 2000, a copy of the Sidgmore paper from the Vortex98 Conference of May 1998.