" It would be nice if some folks on the list could provide some solid information, even if only for one large carrier." One of the busts that I never found much on was the Enron network. Where did it end up? I'd be interested in local route and POP details, offlist would be fine as well. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Andrew Odlyzko via NANOG" <nanog@nanog.org> To: "Dave Taht" <dave.taht@gmail.com> Cc: "NANOG" <nanog@nanog.org> Sent: Monday, November 13, 2023 1:25:39 PM Subject: Re: The rise and fall of the 90's telecom bubble Dave Taht's question about all the redundant fiber that was put down in the telecom bubble is a very interesting one. It would be nice if some folks on the list could provide some solid information, even if only for one large carrier. My impression, from communications with various folks, is that much of that fiber from around 2000 was never lit. The reason is that better fiber came on the market. However, what was used (at least in some cases, again, this is something I would love to get real data on) was that some of the empty conduits that were put down then were used to shoot the new generations of fiber through. (It was quite common for carriers to put down 4 conduits, and only pull fiber through one of them, leaving the other 3 for later use.) Concerning the Doug O'Laughlin post that Dave cites, it is very good. For more on the myth of "Internet doubling every 100 days," my paper "Bubbles, gullibility, and other challenges for economics, psychology, sociology, and information sciences" published in First Monday in Sept. 2010, https://firstmonday.org/ojs/index.php/fm/article/view/3142/2603 Participants of this list were very helpful in providing information for that paper. (Some of the correspondence is in the list archives, most was off-line.) But O'Laughlin is too hard on Global Crossing, for example, when he says it "was essentially a fundraising scheme looking for a problem." Global Crossing had a real business plan, it was the first transatlantic cable that was not built by consortia of incumbent telcos, and it planned to take advantage of the rising demand for transmission by offering capacity to new players, who would otherwise be gouged by incumbents. (It did get into accounting shenanigans later on, as competition arose, but that was later.) What is most interesting is that their business plan was based on an assumption of demand about doubling each year (which is what was taking place), not doubling every 100 days. (This I learned when I was consulted on some of the litigation after the telecom crash, but by now the information is publicly available.) What killed them is that their assumption that it would be difficult for others to get the (special undersea) fiber, the cable-laying ships, the permits, ..., turned out to be wrong, and so a slew of competitors, inspired by the myth of astronomical growth rates, came on the scene. (Global Crossing's expansion into terrestrial fiber networks was also a major contributor.) One of the astounding observations is that while Global Crossing was assuming 100% annual growth rate in traffic, the industry as a whole (as well as the press, the FCC, and so on) were talking of 1,000% growth rates. And the only observer that I was able to find who noted this in print was George Gilder, who drew the wrong conclusion from this! (Details are in the paper cited above.) Andrew P.S. Some interesting materials from telecom bubble era are available at https://www-users.cse.umn.edu/~odlyzko/isources/index.html On Sun, 12 Nov 2023, Dave Taht wrote:

Aside from me pinning the start of the bubble closer to 1992 when commercial activity was allowed, and M&A for ISPs at insane valuations per subscriber by 1995 (I had co-founded an ISP in 93, but try as I might I cannot remember if it peaked at 50 or 60x1 by 1996 (?) and crashed by 97 (?)), this was a whacking good read, seems accurate, and moves to comparing it across that to the present day AI bubble.

https://www.fabricatedknowledge.com/p/lessons-from-history-the-rise-and

In the end we sold (my ISP, founded 93) icanect for 3 cents on the dollar in 99, and I lost my shirt (not for the first time) on it, only to move into embedded Linux (Montavista) after the enormous pop redhat's IPO had had in 99. The company I was part of slightly prior (Mediaplex) went public December 12, 1999 and cracked 100/share, only to crash by march, 2000 to half the IPO price (around $7 as I recall), wiping out everyone that had not vested yet. I lost my shirt again on that and Montavista too and decided I would avoid VCs henceforth.

I am always interested in anecdotal reports of personal events in this increasingly murky past, and in trying to fact check the above link.

So much fiber got laid by 2000 that it is often claimed that it was at least a decade before it was used up, (the article says only 2.7% was in use by 2002) and I have always wondered how much dark, broken, inaccessible fiber remains that nobody knows where it even is anymore due to many lost databases. I hear horror stories...

The article also focuses solely on the us sector, and I am wondering what it looked like worldwide.

I believed in the 90s we were seeing major productivity gains. The present expansion of the internet in my mind should not be much associated with "productivity gains", as, imho, reducing the general population to two thumbs and a 4 inch screen strikes me as an enormous step backwards.

(I have a bad habit of cross posting my mails to where older denizens of the internet reside, sorry! If you end up posting to one of my lists I will add a sender allows filter for you) -- :( My old R&D campus is up for sale: https://tinyurl.com/yurtlab Dave Täht CSO, LibreQos

I saw a map once showing that they seemed to come into town from the west via I-88 (while other westerly routes traveled via the Rock Island). I only know of CenturyLink (then Digital Teleport) going west from 88, so maybe that's where they got their route from? I wonder where their huts\POPs were.... unless they just were in DTI huts too. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Bryan Holloway" <bryan@shout.net> To: "NANOG" <nanog@nanog.org> Sent: Tuesday, November 14, 2023 9:20:24 AM Subject: Re: The rise and fall of the 90's telecom bubble On 11/14/23 15:04, Mike Hammett wrote:

"It would be nice if some folks on the list could provide some solid information, even if only for one large carrier."

One of the busts that I never found much on was the Enron network. Where did it end up? I'd be interested in local route and POP details, offlist would be fine as well.

I ran a facility at 717 S Wells which they used to occupy on the 10th floor back in the day, but they were leveraging other folks' fiber. Level3 and Looking Glass if my addled brain remembers ...

------------------------------------------------------------------------ *From: *"Andrew Odlyzko via NANOG" <nanog@nanog.org> *To: *"Dave Taht" <dave.taht@gmail.com> *Cc: *"NANOG" <nanog@nanog.org> *Sent: *Monday, November 13, 2023 1:25:39 PM *Subject: *Re: The rise and fall of the 90's telecom bubble

Dave Taht's question about all the redundant fiber that was put down in the telecom bubble is a very interesting one. It would be nice if some folks on the list could provide some solid information, even if only for one large carrier.

My impression, from communications with various folks, is that much of that fiber from around 2000 was never lit. The reason is that better fiber came on the market. However, what was used (at least in some cases, again, this is something I would love to get real data on) was that some of the empty conduits that were put down then were used to shoot the new generations of fiber through. (It was quite common for carriers to put down 4 conduits, and only pull fiber through one of them, leaving the other 3 for later use.)

Concerning the Doug O'Laughlin post that Dave cites, it is very good. For more on the myth of "Internet doubling every 100 days," my paper "Bubbles, gullibility, and other challenges for economics, psychology, sociology, and information sciences" published in First Monday in Sept. 2010,

https://firstmonday.org/ojs/index.php/fm/article/view/3142/2603

Participants of this list were very helpful in providing information for that paper. (Some of the correspondence is in the list archives, most was off-line.)

But O'Laughlin is too hard on Global Crossing, for example, when he says it "was essentially a fundraising scheme looking for a problem." Global Crossing had a real business plan, it was the first transatlantic cable that was not built by consortia of incumbent telcos, and it planned to take advantage of the rising demand for transmission by offering capacity to new players, who would otherwise be gouged by incumbents. (It did get into accounting shenanigans later on, as competition arose, but that was later.) What is most interesting is that their business plan was based on an assumption of demand about doubling each year (which is what was taking place), not doubling every 100 days. (This I learned when I was consulted on some of the litigation after the telecom crash, but by now the information is publicly available.) What killed them is that their assumption that it would be difficult for others to get the (special undersea) fiber, the cable-laying ships, the permits, ..., turned out to be wrong, and so a slew of competitors, inspired by the myth of astronomical growth rates, came on the scene. (Global Crossing's expansion into terrestrial fiber networks was also a major contributor.)

One of the astounding observations is that while Global Crossing was assuming 100% annual growth rate in traffic, the industry as a whole (as well as the press, the FCC, and so on) were talking of 1,000% growth rates. And the only observer that I was able to find who noted this in print was George Gilder, who drew the wrong conclusion from this! (Details are in the paper cited above.)

Andrew

P.S. Some interesting materials from telecom bubble era are available at

https://www-users.cse.umn.edu/~odlyzko/isources/index.html

On Sun, 12 Nov 2023, Dave Taht wrote:

...

Aside from me pinning the start of the bubble closer to 1992 when commercial activity was allowed, and M&A for ISPs at insane valuations per subscriber by 1995 (I had co-founded an ISP in 93, but try as I might I cannot remember if it peaked at 50 or 60x1 by 1996 (?) and crashed by 97 (?)), this was a whacking good read, seems accurate, and moves to comparing it across that to the present day AI bubble.

https://www.fabricatedknowledge.com/p/lessons-from-history-the-rise-and

In the end we sold (my ISP, founded 93) icanect for 3 cents on the dollar in 99, and I lost my shirt (not for the first time) on it, only to move into embedded Linux (Montavista) after the enormous pop redhat's IPO had had in 99. The company I was part of slightly prior (Mediaplex) went public December 12, 1999 and cracked 100/share, only to crash by march, 2000 to half the IPO price (around $7 as I recall), wiping out everyone that had not vested yet. I lost my shirt again on that and Montavista too and decided I would avoid VCs henceforth.

I am always interested in anecdotal reports of personal events in this increasingly murky past, and in trying to fact check the above link.

So much fiber got laid by 2000 that it is often claimed that it was at least a decade before it was used up, (the article says only 2.7% was in use by 2002) and I have always wondered how much dark, broken, inaccessible fiber remains that nobody knows where it even is anymore due to many lost databases. I hear horror stories...

The article also focuses solely on the us sector, and I am wondering what it looked like worldwide.

I believed in the 90s we were seeing major productivity gains. The present expansion of the internet in my mind should not be much associated with "productivity gains", as, imho, reducing the general population to two thumbs and a 4 inch screen strikes me as an enormous step backwards.

(I have a bad habit of cross posting my mails to where older denizens of the internet reside, sorry! If you end up posting to one of my lists I will add a sender allows filter for you) -- :( My old R&D campus is up for sale: https://tinyurl.com/yurtlab Dave Täht CSO, LibreQos

Well, and "better" for what purpose? Some glass may not work out well for longhaul 25+ terabit services, but would be fine for metro loops of regular services. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Tim Požar via NANOG" <nanog@nanog.org> To: odlyzko@umn.edu, "Dave Taht" <dave.taht@gmail.com> Cc: "NANOG" <nanog@nanog.org> Sent: Tuesday, November 14, 2023 10:55:19 AM Subject: Re: The rise and fall of the 90's telecom bubble One consideration about older fiber, is it may not be maintained. I know of one large city owned installation that has older fiber that is being abandoned in order to pull higher count and better glass in. The old fiber will end up being cut due to construction, etc. so it is worthless. Tim On 11/13/23 11:25 AM, Andrew Odlyzko via NANOG wrote:

Dave Taht's question about all the redundant fiber that was put down in the telecom bubble is a very interesting one. It would be nice if some folks on the list could provide some solid information, even if only for one large carrier.

My impression, from communications with various folks, is that much of that fiber from around 2000 was never lit. The reason is that better fiber came on the market. However, what was used (at least in some cases, again, this is something I would love to get real data on) was that some of the empty conduits that were put down then were used to shoot the new generations of fiber through. (It was quite common for carriers to put down 4 conduits, and only pull fiber through one of them, leaving the other 3 for later use.)

Concerning the Doug O'Laughlin post that Dave cites, it is very good. For more on the myth of "Internet doubling every 100 days," my paper "Bubbles, gullibility, and other challenges for economics, psychology, sociology, and information sciences" published in First Monday in Sept. 2010,

https://firstmonday.org/ojs/index.php/fm/article/view/3142/2603

Participants of this list were very helpful in providing information for that paper. (Some of the correspondence is in the list archives, most was off-line.)

But O'Laughlin is too hard on Global Crossing, for example, when he says it "was essentially a fundraising scheme looking for a problem." Global Crossing had a real business plan, it was the first transatlantic cable that was not built by consortia of incumbent telcos, and it planned to take advantage of the rising demand for transmission by offering capacity to new players, who would otherwise be gouged by incumbents. (It did get into accounting shenanigans later on, as competition arose, but that was later.) What is most interesting is that their business plan was based on an assumption of demand about doubling each year (which is what was taking place), not doubling every 100 days. (This I learned when I was consulted on some of the litigation after the telecom crash, but by now the information is publicly available.) What killed them is that their assumption that it would be difficult for others to get the (special undersea) fiber, the cable-laying ships, the permits, ..., turned out to be wrong, and so a slew of competitors, inspired by the myth of astronomical growth rates, came on the scene. (Global Crossing's expansion into terrestrial fiber networks was also a major contributor.)

One of the astounding observations is that while Global Crossing was assuming 100% annual growth rate in traffic, the industry as a whole (as well as the press, the FCC, and so on) were talking of 1,000% growth rates. And the only observer that I was able to find who noted this in print was George Gilder, who drew the wrong conclusion from this! (Details are in the paper cited above.)

Andrew

P.S. Some interesting materials from telecom bubble era are available at

https://www-users.cse.umn.edu/~odlyzko/isources/index.html

On Sun, 12 Nov 2023, Dave Taht wrote:

...

Aside from me pinning the start of the bubble closer to 1992 when commercial activity was allowed, and M&A for ISPs at insane valuations per subscriber by 1995 (I had co-founded an ISP in 93, but try as I might I cannot remember if it peaked at 50 or 60x1 by 1996 (?) and crashed by 97 (?)), this was a whacking good read, seems accurate, and moves to comparing it across that to the present day AI bubble.

https://www.fabricatedknowledge.com/p/lessons-from-history-the-rise-and

In the end we sold (my ISP, founded 93) icanect for 3 cents on the dollar in 99, and I lost my shirt (not for the first time) on it, only to move into embedded Linux (Montavista) after the enormous pop redhat's IPO had had in 99. The company I was part of slightly prior (Mediaplex) went public December 12, 1999 and cracked 100/share, only to crash by march, 2000 to half the IPO price (around $7 as I recall), wiping out everyone that had not vested yet. I lost my shirt again on that and Montavista too and decided I would avoid VCs henceforth.

I am always interested in anecdotal reports of personal events in this increasingly murky past, and in trying to fact check the above link.

So much fiber got laid by 2000 that it is often claimed that it was at least a decade before it was used up, (the article says only 2.7% was in use by 2002) and I have always wondered how much dark, broken, inaccessible fiber remains that nobody knows where it even is anymore due to many lost databases. I hear horror stories...

The article also focuses solely on the us sector, and I am wondering what it looked like worldwide.

I believed in the 90s we were seeing major productivity gains. The present expansion of the internet in my mind should not be much associated with "productivity gains", as, imho, reducing the general population to two thumbs and a 4 inch screen strikes me as an enormous step backwards.

(I have a bad habit of cross posting my mails to where older denizens of the internet reside, sorry! If you end up posting to one of my lists I will add a sender allows filter for you) -- :( My old R&D campus is up for sale: https://tinyurl.com/yurtlab Dave Täht CSO, LibreQos

Most of the early public fiber was swept up in the later 2009 BTOP buildout to schools, cop shops, and fire stations https://www.ntia.gov/other-publication/btop-fact-sheet and is now run by each county or sometimes states, who usually manage and allocate it to customer projects with tools like OSP Insight. The vast majority of WAN fiber put down was good old G.652 and should still be usable at 1310 or 1550 nm for dwdm. Dan -----Original Message----- From: NANOG <nanog-bounces+dan.magorian=jhuapl.edu@nanog.org> On Behalf Of Andrew Odlyzko via NANOG Sent: Monday, November 13, 2023 2:26 PM To: Dave Taht <dave.taht@gmail.com> Cc: NANOG <nanog@nanog.org> Subject: [EXT] Re: The rise and fall of the 90's telecom bubble APL external email warning: Verify sender nanog-bounces+dan.magorian=jhuapl.edu@nanog.org before clicking links or attachments  Dave Taht's question about all the redundant fiber that was put down in the telecom bubble is a very interesting one. It would be nice if some folks on the list could provide some solid information, even if only for one large carrier. My impression, from communications with various folks, is that much of that fiber from around 2000 was never lit. The reason is that better fiber came on the market. However, what was used (at least in some cases, again, this is something I would love to get real data on) was that some of the empty conduits that were put down then were used to shoot the new generations of fiber through. (It was quite common for carriers to put down 4 conduits, and only pull fiber through one of them, leaving the other 3 for later use.) Concerning the Doug O'Laughlin post that Dave cites, it is very good. For more on the myth of "Internet doubling every 100 days," my paper "Bubbles, gullibility, and other challenges for economics, psychology, sociology, and information sciences" published in First Monday in Sept. 2010, https://firstmonday.org/ojs/index.php/fm/article/view/3142/2603 Participants of this list were very helpful in providing information for that paper. (Some of the correspondence is in the list archives, most was off-line.) But O'Laughlin is too hard on Global Crossing, for example, when he says it "was essentially a fundraising scheme looking for a problem." Global Crossing had a real business plan, it was the first transatlantic cable that was not built by consortia of incumbent telcos, and it planned to take advantage of the rising demand for transmission by offering capacity to new players, who would otherwise be gouged by incumbents. (It did get into accounting shenanigans later on, as competition arose, but that was later.) What is most interesting is that their business plan was based on an assumption of demand about doubling each year (which is what was taking place), not doubling every 100 days. (This I learned when I was consulted on some of the litigation after the telecom crash, but by now the information is publicly available.) What killed them is that their assumption that it would be difficult for others to get the (special undersea) fiber, the cable-laying ships, the permits, ..., turned out to be wrong, and so a slew of competitors, inspired by the myth of astronomical growth rates, came on the scene. (Global Crossing's expansion into terrestrial fiber networks was also a major contributor.) One of the astounding observations is that while Global Crossing was assuming 100% annual growth rate in traffic, the industry as a whole (as well as the press, the FCC, and so on) were talking of 1,000% growth rates. And the only observer that I was able to find who noted this in print was George Gilder, who drew the wrong conclusion from this! (Details are in the paper cited above.) Andrew P.S. Some interesting materials from telecom bubble era are available at https://www-users.cse.umn.edu/~odlyzko/isources/index.html On Sun, 12 Nov 2023, Dave Taht wrote:

Aside from me pinning the start of the bubble closer to 1992 when commercial activity was allowed, and M&A for ISPs at insane valuations per subscriber by 1995 (I had co-founded an ISP in 93, but try as I might I cannot remember if it peaked at 50 or 60x1 by 1996 (?) and crashed by 97 (?)), this was a whacking good read, seems accurate, and moves to comparing it across that to the present day AI bubble.

https://www.fabricatedknowledge.com/p/lessons-from-history-the-rise-an d

In the end we sold (my ISP, founded 93) icanect for 3 cents on the dollar in 99, and I lost my shirt (not for the first time) on it, only to move into embedded Linux (Montavista) after the enormous pop redhat's IPO had had in 99. The company I was part of slightly prior (Mediaplex) went public December 12, 1999 and cracked 100/share, only to crash by march, 2000 to half the IPO price (around $7 as I recall), wiping out everyone that had not vested yet. I lost my shirt again on that and Montavista too and decided I would avoid VCs henceforth.

I am always interested in anecdotal reports of personal events in this increasingly murky past, and in trying to fact check the above link.

So much fiber got laid by 2000 that it is often claimed that it was at least a decade before it was used up, (the article says only 2.7% was in use by 2002) and I have always wondered how much dark, broken, inaccessible fiber remains that nobody knows where it even is anymore due to many lost databases. I hear horror stories...

The article also focuses solely on the us sector, and I am wondering what it looked like worldwide.

I believed in the 90s we were seeing major productivity gains. The present expansion of the internet in my mind should not be much associated with "productivity gains", as, imho, reducing the general population to two thumbs and a 4 inch screen strikes me as an enormous step backwards.

(I have a bad habit of cross posting my mails to where older denizens of the internet reside, sorry! If you end up posting to one of my lists I will add a sender allows filter for you) -- :( My old R&D campus is up for sale: https://tinyurl.com/yurtlab Dave Täht CSO, LibreQos