On Wed, 20 May 1998, Karl Denninger wrote:
Part of the problem with "public exchanges" is that they get congested. But the REASONS for that congestion are not, I believe, that well understood.
Some of the reasons are well understood. One is that when providers do not upgrade the bandwidth of their pipes into the NAP then packets coming towards them through the NAP get dropped. Thus they create congestion on all these flows. Another reason is that NAP architectures do not scale well past a certain point. A NAP attempts to flatten the Interconnect hierarchy into a fully meshed fabric but when the NAP traffic grows beyond the ability of a specific full-mesh technology or device, then congestion occurs. For instance, 10Mbps and 100Mbps Ethernet and 100 Mbps FDDI are all examples of full-mesh technologies. But when the traffic exceeds their capacity, congestion occurs. The Digital Gigaswitch is a device that can handle a much higher traffic flow but when all the interface slots are full on a single box congestion occurs. We can interconnect multiple devices and/or multiple technologies but now there is no longer a full-mesh and much manual labor is involved in adjusting things like trunk capacity. At this point we appear to be attempting to collapse the entire Internet into an exchange point fabric which is doomed to failure. In some ways, private interconnects appear to be a superior technique as long as both providers keep them upgraded to handle the traffic flows. The ideal exchange point for private interconnects does not have a shared fabric, merely shared power and HVAC in a building with lots of fiber ingress and no restrictions on cross connecting. But even private interconnects can run into a scaling issue because if we attempt to interconnect every pair of providers through a private interconnect, we are attempting to create a full mesh which is not technically feasible. Thus we are led to a solution in which the largest number of the smallest providers use fully-meshed fabrics to interconnect and the larger providers manually build a full-mesh between themselves. This reduces the problem to one of interconnecting the many fully-meshed exchange point fabrics with the larger full-mesh fabric created by the large providers and their private interconnects.
Can we just build faster exchanges? Sure. Will it solve the problem? Not if carriers don't provision fast enough circuits into them!
Karl is right. Faster exchanges may move various limits and bottlenecks around but cannot solve the underlying problem. For instance, a faster exchange raises the traffic level at which a mid-size provider MUST go to private interconnects to handle traffic levels.
If you're seeing poor performance between <X> and <Y> at an exchange, is it due to the exchange fabric's poor performnace, or is one of <X> or <Y> under-provisoned into that fabric? Have either of those carriers DELIBERATELY (or through negligence) failed to provide adequate connectivity to the exchange?
This is the same problem of bandwidth upgrades that I started my message with. But it is also the heart of the issue of interconnecting the single national mesh created by private interconnects with the many smaller meshes created by the exchange points. There is a conflict of interest when a major Internet backbone owns the exchange points because they can attempt to deflect criticism of their connections to the exchange points by pointing out that they are attempting to upgrade the bandwidth capability of the exchange point. They fail to mention that growing the exchange point is like the labors of Sisyphus and cannot succeed.
Likewise, if the *EXCHANGE* operator was negligent (or just unable to keep up with demand) we could hold THEIR feet to the fire as operators.
Sadly, I know of NO exchange currently in operation that subscribes to these operating rules and policies.
Not enough people really understand how the network mesh works to hold anyone's feet to the fire. It doesn't matter whether you collapse portions of the mesh into an exchange point or into a Gigaswitch; it's still the same mesh. And you can even take the opposite tactic and expand an exchange point mesh nationally but the mesh still has to handle the same traffic levels. It is essentially a juggling game where you interconnect various technologies that might be able to handle the traffic flows in a given region of the mesh and hope that you don't drop too many balls. -- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com http://www.memra.com - *check out the new name & new website*