Here is my running summary of this debate, with only mild editorizing:

From: alan@mindvision.com (Alan Hannan)

I believe DIGEX should be congratulated on their forward exploration of technology. Their creative solutions to problems like this are the spirit that will sustain the Internet. DWDM and huge fiber builds will get us part of the way to handling internet growth, but we need to find ways to utilize upper-layer tweaks to do more with less. ---

From: andrew khoo <andrewk@aussie.net>

more aggressive caching techniques would necessitate that the tags be ignored anyway, and the "dynamic" content would still be cached. expires etc are only useful if the caching box decides to honour them. some of the content providers in the US would turn over in their graves if they knew what people who pay heaps more $$$ for traffic are doing to their web sites :). we have done some funky stuff with content (read: caching of dynamic pages, even .asps, "tokenized" HTML, dropping realaudio UDP etc etc etc). --- (I'm stilling waiting for the first successful copyright lawsuit in Australia because the cache isn't honoring expire headers or modifying the data)

From: "Sean M. Doran" <smd@clock.org> Let them turn. The easy fix for them is to duplicate their content elsewhere in the world, so they can control it themselves, and reduce the need for caching by operators like you.

Sean.

From: Paul Vixie <vixie@vix.com> And I wanted to give one thing Sean said more air time so folks won't just gloss over it: ...

That is, a cache which imposes decent long-haul TCP behaviour reduces the number of packets which are delivered all the way from the web server to the terminal server but tail-dropped there rather than being delivered to the end user.

This is rather important, both because the stacks used in last-mile devices (such as the Microsoft Program Loader) are not very modern, and because HTTP persistent connections end up not being very helpful until they are aggregated across multiple last-mile devices. ---

From: Patrick McManus <mcmanus@AppliedTheory.com> In a previous episode Michael Dillon said...

:: [...] Caching is a temporary hack that provides some limited :: benefit at the moment but the need for it is fading except in areas that :: are a couple of years behind in bandwidth deployment. I can't believe you said that. Heirarchical caching reduces latency and increases availability _in addition to_ conserving bandwidth. Those (particularly the second) will remain critically important features no matter if you're traversing a 512k line or OC48... ---

From: Pushpendra Mohta <pushp@CERF.NET>

No amount of capacity upgrades will obviate the need for an intelligent architecture for content distribution on the net. Utilization will explode to consume all bandwidth available and in the absence of an elegant architecture will do so rather wastefully. Most sites on the web are currently misdeployed/incorrectly-located on the network. Techniques that prevents unwarranted replication of data and bring the content to the user rather than the user to the content will bring about more efficient use of bandwidth while increasing performance for all users. These need to be complemented with techniques to address the concerns of other constituents, like marketeers. --pushpendra ---

From: Jon Lewis <jlewis@inorganic5.fdt.net>

Actually, I thought I was the loud one complaining. I'm not a Digex competitor either...I'm the Digex customer that started this thread. --- (Your comments did not compare in volume or tone to some made by Digex's competitors, it would seem a good number of people felt you have a reasonable complaint about notfication but were to polite to rummage through Digex's dirty laundry in this forum.)

From: John Fraizer <John.Fraizer@EnterZone.Net>

Horse-hockey. Purchasing the proper amount of bandwidth from your upstream and your upstream implementing the facilities to deliver that bandwidth is what is necessary to scale the internet into the future. Over-allocation is bad enough on the dialup-ISP side of the equation without backbone providers doing the same thing. If your backbone provider doesn't have the capacity to deliver what they have sole to you, it's time to contact two entities: Your lawyer and another backbone provider. --- (Question: what off the shelf technology goes beyond DWDM OC192? Second Question: When will that bandwidth be needed?)

From: Pete Farmer <pfarmer@strategies-u.com>

As a customer, I don't give a hoot for belief systems and opinions on where things 'ought' to be. Don't talk to me about 'unethical,' talk to me about better ways to solve my business problem. ---

From: "Sean M. Doran" <smd@clock.org>

Yes, and I am sure they fully realize that the loudest noises have been made by DIGEX's *competitors* complaining about DIGEX's customer-notification or service change, and consume a large block of salt. Sean. ---

From: "Patrick W. Gilmore" <patrick@priori.net>

And before anyone goes off once again about how great caching is, I will once again publicly state that I am not at all opposed to caching - even forced caching of customers. It's your network, do as you please. ---(comments relating the the business pratices of Digex and its relations with its customers are not Operational issues and have been deleted. They might explain why everyone is bashing Patrick)

From: Karl Denninger <karl@mcs.net> I have no problem with a proxy where the user has agreed to its use.

I have a major problem with a provider hijacking a stream of data and reprocessing it, then delivering what the client *thinks* is a direct communication session. In fact, I'm not even sure that such an act, absent consent, is legal. --- (This is my favorite, Karl makes all assumptions here about jurisdiction, contracts between two parties, etc. Karl is a competitor of Digex also. I still have this funny image of the Chicago policy showing up in Copenhagen, Denmark, to arrest a Canadian.)

From: Mark Milhollan <mlm@ftel.net>

Karl Denninger writes:

(performance for the FIRST fetch through a proxy is SLOWER - it HAS TO BE, since the proxy must first get the data before it can pass it on).

It doesn't have to -- think cut-thru switching. I've no idea if any/enough of the proxies do this. --- (they do, and Karl should have known better)

From: Hank Nussbacher <hank@ibm.net.il>

For the past 4 years, AS378 has blocked port 80 (all academic network of 8 universities and many colleges) and forced all users thru multiple proxy servers (most recently Squid). Over the years we had to build up a list of sites to bypass. But in addition, one has to provide the ability to bypass based on source IP address (some users just don't want it - even if it speeds up their page downloads and toasts their bread at the same time.) --- (In the US people don't pay extra for un filtered access because it is so cheap. Probably don't want caching because they don't want people to know what type of porn they are downloading. (And don't understand that tcpdump could still monitor them.) So in New Zeeland, Australia, or other places where they pay extra for not going through the cache, its ok to do this, but in the US where the cost is carried by the provider its WRONG. Oh ok, end users have a common law right to dictate the technology used to deliver services to them. I'll remember that when I buy my next car without an emissions control system, because I have the fundemental right to pollute.))

From: Karl Denninger <karl@mcs.net> See, that's the problem.

Proxies are fine WHERE CUSTOMERS HAVE AGREED TO THEIR USE. STEALING someone's packet flow to force it through a proxy is NOT fine. --- (This same arguement could be apply to Ip deliver as specified in RFC 1149 "Standard for the transmission of IP datagrams on avian carriers." Data is data if the transparency is working correctly, the data delivered is identical and delivered faster. The streams aren't being taken anywhere. This is like arguing that echo cancelers can't be used on satilite communications because it breaks some types of transmissions, but helps the majory of customers, and it does in fact modify the STREAM.) So it seems: Against transparent proxying: Karl D. Against Digex transparent proxing: Patrick G. Upset about the problems with transparency not working: Jon L. Against transparency (for operational reasons even!) Mike O'Dell Thinks that bandwidth grows on trees: John F Pro hiarcheial caching, even transparency: Pushpendra M Hank N Patrick M Paul V Sean Doran andrew k Alan H Paul G Rich S Lamenting the lack of a Digex NOC to report transparency problems to: Randy B --- All well. I'd guess that content providers had better start working with cache vendors to make things work right, as I would guess a good bit of the content is going to go through caches, if you like it or not and everyone is going to be better off with it. There should be a real good flame war when people role out CoS/Voice over IP/ RSVP to exist next to best effort delivery. (My contract with provider X clearly implies that my traffic should be treated with equal priority to all other traffic on the network, therefore you are in breach of contract for offering higher priorities than best effort, oh and BTW you are doing unlawful wiretapping because you are classifying traffic by type which means looking past the IP address in the header.) --- Jeremy Porter, Freeside Communications, Inc. jerry@fc.net PO BOX 80315 Austin, Tx 78708 | 512-458-9810 http://www.fc.net