On Sun, 26 Jan 1997, Jonathan Heiliger wrote:

On Sat, 25 Jan 1997, Vadim Antonov wrote:

|} There are no settlements because traffic has little relevance to |} relative worth of connectivity from one provider to another. The large |} ISPs are generally interested in market share or peers, not in volume |} of mutual traffic.

Large ISPs should probably be interested in access to content, without it their users could find the Internet a very boring place.

Yes, and the current peering requirements are enough to keep most small ISPs from growing. I am spending 10s of thousands a month over what I need to spend just because people want to see full DS3 network. I can understand people would want me to be at all NAPs, but why should I need 10X the bandwidth I need for my customers? There are also problems with providers saying that I need to be at every NAP they are at, but what do I do when say a NAP in the east can't give me a connection? They first don't want to let me in at all, then they say that we can't connect until they get a new gigaswitch. I was able to get a gigaswitch for my NAP in 24 hours, why would it take 6 weeks? Nathan Stratton President, NetRail,Inc. ------------------------------------------------------------------------ Phone (888)NetRail NetRail, Inc. Fax (703)534-5033 2007 N. 15 St. Suite 5 WWW http://www.netrail.net/ Arlington, VA 22201 ------------------------------------------------------------------------ "Therefore do not worry about tomorrow, for tomorrow will worry about itself. Each day has enough trouble of its own." Matthew 6:34

On Jan 26, 1997, Nathan Stratton wrote:

Yes, and the current peering requirements are enough to keep most small ISPs from growing. I am spending 10s of thousands a month over what I need to spend just because people want to see full DS3 network. I can understand people would want me to be at all NAPs, but why should I need 10X the bandwidth I need for my customers?

Well, my guess would be that if you don't have a DS3 backbone, why would the big guys want to peer with you anyway? If you don't need that much bandwidth (or don't have it) odds are you don't have enough customers for the big guys to want to peer with you. The DS3 backbone requirement is just a nicer way of saying that (and it is a quantifiable level). Alec -- +------------------------------------+--------------------------------------+ |Alec Peterson - ahp@hilander.com | Erols Internet Services, INC. | |Network Engineer | Springfield, VA. | +------------------------------------+--------------------------------------+

I agree with with the information provider model. Ultimately, entities with attractive content will be selling access to wide area providers, who will sell it to local area providers, who will sell it to customers. This is the old "gatekeeper.dec.com" model extended to fee-based content. I heard that Microsoft was letting providers terminate T3's with them since good access to Microsoft's content is a selling point for an access provider's customer base. Why should such a content provider have to buy peering, or pay wide area telecom costs? On the other hand, right now Microsoft is still effectively buying transit, and at some point they will just charge for access to their content and let other people charge each other for indirect access to that content.

And Microsoft is just the first/largest.

From this point of view, the two main benefactors are the Content Provider and the Content Consumer. How should they split the tab so

Why indeed. Should any content provider pay for distribution costs of its content via Fedex/UPS/USPS or the phone company ? Costs of content distribution are hardly a new problem. Just the distribution transport (Internet) is new. Business people use a simple principle to decide who pays. Its called "Follow The Money" (FTM). Any one who gains from a transaction involving transfer of content has an incentive to share not only in the costs of the content but also in the costs of its distribution. Content is valuable, but has little value without distribution. (Oil has high intrinsic value, but even that value is variable subject to presence or absence of pipelines and pumping stations. Even a limitless supply of oil would be worthless without a distribution system) that they may pay the costs of distribution to a Content Distributor? Does Microsoft (Content Provider) gain by rapid and reliable distribution of its content to its end users ( Content Consumer) ? If so, then the Content Distributor (Internet Access Provider (IAP)) fees should be build into their business model. Does the end user gain by electronic access to Microsoft's content ? The user should be willing to pay for not only the content but the "shipping and handling" cost. This may be both a fee to access Microsoft's site and a subscription fee to an IAP. ( No matter how you split the distribution cost, the user ultimately will and does end up with the tab ) This is just a simplistic illustration of the cost of distribution. No matter which party collects the cost, this should end up as fees to the Content Distributor or Distributors. [How the Content Distributors split these fees among themselves (for example peering vs. transit) should have no conceptual bearing on the economics of content distribution costs between Provider and Consumer. In other worlds, leave the IAPs out of this debate of who pays for the content distribution. Treat this as a cost of doing business and pay the IAP] One may debate the point that the Content Distributor is also a material beneficiary of a transaction involving transfer of content. Where would, for example, an IAP be if they had no content to transfer? There would not ordinarily be a revenue stream for pipeline operators and the pump station and refinery owners if they had no oil to transport or refine. This would be true if the only thing the content distributor was transporting was a specific type of content. IAPs do more. They operate a multi purpose pipeline. Further, even the evolution of the IAP business is going to be such that traditional content bits will only be one kind of bits the evolved entities will transport. Even traditional Internet bits will be a specialized case of the bits that flow on the new transport backbones. I am not arguing that there will be a unifying technology to do this ( there may well be ), but that this is how transport will be sold- a.k.a "The Bundle Sell". This means that Content Distributors are unlikely to subsidize the cost of Generic Content or Commodity Transport, which Internet Access is fast becoming. ] All this works fine when all you have is Generic Content. When this content changes to what I call Compelling Content, things change. Compelling Content attracts Content Consumers in hordes. A large affinity group of Content Consumers can be leveraged in many ways. Compelling Content opens up a a host of additional Revenue Opportunities for both the Content Provider and the Content Consumer. Now instead of just costs involved with distribution of content, there are revenues to offset the costs. For example, now the Compelling Content Provider may advertise on its site for its own products and generate more revenue. Or it may sell advertizing space or content space to other Content Providers and generate revenue that way. (Hence the emergence of a video rental outlet and a bank counter at your local large grocery store or the hordes of free magazines you can pickup while you are there). The Content Distributor could benefit from transporting Compelling Content, i.e the ability to attract more Content Providers to use its distribution system and therefore improve on returns on its capital investments. If the content is particularly compelling and it is the exclusive distributor of that content, it may attract more Content Consumers as customers as well. (The Generic Content would not have ordinarily transitioned to Compelling Content if the distribution system was not performing well. Many content sites have infant mortality because no one budgeted for costs of a decent distribution system) Consumers of Compelling Content, having flocked to the Content Site, may be offered incentives to stay put ( so that the Content Provider may charge a higher CPM rate for advertisers , for example) This is not an exhaustive list, but the point is that Compelling Content creates revenue for many parties and also opportunities for future revenues. The revenue could be used to offset the costs of distribution. "Following the Money" now, there is more of it in a larger number of hands and there are different ways to split it. All Content Providers should start off with budgeting for the costs of the distribution. If their content is compelling, additional revenues will eventually offset these costs. Some costs may be shifted but the Content Consumers will eventually pay for all costs directly or indirectly, and the Content Distributors are unlikely to subsidize what will come a commodity distribution service. Dividing things into Costs and Revenues Opportunities lets the money flow where it deserves and keeps the issues clean. --pushpendra Pushpendra Mohta pushp@cerf.net +1 619 455 3908 Director, CERFnet http://www.cerf.net +1 619 455 3995 (FAX)

all of this pseudo-marketing is enough to make my head spin. the practice of limiting peering agreements has nothing to do with collusion and we all damn well know it. enough of us have been around long enough to know that to draw simple comparisons between the phone business and the internet business is foolish, if not dangerous. settlements, as they exist in the phone industry do not make sense for the internet industry. there may be a way for internet networks to settle on traffic imbalances but i haven't heard a decent one proposed. do we assume that all traffic is web traffic (pull model)? and if that changes? what about multicast traffic (push model sometimes, pull model others)? who derives benefit, and who should pay? sure, mci has a good reason to connect to bell atlantic and all of the LOCAL carriers in the telephone business. their customers are potentially our customers for long distance traffic. we can BILL them. if MCI the phone company decided not to offer its long distance to a certain company's local customers, i would expect that the feds would be all over us/them. if internetMCI decides that we should or should not peer with another network the decision is made purely on traffic flows. if coast to coast traffic moves over our network twice between ourselves and the peer, that's not a peer. if the peer has a number of nap connections and a ds3 ld packet network, that's a peer. the practice that you refer to as 'unwise' keeps us from losing our shirts in an already-too-competitive marketplace. how, you ask? by forcing mci to continually buy bandwidth from ixc's. and the bandwidth is only half the battle, unless the ixc's start offering higher speed pipes, the engineering to keep so many connections going to a single ixc will become very costly. working without such a policy would be the equivalent of allowing a small start-up local phone company to demand and get free long distance for all of it's subscribers from mci. i have no way of billing anyone elses customers, there is no dileniation of service between local and long distance internet so we're left with the 'you move half the packets and i'll move half the packets' model. the end of your post is interesting, however. it may come to the point where larger carriers are forced to tag local as's local and to peer on the 'local' basis. but what benefit for that added complexity? local networks would still need to buy transit from someone. Jeff Young young@mci.net

To: nanog@merit.edu Subject: Re: peering charges? Date: Mon, 27 Jan 1997 09:52:15 -0800 From: Paul A Vixie <paul@vix.com> Sender: owner-nanog@merit.edu

...

[...] say your MCI and you have a smaller provider that wants to peer with you, you had rather have them buy a pipe than let the peer and ride your network for free. It's all about market share, plain and simple.

That's the conventional thinking. (I won't say "conventional wisdom" since this is not a wise practice.) The trouble with this is called the Sherman Act, which has a provision for something like "collusion in restraint of trade." I'm not pointing at MCI -- I have no idea whether they have ever thought as you ascribed above. But in your example, MCI does not know who the nonpeer will ultimately buy transit from -- all it knows is that if MCI refuses to peer, the nonpeer will have to buy transit from someone who *does* peer with MCI. If reasonable suspicion can be raised that this is occurring, then everybody in the "first tier" peering club can be investigated by the feds.

This isn't a "big" deal, Microsoft has been investigated by the feds and they're still in business. In fact, it didn't even hurt their reputation, let alone their bottom line. When the big nets started demanding private peering and refusing to peer with little guys, I had to just shrug and think, "well, they are now big enough that they do not fear gov't intervention."

On the other hand the industry will end up regulated due in part to practices like these. And I received some mail (due to my NIC registrations for past customers) a few weeks ago inviting my ex-customer to join a class action lawsuit against a couple of "first tier peering club" members, that was all about this kind of squeezing.

We really do just need to send eachother local-region routes, which keeps local traffic local, does not give away wide area telecom to noncustomers, takes away some causes for lawsuits and new legislation, and moves us back to a level playing field where folks without wide area networks have to buy transit and do so without complaining.

I want to at the outset that I don't know what MCI does or why, and if I did know I would probably be under NDA and so I wouldn't be able to talk about what I knew. I used MCI as an example in my last note to this list because the person I was replying to had used MCI as an example. For all I know, MCI peers with space aliens, or doesn't, or not. But, getting to the provider independent portion of Jeff Young's recent message here:

the end of your post is interesting, however. it may come to the point where larger carriers are forced to tag local as's local and to peer on the 'local' basis. but what benefit for that added complexity? local networks would still need to buy transit from someone.

Right now we're sucking down a fair amount of backhaul, symmetric to the backhaul of a local Mom & Pop's transit provider, carrying traffic which is both to and from the same local area. If backhaul were free, or even cheap, or even available in the quantities we need, the engineering simplicity would win out -- it's easier to manage a network if there are fewer links and fewer powered boxes inside it. On the other hand we're having a lot of trouble getting enough backhaul on some paths -- at any price. So OK, let's assume that transit providers all come to every local area to pick up local customers. A Mom & Pop ISP buys transit from one of them, on the hopeful assumption that it will peer with the other locally present big providers, thus preventing traffic between endpoints 10 miles away from taking a 500 (or 2000) mile loop through some more distant private peerage or exchange point. This saves on the rarified wide area backhaul, and it is certainly better than what a lot of local markets have now. But if we're assuming the existence local links between big ISP's in each local market, what's wrong with the local exchange concept (since, as I said in an earlier message, sum(1 .. N-1) links are more expensive to build than N links and a GIGAswitch, for reasonable values of N)? And if you're doing a local exchange, why not let Mom & Pop's also connect there, so that their transit link can be a private 10Mb/s or FDDI wire between cages -- saving the end users money indirectly by cutting down the number of bypass carriages? And finally-- if each of the previous steps make sense and you get this far-- why not have big and little ISPs peer directly, sharing only routes which do not result in assymetric wide area expense? (Route segregation between local and wide area was already necessary for an earlier step.) This saves router backplane expense, which while easier (in 1997) to buy than wide area backhaul is still in shorter supply than some people would like. Some L2's are more reliable than some L3's -- and are almost always cheaper per bit carried. This changes "we'll only peer with you if you have a network topology similar to ours" into "we'll be happy to peer with you but be aware that we only send local routes when we peer at public exchanges, and if you want a full routing exchange it'll take 6 T3's worth of private peering -- can you afford it?" I think the network will work better and scale better when this kind of "peering" becomes the norm. If wide area telecom costs are the reason big guys don't like to peer with little guys, then by gum let's see peering and charging all take place exactly where the economics require it.

This sounds like the right approach, although I'd add some flexibility that makes everything in between possible. Ie, if you peer at one NAP, then you get some percentage of the peers routes. If you are at all NAPS, then you get 100%.

I think this is going to be very bilateral and that the details will vary all over the place. And there are some problems with the model, in that CIDR blocks are not internally local to a single area. In order for someone who holds a /13 to send just the routes local to the NAP you're peering at, they have to send you bits and pieces of their /13. This is "hard." I'm not offering this world model without caveats -- doing the right thing is going to take a lot of effort. But I think it's the right direction *anyway*. If I want to send my wife mail and she works at a company 10 miles away, it is *not* going to sit well with me that she can't get it due to a private peering or even public peering point that's dropping packets 2,500 miles from us. (We live with that situation today, and none of us likes it.) I also don't think it is "wise" (there's that word again) to eat wide area line costs with traffic that could stay local. I could go on and on about this, but I see that I already have.

have a seperate routing policy for each local site to prevent local Mom'Pop shop routes from getting into other region's routers. Otherwise, these routes could turn up be BGP best routes, and when you deny them to your

Anyone has good/easy solutions to that?

Sure, you measure the propagation delay and multiply by the speed of light and drop any routes that traveled more than 50 miles. Routers adjust the packet time for any delay they add. Just kidding......

I know that you were using mci as a case in point, i'm just frustrated that we seem to cycle through this same issue at least once a quarter. I wholeheartedly agree that there are things that don't make sense to do with peering, but that's what happens in a regulated environment. whether it's regulation by the ixc's or by the fcc, is no matter. regulations can't optimally fit every scenario. what was wrong with the local exchange model for traffic between large networks? cost, for one. because many large internet providers are now long distance carriers as well, they already have shared infrastructure. why pay someone else (especially a lec) to exchange traffic? if 85% of the traffic i pull out of a nap is to sprint/uunet/ bbn why should i maintain 6 ds3's to that nap? i (personally) like the idea of keeping local routes local at exchanges, but there are issues to work through. it's going to take a non-trivial development effort to make it happen (tag local routes in bgp and only advertise them to the nap local providers). say you're a national sp and i'm a mom&pop. you want to send me only the routes that originate in my local area across the nap. i however, want to send you all of my routes - they're all local. how do you, as the nsp, keep the rest of your infrastructure (outside of the local customers) from knowing that my routes exist? you might localpref them, but then they serve as backup for all of your customers' traffic to me should i lose my transit provider. you might filter them out of the announcements at all sights except for those that you advertised as local to me (yuck). and if a customer get's poor performance through the nap to one of my customers? do you remove the announcements of that network? as i expand into other markets, won't my addressing be necessarily geography based lest i either break large aggregates or advertise routes that really aren't local to you? :-) how do you keep me honest? i like the idea of ixc's allowing zero length circuits within their premisis. is it possible? would ixc's allow it? Jeff Young young@mci.net

Subject: Re: peering charges? In-reply-to: Your message of "Mon, 27 Jan 1997 14:08:44 EST." <199701271908.OAA11437@foghorn.reston.mci.net> Date: Mon, 27 Jan 1997 12:07:08 -0800 From: Paul A Vixie <paul@vix.com>

...

the end of your post is interesting, however. it may come to the point where larger carriers are forced to tag local as's local and to peer on the 'local' basis. but what benefit for that added complexity? local networks would still need to buy transit from someone.

Right now we're sucking down a fair amount of backhaul, symmetric to the backhaul of a local Mom & Pop's transit provider, carrying traffic which is both to and from the same local area. If backhaul were free, or even cheap, or even available in the quantities we need, the engineering simplicity would win out -- it's easier to manage a network if there are fewer links and fewer powered boxes inside it. On the other hand we're having a lot of trouble getting enough backhaul on some paths -- at any price.

So OK, let's assume that transit providers all come to every local area to pick up local customers. A Mom & Pop ISP buys transit from one of them, on the hopeful assumption that it will peer with the other locally present big providers, thus preventing traffic between endpoints 10 miles away from taking a 500 (or 2000) mile loop through some more distant private peerage or exchange point. This saves on the rarified wide area backhaul, and it is certainly better than what a lot of local markets have now.

But if we're assuming the existence local links between big ISP's in each local market, what's wrong with the local exchange concept (since, as I said in an earlier message, sum(1 .. N-1) links are more expensive to build than N links and a GIGAswitch, for reasonable values of N)? And if you're doing a local exchange, why not let Mom & Pop's also connect there, so that their transit link can be a private 10Mb/s or FDDI wire between cages -- saving the end users money indirectly by cutting down the number of bypass carriages?

And finally-- if each of the previous steps make sense and you get this far-- why not have big and little ISPs peer directly, sharing only routes which do not result in assymetric wide area expense? (Route segregation between local and wide area was already necessary for an earlier step.) This saves router backplane expense, which while easier (in 1997) to buy than wide area backhaul is still in shorter supply than some people would like. Some L2's are more reliable than some L3's -- and are almost always cheaper per bit carried.

This changes "we'll only peer with you if you have a network topology similar to ours" into "we'll be happy to peer with you but be aware that we only send local routes when we peer at public exchanges, and if you want a full routing exchange it'll take 6 T3's worth of private peering -- can you afford it?" I think the network will work better and scale better when this kind of "peering" becomes the norm. If wide area telecom costs are the reason big guys don't like to peer with little guys, then by gum let's see peering and charging all take place exactly where the economics require it.