This ignores the fact that both companies have paying customers. If I am a BBN access customer and I cannot get to any sites, then what good is BBN to me? As an access provider, I built a access infrastructure and I get payed by my customer for providing it to them (include in this cost is the price to connect to content providers). If I am a Exodus customer and I pay for hosting, Exodus is no good to me if it doesn't have any viewers. Exodus built a hosting infrastructure, its customers pay for the right to use that infrastructure(included in this cost is the price to connect to viewers). So both companies need each other. Both companies get payed for providing a service. Where is the problem? Why should BBN get a cut of what Exodus's cutomers pay? BBN is trying to get payed to provide something it needs to from Exodus. If BBN needs it, why would Exodus pay for it? Isn't BBN trying to get payed twice?
-----Original Message----- From: Michael Dillon [SMTP:michael@memra.com] Sent: Friday, August 21, 1998 9:13 AM To: list@inet-access.net Subject: Re: BBN/GTEI
On Fri, 21 Aug 1998, Brian Wallingford wrote:
If you don't peer with or buy transit from GTEI, sit back, shut up, and watch and learn (seriously - WATCH and LEARN).
This is seriously good advice. A lot of people have compared this to the UUNet peering flap but the two incidents couldn't be more different. In UUNet's case you had John Sidgemore, an economist, attempting to throw his weight around and push towards a paid peering scenario with a cartel at the core, all of this based solely on an economics viewpoint.
But with BBN, there is a network engineering problem at the core of the issue, that of assymmetric traffic patterns. And BBN realizes that this sort of assymmetry will become more and more common in the future and that the industry needs to find some sort of hybrid peering/settlement mechanism that will work for both parties in an asymmetric arrangement. They are looking at things like what kind of methodology can be used to measure the traffic, what constitutes free balanced peering, how to charge for regional transit on traffic that exceeds the limits of balanced peering, and similar difficult issues.
I believe that the reason we are not hearing many details is that there are NDAs in place about the specifics of the Exodus, AboveNet and CRL peering contracts. But sooner or later those companies will come to some sort of agreement and BBN will explain the rationale behind their thinking. We may not totally agree with that rationale, but I think we can all see that establishing peering between two specialty providers has to be handled a bit differently than between two full-service providers.
And if BBN's ideas can be refined and accepted by the industry, then we will be in a better position because there will be an established methodology and pricing structure for an ISP to transition from full transit to full peering. I know from my experience with Priori that a lot of peering negotiations happen like an old-boys club cartel and if you ain't a member of the club, you can't get in. We need to change this so that there is an open process by which anyone can transition to being a peer based on an open and accepted methodology and pricing structure.
-- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
On Fri, 21 Aug 1998, Goodwin, Dustin wrote:
Both companies get payed for providing a service. Where is the problem? Why should BBN get a cut of what Exodus's cutomers pay? BBN is trying to get payed to provide something it needs to from Exodus. If BBN needs it, why would Exodus pay for it? Isn't BBN trying to get payed twice?
You are asking about specifics and I have no idea what the answers are. Unless somebody speaks to both Exodus and BBN and gets around the NDAs that they have signed, I don't think anyone can know. However, there is a more general issue here. If a company primarily hosts websites, then their traffic will be asymmetric with many more bytes going out than coming in. They must engineer their network to handle this asymmetry and any transit providers they have must also do so. Traditionally, peering only occurred between providers with a national network with at least 4 points of contact, spread out geographically. This requirement is so that each provider carries a roughly equal load of the traffic across expensive national longhaul circuits. It is this balanced transit load that makes them peers. In the case of a web hosting company, the bulk of the traffic is outgoing and if they do the standard shortest-exit routing with their peers, then the peer will be carrying a much larger transit load than the webhosting company. Of course, this could be solved by both parties doing longest-exit which places the largest transit load on the webhosting provider. But there is more. When the webhosting network touches down at only a few exchange points to peer with a provider who has an extensive network you will have a situation in which the peer is providing regional transit for free and must engineer their regional networks to deal with an asymmetric traffic pattern. For instance, if a webhosting provider touches down at San Jose, Chicago, DC and Dallas then they appear to meet the eligibility requirements for peering. But these peers end up providing full transit from Boston to DC, LA to San Jose, Denver to Dallas, and St. Louis to Chicago. This arises because one provider hosts lots of equipment close to an exchange point and the other provider interconnects many POPs in many cities. Somehow we need a way to quantify and measure the traffic and establish what peering is in terms of measurable quantities. A certain amount of asymmetry should be allowable but it can get out of hand. One way to deal with great asymmetry is to deny peering. Another way is to accept peering but measure the asymmetry and have a pricing structure for regional transit that applies after a certain point. Note that this is *NOT* the same as demanding that the webhosting peer become a transit customer. Since they are only using regional transit I would expect that the prices on the transit portion would be less than on a pure transit arrangement. However, for this to happen we would need some way to measure and quantify this regional transit. To date I don't think anyone has attempted to do anything like this except some Australian ISPs who have mapped the IPv4 address space geographically so that they can manage their routing according to the cost of various intercontinental links without relying on somebody else's BGP announcements. I'm sure that we could use some sort of similar geographical mapping of IPv4 addresses to quantify how much regional transit a peer uses and thus establish a sliding scale between a pure transit customer and a pure peer. -- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
Perhaps the best solution here is for BBN to purchase Exodus. On Fri, 21 Aug 1998, Michael Dillon wrote:
Date: Fri, 21 Aug 1998 10:30:47 -0700 (PDT) From: Michael Dillon <michael@memra.com> To: nanog@merit.edu Subject: RE: BBN/GTEI
On Fri, 21 Aug 1998, Goodwin, Dustin wrote:
Both companies get payed for providing a service. Where is the problem? Why should BBN get a cut of what Exodus's cutomers pay? BBN is trying to get payed to provide something it needs to from Exodus. If BBN needs it, why would Exodus pay for it? Isn't BBN trying to get payed twice?
You are asking about specifics and I have no idea what the answers are. Unless somebody speaks to both Exodus and BBN and gets around the NDAs that they have signed, I don't think anyone can know.
However, there is a more general issue here. If a company primarily hosts websites, then their traffic will be asymmetric with many more bytes going out than coming in. They must engineer their network to handle this asymmetry and any transit providers they have must also do so. Traditionally, peering only occurred between providers with a national network with at least 4 points of contact, spread out geographically. This requirement is so that each provider carries a roughly equal load of the traffic across expensive national longhaul circuits. It is this balanced transit load that makes them peers.
In the case of a web hosting company, the bulk of the traffic is outgoing and if they do the standard shortest-exit routing with their peers, then the peer will be carrying a much larger transit load than the webhosting company. Of course, this could be solved by both parties doing longest-exit which places the largest transit load on the webhosting provider. But there is more.
When the webhosting network touches down at only a few exchange points to peer with a provider who has an extensive network you will have a situation in which the peer is providing regional transit for free and must engineer their regional networks to deal with an asymmetric traffic pattern. For instance, if a webhosting provider touches down at San Jose, Chicago, DC and Dallas then they appear to meet the eligibility requirements for peering. But these peers end up providing full transit from Boston to DC, LA to San Jose, Denver to Dallas, and St. Louis to Chicago. This arises because one provider hosts lots of equipment close to an exchange point and the other provider interconnects many POPs in many cities.
Somehow we need a way to quantify and measure the traffic and establish what peering is in terms of measurable quantities. A certain amount of asymmetry should be allowable but it can get out of hand. One way to deal with great asymmetry is to deny peering. Another way is to accept peering but measure the asymmetry and have a pricing structure for regional transit that applies after a certain point. Note that this is *NOT* the same as demanding that the webhosting peer become a transit customer. Since they are only using regional transit I would expect that the prices on the transit portion would be less than on a pure transit arrangement.
However, for this to happen we would need some way to measure and quantify this regional transit. To date I don't think anyone has attempted to do anything like this except some Australian ISPs who have mapped the IPv4 address space geographically so that they can manage their routing according to the cost of various intercontinental links without relying on somebody else's BGP announcements. I'm sure that we could use some sort of similar geographical mapping of IPv4 addresses to quantify how much regional transit a peer uses and thus establish a sliding scale between a pure transit customer and a pure peer.
-- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
=-----------------= = | Charles Sprickman Internet Channel | | INCH System Administration Team (212)243-5200 | | spork@inch.com access@inch.com | = =----------------=
Perhaps the best solution here is for BBN to purchase Exodus.
You mean GTE...
On Fri, 21 Aug 1998, Michael Dillon wrote:
However, there is a more general issue here. If a company primarily hosts websites, then their traffic will be asymmetric with many more bytes going out than coming in. They must engineer their network to handle this asymmetry and any transit providers they have must also do so. Traditionally, peering only occurred between providers with a national network with at least 4 points of contact, spread out geographically. This requirement is so that each provider carries a roughly equal load of the traffic across expensive national longhaul circuits. It is this balanced transit load that makes them peers.
Lets not forget here that shortly after GTE picked up BBN, they also picked up Genuity. Genuity still pretty much exists as an independant entity. So when that is taken into account, GTE ALSO does web hosting.. They just seperate the traffic off from the "BBN" traffic.
In the case of a web hosting company, the bulk of the traffic is outgoing and if they do the standard shortest-exit routing with their peers, then the peer will be carrying a much larger transit load than the webhosting company. Of course, this could be solved by both parties doing longest-exit which places the largest transit load on the webhosting provider. But there is more.
When the webhosting network touches down at only a few exchange points to peer with a provider who has an extensive network you will have a situation in which the peer is providing regional transit for free and must engineer their regional networks to deal with an asymmetric traffic pattern. For instance, if a webhosting provider touches down at San Jose, Chicago, DC and Dallas then they appear to meet the eligibility requirements for peering. But these peers end up providing full transit from Boston to DC, LA to San Jose, Denver to Dallas, and St. Louis to Chicago. This arises because one provider hosts lots of equipment close to an exchange point and the other provider interconnects many POPs in many cities.
When you consider a web hosting company, often there are not the resources to transit traffic to the best exit. Many companies buy transit from one or more companies anyhow to be sure that they have maximum reachability. If, for example, a company buys transit from MCI and uses that to reach BBN, than it will have an impact on the traffic ratios for the BBN/MCI interconnects. If enough similar companies take MCI to get to BBN then it is entirely possible that the BBN/MCI interconnect will become highly asymetric. So takeing into account what just happened with Exodus, does this mean that BBN would try to force MCI to buy transit from BBN? This only goes so far before the whole thing breaks down. Granted that the above is somewhat extreme example but it is most certainly possible. The whole point behind private interconnects is to get traffic to the destination with as few intermediary hops as possible and to be able to better control congestion and loss between the networks. If BBN's customers have trouble getting to certain popular sites, they're going to complain to BBN. If the hosting company's customers can't be accessed from certain sites, they're going to complain to the hosting company. This means that both companies tech support and operations centers get additional calls, additional work, and additional headaches in trying to deal with traffic levels of some third party. In many ways, its the third party that ultimately suffers as they try to deal with both companies to resolve the issues. IMO, asymetry is not a valid requirement for a private interconnect. NOT having the interconnect hurts both companies. ---------------------------------------------------------------------- Wayne Bouchard GlobalCenter web@globalcenter.net Network Engineer (602) 416-6422 800-373-2499 x6422 FAX: (602) 416-9422 http://www.globalcenter.net ----------------------------------------------------------------------
On Fri, 21 Aug 1998, Wayne Bouchard wrote:
So takeing into account what just happened with Exodus, does this mean that BBN would try to force MCI to buy transit from BBN?
No. It means that BBN would bill MCI for the transit component of the traffic exchange that would be measured and priced by an industry agreed measurement methodology and pricing scheme. All peering agreements would be the same in this scenario. Once you get accepted as a peer then you would pay only for the transit that you use that is outside the accepted rule of thumb for balanced peering which could be somthing like a 1.5:1 or 2:1 traffic ratio at the exchange point. Presumably, a pair of peers like MCI and BBN would agree to settle these bills on an aggregate basis so that MCI could pay a $2000 peering charges bill from BBN by offseting it against a $1500 bill sent to BBN along with $500 cash.
If BBN's customers have trouble getting to certain popular sites, they're going to complain to BBN. If the hosting company's customers can't be accessed from certain sites, they're going to complain to the hosting company. This means that both companies tech support and operations centers get additional calls, additional work, and additional headaches in trying to deal with traffic levels of some third party. In many ways, its the third party that ultimately suffers as they try to deal with both companies to resolve the issues.
However, if both companies have a scalable peering agreement with each other, this problem becomes merely a problem of engineering for the traffic levels. The payment aspects are settled automatically by applying the settlement rules. If such scalable peering already existed, I'm convinced that the current situation between Exodus and BBN would not have developped. So while those two companies figure out how to handle their relationship, maybe we could all learn from this and figure out a way to make peering work in a more scalable manner. -- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
On Fri, Aug 21, 1998 at 01:48:14PM -0700, Michael Dillon wrote:
On Fri, 21 Aug 1998, Wayne Bouchard wrote:
So takeing into account what just happened with Exodus, does this mean that BBN would try to force MCI to buy transit from BBN?
No. It means that BBN would bill MCI for the transit component of the traffic exchange that would be measured and priced by an industry agreed measurement methodology and pricing scheme. All peering agreements would be the same in this scenario. Once you get accepted as a peer then you would pay only for the transit that you use that is outside the accepted rule of thumb for balanced peering which could be somthing like a 1.5:1 or 2:1 traffic ratio at the exchange point. Presumably, a pair of peers like MCI and BBN would agree to settle these bills on an aggregate basis so that MCI could pay a $2000 peering charges bill from BBN by offseting it against a $1500 bill sent to BBN along with $500 cash.
Why should they? In fact, what you're advocating is billing the sender for *solicited data* from the recipient's point of view! The end result of this will be discontinuities in the network - since nobody in their right mind is going to allow SOMEONE ELSE to write blank checks on THEIR account, when they have no means to get that back from the party (since they have no contract with the other side). The result? You connect to a web site and get "XXX is asking us to pay for what you already bought - please enter you VISA number at the prompt" in the requestor..... or just a "XXX is being a jerk; go away!"
However, if both companies have a scalable peering agreement with each other, this problem becomes merely a problem of engineering for the traffic levels. The payment aspects are settled automatically by applying the settlement rules.
You've got the problem backwards Michael. -- -- Karl Denninger (karl@MCS.Net)| MCSNet - Serving Chicagoland and Wisconsin http://www.mcs.net/ | T1's from $600 monthly / All Lines K56Flex/DOV | NEW! Corporate ISDN Prices dropped by up to 50%! Voice: [+1 312 803-MCS1 x219]| EXCLUSIVE NEW FEATURE ON ALL PERSONAL ACCOUNTS Fax: [+1 312 803-4929] | *SPAMBLOCK* Technology now included at no cost
On Fri, 21 Aug 1998, Karl Denninger wrote:
In fact, what you're advocating is billing the sender for *solicited data* from the recipient's point of view!
Not at all. I am advocating paying for transit. When A and B use roughly the same amount of each other's transit, there is no point in counting the difference. But when you have an asymmetric situation, rather than cutting off peering altogether because the other guy is too different, why not have a scalable peering situation that directly addresses the asymmetry in traffic flows. The only other solution that I can see is for the network receiving the huge incoming flow to direct all that web traffic through transparent web caches at each exchange point. However that just raises increased barriers to peering and does not deal with non-cacheable dataflows which are increasing over time. Scalable peering would reduce the barriers to peering and make it easier for new players to buy in. They would still have to build a truly national network, but at that point they could not have the door slammed in their faces. Regardless of whether my proposed solution is the correct one or just a bad idea produced by indigestion, you cannot deny that the asymmetry between networks is increasing as network providers specialize the services they offer. The old-fashioned rough-cut peering is becoming more and more unsuitable as the only peering option. We need new ways to do this. Somebody has to take the first step. Somebody has to be a pioneer. The details can always be hashed out later. -- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
On Fri, Aug 21, 1998 at 03:36:08PM -0700, Michael Dillon wrote:
On Fri, 21 Aug 1998, Karl Denninger wrote:
In fact, what you're advocating is billing the sender for *solicited data* from the recipient's point of view!
Not at all. I am advocating paying for transit.
On the contrary. If I buy a DS1 for transit from your network, I'm expecting the person I pay to provide transit - ALL OF THE TRANSIT. That's what I'm buying! Now what you're saying is that "oh, no, that's not really what you bought"! I'm sure the thousands of DS-1 connected CUSTOMERS (that is, transit purchasers) will find this a very, very interesting interpretation.
Regardless of whether my proposed solution is the correct one or just a bad idea produced by indigestion, you cannot deny that the asymmetry between networks is increasing as network providers specialize the services they offer. The old-fashioned rough-cut peering is becoming more and more unsuitable as the only peering option. We need new ways to do this. Somebody has to take the first step. Somebody has to be a pioneer.
No, its actually becoming MORE suitable. Instead of burning the entire circuit in both directions, you're only burning half of it now (one direction).
The details can always be hashed out later.
You're simply wrong, once again. -- -- Karl Denninger (karl@MCS.Net)| MCSNet - Serving Chicagoland and Wisconsin http://www.mcs.net/ | T1's from $600 monthly / All Lines K56Flex/DOV | NEW! Corporate ISDN Prices dropped by up to 50%! Voice: [+1 312 803-MCS1 x219]| EXCLUSIVE NEW FEATURE ON ALL PERSONAL ACCOUNTS Fax: [+1 312 803-4929] | *SPAMBLOCK* Technology now included at no cost
On Fri, 21 Aug 1998, Karl Denninger wrote:
In fact, what you're advocating is billing the sender for *solicited data* from the recipient's point of view!
Not at all. I am advocating paying for transit.
On the contrary.
If I buy a DS1 for transit from your network, I'm expecting the person I pay to provide transit - ALL OF THE TRANSIT.
Of course, and I agree with you 100%. But I was not talking about a transit customer. I was talking about a peer whose traffic interchange is asymmetric and who therefore uses some regional transit in the other guy's network. I'm saying that instead of slamming the door in his face and telling him to buy transit, we need to have a scalable peering option that is a blend. Maybe I am headed in the wrong direction with this but I do believe we need a better solution for peering with asymmetric peers that reduces the barriers to entry to $$$. Right now there are barriers to entry that probably will not pass the scrutiny of the DOJ.
No, its actually becoming MORE suitable. Instead of burning the entire circuit in both directions, you're only burning half of it now (one direction).
You still have to pay for the whole circuit. -- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
On Fri, Aug 21, 1998 at 04:43:51PM -0700, Michael Dillon wrote:
On Fri, 21 Aug 1998, Karl Denninger wrote:
In fact, what you're advocating is billing the sender for *solicited data* from the recipient's point of view!
Not at all. I am advocating paying for transit.
On the contrary.
If I buy a DS1 for transit from your network, I'm expecting the person I pay to provide transit - ALL OF THE TRANSIT.
Of course, and I agree with you 100%. But I was not talking about a transit customer.
YES YOU ARE! Let's look at the instant case - BBN <> Exodus in a PEER relationship BBN claims that Exodus sends more traffic (by a LOT) than they send. To which I say "so what" - BBN'S TRANSIT CUSTOMERS ASKED FOR IT! Now, if BBN wishes to claim that they have the right to bill Exodus for that which THEIR TRANSIT CUSTOMERS ASKED FOR, I say "bullshit". Further, I suspect that BBN's transit customers, upon learning that Exodus refuses to pay, will ALSO say *bullshit* and demand that BBN either carry the traffic, as they PAID for it be carried, or they will find a provider that will! Now if BBN has MISPRICED their transit, that's THEIR problem. To try to offload that problem on their PEERS, when those peers are sourcing that traffic at the EXPLICIT request of their customers, is IMHO nuts.
I was talking about a peer whose traffic interchange is asymmetric and who therefore uses some regional transit in the other guy's network. I'm saying that instead of slamming the door in his face and telling him to buy transit, we need to have a scalable peering option that is a blend.
You're simply wrong here. No traffic passing over a peer connection is "using some transit that is not paid for". That's an argument that in its logical conclusion (two or three levels down the tree of logic) means that someone has been ripped off and/or deceived - that they paid for something that the provider in question doesn't intend to provide! ALL traffic passing between peered providers is BY DEFINITION sourced by a TRANSIT customer of one provider and sunk by a TRANSIT CUSTOMER of the other network. The correlary to this is that all traffic passing between peered providers was REQUESTED by one of the transit customers of one of those networks, who PAID for that transit to be provided to them! THIS is why claiming that "well, you're asymmetric so you pay!" is baloney. Were I to find myself in that position my response would be "Bite me!", followed same-day by a letter to EVERY ONE of the denying provider's customers - on paper - announcing that their provider had just tried to bill me for what the target of those letters might think they had already bought and paid for! The fallout from THAT would be immediate and severe.
Maybe I am headed in the wrong direction with this but I do believe we need a better solution for peering with asymmetric peers that reduces the barriers to entry to $$$. Right now there are barriers to entry that probably will not pass the scrutiny of the DOJ.
NO barrier to peering will pass the scrutiny of the DOJ. Sorry, that's the facts. A peer relationship is by *definition* not assignable as something which has disparate value to one end or the other, since by definition each provider involved is being paid to provide transport by their customers, and those customers won't be if they can't get where they want to go.
No, its actually becoming MORE suitable. Instead of burning the entire circuit in both directions, you're only burning half of it now (one direction).
You still have to pay for the whole circuit.
And you have the unused half over which you can run other traffic. -- -- Karl Denninger (karl@MCS.Net)| MCSNet - Serving Chicagoland and Wisconsin http://www.mcs.net/ | T1's from $600 monthly / All Lines K56Flex/DOV | NEW! Corporate ISDN Prices dropped by up to 50%! Voice: [+1 312 803-MCS1 x219]| EXCLUSIVE NEW FEATURE ON ALL PERSONAL ACCOUNTS Fax: [+1 312 803-4929] | *SPAMBLOCK* Technology now included at no cost
On Fri, Aug 21, 1998 at 08:42:34PM -0500, Karl Denninger wrote:
ALL traffic passing between peered providers is BY DEFINITION sourced by a TRANSIT customer of one provider and sunk by a TRANSIT CUSTOMER of the other network. The correlary to this is that all traffic passing between peered providers was REQUESTED by one of the transit customers of one of those networks, who PAID for that transit to be provided to them!
There can be no 'request' without associated 'acceptance' of that requet. Therefore we have two 'half-links' where the content provider thinks it beneficial to pay for the other half-link in order to get its traffic delivered to the content consumer paying the other half. A corporate website covers its cost of the half-link in product sales while a more serious content provider sells subscriptions. A colo charges the website publisher not only for rack space etc. but also to provide the half-link to content consumer. So the colo has the fees and will pay if peering is asymmetric. -- tuomas.toivonen@fishpool.fi fishpool creations ltd http://www.kasvua.org/~toivotuo/ http://www.fishpool.fi/
All: The issue of asymmetric data flows is one of the reasons why we are building an optical Internet here in Canada. With an optical Internet you can traffic engineer individual wavelengths to support Tx/Rx asymmetric data flows in any way you want. This makes much more efficient use of the underlying fiber indrastructure. For more information www.canarie.ca www.canet2.net Bill ------------------------------------------- Bill St Arnaud Director Network Projects CANARIE bill.st.arnaud@canarie.ca http://www.canarie.ca/bstarn  Â
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Michael Dillon Sent: Friday, August 21, 1998 7:44 PM To: nanog@merit.edu Subject: Re: BBN/GTEI
On Fri, 21 Aug 1998, Karl Denninger wrote:
In fact, what you're advocating is billing the sender for *solicited data* from the recipient's point of view!
Not at all. I am advocating paying for transit.
On the contrary.
If I buy a DS1 for transit from your network, I'm expecting the person I pay to provide transit - ALL OF THE TRANSIT.
Of course, and I agree with you 100%. But I was not talking about a transit customer. I was talking about a peer whose traffic interchange is asymmetric and who therefore uses some regional transit in the other guy's network. I'm saying that instead of slamming the door in his face and telling him to buy transit, we need to have a scalable peering option that is a blend.
Maybe I am headed in the wrong direction with this but I do believe we need a better solution for peering with asymmetric peers that reduces the barriers to entry to $$$. Right now there are barriers to entry that probably will not pass the scrutiny of the DOJ.
No, its actually becoming MORE suitable. Instead of burning the entire circuit in both directions, you're only burning half of it now (one direction).
You still have to pay for the whole circuit.
-- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
Has anyone yet come up with any good metrics for route-miles a packet must pass through in one direction or another? Right now, the closest thing I have seen to that is MEDs in a BGP announcement, set by a provider who knows a little something about how his network is built. The problem I see is that if we base settlement on how the MEDs are set, an unscrupulous peer could change his MEDs to offset a portion of his transit or something along those lines. Or every content provider could buy a same-sized access provider or vice versa, and solve the problem that way. -Deepak. On Fri, 21 Aug 1998, Michael Dillon wrote:
On Fri, 21 Aug 1998, Karl Denninger wrote:
In fact, what you're advocating is billing the sender for *solicited data* from the recipient's point of view!
Not at all. I am advocating paying for transit. When A and B use roughly the same amount of each other's transit, there is no point in counting the difference. But when you have an asymmetric situation, rather than cutting off peering altogether because the other guy is too different, why not have a scalable peering situation that directly addresses the asymmetry in traffic flows. The only other solution that I can see is for the network receiving the huge incoming flow to direct all that web traffic through transparent web caches at each exchange point. However that just raises increased barriers to peering and does not deal with non-cacheable dataflows which are increasing over time. Scalable peering would reduce the barriers to peering and make it easier for new players to buy in. They would still have to build a truly national network, but at that point they could not have the door slammed in their faces.
Regardless of whether my proposed solution is the correct one or just a bad idea produced by indigestion, you cannot deny that the asymmetry between networks is increasing as network providers specialize the services they offer. The old-fashioned rough-cut peering is becoming more and more unsuitable as the only peering option. We need new ways to do this. Somebody has to take the first step. Somebody has to be a pioneer.
The details can always be hashed out later.
-- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
Isn't being a pioneer doing best-exit to eliminate the cost imbalances? I thought that was the whole idea behind what BBN/Exodus were doing.
On Fri, 21 Aug 1998, Karl Denninger wrote:
In fact, what you're advocating is billing the sender for *solicited data* from the recipient's point of view!
Not at all. I am advocating paying for transit. When A and B use roughly the same amount of each other's transit, there is no point in counting the difference. But when you have an asymmetric situation, rather than cutting off peering altogether because the other guy is too different, why not have a scalable peering situation that directly addresses the asymmetry in traffic flows. The only other solution that I can see is for the network receiving the huge incoming flow to direct all that web traffic through transparent web caches at each exchange point. However that just raises increased barriers to peering and does not deal with non-cacheable dataflows which are increasing over time. Scalable peering would reduce the barriers to peering and make it easier for new players to buy in. They would still have to build a truly national network, but at that point they could not have the door slammed in their faces.
Regardless of whether my proposed solution is the correct one or just a bad idea produced by indigestion, you cannot deny that the asymmetry between networks is increasing as network providers specialize the services they offer. The old-fashioned rough-cut peering is becoming more and more unsuitable as the only peering option. We need new ways to do this. Somebody has to take the first step. Somebody has to be a pioneer.
The details can always be hashed out later.
-- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
On Fri, 21 Aug 1998, Robert Bowman wrote:
Isn't being a pioneer doing best-exit to eliminate the cost imbalances? I thought that was the whole idea behind what BBN/Exodus were doing.
I don't know your network and BBN's well enough to do anything but guess here, but wouldn't there still be asymmetry in the number of bits of traffic exchanged, even with best exit? I'm not sure if the things I am discussing are directly applicable to your situation with BBN but if you think some of this stuff could work and are willing to try it, why not ask BBN if they would consider some sort of interim agreement while you work out the details. I guess I'm the kind of guy who always looks for a creative, off-the-wall, "think different" kind of solution to a problem in negotiations. The worst the other folks can say is no. But sometimes they say hmmm... -- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
why is assymetry the issue?? I'm very confused. It's the imbalance of costs.. right???
On Fri, 21 Aug 1998, Robert Bowman wrote:
Isn't being a pioneer doing best-exit to eliminate the cost imbalances? I thought that was the whole idea behind what BBN/Exodus were doing.
I don't know your network and BBN's well enough to do anything but guess here, but wouldn't there still be asymmetry in the number of bits of traffic exchanged, even with best exit?
I'm not sure if the things I am discussing are directly applicable to your situation with BBN but if you think some of this stuff could work and are willing to try it, why not ask BBN if they would consider some sort of interim agreement while you work out the details. I guess I'm the kind of guy who always looks for a creative, off-the-wall, "think different" kind of solution to a problem in negotiations. The worst the other folks can say is no. But sometimes they say hmmm...
-- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
On Fri, Aug 21, 1998 at 03:59:30PM -0700, Robert Bowman wrote:
Isn't being a pioneer doing best-exit to eliminate the cost imbalances? I thought that was the whole idea behind what BBN/Exodus were doing.
That is nice, but it doesn't change the fundamentals of what is going on. That is, even if someone is NOT doing "best exit" (or there is only one exit) that doesn't change the fact that the transit customers on the other network paid in part so they could get there! -- -- Karl Denninger (karl@MCS.Net)| MCSNet - Serving Chicagoland and Wisconsin http://www.mcs.net/ | T1's from $600 monthly / All Lines K56Flex/DOV | NEW! Corporate ISDN Prices dropped by up to 50%! Voice: [+1 312 803-MCS1 x219]| EXCLUSIVE NEW FEATURE ON ALL PERSONAL ACCOUNTS Fax: [+1 312 803-4929] | *SPAMBLOCK* Technology now included at no cost
On Fri, 21 Aug 1998, Karl Denninger wrote:
On Fri, Aug 21, 1998 at 01:48:14PM -0700, Michael Dillon wrote:
On Fri, 21 Aug 1998, Wayne Bouchard wrote:
So takeing into account what just happened with Exodus, does this mean that BBN would try to force MCI to buy transit from BBN?
No. It means that BBN would bill MCI for the transit component of the traffic exchange that would be measured and priced by an industry agreed measurement methodology and pricing scheme. All peering agreements would be the same in this scenario. Once you get accepted as a peer then you would pay only for the transit that you use that is outside the accepted rule of thumb for balanced peering which could be somthing like a 1.5:1 or 2:1 traffic ratio at the exchange point. Presumably, a pair of peers like MCI and BBN would agree to settle these bills on an aggregate basis so that MCI could pay a $2000 peering charges bill from BBN by offseting it against a $1500 bill sent to BBN along with $500 cash.
Since I don't have Michael's original post, I'll just respond to this quoted portion: And any suitably informed party to a peering arrangement like this can simply offer free hosting to search engine crawlers until they are extracting the desired amount cash from their peer via settlements. HECK with peering agreements like this, you could PAY search engine crawlers to host with you. :) Cool! Mike. ps. If it isn't complete obvious at this point, the moral of the story is that if you decide to pay people for any measure of traffic flow you have given them incentive to provide you with exactly what you want and they will make sure you end up paying them. pps. In otherwords, if it involves money you can't help incentivizing some form of behavior. +------------------- H U R R I C A N E - E L E C T R I C -------------------+ | Mike Leber Direct Internet Connections Voice 408 282 1540 | | Hurricane Electric Web Hosting & Co-location Fax 408 971 3340 | | mleber@he.net http://www.he.net | +---------------------------------------------------------------------------+
At 08:47 PM 08/21/1998 -0700, Mike Leber wrote:
... And any suitably informed party to a peering arrangement like this can simply offer free hosting to search engine crawlers until they are extracting the desired amount cash from their peer via settlements.
Folks, Think very carefully about what happens as traffic increases... Given that the context is the introduction of "settlement-based" peering for traffic imbalance in a shortest-exit world, then all of the various schemes for generation of traffic (web crawlers, backup services, etc) are no problem whatsoever, and may actually been seen as desirable... Background: Imagine a nationwide Internet provider with customer A on the west coast and customer B on the east coast. Customer B sends queries to customer A's server, and receives quite a bit of data in return. (all of this traffic stays on the provider's backbone in this case) Presume for the moment that the provider charges the customers (A&B) on a usage-basis which takes both sending and received traffic levels into consideration. Now, one could charge each of A and B the full cost of the traffic exchanged (which would effectively be "double dipping" and quite profitable :-) but a competitive market makes it far more likely that the rate per traffic measure will be such that A & B together pay the costs associated with their traffic exchange Now, given that A & B are both on the provider's network, everything is fine. When we hypothesize that A is instead connected to a peer network, we don't know whether costs will be recovered. In a shortest- exit world, traffic is quickly exchanged to the original provider's network and hence the costs of transmission to B are almost identical to the case where the A was a customer. Given that there's only one customer (B) now paying for these costs, this presents a potential problem. Customer B expects to pay their fair share of the costs, but doesn't expect to pay a larger portion of the total costs simply because A is connected to another provider. The truth is, there is actually nothing wrong with this situation if the traffic to/from the peer is of the same order of magnitude. To elaborate: a peer provides traffic which must be carried without corresponding sender-side revenue. On the other hand, the peer also allows the Internet provider to provide a similar level of sender-paid traffic which (by nature of being handed off) has very little actual transmission costs. This is why typical peers are overall neutral to cost recovery, and why peers which send far more traffic than received are problematic. There are a few options that have been mentioned to this situation, including the "receiver pays full freight / sending is free" model, the use of something other than shortest-exit routing, and introduction of settlements for such traffic. For purposes of this thread, we're trying to determine if the settlement approach is doomed to failure due to ability to generate additional traffic flows through innovation. Assessment: Presuming that a peer which is a net exporter of traffic begins to address this by encouraging additional traffic from customers to its network, this would result in more sender-paid originating traffic which quickly moves to the peer. Such traffic is (as noted above) quite profitable, as it includes full cost recover but little actual costs due to shortest to the peer. This is actually good for both networks, and us back to an Internet where more traffic is considered a good thing by ISP's at both ends of the connection. /John p.s. The fact that the sender of traffic should be paying some portion of the resulting costs is not a surprise to anyone; many of the content companies that I've spoken to believe they already are paying more as traffic increases, and were quite surprised to find that it doesn't actually make it to the networks which bear the brunt of the traffic carriage. p.p.s. As noted, departure from shortest-exit is also another approach which may provide some answers to this situation, but that's a different topic which deserves its own thread. This message is simply noting that settling for peering traffic is quite viable, despite assertions to the contrary regarding traffic generation. As long as you're billing the senders on your network for increased usage (and handing it off shortest-exit), increased traffic is good thing.
p.s. The fact that the sender of traffic should be paying some portion of the resulting costs is not a surprise to anyone; many of the content companies that I've spoken to believe they already are paying more as traffic increases, and were quite surprised to find that it doesn't actually make it to the networks which bear the brunt of the traffic carriage.
Why don't you just resort to a bit-counting method that I suggested earlier for settlement peering, and make the *SENDER* carry the traffic to the 'local' destination ? That way, the sender has to build a large enough network to distribute its traffic to the destination networks, without just presenting at one network point and expecting destination networks to foot the inter-state / inter-country traffic .. ? Adrian
On Fri, 21 Aug 1998, Michael Dillon wrote:
On Fri, 21 Aug 1998, Wayne Bouchard wrote:
So takeing into account what just happened with Exodus, does this mean that BBN would try to force MCI to buy transit from BBN?
No. It means that BBN would bill MCI for the transit component of the traffic exchange that would be measured and priced by an industry agreed measurement methodology and pricing scheme.
Hey Michael, Wasn't it you on the inet-access list suggesting:
If you don't peer with or buy transit from GTEI, sit back, shut up, and watch and learn (seriously WATCH and LEARN).
So, were the above comments meant to reference everyone *except* you? /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ Patrick Greenwell (800) 299-1288 v Systems Administrator (925) 377-1212 v NameSecure (925) 377-1414 f \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
On Fri, 21 Aug 1998, Patrick Greenwell wrote:
Wasn't it you on the inet-access list suggesting:
If you don't peer with or buy transit from GTEI, sit back, shut up, and watch and learn (seriously WATCH and LEARN).
Nope. Wasn't me. The above was quoted by me in a message.
So, were the above comments meant to reference everyone *except* you?
Nope. They were for everyone except *YOU*. -- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
What I am unclear about is why this type of network(s) is being shutoff. It seems that this is exactly the kind of network that should not lose peering. It is traffic that BBN customers are requesting/buying. For example what would happen if customers could not get to yahoo they would riot. That gives the content providers some pull in my book esp if they are handing off traffic at several exchange points. Anyone else have this point of view? Dave Thank you, David Diaz Chief Technical Officer Netrail, Inc. 888-NETRAIL On Fri, 21 Aug 1998, Goodwin, Dustin wrote:
This ignores the fact that both companies have paying customers. If I am a BBN access customer and I cannot get to any sites, then what good is BBN to me? As an access provider, I built a access infrastructure and I get payed by my customer for providing it to them (include in this cost is the price to connect to content providers). If I am a Exodus customer and I pay for hosting, Exodus is no good to me if it doesn't have any viewers. Exodus built a hosting infrastructure, its customers pay for the right to use that infrastructure(included in this cost is the price to connect to viewers). So both companies need each other. Both companies get payed for providing a service. Where is the problem? Why should BBN get a cut of what Exodus's cutomers pay? BBN is trying to get payed to provide something it needs to from Exodus. If BBN needs it, why would Exodus pay for it? Isn't BBN trying to get payed twice?
-----Original Message----- From: Michael Dillon [SMTP:michael@memra.com] Sent: Friday, August 21, 1998 9:13 AM To: list@inet-access.net Subject: Re: BBN/GTEI
On Fri, 21 Aug 1998, Brian Wallingford wrote:
If you don't peer with or buy transit from GTEI, sit back, shut up, and watch and learn (seriously - WATCH and LEARN).
This is seriously good advice. A lot of people have compared this to the UUNet peering flap but the two incidents couldn't be more different. In UUNet's case you had John Sidgemore, an economist, attempting to throw his weight around and push towards a paid peering scenario with a cartel at the core, all of this based solely on an economics viewpoint.
But with BBN, there is a network engineering problem at the core of the issue, that of assymmetric traffic patterns. And BBN realizes that this sort of assymmetry will become more and more common in the future and that the industry needs to find some sort of hybrid peering/settlement mechanism that will work for both parties in an asymmetric arrangement. They are looking at things like what kind of methodology can be used to measure the traffic, what constitutes free balanced peering, how to charge for regional transit on traffic that exceeds the limits of balanced peering, and similar difficult issues.
I believe that the reason we are not hearing many details is that there are NDAs in place about the specifics of the Exodus, AboveNet and CRL peering contracts. But sooner or later those companies will come to some sort of agreement and BBN will explain the rationale behind their thinking. We may not totally agree with that rationale, but I think we can all see that establishing peering between two specialty providers has to be handled a bit differently than between two full-service providers.
And if BBN's ideas can be refined and accepted by the industry, then we will be in a better position because there will be an established methodology and pricing structure for an ISP to transition from full transit to full peering. I know from my experience with Priori that a lot of peering negotiations happen like an old-boys club cartel and if you ain't a member of the club, you can't get in. We need to change this so that there is an open process by which anyone can transition to being a peer based on an open and accepted methodology and pricing structure.
-- Michael Dillon - Internet & ISP Consulting Memra Communications Inc. - E-mail: michael@memra.com Check the website for my Internet World articles - http://www.memra.com
participants (14)
-
Adrian Chadd -
Bill St. Arnaud -
Charles Sprickman -
Dave Diaz -
Deepak Jain -
Goodwin, Dustin -
John Curran -
Karl Denninger -
Michael Dillon -
Mike Leber -
Patrick Greenwell -
Robert Bowman -
Tuomas Toivonen -
Wayne Bouchard