"...the companies have agreed to the settlement-free exchange of traffic subject to specific payments if certain obligations are not met." So it does look like Cogent bent somwhat...I'm guessing they agreed to pay some sort of "traffic imbalance fee"? Anyone know of any other peering arrangements that have similar terms? I'll admit, that's a new one for me... -C On Oct 28, 2005, at 2:31 PM, Eric Louie wrote:

Now, one really needs to wonder why the agreement could not be reached *prior* to the depeering on 10/5

It's not rocket science.

It's only as complex as one makes it out to be. (one can attempt to explain away the complexities, but they apparently were able to *finalize* an agreement in 3 weeks, perhaps the agreement happened in it's entirety in 3 weeks - no speculation on the agreement is required unless you have nothing better to do)

Who are the next discontent couples?

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Jared Mauch Sent: Friday, October 28, 2005 11:08 AM To: nanog@merit.edu Subject: cogent+ Level(3) are ok now

http://biz.yahoo.com/prnews/051028/laf022.html?.v=27

The internet will not end on November(9)th :)

- jared

-- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.

-- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.1.362 / Virus Database: 267.12.5/150 - Release Date: 10/27/2005

On 10/28/05 5:45 PM, "JC Dill" <lists05@equinephotoart.com> wrote:

Christopher Woodfield wrote:

...

"...the companies have agreed to the settlement-free exchange of traffic subject to specific payments if certain obligations are not met."

So it does look like Cogent bent somwhat...I'm guessing they agreed to pay some sort of "traffic imbalance fee"?

There are other possibilities.

Maybe they agreed to pay a transit fee should they fail to carry the L3 user's requested traffic as far as possible before handing it off (cold potato routing) and hand it off at the earliest possibility (hot potato routing) leaving L3 to backhaul it across the L3 network to the user who requested the data.

I doubt it. Cold potato is normally the first thing Cogent offers in a situation like this. I'm guessing this went something beyond that. Cogent would have offered cold potato well before the original depeering. I have no specific information, but I'm guessing there is a per-mbps charge that kicks in at certain ratio levels. Or, there may be a flat "port charge" per month under certain conditions - Sprint did this many years ago.

Etc.

jc

I'm having a bit of trouble figuring out Level(3)'s goal in all this. A bit of incremental revenue? For all of this trouble? I could understand feeling that Cogent's ratios are a violation of their peering requirements and depeering them on principle, but if that's the case, why back down for a little cash? Of course, various external pressures may have been brought to bear on Level(3). Customers, regulators, press, creditors, etc. - Dan

Hi John,

Even with cold-potato routing, there is an expense in handling increased levels of traffic that is destined for your network. This increase in traffic often has no new revenue associated with it, because it is fanning out to thousands of flat-rate consumer/small-business connections (e.g. DSL) where billing is generally by peak capacity not usage.

not true for cogent tho, we know that virtually all their traffic is usage based transit customers Steve

On Nov 1, 2005, at 7:53 AM, John Curran wrote:

At 12:27 PM +0000 11/1/05, Stephen J. Wilcox wrote:

...

Hi John,

...

Even with cold-potato routing, there is an expense in handling increased levels of traffic that is destined for your network. This increase in traffic often has no new revenue associated with it, because it is fanning out to thousands of flat-rate consumer/small-business connections (e.g. DSL) where billing is generally by peak capacity not usage.

not true for cogent tho, we know that virtually all their traffic is usage based transit customers

The traffic from Cogent creates additional infrastructure requirements on L3. L3 may (or may not) be able to recover these costs as incremental revenue from the recipients, depending on the particulars of their agreements. One way of mitigating their exposure is to set an upper bound on uncompensated inbound traffic.

Mind you, this is entirely hypothetical, as specifics of the Cogent/ L3 agreement are not available. However, it is one way to let everyone "bill and keep" for Internet traffic without an unlimited exposure, and it is an approach that has been used successfully in the past.

Taking L3 & Cogent completely out of this discussion, I'm not sure I agree with your assessment. I think everyone agrees that unbalanced ratios can create a situation where one side pays more than the other. However, assuming something can be used to keep the costs equal (e.g. cold-potato?), I do not see how one network can tell another: "You can't send me what my customers are requesting of you." If your business model is to provide flat-rate access, it is not _my_ responsibility to ensure your customers do not use more access than your flat-rate can compensate you to deliver. -- TTFN, patrick

John Curran wrote:

Cold-potato only addresses the long-haul; there's still cost on the receiving network even if its handed off at the closest interconnect to the final destination(s).

And there's still revenue, as the traffic is going to customers (we all filter our prefixes carefully, right?). What's the problem with cold-potato again, or should we all just try to double-dip? pt

On Tue, Nov 01, 2005 at 11:16:58AM -0500, vijay gill wrote:

Pete Templin wrote:

...

John Curran wrote:

...

Cold-potato only addresses the long-haul; there's still cost on the receiving network even if its handed off at the closest interconnect to the final destination(s).

And there's still revenue, as the traffic is going to customers (we all filter our prefixes carefully, right?). What's the problem with cold-potato again, or should we all just try to double-dip?

pt

ah yes, double dipping. On-net traffic should be charged a lot less, because after all, it is double dipping.

I can almost smell your sarcasm from here. :) The problem here is that people naively assume all traffic is the same, and costs the same to deliver, which is just not the case. On-net traffic costs significantly more to deliver than outbound traffic, because you are virtually guaranteed that you are going to have to haul it somewhere at your expense. People expect their sub $10/Mbps transit pricing for all services across the board now, without understanding that those rates are ONLY sustainable because of negligible longhaul costs for the outbound traffic. On-net traffic is not "double dipping", it is the ony way that transit can be sold for a particular price. So does that mean that anyone with outbound heavy traffic is automatically taking advantage of a peer? Of course not, because while some types of traffic may indeed cost more to deliver, that traffic is usually *gasp* billed at a higher rate too. Other than spot markets like Cogent trying to prop up its ratios or a small tier 2/3 taking advantage of a 95th percentile billing trick to give away "free" inbound, I would challange folks to find ordinary markets where inbound traffic is not priced substantially higher than outbound, especially in areas outside of the "big tier 1 bandwidth cities". Numbers close to $100/Mbps (or higher) are still perfectly common on OC3's, even on cities which are on major longhaul fiber routes. Remember that content can be moved in order to reduce the cost, eyeballs can not. CDN's deliver bits to the right areas to bypass transport costs, and even ordinary folks choose where to install their servers in order to maximize quality and lower price. Content people who buy transit routinely put their servers at or near major ix facilities in order to get a lower price for the traffic ("hey look my content goes in and out the same pop, or even the same router"). Yes there is an associated cost to deliver access traffic to far-flung regions, but your customers are paying you a higher rate to do it too. So, what is inherently wrong with content customers paying $10/Mbps for a service which is substantially cheaper to provide, and the access customers paying $70/Mbps for the same thing? A lot of people seem to be taking the position of silent resentment towards the folks who are selling content heavy bandwidth at what can only be described as competetive market pricing (meaning, you can buy it at that price from almost anyone). They see such a large volume of traffic and think: a) crap, our network design can't possibly deliver that many bits at those prices in order to compete with them. and b) but man if we were billing all that at $70/Mbps we could, and we would be if not for that damn content-heavy network who is getting "free peering" in to our network in order to sell it for so cheap. We're paying more of the cost for that traffic than they are too, clearly we need to depeer them. Unfortunately they often do so without understanding the symbiotic relationship between the two kinds of traffic, and the two types of networks. If you look at a network like Cogent, it is designed from the ground up to be efficient and cheap at delivering bulk bits from a few customers at a few key points to the rest of the Internet, which is how Cogent is able to erm lose as little money as they do. Their network design looks almost nothing like a network who is optimized to deliver access circuits to a large number of smaller customers across a large number of locations, and it would be far less efficient at it if called upon to do so. In this case, jealousy is blinding a lot of people to the fact that there is room for networks who specialize in content to co-exist with networks who specialize in access, and for them both to add value to each other through interconnection. Specializing in a specific area leads to optimized network designs and reduced costs, and networks who don't may find that they aren't very good (or at least, cost competetive) at either. This naturally leads into two camps: 1) Networks who are more efficient, who end up paying a lot less, and who end up moving a very large amount of bits because of it (but at a much lower price/meg). 2) Networks who are less efficient, who pay a lot more, and who therefore have to charge their customers a lot more in order to survive. These networks face constant attrition from more competetive providers, and quickly realize that access to their single homed customers is one of their only bargining chips left to make people pay a higher price. Yes these are financially trying times for everyone, content and access networks alike. Everyone wants to "take action" to generate additional revenue, or to strike out at networks who are perceived as offering prices which are "too low" based on "dumping" of push heavy traffic and "unfair" cost burdons. Unfortunately in the rush to do that, many networks are actually creating exactly the situation they want to avoid. How much business do you think has been lost to Cogent giving away free or super-low-cost inbound in order to prop up its ratios? How much revenue has been generated by depeering Cogent because of ratio imbalances? The numbers speak for themselves, regardless of who does or doesn't pay a higher amount to deliver the traffic. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Wed, Nov 02, 2005 at 08:22:20AM -0600, Pete Templin wrote:

Time out here. John set the stage: cold potato addressed the long haul (or at least that's the assumption in place when I hopped on board). If NetA and NetB are honoring MED (or other appropriate knob), NetA->NetB traffic has already arrived at the closest mutual peering point in the A->B direction. The rest of the infrastructure would be the responsibility of NetB to get the traffic to CustomerPortXYZ, no? How could CustomerXY get any closer to NetA without cutting NetB out of the middle, and if NetB is out of the middle, why should CustomerXY pay NetB anything?

You're forgetting that MEDs suck. When used on real complex production networks, they almost always degrade the quality of the routing. Yes with enough time and energy (or a small enough network) you *can* beat perfect MEDs out of the system (and your customers). You can selectively deaggregate the hell out of your network, then you can zero out all the known aggregate blocks and regions that are in the middle of two MED-speaking interconnection points, and get your customers to tag aggregate blocks announced in multiple locations so that you can zero out those MEDs. With enough time and energy anything is possible, the point is that most folks don't consider it to be worth the time, let alone the customer anger when it degrades your traffic. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Wed, Nov 02, 2005 at 02:44:20PM -0600, Pete Templin wrote:

I came up with a reasonably scalable solution using communities and route-map continue, but:

For what value of "scalable"? --Jeff

Jeff Aitken wrote:

On Wed, Nov 02, 2005 at 02:44:20PM -0600, Pete Templin wrote:

...

I came up with a reasonably scalable solution using communities and route-map continue, but:

For what value of "scalable"?

For me, plenty, but a four-POP single-state network usually has different constraints on "scalable". However, I'd categorize it as one community-list per MED tier (i.e. if you just want near/far, that's two tiers, etc.) and one community-list entry per POP (or group of POPs, if you have some grouping logic embedded in your internal communities). pt

On Wed, Nov 02, 2005 at 05:13:27PM -0600, Pete Templin wrote:

For me, plenty, but a four-POP single-state network usually has different constraints on "scalable".

Right. On Wed, Nov 02, 2005 at 06:20:39PM -0500, Deepak Jain wrote:

I think Pete is saying that as long as you aren't a control-nazi, its a good system. :)

My point wasn't that his system doesn't work for him; presumably it does. My point was that in the context of global peering, what works for a small network may not (and probably does not) work for someone the size of, say, Level3. There are a lot of operational issues that arise if you listen to MEDs from peers. Apart from the "minor" issues like oscillations, ratchet-down, and packing inefficiencies, there is the fundamental problem that you will see potentially significant churn as a result of changes in your peers' networks. There is also the problem that there is no single "best" exit point for many large prefixes (e.g., if you peer with L3, there is no one "best" place to send traffic destined for something inside 4/8). --Jeff

On Nov 1, 2005, at 10:04 AM, John Curran wrote:

At 9:40 AM -0500 11/1/05, Patrick W. Gilmore wrote:

...

I think everyone agrees that unbalanced ratios can create a situation where one side pays more than the other. However, assuming something can be used to keep the costs equal (e.g. cold- potato?),

Cold-potato only addresses the long-haul; there's still cost on the receiving network even if its handed off at the closest interconnect to the final destination(s).

Which is COMPLETELY AND TOTALLY irrelevant to the peer network. If your network can't cover the cost of delivering bits from the DSLAM to the CPE, why in the hell are you in this business? You've been doing this for a very, very long time John. I know you know better. Stop trying to confuse the newbies.

...

I do not see how one network can tell another: "You can't send me what my customers are requesting of you."

Depeering seems to say exactly that, no?

Only if you are Cogent / L3 (or Cogent / FT, or Cogent / Teleglobe, or Cogent / $NEXT-DEPEER). Any other time a network gets de-peered, the bits still flow. So I repeat, how can an eyeball network tell a content provider: "You can't send me what my customers are requesting of you." The only way I can think to do that is to intentionally congest the path. (Which many eyeball networks actually do, now that I think about it.) But that might have an adverse affect on your customer growth.

...

If your business model is to provide flat-rate access, it is not _my_ responsibility to ensure your customers do not use more access than your flat-rate can compensate you to deliver.

Agreed... I'm not defending the business model, only pointing out that some folks may find it easier to bill their "peers" than customers.

I doubt they will succeed - at least in the long run, or even in the majority of cases. But stranger things have happened. Just remember, turn-about it fair play. So they should be careful what they wish for. -- TTFN, patrick

On Tue, 1 Nov 2005, John Payne wrote:

What am I missing?

That it's a pure power play. Peering is only distantly associated with costs or responsibilities. It has to do with what company has the intestinal fortitude to draw a line in the sand and stick with it no matter how many customers cancel their service. Those with a critical mass of traffic and the right amount of guts win. Everyone else loses the peering game. -- Brandon Ross AIM: BrandonNRoss Director, Network Engineering ICQ: 2269442 Internap Skype: brandonross Yahoo: BrandonNRoss

On Tue, 1 Nov 2005, Stephen J. Wilcox wrote:

On Tue, 1 Nov 2005, Brandon Ross wrote:

...

On Tue, 1 Nov 2005, John Payne wrote:

...

What am I missing?

That it's a pure power play.

market position is important

If by market position you are referring to who needs/wants/can do without the traffic more, yes.

...

Peering is only distantly associated with costs or responsibilities.

no, peering is entirely associated with costs or responsibilities.. what other reason is there to peer ?

I was probably being a bit too dramatic with that statement. What I'm trying to get across is that it doesn't matter who is "supposed to" pay for "their customers'" traffic. It doesn't matter that I have a million dialup users, if I can use my market position to get someone else to peer with me "for free" that's all that matters. The fact that those 1 million customers pay me is irrelevant.

...

It has to do with what company has the intestinal fortitude to draw a line in the sand and stick with it no matter how many customers cancel their service.

have to weigh up the gains and losses to see if that is a good or bad thing tho.

Of course.

...

Those with a critical mass of traffic and the right amount of guts win.

markets are always stacked in favour of the larger players in that way.. saying 'hey i'm a little guy, give me chance' generally goes unheard

Quite true.

...

Everyone else loses the peering game.

not peering isnt necessarily losing, there are networks who would peer with me if i turned up in asia or the west coast, but my cost to get there is greater than sticking to transit.

You don't have to tell me that, I work for Internap, we've made a business out of not peering, and doing quite well at it. I said "loses the peering game". I didn't say they lost the game in entirety. Similarly, just because a company "wins" the peering game (fully peered with all other default free networks) doesn't mean it wins the business game. Just take a look at a former employer of mine, 4006 was default free, but that doesn't mean that we made any money.

to get a new peer, both sides need to feel they are gaining value

Or one side needs to be more scared of the other side cutting them off. -- Brandon Ross AIM: BrandonNRoss Director, Network Engineering ICQ: 2269442 Internap Skype: brandonross Yahoo: BrandonNRoss

* John Payne:

That is something that has always confused me about ratio based peering disputes.

I don't understand them, either. However, if you define incoming traffic as "bad", it encourages depeering by the receiving side if the incoming/outgoing ratio exceeds a certain value, especially among close-to-tier-1 carriers: the traffic does not automatically disappear just because you depeer. Now suppose that the sending side doesn't want to play games and buys transit from one of your other peers. Given the tier-1 status, there is some chance that this has a measurable impact on the traffic ratio with that other peer. Essentially, this is a self-fulfilling prophecy, and it works equally well if you define outgoing traffic as "bad".

On Tue, 01 Nov 2005 11:46:20 EST, John Payne said:

That is something that has always confused me about ratio based peering disputes. Surely it is the responsibility of the content-sucking network to build and engineer to meet the demands of *their* customers (and build the cost of doing that into the pricing model). It appears to me that the content heavy networks are going above and beyond to work around the broken model that the content-suckers have.

What am I missing?

Obviously, the same thing that management at SBC is missing: http://www.marketwatch.com/news/story.asp?guid=%7B5A606A5A%2D18D7%2D4FC9%2DA... WASHINGTON (MarketWatch) --- The chief executive of SBC Communications Inc. thinks companies doing business on the Internet, such as Microsoft Corp. and Vonage Inc., are due for a wake-up call. "How do you think they're going to get to customers? Through a broadband pipe. Cable companies have them. We have them," said Ed Whitacre in a BusinessWeek Online interview. "What they would like to do is use my pipes for free. I ain't going to let them do that." He argued that because SBC and others have invested to build high-speed networks, they are due a return. "There's going to have to be some mechanism for these people ... to pay for the portion they're using. Why should they be allowed to use my pipes?" He offered no details how his idea could be accomplished. For an Internet company to "expect to use these pipes free is nuts!" Whitacre added for good measure.

Sounds like an extremely short-sighted view of the Net and it's economics. Claiming content providers should be charged for "using" broadband access-pipes is fine and dandy, but coveniently forgetting that without content there probably wouldn't be a great deal of customers wanting broadband in the first place is a bit sloppy, no?

while valid, this argument plays into the power play game. the key point is that the content providers already paid once for transport [0], as did the content users. we may need more gummint support to keep the rbocs from abusing their subsidized monopoly ownership of the last mile. two years ain't enough to get the cartel-minded out of control of the fcc. randy --- [0] - whether to transit upstreams or by deploying a large network themselves (aol)

Eric Louie wrote:

Now, one really needs to wonder why the agreement could not be reached *prior* to the depeering on 10/5

It's not rocket science.

As people have pointed out repeatedly, this was surely not rocket science since it wasn't a technical problem at all. It was a business conflict. It seems clear to me what probably happened. First-round negotaitions failed 'cause Level 3 thought Cogent was bluffing (and perhaps vice versa). Level 3 called the bluff, but it wasn't a bluff, and Level 3 then blinked (or so it appears from reading between the lines of what I've seen). They both got back to negotiation, and with a better understanding of to how much pain the other willing to take to get what they want, this time they came out with an agreement. Doesn't seems mysterious. [snip]

Who are the next discontent couples?

And how do I protect myself and my customers from any problems these kinds of events cause regardless of who the next players might be? -- Crist J. Clark crist.clark@globalstar.com Globalstar Communications (408) 933-4387