Patrick, To further your point about the dynamics of peering: Not to sound overly altruistic, but nowhere in there did I see, "it's good for the Internet". If peering was less of a raw business decision, the Internet would be a better place. In this case, if they left it status quo and congested, at least users could send smoke signals across the two networks and could at least communicate. The implications of this depeering could be pretty severe. If you dive into business ethics, there's some pretty serious moral dilemmas involved with cutting communications between two major networks. This could be taken to an extreme if it causes VoIP calls to fail and ultimately disrupts someone's 911 calling. In less reaching situations, someone can't SMS with their wife to know what to bring home from the grocery store. Regardless of how stupid relying on the Internet is (I know it's sad we can't) I do feel networks have a moral responsibility to provide continuity. As you know, beyond business, peering can decrease latency, increase throughput, and provide better network engineering, thus increasing the scale of the overall Internet. As a technologist and entrepreneur I try to do what's best for the Internet in my peering decisions rather than what Bill Lumbergh would say, "umm yeah, do what's best for the company". In this case, it's very clear that customers are impacted and the Internet as a whole suffers, which is really unfortunate. The end result of a business decision has been to sacrifice the customer's needs, trust, and ability to communicate. It's a bad maxim to subscribe to! I really hope that other networks do apply more thinking into peering than just what's best for business -- it sure shows off an ugly underbelly. -Barrett On Nov 1, 2008, at 9:45 AM, Patrick W. Gilmore wrote:

On Nov 1, 2008, at 12:05 PM, Chris Adams wrote:

...

Once upon a time, bas <kilobit@gmail.com> said:

...

I've heard eyeball networks refer to traffic flows as sending too.. "You content hosters are sending us too much traffic, we want money to upgrade ports and transport all that traffic" Complete reverse logic imho. It is always eyeball network customers that request data. (except for a small portion of iphone/blackberry push email, but that can't account for much.)

Traffic sources tend to be concentrated in large data centers (easier to service), while traffic sinks (DSL, cable, wireless) are widespread and costly to upgrade. The sink customers don't want to pay more (and there's at least some competition), so the sink providers look to see where else they get income to pay for their needed network upgrades.

Combined with hot-potato routing, the first part of that paragraph is a fancy way of saying "I have to carry the large packet a long way, you have to carry the small packet a long way". It is not "fair". This is almost a good reason, but not quite. (It can also be offset by moving the source next to the sink, through cold-potato routing / MEDs, anycast, CDNs, etc.)

The second part is a good business reason. Profitable revenue is good, costs are bad.

There are good business reasons not to pay the sink as well. But neither decision is obvious or the same for everyone.

Peering is complicated, people should stop trying to generalize it.

Peering is a business tool. For years & years many people have claimed that to "peer" you must be equal. Bullshit. If I can make more or spend less by peering, I should do it. If not, I should not. Full stop. Notice the complete lack of regard for how big you are, how much capacity your backbone has, how many ASes are downstream of you, etc.? When I go to buy routers or hire employees or any other business transaction, I don't say "that router vendor is making more money than I am, so I won't buy from him". If people applied "peering" logic to anything else, they'd be laughed out of a job.

Don't know about you, but I am in business to make money, not measure my anatomy. How big the next guy is doesn't enter into my equation - other than how it affects my bottom line.

To be clear, it is entirely possible that peering does not save you money. Vijay is right, most people can't measure their COGS to save their life. And if the network in question cannot, there's no way in hell the prospective peer can. If you are a huge point source of traffic and want to peer, I may save money by saying no and paying a transit provider to deliver the packet to me where I want it (especially at today's prices). Fiber, routers, colo, NOC employees, engineers, etc., are all not free ya know.

You can claim my customers asked for the data and therefore I have a requirement to peer, but you would be deluded. What my customer and I have agreed has _nothing_ to do with you or your needs. You don't tell me how to run my network, and I won't tell you how to run yours. Deal?

On the flip side, saying "you are not on 3 continents so I will not peer" is stupid of not peering costs you millions a year. Stupid decisions abound in the peering ecosystem.

There are tons of other _business_ reasons to peer, or not to peer. But "we're equal" or "your customer asked for it" are not reasons, stop using them.

-- TTFN, patrick

True... however this depeering may have created more of a mess for Sprint's marketing and their customers than they predicted, which has a negative impact on business and would not be fun to explain at a board meeting. I guess it's hard for sweater vests to understand that until it smacks them in the face. -Barrett On Nov 1, 2008, at 5:00 PM, Matthew Petach wrote:

On 11/1/08, Barrett Lyon <blyon@blyon.com> wrote: ...

...

In this case, it's very clear that customers are impacted and the Internet as a whole suffers, which is really unfortunate. The end result of a business decision has been to sacrifice the customer's needs, trust, and ability to communicate. It's a bad maxim to subscribe to! I really hope that other networks do apply more thinking into peering than just what's best for business -- it sure shows off an ugly underbelly.

-Barrett

Unfortunately, as I'm sure you're all too aware, for public companies, it's very hard to get away with saying "I was doing what was right for the Internet, not what would make my business the most money" at a shareholder meeting, or during an earning's call with Wall Street analysts; they tend to be very unforgiving of actions that aren't in line with the short-term profit-making goal, to the point where CEOs have been ousted and class-action lawsuits threatened when it seems the actions being taken weren't geared to optimize profits for the shareholders.

Converging two threads together, I think the same pressure affects IPv6 deployment and will affect IPv6 peering; while it would be *best* for the Internet if everyone put the time and resources forward to get dual-stacked now, bring up widespread peering, and get IPv6 to a point where it's a viable transport mechanism, the real fact of the matter is that there's no profit motive in moving to IPv6 at the moment, so people just aren't going to do it, no matter how much it may be "best for the Internet".

Fix the market drivers for public companies, and then we can fix the Internet; otherwise, we'll always be steering towards that which makes sense for the business, regardless of which customers of other networks it hurts, or which resource exhaustion cliff it hurtles us towards.

Putting on a devil's advocate hat for a moment...if various international regulatory bodies and government agencies mandated universal connectivity via both IPv4 and IPv6, depeering would cease to be an issue; regardless of what the business side said, companies would not be allowed to partition the Internet, and widespread adoption of IPv6 would be forced, rather than being a "maybe someday" case. Downside is that prices would go up, and expansion into new regions would slow down, as the costs associated with developing new areas would bring a much higher price tag. It would be better for the Internet as a whole, but worse for most of the individual users of it, from a cost perspective. Would you still say things were better for the overall Internet at that point, if it meant everyone had to pay more in order to ensure that universal connectivity?

I'm cheap, so I'm leaning towards the side of letting the market work these issues out; but I'm always willing to listen to other thoughts on the matter. :)

Matt

Lamar Owen wrote:

On Saturday 01 November 2008 20:00:46 Matthew Petach wrote:

...

Unfortunately, as I'm sure you're all too aware, for public companies, it's very hard to get away with saying "I was doing what was right for the Internet, not what would make my business the most money" at a shareholder meeting, or during an earning's call with Wall Street analysts; they tend to be very unforgiving of actions that aren't in line with the short-term profit-making goal, to the point where CEOs have been ousted and class-action lawsuits threatened when it seems the actions being taken weren't geared to optimize profits for the shareholders.

The next Great Compromise could be over multihoming of endusers (whether the enduser is a content consumer or a content provider is irrelevant; IP at Layer 3 is peer-to-peer anyway, between end systems). After all, if providers are not willing to 'budge' a little 'for the good of the Internet' then why should endusers reduce or give up multihoming aspirations 'for the good of the Internet?'

After all, if access to the 'Internet' is a metric for making money, and money is being lost by access to content consumers being cut due to depeering events outside the enduser's control, then it makes business sense for an enduser to multihome, to bring connectivity events closer to the enduser's control. Want to prevent multihoming exploding routing tables? Prevent the desire to multihome by limiting 'partitioning' events.

So now I'm going to join Randy and say this is rehashing. We know the "interwebz" are a collection of AS. We know they can partition at any time. We know that certain players have a history of causing this to happen more then others. What I haven't seen discussed in any great detail, is how to limit those events. Some are asking for regulation. Ok. What will that regulation look like? As the technical network operations community, what do we want to see? What stipulations etc? Are a few interruptions that affect a small subset of IP addresses and AS on the net really worth going through the effort to pass regulation around the world? If so, then let's take advantage of a long and rich history of reaching consensus (see the RFC process which I think has work pretty well) and hash out what we want to see. Otherwise let's move on. Charles

I really doubt Sprint's purpose here is to "hurt the Internet" or to harm Cogent either in terms of costs or reputation. Here are my views on the topic: Every time Cogent gets de-peered (at least 5 times now since 2003), this discussion comes up again and it seems that some people forget (or don't know) how many times it's happened to them before. There must be a reason it keeps happening, right? Are there any other large ISPs that have had this type of problem 5 times? As someone was saying earlier, in the PSTN world carriers generally pay for every call terminated to another carrier's network and pay each other back and forth. In IP peering, these types of costs are "eliminated" by settlement-free peering relationships where carriers feel there is a benefit to do so. These are relationships or contracts between the two carriers, and most of us have no idea how these are written or what clauses are included about how and when one carrier can end that contract. Regardless of the exact terms, there will certainly be actions or other situations that would be viewed as a breach of contract, resulting in ending or changing the relationship. In the case of Cogent, they seem to want to be a Tier 1 carrier (usually loosely defined as an carrier that does not pay for transit or access from/to any other carrier), but they are not usually considered one by many in the industry. Technically at this point they are not since they are believed to pay Level 3 and Sprint. Now I really can't speak to exactly why each carrier that has de-peered Cogent in the past has done so, but based on conversations I've had with higher-ups at one of these ISPs, their major issue with Cogent was a huge discrepancy in the volume of inbound vs. outbound traffic. To that carrier, based on the traffic patterns, they believed that Cogent should be paying for their connections and was not keeping to the spirit of their relationship or breaking the contract if there was one. They supposedly attempted for some time to resolve the issue amicably, but when that failed they chose to take action as a last chance to resolve the dispute to their liking. Now as to the "harmful" effect to Cogent's customers, that effect would be easily mitigated if Cogent would choose to buy transit from any other ISP. Instead, they try to avoid that by offering affected customers free circuits for some period of time, which hopefully turn into paying customers at a later date. Also, anyone running any important site or network knows never to be single-homed, and therefore should not be effected in the long run. Anyone single homed accepts the risks associated with that by not having redundant connections, especially if that single home is Cogent based on their history of peering arguments. So based on that the only "difference" I'd expect this to make is in the relationship between Sprint and Cogent in the future. I doubt this will change Sprint's, Cogent's, or any other ISP's corporate views/policies on peering in the long term. Just my 2 cents, -Scott -----Original Message----- From: Matthew Moyle-Croft [mailto:mmc@internode.com.au] Sent: Saturday, November 01, 2008 10:07 PM To: bas Cc: nanog@nanog.org Subject: Re: Sending vs requesting. Was: Re: Sprint / Cogent bas wrote:

Why does everyone keep referring to traffic flows as sendng? In this case it's not as if Cogent just randomly sends data to Sprint.

I think it's a really odd reinterpretation of telephony concepts. In telephony interconnects are typically settlement based, sender pays receiver, in the settlement based world it seems to have gotten confused. I'm still trying to come to terms with what Sprint is trying to achieve here. I can only assume it's (and I'm stealing from Vijay here) to raise Cogent's cost of doing business by forcing them to do settlement based or paid peering and thus trying to force the cost of their transit to rise. Maybe it's to damage Cogent's reputation as well? The cost of doing this seems to be high (ie. upsetting high paying (single homed) transit and mobile customers) and getting negative media coverage. Is this really going to make a substantial kind of difference? MMC -- Matthew Moyle-Croft - Internode/Agile - Networks