Even better by Verizon - http://publicpolicy.verizon.com/blog/entry/unbalanced-peering-and-the-real-s...

Some may recognize the name of the author for the WSJ article given she attended NANOG in Orlando - http://online.wsj.com/article_email/SB10001424127887323836504578553170167992...

...

http://gigaom.com/2013/06/19/peering-pressure-the-secret-battle-to-control-t...

as someone who does not really buy the balanced traffic story, some are eyeballs and some are eye candy and that's just life, seems like a lot of words to justify various attempts at control, higgenbottom's point. randy

On Wed, Jun 19, 2013 at 7:12 PM, Blake Dunlap <ikiris@gmail.com> wrote:

Verizon wishes money to accept data it requested from other vendors, film at 11.

The phrase you're looking for is, "double billing." Same byte, two payers. -Bill -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Wed, Jun 19, 2013 at 06:39:48PM -0500, Leo Bicknell wrote:

On Jun 19, 2013, at 6:03 PM, Randy Bush <randy@psg.com> wrote:

...

as someone who does not really buy the balanced traffic story, some are eyeballs and some are eye candy and that's just life, seems like a lot of words to justify various attempts at control, higgenbottom's point.

I agree with Randy, but will go one further.

Requiring a balanced ratio is extremely bad business because it incentivizes your competitors to compete in your home market.

You're a content provider who can't meet ratio requirements? You go into the eyeball space, perhaps by purchasing an eyeball provider, or creating one.

Google Fiber, anyone?

Having a requirement that's basically "you must compete with me on all the products I sell" is a really dumb peering policy, but that's how the big guys use ratio.

At the end of the day though, this comes down to a clash of business models and the reason why it's a public spectacle, and of public policy interest is due to the wide spread legacy of monopoly driven public investment in the last mile infrastructure. -dorian

Hi Wayne, Another important point not to be missed is that these days, thanks to CDN technology, a heavy inbound ratio does not necessarily indicate a high cost burden like it did pre-CDN tech. Even more ironically, the unwillingness of a peer to upgrade connections due to the ratio excuse results in the CDN having to source traffic from non-optimal locations just to get the bits into the other network, thereby increasing the cost burden of the broadband network. If it were true that these issues were only about cost there would be plenty of common ground to negotiate acceptable peering terms, don't you think? Dave -----Original Message----- From: Wayne E Bouchard [mailto:web@typo.org] Sent: Wednesday, June 19, 2013 6:03 PM To: Dorian Kim Cc: North American Network Operators' Group Subject: Re: net neutrality and peering wars continue On Wed, Jun 19, 2013 at 07:44:15PM -0400, Dorian Kim wrote:

On Wed, Jun 19, 2013 at 06:39:48PM -0500, Leo Bicknell wrote:

...

On Jun 19, 2013, at 6:03 PM, Randy Bush <randy@psg.com> wrote:

...

as someone who does not really buy the balanced traffic story, some are eyeballs and some are eye candy and that's just life, seems like a lot of words to justify various attempts at control, higgenbottom's point.

I agree with Randy, but will go one further.

Requiring a balanced ratio is extremely bad business because it incentivizes your competitors to compete in your home market.

You're a content provider who can't meet ratio requirements? You go into the eyeball space, perhaps by purchasing an eyeball provider, or creating one.

Google Fiber, anyone?

Having a requirement that's basically "you must compete with me on all the products I sell" is a really dumb peering policy, but that's how the big guys use ratio.

At the end of the day though, this comes down to a clash of business models and the reason why it's a public spectacle, and of public policy interest is due to the wide spread legacy of monopoly driven public investment in the last mile infrastructure.

-dorian

At the risk of inflaming passions, I'll share my opinion on this whole topic and then disappear back into my cubicle. For my part, peering ratios never made sense anyway except in the pure transit world. I mean, content providers are being punished by eyeball networks because the traffic is one way. Well, DUH! But everyone overlooks two simple facts: 1) Web pages don't generate traffic, users do. Content sits there taking up disk space until a user comes to grab it. (Not quite the case with data miners such as Google, but you get the idea.) 2) Users would not generate traffic unless there were content they want to access. Whether that is web pages, commerce pages such as Amazon or ebay, streams, or peer-to-peer game traffic, if there's nothing interesting, there's nothing happening. So both sides have an equal claim to "it's all your fault" and one seeking to punish the other is completely moronic. Traffic interchange is good. Period. It puts the users closer to the content and the content closer to the user and everyone wins. So I never once understood why everyone was all fired up about ratios. It just never made any sense to me from the get-go. To have government get into this will certainly not help the problem, it will just make it a hundred times worse. Remember the old saying that the eight most terrifying words in the English language are, "I'm from the government. I'm here to help." and boy will they try to "help". You'll be lucky if you as a company can keep still your doors open after they get done "helping" you. Anyhow, just my two bits. -Wayne --- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/

On Jun 19, 2013, at 7:31 PM, Benson Schliesser <bensons@queuefull.net> wrote:

What do you mean "not really buy the balanced traffic story"? Ratio can matter when routing is asymmetric. (If costs can be approximated as distance x volume, forwarding hot-potato places a higher burden on the recipient...) And we've basically designed protocols that route asymmetrically by default. Measuring traffic ratios is the laziest solution to this problem, and thus the one we should've expected.

That was a great argument in 1993, and was in fact largely true in system that existed at that time. However today what you describe no longer really makes any sense. While it is technically true that the protocols favor asymmetric routing, your theory is based on the idea that a content site exists in one location, and does not want to optimize the user experience. That really doesn't describe any of the large sources/sinks today. When you access "www.majorwebsite.com" today a lot of science (hi Akamai!) goes into directing users to servers that are close to them, trying to optimize things like RTT to improve performance. Content providers are generally doing the exact opposite of hot potato, they are cold potatoing entire racks into data centers close to the eyeballs at great cost to improve performance. But to the extent a few people still have traffic patterns where they can asymmetrically route a large amount of traffic, the situation has also changed. In 1993 this was somewhat hard to detect, report, and share. Today any major provider has a netflow infrastructure where they can watch this phenomena in real time, no one is pulling the wool over their eyes. There are also plenty of fixes, for instance providers can exchange MED's to cold potato traffic, or could charge a sliding fee to recover the supposed differences. The denial of peering also makes bad business sense from a dollars perspective. Let's say someone is asymmetric routing and causing an eyeball network extra long haul transport. Today they deny them peering due to ratio. The chance that the content network will buy full-priced transit from the eyeball network? Zero. It doesn't happen. Instead they will buy from some other provider who already has peering, and dump off the traffic. So the eyeball network still gets the traffic, gets it hidden in a larger traffic flow where they can't complain if it comes from one place, and get $0 for the trouble. A much better business arrangement would be to tie a sliding fee to the ratio. Peering up to 2:1 is free. Up to 4:1 is $0.50/meg, up to 6:1 is $1.00/meg, up to 10:1 is $1.50 a meg. Eyeball network gets to recover their long haul transport costs, it's cheaper to the CDN than buying transit, and they can maintain a direct relationship where they can keep up with each other using things like Netflow reporting. While I'm sure there's some network somewhere that does a sane paid peering product like this, I've sure never seen it. For almost all networks it's a pure binary decision, free peering or full priced transit. Quite frankly, if the people with MBA's understood the technical aspects of peering all of the current peering policies would be thrown out, and most of the peering coordinators fired. "Settlement" is a dirty word in the IP realm, but the basic concept makes sense. What was a bad idea was the telco idea of accounting for every call, every bit of data. Remember AT&T's 900 page iPhone bills when they first came out? Doing a settlement based on detailed traffic accounting would be stupid, but doing settlements based on traffic levels, and bit-mile costs would make a lot of sense, with balanced traffic being free. Oh, and guess what, if people interconnected between CDN and eyeball networks better the users would see better experiences, and might be more likely to be satisfied with their service, and thus buy more. It's good business to have a product people like. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/

Well, with net flow Analytics, it's not really the case that we don't have a way of evaluating the relative burdens. Every major net flow Analytics vendor is implementing some type of distance measurement capability so that each party can calculate not only how much traffic they carry for each peer, but how far. Dave -- 520.229.7627 cell On Jun 19, 2013, at 8:23 PM, "Benson Schliesser" <bensons@queuefull.net> wrote:

On 2013-06-19 8:46 PM, Leo Bicknell wrote:

...

That was a great argument in 1993, and was in fact largely true in system that existed at that time. However today what you describe no longer really makes any sense.

While it is technically true that the protocols favor asymmetric routing, your theory is based on the idea that a content site exists in one location, and does not want to optimize the user experience. ...

A much better business arrangement would be to tie a sliding fee to the ratio. Peering up to 2:1 is free. Up to 4:1 is $0.50/meg, up to 6:1 is $1.00/meg, up to 10:1 is $1.50 a meg. Eyeball network gets to recover their long haul transport costs, it's cheaper to the CDN than buying transit,

Agreed that CDN, traffic steering, etc, changes the impact of routing protocols. But I think you made my point. The sending peer (or their customer) has more control over cost. And we don't really have a good proxy for evaluating relative burdens.

That's not to suggest that peering disputes are really about technical capabilities. Nor fairness, even...

Cheers, -Benson

On Jun 19, 2013, at 23:41, "Siegel, David" <David.Siegel@level3.com> wrote:

Well, with net flow Analytics, it's not really the case that we don't have a way of evaluating the relative burdens. Every major net flow Analytics vendor is implementing some type of distance measurement capability so that each party can calculate not only how much traffic they carry for each peer, but how far.

Admittedly, it's been a few years since I looked at such tools... So please help me understand: does the tool evaluate distance (and therefore burden) as it extends into the peer's network, or just into the local network? And in either case, is this kind of data normalized and shared between peers? It seems like there could be a mechanism here to evaluate fairness of burdens, but I'm skeptical that these tools are used in such a way. I'd be glad to be incorrect. ;) Cheers, -Benson

The tools cannot estimate burden into the peers network very well, particularly when longest-exit routing is implement to balance the mileage burden, so each party shares their information with each other and compares data in order to make decisions.

It's not common, but there are a handful of peers that share this information with each other.

i have not been able to find it easily, but some years back rexford and others published on a crypto method for peers to negotiate traffic adjustment between multiple peering points with minimal disclosure. it was a cool paper. randy

On Jun 20, 2013, at 5:47 PM, Robert M. Enger <NANOG@enger.us> wrote:

Perhaps last-mile operators should A) advertise each of their metropolitan regional systems as a separate AS B) establish an interconnection point in each region where they will accept traffic destined for their in-region customers without charging any fee

C) Buck up and carry the traffic their customers are paying them to carry. Least I just sound like a complainer, I actually think this makes rational business sense. The concept of peering was always "equal benefit", not "equal cost". No one ever compares the price of building last mile transport to the cost of building huge data centers all over with content close to the users. The whole "bit-mile" thing represents an insignificant portion of the cost, long haul (in large quantities) is dirt cheap compared to last mile or data center build costs. If you think of a pure content play peering with a pure eyeball play there is equal benefit, in fact symbiosis, neither could exist without the other. The traffic flow will be highly asymmetric. Eyeball networks also artificially cap their own ratios with their products. Cable and DSL are both 3x-10x down, x up products. Their TOS policies prohibit running servers. Any eyeball network with a asymmetric edge technology and no-server TOS need only look in the mirror to see why their aggregate ratio is hosed. Lastly, simple economics. Let's theorize about a large eyeball network with say 20M subscribers, and a large content network with say 100G of peering traffic to go to those subscribers. * Choice A would be to squeeze the peer for bad ratio in the hope of getting them to pay for, or be behind some other transit customer. Let's be generous and say $3/meg/month, so the 100G of traffic might generate $300,000/month of revenue. Let's even say you can squeeze 5 CDN's for that amount, $1.5M/month total. * Choice B would be to squeeze the subscribers for more revenue to carry the 100G of "imbalanced traffic". Perhaps an extra $0.10/sub/month. That would be $2M/month in extra revenue. Now, consider the customer satisfaction issue? Would your broadband customers pay an extra $0.10 per month if Netflix and Amazon streaming never went out in the middle of a movie? Would they move up to a higher tier of service? A smart end user ISP would find a way to get uncongested paths to the content their users want, and make it rock solid reliable. The good service will more than support not only cost recovery, but higher revenue levels than squeezing peers. Of course we have evidence that most end user ISP's are not smart, they squeeze peers and have some of the lowest customer satisfaction rankings of not just ISP's, but all service providers! They want to claim consumers don't want Gigabit fiber, but then congest peers so badly there's no reason for a consumer to pay for more than the slowest speed. Squeezing peers is a prime case of cutting off your nose to spite your face. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/

Maybe someone could enlighten my ignorance on this issue. Why is there a variable charge for bandwidth anyways? In a very simplistic setup, if I have a router that costs $X and I run a $5 CAT6 cable to someone elses router which cost them $Y, plus a bit of maintenance time to set up the connections, tweak ACLs, etc... So now there's an interconnect between two providers at 1 gigabit, and the only issue I see is the routers needing to be replaced within Z years when it dies or when it needs to handle a 10 gigabit connection. So it seems I should be able to say "Here's a 1 gigabit connection. It will cost $Q over Z years or you can pay $Q/Z yearly", etc... And wouldn't the costs go down if I had a bunch of dialup/DSL/cable/fiber users as they are paying to lower the costs of interconnects so they get content with less latency and fewer bottlenecks? -A On Thu, Jun 20, 2013 at 4:18 PM, Leo Bicknell <bicknell@ufp.org> wrote:

On Jun 20, 2013, at 5:47 PM, Robert M. Enger <NANOG@enger.us> wrote:

...

Perhaps last-mile operators should A) advertise each of their metropolitan regional systems as a separate AS B) establish an interconnection point in each region where they will accept traffic destined for their in-region customers without charging any fee

C) Buck up and carry the traffic their customers are paying them to carry.

Least I just sound like a complainer, I actually think this makes rational business sense.

The concept of peering was always "equal benefit", not "equal cost". No one ever compares the price of building last mile transport to the cost of building huge data centers all over with content close to the users. The whole "bit-mile" thing represents an insignificant portion of the cost, long haul (in large quantities) is dirt cheap compared to last mile or data center build costs. If you think of a pure content play peering with a pure eyeball play there is equal benefit, in fact symbiosis, neither could exist without the other. The traffic flow will be highly asymmetric.

Eyeball networks also artificially cap their own ratios with their products. Cable and DSL are both 3x-10x down, x up products. Their TOS policies prohibit running servers. Any eyeball network with a asymmetric edge technology and no-server TOS need only look in the mirror to see why their aggregate ratio is hosed.

Lastly, simple economics. Let's theorize about a large eyeball network with say 20M subscribers, and a large content network with say 100G of peering traffic to go to those subscribers.

* Choice A would be to squeeze the peer for bad ratio in the hope of getting them to pay for, or be behind some other transit customer. Let's be generous and say $3/meg/month, so the 100G of traffic might generate $300,000/month of revenue. Let's even say you can squeeze 5 CDN's for that amount, $1.5M/month total.

* Choice B would be to squeeze the subscribers for more revenue to carry the 100G of "imbalanced traffic". Perhaps an extra $0.10/sub/month. That would be $2M/month in extra revenue.

Now, consider the customer satisfaction issue? Would your broadband customers pay an extra $0.10 per month if Netflix and Amazon streaming never went out in the middle of a movie? Would they move up to a higher tier of service?

A smart end user ISP would find a way to get uncongested paths to the content their users want, and make it rock solid reliable. The good service will more than support not only cost recovery, but higher revenue levels than squeezing peers. Of course we have evidence that most end user ISP's are not smart, they squeeze peers and have some of the lowest customer satisfaction rankings of not just ISP's, but all service providers! They want to claim consumers don't want Gigabit fiber, but then congest peers so badly there's no reason for a consumer to pay for more than the slowest speed.

Squeezing peers is a prime case of cutting off your nose to spite your face.

-- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/

On 6/20/2013 10:26 PM, Jared Mauch wrote:

Many things aren't as obvious as you state above. Take for example routing table growth. There's going to be a big boom in selling routers (or turning off full routes) when folks devices melt at 512k routes in the coming years.

Indeed. We're running PFC3CXL's and had already reallocated FIB TCAM to 768K IPv4s in anticipation. We also had maximum-prefix 500000 with a warning at 90%, and today it triggered (or at least first time I noticed it)... we ran > 450K prefixes from 3 providers about 1:30 EDT today and got the warnings. The end is near :) If you haven't made provisions, please do so now :) Jeff

On Thu, Jun 20, 2013 at 6:47 PM, Robert M. Enger <NANOG@enger.us> wrote:

Perhaps last-mile operators should A) advertise each of their metropolitan regional systems as a separate AS B) establish an interconnection point in each region where they will accept traffic destined for their in-region customers without charging any fee

What would be the point of (A)? They can just set a BGP community based on where a route originates and then match by BGP community for the sufficiently-local routes when they peer. They don't do B because the complaint about "you're abusing my long haul bandwidth" is basically a lie. They want to get paid twice for each byte and if they think they can then they won't do settlement free peering with you. Period. On Thu, Jun 20, 2013 at 9:10 PM, Aaron C. de Bruyn <aaron@heyaaron.com> wrote:

Maybe someone could enlighten my ignorance on this issue. Why is there a variable charge for bandwidth anyways?

Some equipment is used to connect to you. A cable. A port on a card in a router. Whatever. Also, that router can only have so many ports, so when you connect to it a fraction of that router's equipment, maintenance and management cost is attributable to your specific connection. That's the monthly port charge. Your packets are then multiplex with lots of other folks' packets an a variety of cables and through a variety of routers as they travel between you and the machines you're talking to. That infrastructure has a cost $X. It's used by all of their customers (packets cross it) at a total of Y gbps. Your consumption divided by Y is your fraction of that usage. That fraction times $X is the service provider's variable cost of moving your packets. Variable cost, variable charge. On Thu, Jun 20, 2013 at 11:42 PM, Jon Lewis <jlewis@lewis.org> wrote:

At this rate, if they do produce a PFC that takes the 6500 to several million routes, it's probably going to be too late for those to be available in any real quantity on the secondary market. Maybe that's the plan.

"Maybe"? Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On 20 June 2013 13:07, Bill Woodcock <woody@pch.net> wrote:

On Jun 19, 2013, at 7:21 PM, Benson Schliesser <bensons@queuefull.net> wrote:

...

The sending peer (or their customer) has more control over cost.

I'll assume that, by "sending peer," you mean the content network. If so, I disagree. The content network has no control whatsoever over the location of the eyeball customer. The eyeball customer has sole control over his or her own location, while the content network has sole control over the location from which they reply to requests.

Therefore, control is shared between the two sides. And both are incentivized to minimize costs. If both minimize their costs, overall costs are minimized. That's why this system works.

I think his point was that the receiving side can massage their BGP announcements all they like but the sending network has more instantaneous control over how the traffic will flow. This is before analysis, communication, application of policies / contractual arrangements, de-peering etc.etc. kick in. cheers Marty

On Jun 20, 2013, at 5:37 AM, Benson Schliesser <bensons@queuefull.net> wrote:

Right. By "sending peer" I meant the network transmitting a packet, unidirectional flow, or other aggregate of traffic into another network. I'm not assuming anything about whether they are offering "content" or something else - I think it would be better to talk about peering fairness at the network layer, rather than the business / service layer.

In that case, it's essentially never an issue, since essentially every packet in one direction is balanced by a packet in the other direction, so rotational symmetry takes care of the "fairness." I think you may be taking your argument too far, though, since by this logic, the sending and receiving networks also have control over what they choose to transit and receive, and I think that discounts too far the reality that it is in fact the _customers_ that are making all of these decisions, and the networks are, in the aggregate, inflexible in their need to service customers. What a customer will pay to do, a service provider will take money to perform. It's not really service providers (in aggregate) making these decisions. It's customers. -Bill

On Thu, 20 Jun 2013 22:39:56 +0200, Niels Bakker said:

You're mistaken if you think that CDNs have equal number of packets going in and out.

And even if the number of packets match, there's the whole "1500 bytes of data, 64 bytes of ACK" thing to factor in...

On 22/06/13 13:08, Matthew Petach wrote:

On Thu, Jun 20, 2013 at 2:29 PM, <Valdis.Kletnieks@vt.edu> wrote:

...

On Thu, 20 Jun 2013 22:39:56 +0200, Niels Bakker said:

...

You're mistaken if you think that CDNs have equal number of packets going in and out. And even if the number of packets match, there's the whole "1500 bytes of data, 64 bytes of ACK" thing to factor in...

That's easily solved by padding the ACK to 1500 bytes as well.

Matt

Or indeed by the media player sending large amounts of traffic back to the CDN via auxiliary HTTP POST requests? Neil

On 22/06/13 16:34, Owen DeLong wrote:

...

...

That's easily solved by padding the ACK to 1500 bytes as well.

Matt

Or indeed by the media player sending large amounts of traffic back to the CDN via auxiliary HTTP POST requests?

Neil

That would assume that the client has symmetrical upstream bandwidth over which to send such datagrams. At least in the US, that is the exception, not the rule.

Owen

Hi Owen, You only need to match the video stream bandwidth, not the full download speed of the link. Given that current multicore CPUs are now fast enough to decode HEVC in software, and with HEVC being roughly twice as efficient as H.264, that means you should be able to do quite decent full HDTV quality video with an average bandwdith of about 5 Mbps, given sufficient buffering to smooth out the traffic. Less, if you're willing to compromise on picture quality a bit, and go for, say, 720p. So, given an HEVC-capable decoder, this strategy should work for any connection with an upstream speed of better than about 4 to 5 Mbps, which is becoming more and more common on cable Internet service, as DOCSIS 3.0 is rolled out and faster links become more common, Neil

On Sat, Jun 22, 2013 at 9:19 AM, Neil Harris <neil@tonal.clara.co.uk> wrote:

On 22/06/13 13:08, Matthew Petach wrote:

...

That's easily solved by padding the ACK to 1500 bytes as well.

Matt

Or indeed by the media player sending large amounts of traffic back to the CDN via auxiliary HTTP POST requests?

ah... botnet... how I love thee?

When you convert your botnet to a business model, you have to change the name. If it's a business, the politically correct term is "Elastic Cloud Computing" Owen On Jun 22, 2013, at 6:19 PM, jim deleskie <deleskie@gmail.com> wrote:

Botnets to help with peering ratio's could be a new business model? :)

On Sat, Jun 22, 2013 at 1:00 PM, Christopher Morrow <morrowc.lists@gmail.com

...

wrote:

...

...

On 22/06/13 13:08, Matthew Petach wrote:

...

That's easily solved by padding the ACK to 1500 bytes as well.

Matt

Or indeed by the media player sending large amounts of traffic back to

On Sat, Jun 22, 2013 at 9:19 AM, Neil Harris <neil@tonal.clara.co.uk> wrote: the

...

CDN via auxiliary HTTP POST requests?

ah... botnet... how I love thee?

* owen@delong.com (Owen DeLong) [Thu 20 Jun 2013, 23:38 CEST]:

On Jun 20, 2013, at 10:39 PM, Niels Bakker <niels=nanog@bakker.net> wrote:

...

* woody@pch.net (Bill Woodcock) [Thu 20 Jun 2013, 16:59 CEST]:

...

On Jun 20, 2013, at 5:37 AM, Benson Schliesser <bensons@queuefull.net> wrote:

...

Right. By "sending peer" I meant the network transmitting a packet [...] every packet in one direction is balanced by a packet in the other direction

You're mistaken if you think that CDNs have equal number of packets going in and out.

They are roughly equal (modulo delayed acks, etc.). However, the number of octets is very different from the number of packets. There is much greater asymmetry in number of octets than in number of packets.

Thank you, Captain Obvious. Also, if you don't have data, best to keep your opinion to yourself, because you might well be wrong. -- Niels.

On Jun 20, 2013, at 1:39 PM, Niels Bakker <niels=nanog@bakker.net> wrote:

You're mistaken if you think that CDNs have equal number of packets going in and out.

I'm aware that neither the quantity nor the size of packets in each direction are equal. I'm just hard-pressed to think of a reason why this matters, and so tend to hand-wave about it a bit… To a rough approximation, flows are balanced. Someone requests something, and an answer follows. Requests tend to be small, but if someone requests something large, a large answer follows. Conversely, people also send things, which are followed by small acknowledgements. Again, this only matters if you place a great deal of importance both on the notion that size equals fairness, and that fairness is more important than efficiency. I would argue that neither are true. I'm far more interested in seeing the cost of Internet service go down, than seeing two providers saddled with equally high costs in the name of fairness. And costs go down most quickly when each provider retains the full incentivization of its own ability to minimize costs. Not when they have to worry about "fairness" in an arbitrary metric, relative to other providers. The only occasion I can think of when traffic flows of symmetric volume have an economic benefit are when a third party is imposing excess rent on circuits, such that the cost of upgrading capacity is higher than the cost of "traffic engineering" flows to fill reverse paths. And that's hardly the sort of mental pretzels I want carriers to be having to worry about, instead of moving bits to customers.

I think the point is here that networks are nudging these decisions by making certain services suck more than others by way of preferential network access.

I agree completely that that's the problem. But it didn't appear to be what Benson was talking about. -Bill

On Jun 21, 2013, at 4:20 PM, Benson Schliesser <bensons@queuefull.net> wrote:

On 2013-06-21 4:54 AM, Bill Woodcock wrote:

...

Again, this only matters if you place a great deal of importance both on the notion that size equals fairness, and that fairness is more important than efficiency. ...

...

I think the point is here that networks are nudging these decisions by making certain services suck more than others by way of preferential network access. I agree completely that that's the problem. But it didn't appear to be what Benson was talking about.

It's clear to me that you don't understand what I've said. But whether you're being obtuse or simply disagreeing, there is little value in repeating my specific points. Instead, in hope of encouraging useful discussion, I'll try to step back and describe things more broadly.

The behaviors of networks are driven (in almost all cases) by the needs of business. In other words, decisions about peering, performance, etc, are all driven by a P&L sheet.

This isn't exactly true and it turns out that the subtle difference from this fact is very important. They are driven not by a P&L sheet, but by executive's opinions of what will improve the P&L sheet. There is ample evidence that promiscuous peering can actually reduce costs across the board and increase revenues, image, good will, performance, and even transit purchases. There is also evidence that turning off peers tends to hamper revenue growth, degrade performance, create a negative image for the organization, reduce good will, etc. One need look no further than the history of SPRINT for a graphic example. In the early 2000's when SPRINT started depeering, they were darn near the epicenter of internet transit. Today, they're yet another also ran among major telco-based ISPs. Sure, their peering policy alone is likely not the only cause of this decline in stature, but it certainly contributed.

So, clearly, these networks will try to minimize their costs (whether "fair" or not). And any imbalance between peers' cost burdens is an easy target. If one peer's routing behavior forces the other to carry more traffic a farther distance, then there is likely to be a dispute at some point - contrary to some hand-wave comments, carrying multiple gigs of traffic across the continent does have a meaningful cost, and pushing that cost onto somebody else is good for business.

Reasonable automation means that it costs nearly nothing to add peers at public exchange points once you are present at that exchange point. The problem with looking only at the cost of moving the bits around in this equation is that it ignores where the value proposition for delivering those bits lies. In reality, if an eyeball ISP doesn't maintain sufficient peering relationships to deliver the traffic the eyeballs are requesting, the eyeballs will become displeased with said ISP. In many cases, this is less relevant than it should be because the eyeball network is either a true monopoly, an effective monopoly (30/10Mbps cable vs. 1.5Mbps/384k DSL means that cable is an effective monopoly for all practical purposes), or a duopoly where both choices are nearly equally poor. In markets served by multiple high speed providers, you tend to find that consumers gravitate towards the ones that don't engage in peering wars to the point that they degrade service to those customers. On the other hand, if a content provider does not maintain sufficient capacity to reach the eyeball networks in a way that the eyeball networks are willing to accept said traffic, the content provider is at risk of losing subscribers. Since content tends to have many competitors capable of delivering an equivalent service, content providers have less leverage in any such dispute. Their customers don't want to hear "You're on Comcast and they don't like us" as an excuse when the service doesn't work. They'll go find a provider Comcast likes. The bottom line is that these ridiculous disputes are expensive to both sides and degrade service for their mutual customers. I make a point of opening tickets every time this becomes a performance issue for me. If more consumers did, then perhaps that cost would help drive better decisions from the executives at these providers. The other problem that plays into this is, as someone noted, many of these providers are in the internet business as a secondary market for revenue added to their primary business. They'd rather not see their primary business revenues driven onto the internet and off of their traditional services. As such, there is a perceived P&L gain to the other services by degrading the performance of competing services delivered over the internet. Attempting to use this fact to leverage (extort) money from the content providers to make up those revenues also makes for an easy target in the board room.

This is where so-called "bit mile peering" agreements can help - neutralize arguments about balance in order to focus on what matters. Of course there is still the "P" side of a P&L sheet to consider, and networks will surely attempt to capture some of the success of their peers' business models. But take away the legitimate "fairness" excuses and we can see the real issue in these cases.

The problem I see with "bit mile peering" agreements is that the measurement of traffic that would be necessary to make such an agreement function reliably and verifiably by both sides would likely cost more than the moving of the traffic in question. I'd hate to see the internet degrade to telco style billing where it often cost $0.90 of every $1 collected to cover the costs of the call accounting and billing systems.

Not that we have built the best (standard, interoperable, cheap) tools to make bit-mile peering possible... But that's a good conversation to have.

It might be an interesting conversation to have, but at the end of the day, I am concerned that the costs of the tools and their operations exceeds the cost being accounted. In such a case, it is often better to simply write off the cost. It's like trying to recover all the screws/nuts/washers that fall on the floor in an assembly plant in order to save money. The cost of retrieving and sorting them vastly exceeds their value. Owen