Hello, I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught. All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy. Thanks, Rucas PS: sorry if I sent this twice; client lagged a bit.
On Mon, 6 Jun 2011, rucasbrown@hushmail.com wrote:
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Not necessarily. Peering with an ISP who wants to take the traffic between your network and theirs through a saturated pipe, an overloaded router, or across an MPLS pipe with 13 underlying hops (each of which could be a choke point themselves) will not make your end-to-end latencies any better. As others have mentioned, some ISPs do have friendly peering policies. This is particularly true for ISPs that are co-located at the same IXP, because much of the opex is already baked into the ISP's relationship with the IXP. The reason most of the larger ISPs, particularly those who live in the DFZ, have peering policies (especially for settlement-free peering) that could be construed as less friendly to smaller networks is because those guys want to sell you transit, rather than let you peer for free, or for less than a the full transit rate. It doesn't make financial sense for them to exchange bits with you for free, when they can make money off of those same bits if you buy transit instead. jms
On Jun 6, 2011, at 12:53 PM, Justin M. Streiner wrote:
On Mon, 6 Jun 2011, rucasbrown@hushmail.com wrote:
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Not necessarily. Peering with an ISP who wants to take the traffic between your network and theirs through a saturated pipe, an overloaded router, or across an MPLS pipe with 13 underlying hops (each of which could be a choke point themselves) will not make your end-to-end latencies any better.
As others have mentioned, some ISPs do have friendly peering policies. This is particularly true for ISPs that are co-located at the same IXP, because much of the opex is already baked into the ISP's relationship with the IXP.
The reason most of the larger ISPs, particularly those who live in the DFZ, have peering policies (especially for settlement-free peering) that could be construed as less friendly to smaller networks is because those guys want to sell you transit, rather than let you peer for free, or for less than a the full transit rate. It doesn't make financial sense for them to exchange bits with you for free, when they can make money off of those same bits if you buy transit instead.
carrying packets long distances cost more than carrying them short distances... large networks have an incentive to have the cost of that conveyance be reflected in peering relationship figuring out what if relationship makes sense in the marginal sense implies both parties see mutual benifit.
jms
Nope. It is because who pay the money, and somebody wants to earn the money because they have more control. So it is because of "money". Welcome to the world of capitalism. Alex On Mon, Jun 6, 2011 at 5:19 PM, <rucasbrown@hushmail.com> wrote:
Hello,
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Thanks, Rucas
PS: sorry if I sent this twice; client lagged a bit.
its not always about money. sometimes its reputation. /bill On Mon, Jun 06, 2011 at 05:24:46PM -0500, Alex Ryu wrote:
Nope.
It is because who pay the money, and somebody wants to earn the money because they have more control. So it is because of "money".
Welcome to the world of capitalism.
Alex
On Mon, Jun 6, 2011 at 5:19 PM, <rucasbrown@hushmail.com> wrote:
Hello,
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Thanks, Rucas
PS: sorry if I sent this twice; client lagged a bit.
On Mon, Jun 6, 2011 at 3:39 PM, <bmanning@vacation.karoshi.com> wrote:
its not always about money. sometimes its reputation.
And also reasonably hygene, and both individual and community self defense. There are some less competent network operators out there (and even good ones have bad days). And some of the people out there speaking BGP want to do really malign things with internet traffic, like hijack and snoop, inject spam, sometimes injecting spam by hijacking someone else's net temporarily, create malware sites, hack others, etc. -- -george william herbert george.herbert@gmail.com
FWIW, Hurricane Electric has an aggressively open peering policy and we will peer with anyone who is willing to peer at any exchange where we are connected. We believe as stated by Rucas that this only serves to enhance the internet experience for our customers as well as our peers. So far, it seems to be working pretty well for us. I encourage others to follow our lead in this regard as it truly does make a more functional internet. Owen On Jun 6, 2011, at 3:24 PM, Alex Ryu wrote:
Nope.
It is because who pay the money, and somebody wants to earn the money because they have more control. So it is because of "money".
Welcome to the world of capitalism.
Alex
On Mon, Jun 6, 2011 at 5:19 PM, <rucasbrown@hushmail.com> wrote:
Hello,
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Thanks, Rucas
PS: sorry if I sent this twice; client lagged a bit.
I agree, HE's peering policy makes them an attractive transit provider. However, money and strategy still come into play here. For example, ISP Z will think "I need some peering and transit. But if I get HE transit then some people may not peer with me at X-exchange because they will already see my routes via their HE peering" So then they get some transit from a network who is useless with their settlement free peering, then get the peers on the X-exchange and only when they are happily peered will they go to HE. -- Leigh Porter ________________________________________ From: Owen DeLong [owen@delong.com] Sent: 07 June 2011 06:43 To: Alex Ryu Cc: nanog@nanog.org Subject: Re: Why don't ISPs peer with everyone? FWIW, Hurricane Electric has an aggressively open peering policy and we will peer with anyone who is willing to peer at any exchange where we are connected. We believe as stated by Rucas that this only serves to enhance the internet experience for our customers as well as our peers. So far, it seems to be working pretty well for us. I encourage others to follow our lead in this regard as it truly does make a more functional internet. Owen On Jun 6, 2011, at 3:24 PM, Alex Ryu wrote:
Nope.
It is because who pay the money, and somebody wants to earn the money because they have more control. So it is because of "money".
Welcome to the world of capitalism.
Alex
On Mon, Jun 6, 2011 at 5:19 PM, <rucasbrown@hushmail.com> wrote:
Hello,
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Thanks, Rucas
PS: sorry if I sent this twice; client lagged a bit.
______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
----- Original Message -----
From: "Owen DeLong" <owen@delong.com>
So far, it seems to be working pretty well for us. I encourage others to follow our lead in this regard as it truly does make a more functional internet.
I concur, and I specifically would like to see a lot more *geographically* local peering, so packets from Roar Runner[1] Tampa Bay to FiOS Tampa Bay don't have to clog up an exchang point in Reston or Dallas; this stuff *will* eventually bite us in another Katrina-scale event. Cheers, -- jra [1]Roar Runner was a typo, but given most of what the Internet is used for these days[2], it's so funny I'm going to leave it in. [2]"Recreational Indignation". -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
On Jun 7, 2011, at 11:42 AM, Jay Ashworth wrote:
I concur, and I specifically would like to see a lot more *geographically* local peering, so packets from Roar Runner[1] Tampa Bay to FiOS Tampa Bay don't have to clog up an exchang point in Reston or Dallas; this stuff *will* eventually bite us in another Katrina-scale event.
What I've found interesting is the cost of circuits seem to not be distance-sensitive. I think this will contribute to mega-regional peering for the foreseeable future. (ie: dc, sj, dfw, chi, nyc, etc…) Unless these costs come closer to reflecting a balance then I suspect we will continue to see this regional networking. I had a hard time getting people to interconnect even in the CLEC COLO spaces. very few people had bgp capable devices in those locations, while they were big and had traffic, the gear for running bgp just wasn't there. - Jared
On Tue, Jun 07, 2011 at 11:52:31AM -0400, Jared Mauch wrote:
What I've found interesting is the cost of circuits seem to not be distance-sensitive. I think this will contribute to mega-regional peering for the foreseeable future.
(ie: dc, sj, dfw, chi, nyc, etcb&)
Unless these costs come closer to reflecting a balance then I suspect we will continue to see this regional networking. I had a hard time getting people to interconnect even in the CLEC COLO spaces. very few people had bgp capable devices in those locations, while they were big and had traffic, the gear for running bgp just wasn't there.
- Jared
well - no BGP, != an ISP :) this sounds very much like the folks who wanted to put up a south asian IX in guam. lots and lots of fiber pairs landed there, but it was just repeaterd and pushed back into the water. No kit for peering there. (other problems w/ Guam left as an academic eercise) /bill
Network utopia. ~Jay “Engineering is about finding the sweet spot between what's solvable and what isn't." “Good engineering demands that we understand what we’re doing and why, keep an open mind, and learn from experience.” Radia Perlman "If human beings are perceived as potentials rather than problems, as possessing strengths instead of weaknesses, as unlimited rather than dull and unresponsive, then they thrive and grow to their capabilities." Please consider the environment before printing e-mail -----Original Message----- From: rucasbrown@hushmail.com [mailto:rucasbrown@hushmail.com] Sent: Monday, June 06, 2011 4:20 PM To: nanog@nanog.org Subject: Why don't ISPs peer with everyone? Hello, I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught. All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy. Thanks, Rucas PS: sorry if I sent this twice; client lagged a bit.
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
Some ISPs have very friendly peering policies, but some obstacles facing even the friendliest ISPs are: *Poor operator reputation or significantly different networking mindsets may make some peers undesirable *Potential peer is attempting to become a tier-1 and demands paid-peering *Potential peers do not have similar POPs or budget for transport between POPs for peering *Some ISPs do not have the ability to easily determine the destination of their traffic and which peers would be most advantageous in terms of transit reduction *Potential peer is lazy or reluctant to make changes I'm sure I'm missing a few, but I believe these are a couple significant obstacles to a more 'meshy' internet. Nathan
I'll answer with some questions: Where should they peer? Who should/will pay for the routers and aggregation ports? How about the power, racks, and building space? Who should/will pay for the network engineers to do the configuration for the peering? In short, peering isn't free for anyone. It _can_ be efficient in some cases but in others its damn pita and you never really know which one a given case will turn into. (its not always a problem of technical competence) On 6/6/2011 6:19 PM, rucasbrown@hushmail.com wrote:
Hello,
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Thanks, Rucas
PS: sorry if I sent this twice; client lagged a bit.
-- Scott Helms Vice President of Technology ISP Alliance, Inc. DBA ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------
From nanog-bounces+bonomi=mail.r-bonomi.com@nanog.org Mon Jun 6 17:20:16 2011 Date: Mon, 06 Jun 2011 18:19:37 -0400 To: nanog@nanog.org Subject: Why don't ISPs peer with everyone? From: rucasbrown@hushmail.com
Hello,
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
The answer to _every_ question that starts of "why don't they..." is "money". Who pays for the circuits to establish a 'peering connection' with an ISP half-the world away? How much does traffic does "Joes Bait Shop and ISP" in Painted Privvy, Nebraska have with a community ISP in Honshu, JP?"
Hello,
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
The answer to _every_ question that starts of "why don't they..." is "money".
Who pays for the circuits to establish a 'peering connection' with an ISP half-the world away? How much does traffic does "Joes Bait Shop and ISP" in Painted Privvy, Nebraska have with a community ISP in Honshu, JP?"
There are a lot of considerations. How many peering sessions can your hardware support? How many peering locations are you going to need? What will the internal network to tie all those together look like. Will you now need to upgrade your core? Will adding a new peer place another peering agreement at risk by changing the traffic balance? So even if the peering itself is "free", the infrastructure required to support large scale peering at multiple locations can be quite expensive. Are you going to want to haul traffic from New Jersey to California to hand it to a peer who hauls the traffic all the way back to New Jersey again? Does your router in Kansas City want to hand the traffic to the peer in New York or in Seattle? For a small regional network, peering can be easy. For a large network that spans a continent, it can be pretty hard.
On Mon, 06 Jun 2011 18:19:37 -0400 rucasbrown@hushmail.com wrote:
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It
It depends on the ISP, but there are a variety of reasons for not wanting to peer with any potential peer or in this case "every other ISP". Also let's distinguish between paid-peering and settlement-free peering. I think we can agree that if there were only paid-peering, then a complete mesh would be not only technologically impractical, but also economically as well. Plug the terms "economics", "internet" and "peering" into your favorite search engine and you should come up with some relevant reading material.
Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
Can you explain why this is a bigger issue in your scenario?
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Certainly most ISPs care about that to some degree, but to get to the heart of the matter, consider the mindset of any profit-motivated ISP, especially where one is "larger" in some sense of the word than the other who wants to peer. If I'm the larger ISP, and you're the smaller ISP coming to me to peer settlement-free, why should I peer with you? So our customers can get better performance to each other? Why don't your customers just connect to me instead? What do I lose if we don't peer? If you're small, probably not too much. John
On Mon, Jun 6, 2011 at 5:19 PM, <rucasbrown@hushmail.com> wrote:
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I
ISPs are often concerned about: Cost of Peering, and Loss of Revenue due to peering -- ISPs usually like to charge for internet services they provide. Free peering is only beneficial to both sides of a peering relationship when it does reduce costs more than it reduces expected revenue. (a) Costs of peering; both in terms of administrative overhead, ports, circuits, cabinet space, and system resources on existing equipment. Creating a presence in an exchange or building media connections from one ISP to another is not free, ISPs don't naturally all have equipment within range of a free patch cable. Every peering connection a router deals with requires some computing power, some memory, table entries on the router, and, depending on the exchange, possibly additional physical connections. And of course, there are man hours to maintain peering sessions. ISPs are more likely to peer when cost is low relative to advantages after all considered. (B) Loss of revenue due to peering. An extreme example is a very large ISP peering with a small ISP, to allow the small ISP to reach large ISP's customers. The large ISP loses revenue, if they provide the peering for free, since it would mean the small ISP is not paying for that transit. Example: If Level3 peered with anyone who wanted, for free... that would mean noone would have to buy transit from Level3 to send traffic to Level3 customers. There is an analogous situation for ISPs of all sizes though. And if they do agree to peer, there is usually some stipulation about the ratio of traffic beng sent versus received. ISPs do not want to peer for free, if there is a chance their partner would need to buy services from them directly, or indirectly (without the peering), that exceed the benefit/cost reduction of peering. And once a customer, never a peer. -- -JH
2011/6/7 Jimmy Hess <mysidia@gmail.com>:
(a) Costs of peering; both in terms of administrative overhead, ports, circuits, cabinet space,...
The cost of peering on an IXP is roughly the same as setup fees for a new transit, and a BGP session to an IXP route server is not far from what will a full view cost in RAM and CPU on your edges.
(B) Loss of revenue due to peering. An extreme example is a very large ISP peering with a small ISP, to allow the small ISP to reach large ISP's customers. The large ISP loses revenue, if they provide the peering for free, since it would mean the small ISP is not paying for that transit.
Large ISPs do buy transit too. On a financial perspective, it can be considered as "outsourcing the peering function", with a paid SLA for this connectivity...
And once a customer, never a peer.
Never peer with one of your peer's customer is one basic rule of peering agreements between tier-2 and 1 networks. It's a shame financial pragmatism makes the Internet less "meshy", and thus more fragile... -- Jérôme Nicolle
On Mon, 6 Jun 2011, rucasbrown@hushmail.com wrote: Please define ISP. -Hank
Hello,
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Thanks, Rucas
PS: sorry if I sent this twice; client lagged a bit.
in this context, anyone who is a BGP speaker is an ISP. /bill On Tue, Jun 07, 2011 at 07:34:25AM +0300, Hank Nussbacher wrote:
On Mon, 6 Jun 2011, rucasbrown@hushmail.com wrote:
Please define ISP.
-Hank
Hello,
I wouldn't consider myself a network engineer, nor do I have any formal training, but why don't ISPs peer with every other ISP? It would only save EVERYONE money if they did this, no? Only issue I see is with possibly hijacked / malicious AS owners, but that's not very common to do without being caught.
All the whole "don't peer with this guy" only makes your customers have worse latencies and paths to other people, making the Internet less healthy.
Thanks, Rucas
PS: sorry if I sent this twice; client lagged a bit.
On Tue, 7 Jun 2011 bmanning@vacation.karoshi.com wrote:
in this context, anyone who is a BGP speaker is an ISP.
Peering costs money. The transit bandwidth saved by peering with another network may not be sufficient to cover the cost of installing and maintaining whatever connections are necessary to peer. Then there's the big networks who really don't want to peer with anyone other than similarly sized big networks...everyone else should be their transit customer. I manage a network that's primarily a hosting network. There's a similar hosting network at the other end of the building. We both have multiple gigs of transit. We don't peer with each other. Perhaps we should, because the cost of the connection would be negligible (I think we already have multiple fiber pairs between our suites), but looking at my sampled netflow data, I'm guessing we average about 100kbit/s or less traffic in each direction between us. At that low a level, is it even worth the time and trouble to coordinate setting up a peering connection, much less tying up a gigE port at each end? Anyone from hostdime reading this? :) If so, what are your thoughts? ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
-----Original Message----- From: Jon Lewis [mailto:jlewis@lewis.org] Sent: Tuesday, June 07, 2011 10:00 AM -snip- I manage a network that's primarily a hosting network. There's a similar hosting network at the other end of the building. We both have multiple gigs of transit. We don't peer with each other. Perhaps we should, because the cost of the connection would be negligible (I think we already have multiple fiber pairs between our suites), but looking at my sampled netflow data, I'm guessing we average about 100kbit/s or less traffic in each direction between us. At that low a level, is it even worth the time and trouble to coordinate setting up a peering connection, much less tying up a gigE port at each end? ----- 100kbit/s at <1ms is better than 100kbit/s at > 1ms. We are hosting as well and some of our top 25 ASNs are other hosting networks, YMMV. -Drew
On Tue, Jun 07, 2011 at 10:15:48AM -0400, Drew Weaver wrote:
-----Original Message----- From: Jon Lewis [mailto:jlewis@lewis.org] Sent: Tuesday, June 07, 2011 10:00 AM
-snip-
I manage a network that's primarily a hosting network. There's a similar hosting network at the other end of the building. We both have multiple gigs of transit. We don't peer with each other. Perhaps we should, because the cost of the connection would be negligible (I think we already have multiple fiber pairs between our suites), but looking at my sampled netflow data, I'm guessing we average about 100kbit/s or less traffic in each direction between us. At that low a level, is it even worth the time and trouble to coordinate setting up a peering connection, much less tying up a gigE port at each end? -----
100kbit/s at <1ms is better than 100kbit/s at > 1ms.
True, but the point being made is: how *much* better? Is it enough better to justify the cost of installing and maintaining another peering link? - Matt -- "Ah, the beauty of OSS. Hundreds of volunteers worldwide volunteering their time inventing and implementing new, exciting ways for software to suck." -- Toni Lassila, in the Monastery
On Wed, 8 Jun 2011, Matthew Palmer wrote:
netflow data, I'm guessing we average about 100kbit/s or less traffic in each direction between us. At that low a level, is it even worth the time and trouble to coordinate setting up a peering connection, much less tying up a gigE port at each end? -----
100kbit/s at <1ms is better than 100kbit/s at > 1ms.
True, but the point being made is: how *much* better? Is it enough better to justify the cost of installing and maintaining another peering link?
Additionally, we share at least one common transit provider, so we'd be trading <1ms for 1-2ms. Obviously, if we were talking about a leased line with any MRC, the answer would be hell no. Since we're able to utilize fiber inside the building with no MRC, the answer is more of a "why bother?" It's not going to save either of us any meaningful amount of transit bandwidth $/capacity. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On 6/7/2011 11:38 AM, Jon Lewis wrote:
Additionally, we share at least one common transit provider, so we'd be trading <1ms for 1-2ms. Obviously, if we were talking about a leased line with any MRC, the answer would be hell no. Since we're able to utilize fiber inside the building with no MRC, the answer is more of a "why bother?" It's not going to save either of us any meaningful amount of transit bandwidth $/capacity.
That's what it really boils down to. How much money can be saved versus performance. If I'm doing a lot of throughput to a specific network, it makes sense that I might want to connect to them, especially if that connection either 1) saves me money or 2) gives me superior QOS/load balancing without a cost increase. Anything less than 200mbit of traffic isn't even worth me considering these days, and as I grow, I'm sure that number will increase. Content providers generally won't peer unless you meet certain traffic requirements for the same reason. Jack
2011/6/8 Jack Bates <jbates@brightok.net>:
That's what it really boils down to. How much money can be saved versus performance. If I'm doing a lot of throughput to a specific network, it makes sense that I might want to connect to them, especially if that connection either 1) saves me money or 2) gives me superior QOS/load balancing without a cost increase.
Anything less than 200mbit of traffic isn't even worth me considering these days, and as I grow, I'm sure that number will increase. Content providers generally won't peer unless you meet certain traffic requirements for the same reason.
That's certainly a valid approach for direct (private) peering, it's not applicable to IXPs offering route servers. -- Jérôme Nicolle
On 6/7/2011 6:39 PM, Jérôme Nicolle wrote:
That's certainly a valid approach for direct (private) peering, it's not applicable to IXPs offering route servers.
In my case, I have to justify the long haul to an IXP as appropriate cost savings, and given that haul often costs more than I pay for transit, it still hasn't justified. Perhaps when I get to multiple 10GE traffic loads and justify leasing a 600 mile dark fiber ring to DFW. Jack
I'd like to foster a discussion here to better understand this, not rile anyone up. That said, what I see so far is a representation of those who do not recall the halcyon days before a rabid profit motive was the driving force behind ISPs. Peering in it's original sense is/was free. It was a swap of traffic. That profit motive has created the phrase "settlement free peering" to refer to the original definition so it seems like the free swap of traffic is the aberration. The big ISPs used to seek to balance content hosting and the customer load to avoid having to pay for any sort of transit. AOL was known to acquire companies which had huge downstream traffic for this purpose. Now we see ISPs waging an economic war with content providers wanting to find a way to charge, say, Google for allowing them to to pass their YouTube content along to the ISP's subscribers. This is the result of letting non-technical, profit-driven managers run the show and not the usually eager to cooperate network engineers who actually understand how this stuff works. The problem here is that the closer you are to the end user, the harder you're getting screwed, and not in a good way. The very large ISPs are doing real peering, and charging smaller, end-user focused ISPs high transit rates so that they can't possibly compete on price with the inferior, customer-service-impaired ISP end-user offerings. The US government has declined to enforce any sort of rule which might require the huge ISPs to grant wholesale-type access to their physical networks (for better or worse depending on your POV) or examine any of this cartel-type behavior under the light of monopoly rules. So please, short of socialism, and in light of the rampant legislation-for-sale culture in our government (how many FCC commissioners get jobs with huge ISPs?) how do we fix this? Please note: I'm not advocating socialism. I might advocate regulation a la public utilities. There is universal agreement that the internet is "critical infrastructure." deregulating other utilities hasn't been uniformly successful, especially when measured from the consumers' point of view. Thoughts? Rob Sent from my iPad, so I can't have a fun sig. On Jun 7, 2011, at 10:00 AM, "Jon Lewis" <jlewis@lewis.org> wrote:
On Tue, 7 Jun 2011 bmanning@vacation.karoshi.com wrote:
in this context, anyone who is a BGP speaker is an ISP.
Peering costs money. The transit bandwidth saved by peering with another network may not be sufficient to cover the cost of installing and maintaining whatever connections are necessary to peer. Then there's the big networks who really don't want to peer with anyone other than similarly sized big networks...everyone else should be their transit customer.
I manage a network that's primarily a hosting network. There's a similar hosting network at the other end of the building. We both have multiple gigs of transit. We don't peer with each other. Perhaps we should, because the cost of the connection would be negligible (I think we already have multiple fiber pairs between our suites), but looking at my sampled netflow data, I'm guessing we average about 100kbit/s or less traffic in each direction between us. At that low a level, is it even worth the time and trouble to coordinate setting up a peering connection, much less tying up a gigE port at each end?
Anyone from hostdime reading this? :) If so, what are your thoughts?
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Content providers (e.g. Netflix, Hulu, YouTube) will always try to get their content serviced for little to no cost. The low cost, web-only plan isn't sustainable, and the amount of Netflix traffic around the globe is a good example; There's a lot of traffic that they aren't paying for. The free market only works if entities self-police. But as has been expertly stated, there's no money in that. I had an idea, I'm sure it's been said before: If we actually had solid "Tier 1 vs Tier 2 vs Tier 3" thresholds, and we could come up with an agreeable metric, we might be able to minimize the impact of bandwidth hogs (sorry Netflix, pointing at you). So, if you are a Tier 1, you are required to have at least 10 piers in 10 locations, 5 of which must be Tier 1 providers. If you are Tier 2, that number is halved. It could be a combination of having the "status" of being a Tier 1 provider, but the major benefit is a reduction of the diameter of the Internet. Even done by continent, this could offer enough parallel paths to help address (potentially) the cost of doing business. I think we would need to have something similar for content providers. To reach Tier 1 status, you are required to have 10 piers in 10 locations, which should cover a set multiple of your total bandwidth (1 TB if it is 500 GB, etc....) For reaching different tiers, they could receive a price break on the cost of Internet circuits. There would also need to be a middle ground somewhere. Circuits would either need to stop being unlimited or have service thresholds. For exceeding, the content provider would be liable to pay X amount per Gigabit of bandwidth. This would then force Content providers to scale their business rather than relying on the upstream providers' upstream provider to do so. Not perfect by a great margin, but I think something like that could help. Sincerely, Brian A . Rettke -----Original Message----- From: Robert F Maxwell [mailto:rmaxwell@umd.edu] Sent: Tuesday, June 07, 2011 7:45 AM To: Jon Lewis Cc: bmanning@vacation.karoshi.com; nanog@nanog.org Subject: Re: Why don't ISPs peer with everyone? I'd like to foster a discussion here to better understand this, not rile anyone up. That said, what I see so far is a representation of those who do not recall the halcyon days before a rabid profit motive was the driving force behind ISPs. Peering in it's original sense is/was free. It was a swap of traffic. That profit motive has created the phrase "settlement free peering" to refer to the original definition so it seems like the free swap of traffic is the aberration. The big ISPs used to seek to balance content hosting and the customer load to avoid having to pay for any sort of transit. AOL was known to acquire companies which had huge downstream traffic for this purpose. Now we see ISPs waging an economic war with content providers wanting to find a way to charge, say, Google for allowing them to to pass their YouTube content along to the ISP's subscribers. This is the result of letting non-technical, profit-driven managers run the show and not the usually eager to cooperate network engineers who actually understand how this stuff works. The problem here is that the closer you are to the end user, the harder you're getting screwed, and not in a good way. The very large ISPs are doing real peering, and charging smaller, end-user focused ISPs high transit rates so that they can't possibly compete on price with the inferior, customer-service-impaired ISP end-user offerings. The US government has declined to enforce any sort of rule which might require the huge ISPs to grant wholesale-type access to their physical networks (for better or worse depending on your POV) or examine any of this cartel-type behavior under the light of monopoly rules. So please, short of socialism, and in light of the rampant legislation-for-sale culture in our government (how many FCC commissioners get jobs with huge ISPs?) how do we fix this? Please note: I'm not advocating socialism. I might advocate regulation a la public utilities. There is universal agreement that the internet is "critical infrastructure." deregulating other utilities hasn't been uniformly successful, especially when measured from the consumers' point of view. Thoughts? Rob Sent from my iPad, so I can't have a fun sig. On Jun 7, 2011, at 10:00 AM, "Jon Lewis" <jlewis@lewis.org> wrote:
On Tue, 7 Jun 2011 bmanning@vacation.karoshi.com wrote:
in this context, anyone who is a BGP speaker is an ISP.
Peering costs money. The transit bandwidth saved by peering with another network may not be sufficient to cover the cost of installing and maintaining whatever connections are necessary to peer. Then there's the big networks who really don't want to peer with anyone other than similarly sized big networks...everyone else should be their transit customer.
I manage a network that's primarily a hosting network. There's a similar hosting network at the other end of the building. We both have multiple gigs of transit. We don't peer with each other. Perhaps we should, because the cost of the connection would be negligible (I think we already have multiple fiber pairs between our suites), but looking at my sampled netflow data, I'm guessing we average about 100kbit/s or less traffic in each direction between us. At that low a level, is it even worth the time and trouble to coordinate setting up a peering connection, much less tying up a gigE port at each end?
Anyone from hostdime reading this? :) If so, what are your thoughts?
---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Tue, Jun 7, 2011 at 9:08 AM, Rettke, Brian <Brian.Rettke@cableone.biz>wrote:
Content providers (e.g. Netflix, Hulu, YouTube) will always try to get their content serviced for little to no cost. The low cost, web-only plan isn't sustainable, and the amount of Netflix traffic around the globe is a good example; There's a lot of traffic that they aren't paying for. The free market only works if entities self-police. But as has been expertly stated, there's no money in that.
I had an idea, I'm sure it's been said before:
If we actually had solid "Tier 1 vs Tier 2 vs Tier 3" thresholds, and we could come up with an agreeable metric, we might be able to minimize the impact of bandwidth hogs (sorry Netflix, pointing at you).
First - I don't work for Netflix! I'm a consumer of their product and a network engineer who mostly gets stuff. So I'd like to offer a point of distinction that's kinda bugged me whenever these conversations pop up here: Netflix the company doesn't consume bandwidth nor are *they* a bandwidth hog. The consumer is the bandwidth hog. And the consumer pays their ISP for that bandwidth. ISP's over provision in the hopes that most folks won't use what they are paying for and to help keep costs down (very valid). Companies like Netflix and even Google (I don't know this for a fact - just making logical assumptions) are not going to rely on peering arrangements of ISPs to deliver 100% of their traffic. If they did they'd place their business model in the hands of network operators who don't have Netflix's best interests in mind. They are going to use caching like products or services to bring their content closer to the consumer, develop them to be bandwidth and latency aware, or even make peering arrangements on their own (to your point). These peering arrangements and products they purchase / pay for are most likely located within Tier 1 networks in the USA. So technically, if my assumptions are correct, Netflix probably is paying for their bandwidth that exits their network. And the consumer is paying for their bandwidth. Now - Netflix like content providers may cause some of the ISP's to rethink their over provisioning strategies, but that's not my problem. I'm paying for my bandwidth, therefore, I want to use it for what I want when I want. It's my ISP's job to deliver what I'm paying for. This is just my .02 and that tangent is over for now! To the original poster - I think it'd be technically impossible to have every ISP plugged into every ISP, physically ($$ issues aside). How many ISPs are there and how many routers / ports would you need? And I'm pretty sure that most Tier 1 ISP's peer with each other - but that's an assumption not made of fact. Maybe someday when there really are no bandwidth or latency limitations an overlay routing model could abstract the physical issues we all deal with and everyone can logically peer with everyone (although I'm not sure even that would make sense) but until then a hierarchical model (Tier 1 vs Tier2 etc..) seems to me to make the most sense. Anyway, the implementation of that hierarchical Internet is driven by $$ of course. Kenny
On Mon, Jun 6, 2011 at 6:19 PM, <rucasbrown@hushmail.com> wrote:
why don't ISPs peer with every other ISP?
1. For those who can pull it off, getting paid twice for each packet is better than getting paid once. 2. Your service has a value per byte and a cost per byte. If your value is less than your cost, you go out of business. Open peering facilitates greater consumption on the part of your customers. Unless you're structured to charge them more for that increased consumption, it reduces the value of each byte you pass. Unless you're peering with someone in the same or higher tier (who you'd otherwise have to pay for transit) the odds are you're reducing the value of your bytes faster than you're reducing your cost. Personally, I'd love to see 95th percentile billing applied universally with everybody getting a large pipe the same way everybody gets a 200 amp electrical service. The problem with that notion is that A) consumers are hooked on "unlimited," and B) your toaster doesn't get hacked and start consuming 200 amps all day without your knowledge. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Tue, Jun 7, 2011 at 7:10 PM, William Herrin <bill@herrin.us> wrote: [snip]
gets a 200 amp electrical service. The problem with that notion is that A) consumers are hooked on "unlimited," and B) your toaster Consumers aren't getting "unlimited right now". They're getting (unknown number of databytes)/month, before the ISP speed caps, throttles, rate limits them or turns them off for "excessive usage".
doesn't get hacked and start consuming 200 amps all day without your knowledge.
Your toaster is plugged into an outlet that probably has a 20 amp circuit breaker on it. If someone hacks it without your knowledge to eat 200 amps, it will get turned off. A similar mechanism could be built into network CPEs. -- -JH
On Tue, Jun 7, 2011 at 8:41 PM, Jimmy Hess <mysidia@gmail.com> wrote:
On Tue, Jun 7, 2011 at 7:10 PM, William Herrin <bill@herrin.us> wrote: [snip]
gets a 200 amp electrical service. The problem with that notion is that A) consumers are hooked on "unlimited," and B) your toaster Consumers aren't getting "unlimited right now". They're getting (unknown number of databytes)/month, before the ISP speed caps, throttles, rate limits them or turns them off for "excessive usage".
They're being told they're getting unlimited and for 99% of them it's true in the sense that their usage does not induce their ISP to impose its cap. Point is: they expect unlimited and a service which doesn't claim to be unlimited is, therefore, a non-starter. Back in the day I faced this problem at my dialup ISP. We had a 240 hour per month cap on dialup usage so that the 24/7 users would buy a 24/7 account or go elsewhere. We started losing business from folks using 30 and 40 hours a month because the other guy was "unlimited." So we did some fancy wordsmithing and came up with "unlimited _attended_ hours" meaning you had to be in front of your computer. How did we know? Because you sleep too so if you're online for 23+ hours per day every day, your usage isn't "attended." Our salesfolk tested the waters, but we couldn't sell a $5/month plus $0.10/hour product even though that would have resulted in most customers paying less. When I say consumers are hooked on unlimited, that's what I'm talking about.
Your toaster is plugged into an outlet that probably has a 20 amp circuit breaker on it. If someone hacks it without your knowledge to eat 200 amps, it will get turned off.
A similar mechanism could be built into network CPEs.
A similar mechanism is built in to network CPEs. It's called the port speed and the choices are 10, 100 and 1000. The electrical company metaphor breaks down here. Wiring an appliance so it can consume your entire electrical service has no desirable traits. Wiring your computing equipment so they can communicate at higher speeds within the building than leaving the building is the opposite. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
participants (27)
-
Alex Ryu -
bmanning＠vacation.karoshi.com -
Drew Weaver -
George Bonser -
George Herbert -
Hank Nussbacher -
Jack Bates -
Jared Mauch -
Jay Ashworth -
Jimmy Hess -
Joel Jaeggli -
John Kristoff -
Jon Lewis -
Justin M. Streiner -
Jérôme Nicolle -
Kenny Sallee -
Leigh Porter -
Matthew Palmer -
Murphy, Jay, DOH -
Nathan Eisenberg -
Owen DeLong -
Rettke, Brian -
Robert Bonomi -
Robert F Maxwell -
rucasbrown＠hushmail.com -
Scott Helms -
William Herrin