I know a couple networks that offer to sell transit over exchanges that permit it, but require that you take a private VLAN on the exchange. Some exchanges offer private VLANs, others don't. Regards, Chris Rogers +1.302.357.3696 x2110 http://inerail.net/ On Tue, Nov 25, 2014 at 1:51 PM, Nick Hilliard <nick@foobar.org> wrote:

On 25/11/2014 18:47, Colton Conor wrote:

...

Is this possible?

it depends. Some transit providers will decline to do this because it can impact on their margin. Most IXPs don't have a problem with it, but some do - although it's not clear how they can tell which packets are transit and which are peering.

Nick

Hi Colton, The primary challenge in buying IP Transit across a Peering Exchange is not so much of a technical configuration challenge, but rather a 'how do we keep track of how much IP Transit you are using' ..a billing challenge. and additionally, one is making the assumption that there is capacity to do so on the IP Transit Providers Peering Port Connection. While it is possible to deal with such issue, but you need someone willing and able to do so, on the other side. ------------------------

I think the way most providers would do this would be to get a rack and power with Equinix. Pay a cross connect fee from the wave provider to our rack. Pay for an exchange port (which includes a cross connect to the exchange) for the 5GBPS of traffic going to Netflix, Google, etc. And then pay for yet another cross connect going to HE.net's cage to get pure IP from them.

Yes, you are right, this is the traditional way of doing so, and yes, it can get expensive.. For this exact reason, folks such as us and others who are willing to provide access via their existing resources at different facilities. We are facilitating flexible connectivity needs of folks who are running remote (from major metro areas) such as yours, in Miami, Atlanta, and I know others who are doing so in Equinox Chicago, one in Texas and a couple of the West Coast. Feel free to ping me off list if you are interested in additional details. Regards Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net ----- Original Message -----

From: "Colton Conor" <colton.conor@gmail.com> To: "Ammar Zuberi" <ammar@fastreturn.net> Cc: "NANOG" <nanog@nanog.org> Sent: Tuesday, November 25, 2014 2:51:47 PM Subject: Re: Buying IP Bandwidth Across a Peering Exchange

The exchange in question is Equinix. Their sales team is leading me to believe there are multiple exchange products. One where you can peer with providers (Google, Netflix for example) and then one where you can create virtual private layer 2 vlans between providers. Then there is also the traditional cross connect fee of $350 if you want to go from one cage/rack to the other.

So in a situation where we are getting a 10Gig transport wave to Equinix, we would ideally like to split this wave's use to 5Gbps of traffic going to the peering exchange for traffic going directly to Google, Netflix, and other CDN's, and then 5Gbps of pure IP transit going to a low cost provider like HE.net. Of course providers like HE.NET are also peers on the peering exchange, so it seems possible that we could just opening a peering conenction with them.

I think the way most providers would do this would be to get a rack and power with Equinix. Pay a cross connect fee from the wave provider to our rack. Pay for an exchange port (which includes a cross connect to the exchange) for the 5GBPS of traffic going to Netflix, Google, etc. And then pay for yet another cross connect going to HE.net's cage to get pure IP from them.

If I can buy transit directly I avoid the expenses of having to pay for space, power, another router/switch, plus a second cross connect. Thats quite a bit of money saved.

Are exchanges really that unreliable compared to a traditional cross connect?

On Tue, Nov 25, 2014 at 12:52 PM, Ammar Zuberi <ammar@fastreturn.net> wrote:

...

Hi Conor,

I know this is possible since Hurricane Electric does it for IPv6 transit, however, I'm not sure if it violates any exchange rules or if it's even a good idea.

...

On 25 Nov 2014, at 10:47 pm, Colton Conor <colton.conor@gmail.com> wrote:

I know typically peering exchanges are made for peering traffic between providers, but can you buy IP transit from a provider on an exchange? An example, buy a 10G port on an exchange, peer 5Gbps of traffic with multiple providers on the exchange, and buy 5Gbps of IP transit from others on the exchange?

Some might ask why not get a cross connect to the provider. It is cheaper to buy an port on the exchange (which includes the cross connect to the exchange) than buy multiple cross connects. Plus we are planning on getting a wave to the exchange, and not having any physical routers or switches at the datacenter where the exchange/wave terminates at. Is this possible?

On Tuesday, November 25, 2014 09:51:47 PM Colton Conor wrote:

Are exchanges really that unreliable compared to a traditional cross connect?

Not necessarily. It's just that when money is changing hands, folk tend to find (passive) x-connects within the data centre to be far more reliable (even though they are not infallible) than passing traffic across another (active) system being run by someone else in the same physical facility. Plus, some service providers will drastically reduce or eliminate SLA's (for whatever they may be worth) if there is another active system in between you and their service. Mark.

I agree with Bill...going it on the cheap is risky. DOn't consider it for primary. It may be good for backup. I have sold small amounts of transit to non-ISP companies on exchanges (100-200 meg). It's a good extra backup for ISPs, if you setup your local pref, MED and then prepend your AS an extra time or two to the prefixes you transmit. Then if you ever need to use it, it's sitting there waiting to send and receive traffic. I let ISPs customers do that with us for real low cost backup fees. Bob Evans

On Nov 25, 2014, at 10:47 AM, Colton Conor <colton.conor@gmail.com> wrote:

...

I know typically peering exchanges are made for peering traffic between providers, but can you buy IP transit from a provider on an exchange? An example, buy a 10G port on an exchange, peer 5Gbps of traffic with multiple providers on the exchange, and buy 5Gbps of IP transit from others on the exchange?

Some IXPs have a rule that explicitly disallows this, others encourage it, most don’t care. I don’t know of any that have a mechanism to enforce a rule prohibiting it.

PCH’s guidance in the IXP formation process is to avoid creating rules which are, practically, unenforceable. So we generally counsel IXPs against having a rule precluding transit across the switch fabric. That said, a crossconnect is a _much better idea_.

...

Some might ask why not get a cross connect to the provider. It is cheaper to buy an port on the exchange (which includes the cross connect to the exchange) than buy multiple cross connects. Plus we are planning on getting a wave to the exchange, and not having any physical routers or switches at the datacenter where the exchange/wave terminates at. Is this possible?

Yes, it’s possible, but what you describe is a pretty fragile setup. Lots of common points of failure between peering and transit; places where screwing one up would screw both up. If all of this is really tangential to whatever you’re doing, and you don’t mind looking a little out-of-step with best practices, and you don’t mind it all being down at once, any time anything breaks, then it may be a reasonable economy. If you’re planning on actually depending on it, you need to do better engineering, and either spend more money, or allocate your money more conservatively.

Doing everything the cheapest possible way, regardless of the fragility or complexity, is very short-sighted, and is unlikely to be an economy in the long run.

-Bill

On Nov 25, 2014, at 10:56 AM, Bill Woodcock <woody@pch.net> wrote:

On Nov 25, 2014, at 10:47 AM, Colton Conor <colton.conor@gmail.com> wrote:

...

I know typically peering exchanges are made for peering traffic between providers, but can you buy IP transit from a provider on an exchange? An example, buy a 10G port on an exchange, peer 5Gbps of traffic with multiple providers on the exchange, and buy 5Gbps of IP transit from others on the exchange?

Some IXPs have a rule that explicitly disallows this, others encourage it, most don’t care. I don’t know of any that have a mechanism to enforce a rule prohibiting it.

PCH’s guidance in the IXP formation process is to avoid creating rules which are, practically, unenforceable. So we generally counsel IXPs against having a rule precluding transit across the switch fabric. That said, a crossconnect is a _much better idea_.

...

Some might ask why not get a cross connect to the provider. It is cheaper to buy an port on the exchange (which includes the cross connect to the exchange) than buy multiple cross connects. Plus we are planning on getting a wave to the exchange, and not having any physical routers or switches at the datacenter where the exchange/wave terminates at. Is this possible?

Yes, it’s possible, but what you describe is a pretty fragile setup. Lots of common points of failure between peering and transit; places where screwing one up would screw both up. If all of this is really tangential to whatever you’re doing, and you don’t mind looking a little out-of-step with best practices, and you don’t mind it all being down at once, any time anything breaks, then it may be a reasonable economy. If you’re planning on actually depending on it, you need to do better engineering, and either spend more money, or allocate your money more conservatively.

Doing everything the cheapest possible way, regardless of the fragility or complexity, is very short-sighted, and is unlikely to be an economy in the long run.

-Bill

I’d say that depends… If it’s an equal cost choice, for example, between getting waves to multiple exchanges and peering with multiple providers at each exchange that way vs. putting a router at one exchange and getting cross-connects there, then I would argue that the former is actually more robust. Owen

On Tuesday, November 25, 2014 10:34:14 PM Eric Van Tol wrote:

It's been a while since I've checked the Equinix Customer Agreement and Policies documents, but I know at one time they required a physical presence in the in the IDC for an Exchange cross-connect. This may have changed in the past several years.

Several exchange points now support some kind of resale model, where peering members are transported into the exchange point via network, without the need for physical presence at the exchange point location. I'm not sure whether Equinix's exchange points do this. Mark.

On Wednesday, November 26, 2014 02:42:39 PM Ammar Zuberi wrote:

I’m pretty sure IX Reach can take you into an Equinix exchange, so it is probably possible that they allow this kind of stuff to happen.

I meant in terms of a reseller model between the exchange point and preferred service providers on behalf of the exchange point members. Of course, anyone can transport anyone anywhere, as long as the right people are paid. But exchange points have been getting into reseller models with transport providers as a way to discount what would be a normal transport service between two or more points. Mark.

Well, we would have a BGP router in another town. Then get a wave from a transport provider from the other town to the town that equinix or the peering exchange was located at. The cross connect would go from the transport providers Z location to the port on the exchange. I have confirmed that Equinix is willing to sell us a port on the exchange even if we don't have a physical presence there. On Wed, Nov 26, 2014 at 8:45 AM, <Valdis.Kletnieks@vt.edu> wrote:

On Tue, 25 Nov 2014 15:34:14 -0500, Eric Van Tol said:

...

but I know at one time they required a physical presence in the in the IDC for an Exchange cross-connect.

At the risk of being snarky, if somebody doesn't have a presence where do you connect the other end of the cross-connect cable? :)

(Note that's different than "I'm in a PoP on the west side of town, and the logical place to land my uplink is blade 2, port 3 of a router belonging to $upstream over on the east side of town" - that's an external connection not a cross-connect)

Yes, we could of course pay for some space and power with a shared hosting provider, but buying a full rack and power for a single router seems silly. The ideal person to buy the small amount of space and power from would be the transport provider that is transporting us to Equinix, but in most cases those are large providers like Level3 who aren't interested nor willing to sell us a quarter of a cabinet. If we were to get space directly with Equnix then we would have yet another cross connect from the transport provider to our rack. The whole point on getting to the exchange is to peer with providers like Netflix and Google as this would be for an eyeball network, but after you add up the cost of the rack, power, cross connects, exchange ports, etcs I am not seeing the value in doing so. Espically with some of the Tier 1's willing to sell us Gbps IP transit links for almost the cost of the transport back to an peering location. On Wed, Nov 26, 2014 at 12:13 PM, Rob Seastrom <rs@seastrom.com> wrote:

Colton Conor <colton.conor@gmail.com> writes:

...

Some might ask why not get a cross connect to the provider. It is cheaper to buy an port on the exchange (which includes the cross connect to the exchange) than buy multiple cross connects. Plus we are planning on getting a wave to the exchange, and not having any physical routers or switches at the datacenter where the exchange/wave terminates at. Is this possible?

"Technically possible" and "advisable" are two different things. If you enjoy finger-pointing on the occasions where you are trying to smoke out performance issues, I encourage as many third, fourth, and fifth-party-managed network layers in the mix as possible. A wave with no way to test to the handoff point would of course be the icing on the cake.

Are you sure you can't afford to sublet a few ru of space from someone and pay for a couple extra cross connects?

-r

Having run an exchange, I can speak to a couple of points. 1.An exchange is only as good as any other provider. If they don¹t have a redundant design then you have more room for failure. Same can be said about good staff behind it. If they know what they are doing and keep it simple, then it can be a very advantageous thing to you. 2.An exchange really shines in data centers where you have to pay for cross connects. I can buy 1 10 gig connection to an exchange in Chicago and peer with 30 other providers easy. If I were to run a cross connect to each provider directly that would cost me between 7 and 9 grand a month in an Equinix data center. Instead I am paying $1300 for the same thing. If I want a private vlan add on a little extra. 3.Some exchanges carry weight with the folks like Netflix and others. In most cases, they will peer with you easier on an exchange than a direct connection. Keep in mind switch ports cost money, especially in a dense data center. Justin -- Justin Wilson <j2sw@mtin.net> http://www.mtin.net <http://www.mtin.net/blog> Managed Services ­ xISP Solutions ­ Data Centers http://www.thebrotherswisp.com Podcast about xISP topics http://www.midwest-ix.com Peering ­ Transit ­ Internet Exchange

Hi Clayton, Putting on my TorIX hat, I'll address what you've brought up: 1. We implemented port security because MAC ACL's were not effectively blocking certain types of bad traffic, which was a problem with the hardware in place at the time. As you are certainly aware, getting vendors to work on esoteric problems faced by a small number of their customers can be frustrating. 2. Port security effectively logs rogue MAC's received on the port, which was/is not always the case when certain types of "bad or unwanted traffic are received. This has proven invaluable for trouble-shooting and is very easy to pass along that info to the peer for further investigation without having to begin a separate trouble-shooting process with all parties online and aligned, and hoping the problem reappears. 3. Since we implemented port security, the stability of TorIX has been excellent. No more sudden outages due to peer human error or bad peer architecture. (some of which is mind blowing). 4. If you think the 60 minute lock-down is excessive, bring it up on torix-members and begin a discussion, which we're certainly open to when it will not adversely affect the integrity of the IX. 5. If Netflix was at TorIX, I guarantee you would see traffic shoot through the roof, and that's why we'd welcome NF and others like FB, Edgecast etc. joining TorIX. We are one of the largest IX'es in terms of number of peers in the world after all. Back onto the original topic, a number of peers sell transit over the IX. TorIX does not offer SLA's, but we do not stop peers from maximizing their value of the IX. -- Stephen (volunteer at TorIX) On 2014-11-30 6:51 PM, Clayton wrote:

We peer at TorIX and Equinix. I have to say that despite the fact that Equnix charges us more for our port, we're getting far more value from it than TorIX. Around double the traffic, and they don't have silly punative measures like locking your port if you leak a MAC address other than the one you registered with them (Equnix filters the MAC, but doesn't apply a 60 minute port shut down penalty if you leak like TorIX does).