On Tuesday, June 14, 2016, Patrick W. Gilmore <patrick@ianai.net> wrote:

On Jun 14, 2016, at 11:50 AM, Hugo Slabbert <hugo@slabnet.com <javascript:;>> wrote:

...

On Tue 2016-Jun-14 10:12:10 -0500, Matt Peterson <matt@peterson.org <javascript:;>> wrote:

...

This week at NANOG67, a presentation was given early on that did not reflect well for our community at large. Regardless of the content or accuracy of the data presented (not the intention of this thread), specific members of the community (some of which are sponsors) were clearly targeted in a hurtful manner. The delivery of the content did not seem within the spirit of NANOG, but instead a personal opinion piece. While no specific rules of the speaking guidelines <https://www.nanog.org/meetings/presentation/guidelines> were likely broken, this does bring up a point of where the acceptable threshold exists (if at all). To be abundantly clear - I have nothing against the content itself, the presenter, the PC's choice of allowing this talk, etc. - I only wish to clarify if our guidelines need modernization.

As a community, how do we provide constructive criticism to industry suppliers (that may also be fellow competitors, members, and/or suppliers)? For example, router vendors are routinely compared without specific names mentioned (say in the case of a unpublished vulnerability) - how is a service provider any different?

I understand the discretion involved in your question, but could we clarify exactly what presentation is being discussed so those of us who were not present at NANOG67 can also participate in an informed way?

I personally think the meta-question Matt asked is more important than opinions on a specific presentation. Plus I worry about devolving into a “that was a good preso” / “no it wasn’t!!” flamefest.

Harassment policy is a good idea https://www.ietf.org/iesg/statement/ietf-anti-harassment-policy.html Walking on eggshells because sponsors don't appreciate the message and find posting pictures of their dance parties while discussing non-profit financials is ... Or is that a different subtweet? We are talking about dnssec? To that end, let a million flowers bloom. It was a good relevant talk. Regards, C&J --

TTFN, patrick

I'd suggest that this is not an operation discussion and should be moved to the NANOG Membership list. I don't see any violation of the presentation guidelines. Also, the day we decide to censor ourselves to avoid offending vendors is the end of my involvement in NANOG - and I suspect that is the case for many others. Matt is being coy, for some reason. He didn't like Dave Temkin's talk about IXP costs. I listened very carefully and did not hear any specific members or people targeted - only organizations and companies. NANOG is not and has never been a "safe space" for sponsors or organizations that exist in the network space. It never should be. If LINX or AMSIX or anyone else didn't like what was said, they should have rocked the mic (which they did!) and they should come to the next NANOG and present a counterpoint. Daniel Golding (speaking in my personal capacity) On Tue, Jun 14, 2016 at 11:10 AM Ca By <cb.list6@gmail.com> wrote:

On Tuesday, June 14, 2016, Patrick W. Gilmore <patrick@ianai.net> wrote:

...

...

On Tue 2016-Jun-14 10:12:10 -0500, Matt Peterson <matt@peterson.org <javascript:;>> wrote:

...

This week at NANOG67, a presentation was given early on that did not reflect well for our community at large. Regardless of the content or accuracy of the data presented (not the intention of this thread), specific members of the community (some of which are sponsors) were clearly targeted in a hurtful manner. The delivery of the content did not seem within

On Jun 14, 2016, at 11:50 AM, Hugo Slabbert <hugo@slabnet.com <javascript:;>> wrote: the

...

...

spirit of NANOG, but instead a personal opinion piece. While no specific rules of the speaking guidelines <https://www.nanog.org/meetings/presentation/guidelines> were likely broken, this does bring up a point of where the acceptable threshold exists (if at all). To be abundantly clear - I have nothing against the content itself, the presenter, the PC's choice of allowing this talk, etc. - I only wish to clarify if our guidelines need modernization.

As a community, how do we provide constructive criticism to industry suppliers (that may also be fellow competitors, members, and/or suppliers)? For example, router vendors are routinely compared without specific names mentioned (say in the case of a unpublished vulnerability) - how is a service provider any different?

I understand the discretion involved in your question, but could we clarify exactly what presentation is being discussed so those of us who were not present at NANOG67 can also participate in an informed way?

I personally think the meta-question Matt asked is more important than opinions on a specific presentation. Plus I worry about devolving into a “that was a good preso” / “no it wasn’t!!” flamefest.

Harassment policy is a good idea

https://www.ietf.org/iesg/statement/ietf-anti-harassment-policy.html

Walking on eggshells because sponsors don't appreciate the message and find posting pictures of their dance parties while discussing non-profit financials is ... Or is that a different subtweet?

We are talking about dnssec?

To that end, let a million flowers bloom.

It was a good relevant talk.

Regards, C&J

--

...

TTFN, patrick

On Jun 14, 2016, at 11:20 AM, Daniel Golding <dgolding@gmail.com> wrote:

John,

We've had this for years. https://www.nanog.org/governance/attendance

If you notice similarities - they are intentional.

<chuckle>

If you notice differences - NANOG has always had a higher threshold for a frank exchange of views between participants. We have no desire to stifle that.

Makes perfect sense to me - thanks for the pointer! So, you’ve set expectations, and those include a clear reporting and enforcement process, so is discussion of the session in question (I actually have no idea which one it is ) on a mailing list really the right approach? Alternatively should folks who feel there was an issue just follow the reporting process? (rhetorical question) /John Disclaimer: my views alone.

So I just watched the video of Dave’s talk. While my overall impression is that it’s 51 minutes of my life that I’d rather have back and I don’t agree with several of his conclusions, I don’t see anything inherently wrong or hurtful about it. I don’t think it painted anyone in a bad light beyond fair criticism. He was clear about what was his opinion and what wasn’t. I do think he’s flat out wrong in his belief about marketing budgets of regulated utilities. I know for a fact, that PG&E spends tons of money on marketing. Sure, they aren’t sponsoring coffee bars at NANOG, that’s not their target market. Neither are the IXPs running television ads (at least none that I know of) since that’s not their target market. PG&E does do lots of TV ads, billboards, and other consumer-oriented marketing. I’m sure if you were to attend electrical industry trade shows, you’d find they have a presence there as well, though I have no idea, it’s not my industry. I do think we could all benefit from more cost-effective ISPs and I do think that models like MICE, SIX, and SFMIX are better for most of their participants than the ultra-expensive premium exchanges with branch offices in all the major markets. However, I suspect that in the long run, market demand will actually sort that out. Owen

On Wed, Jun 15, 2016 at 3:43 AM, Aled Morris <aledm@qix.co.uk> wrote:

Me too and I was confused about what the point of it was.

I had always assumed the customers of those IXs he singled out were generally happy with the service they were getting and the money they are paying.

Is Dave trying to say they are being duped? Is he trying to identify a need for regulation?

I was pointing out facts about IXPs that many did not know, including the actual organizational structure. I was also opining on how these IXPs could be better; mainly, how they choose to spend money.

Perhaps Dave was advocating the SIX model and suggesting the customers of the existing exchanges should be looking to organise an alternative in their localities.

Absolutely correct (which should answer Hank's question, as well).

Or perhaps this is a wakeup call for LoNAP and the smaller exchanges who "compete" with AMS-IX, DE-CIX and NetNod - stop trying to mimic their commercial models (big fees which pay for staff and marketing) and look instead at the lean SIX as the way of offering a service at a price competitive to transit.

Also absolutely correct. I don't want to see them falling into a trap of conflating marketing and outreach and/or offering an overly rich product set at the cost of price and operational simplicity.

Or was there a hidden message in Dave's presentation that I missed?

Seems like you got it.

Aled

General, with the four being used as varying examples. I could have included US IXP's, but almost none publish their prices and the ones that do only started recently, so the comparison wasn't worthwhile.  On Wed, Jun 15, 2016 at 8:39 AM -0500, "Nick Hilliard" <nick@foobar.org> wrote: Dave Temkin wrote:

I was pointing out facts about IXPs that many did not know, including the actual organizational structure.

Dave, was this talk about IXPs in general, or the 4 IXPs you named in your talk? Nick

I hope you'll excuse the aggressive snipping, as I wanted to try to address as many of your points without repeating myself as possible. On Wed, Jun 15, 2016 at 8:48 AM, Nick Hilliard <nick@foobar.org> wrote:

Hi Dave,

Dave Temkin wrote:

With respect to all parties involved in this discussion, I'd suggest that these four IXPs are not representative of the IXP community in the areas that you talked about, namely size, marketing budgets, corporate profit / surplus or expansion intentions.

They are representative of the most important IXPs to deliver traffic from in Western Europe. I would posit that what defines important to me may not be what defines important to you and the same can be said when you look at how various "internet" companies look at what's important in their vertical.

...

https://docs.google.com/spreadsheets/d/18ztPX_ysWYqEhJlf2SKQQsTNRbkwoxPSfaC6ScEZAG8/edit?pref=2&pli=1#gid=0

I have heard people from other large US multinationals say that LINX's outreach and policy representation alone were worth the port charges they pay.

I dedicated an entire slide to this. I think Malcom does great work.

Netnod runs a dns root server system (i.root-servers.net) as well as a heavy duty time service.

There are others who do this for no cost and some who do it for government money. Whether or not my port fees should subsidize this is a valid question, and was brought up in the Q&A afterwards.

Regarding the pricing reduction on page 16 of your preso, the US$ and UK£ are not much different than what they were 5 years ago, but the € has dropped by 30% against the US$.

You speak to this below, however if my business is primarily run in USD (which was the relevant use case presented: I'm a US company deciding if I should peer in Europe or buy transit) then those currency fluctuations have a very different impact than if I'm a European company functioning primarily in local currency.

You made the point that some US companies buy services in Europe using US$, but not all do.

Not all do. Again, this wasn't an exhaustive list of what every IXP and every member does. This is what I see, and the entire presentation was framed as that. How currency fluctuations impact my business will likely vary significantly from how they impact yours.

Regardless of all that, Job's pricing spreadsheet suggests that the pricing models are substantially lower for the other IXPs in his list, and have seen proportional reductions at least equal to transit pricing drops, if not greater. If your talk was about IXPs in general, this was an important omission.

There are absolutely some great pricing models on IXPs. There are also some terrible ones. I highlighted the ones I find to be bad. Again, my presentation, my opinion, in the same way that someone might stand up and say "Cisco sucks because they don't have the CLI I want" or "Juniper sucks because they charge too much for ports with TCAM". I don't have to then present an exhaustive list of those that are better in order to validate my claim. I did purposefully mention SIX as a polar opposite example - there is definitely a happy medium to be found.

Two of the organisations you mentioned are member-owned and are bound by formal votes from their membership. Member votes are, in fact, legally binding in most if not all member-owned IXPs that I'm aware of in Europe. Netnod, DECIX and many others are not participant-owned, so this does not apply.

Which I mentioned in my presentation. I refrained from exhaustively explaining each model as IANAL and didn't feel it made sense for me to attempt to explain four different nations laws. I did invite comment from the floor for the organizations mentioned to do so, and they declined. To be clear: I did apologize both in the Plenary and then later on Twitter if the "Membership" slide was misleading, as I did not intend on implying that LINX is not a member owned organization.

In the area of marketing budgets and expansion, there is probably a correlation between the two, but I think it's worth mentioning that pretty much no other IXP - at least out of those mentioned in Job's spreadsheet - have anything close to that % of their overall budgets dedicated to marketing

It wasn't. It was about the IXPs that mattered to me. If we had an entire day to talk about this I could've been way more exhaustive (and everyone would've been way more exhausted...). LONAP and INEX are great counter examples.

If people don't like the idea of LINX, DE-CIX and AMS-IX expanding outside their current markets, they don't have to connect to them and that's ok because this is a fully unregulated market: no-one has ever forced anyone to connect to any IXP.

Completely agree!

Otherwise, thanks for giving a great talk - it's both refreshing and stimulating to have this discussion, and it's great to get feedback from the community about it.

Thank you! That was the point of the talk: to start the discussion. It definitely has, and I'm glad for that.

I think the popularity of the donation-based IX largely a violent reaction to the over-priced major IX operators in the US. People didn't like what was happening, so went to the polar opposite. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Nick Hilliard" <nick@foobar.org> To: "Dave Temkin" <dave@temk.in> Cc: "NANOG list" <nanog@nanog.org> Sent: Thursday, June 16, 2016 6:45:22 PM Subject: Re: NANOG67 - Tipping point of community and sponsor bashing? Dave Temkin wrote:

They are representative of the most important IXPs to deliver traffic from in Western Europe.

I don't doubt that they are important IXPs for delivering traffic. However, no other IXP in europe (both eastern and western) is doing expansion outside the countries that they operate in, other than three out of the four that you mentioned; none of the member-owned organisations in the region are making large profits or in most cases anything more than marginal profits, and all of them have lower port costs. Also, none of their activities suggest that their marketing budgets are large. These, I think, were the main points of contention you were concerned about.

I would posit that what defines important to me may not be what defines important to you and the same can be said when you look at how various "internet" companies look at what's important in their vertical.

We're not talking about relative importance; we're talking about whether the problems you identified with the four IXPs named in your talk are representative of problems with the larger IXP community. I cannot find evidence that they are, at least not in the areas that you identified as problems.

Netnod runs a dns root server system (i.root-servers.net <http://i.root-servers.net>) as well as a heavy duty time service.

There are others who do this for no cost and some who do it for government money. Whether or not my port fees should subsidize this is a valid question, and was brought up in the Q&A afterwards.

All root operators do this for no charge, but at substantial cost. Running a root dns server system is one of the things what Netnod does because that's one of the things that the organisation is chartered to do.

Regarding the pricing reduction on page 16 of your preso, the US$ and UK£ are not much different than what they were 5 years ago, but the € has dropped by 30% against the US$.

You speak to this below, however if my business is primarily run in USD (which was the relevant use case presented: I'm a US company deciding if I should peer in Europe or buy transit) then those currency fluctuations have a very different impact than if I'm a European company functioning primarily in local currency.

Oh sure, but this is a matter that you need to take up with your financial people. I have no doubt that Netflix employs smart financial people, and that their decisions are the right thing for Netflix. IXPs are going to operate in their local currency and they cannot be held responsible for international currency fluctuations. From this respect, I don't think it's useful to bring this up in a critical context because it's not something that they can influence in any way whatever.

I did purposefully mention SIX as a polar opposite example - there is definitely a happy medium to be found.

This edges into one of the things that is crucial to this discussion, and it was unfortunate that it wasn't explored more. The crux is that there is a substantial cultural difference between how US people view IXPs and how european people view IXPs. As far as I can tell there are, for the most part, two types of IXPs in the US: commercial and co-operative. How they differ from european IXPs is that the commercials are almost all run by the data centres and are tied to those data centres. Most if not all of the co-operative IXPs are to some degree or other financed by donations or sponsorship and the donation types are: cash, equipment and manpower. In europe, there are three types of IXP: commercial, member based and non-member, non-profit. Many of the commercial IXPs are not owned by the data centres (e.g. NL-IX, ECIX, etc). The member-owned IXPs are answerable fully to their membership (e.g. LINX, INEX), and the non-member, non-profit IXPs (Netnod, VIX, etc) provide a service to the community as they see fit, but are not required to answer to the organisations who use them for peering services, even if they are likely to listen to what those organisations say. Crucially, almost all of the european non-profit IXPs are 100% self-funded without donations, sponsorship or subsidisation of manpower. They have offices, admin people, support staff and everything else that a normal business has. IXP staff are paid market rates because if you don't do this, they walk. They do this because this having a dependable income source ensures long term stability and their members / customers / peering participants need to know that the organisation that supports their business is built on a sound structural foundation. This matters to them and they are prepared to pay for it. The flip side of it is that European non-profit IXPs are, by design, more expensive than US non-profit IXPs and there are almost no free / zero-recurrent-cost IXPs in the region. So I would question comparing a european IXP with e.g. SIX or CommunityIX and would suggest that this is a case of apples and bananas. Both are yummy, but whether you prefer one or another is a matter of taste. How IXPs spend revenue is a different matter. You're right to bring this topic up because scrutiny is what keeps things honest. Nick

Hi Nurani, Much of what you've asked me below is answered up-thread, so I'm not going to rehash it for the sanity of the others following this discussion. I have snipped what hasn't been. On Thu, Jun 16, 2016 at 8:52 AM, Nurani Nimpuno <nurani@netnod.se> wrote:

I take your point about the Netnod fees (even though I would also like to point out that we have actually reduced our other port fees for 100mbps, 1G, remote peering). But I’m not sure why you haven’t brought it to us directly. Netflix has been at several Netnod meetings in the past, so we have had plenty of opportunity to discuss this.

Nothing in my presentation said "Netflix seeks to get better port fees". You'll find that I, not once, in my deck or oral presentation, mentioned Netflix. I spoke at length with LINX after the presentation and pointed out that I seek to help the entire market, not just my employer, better understand how IXPs price their services, what things are negotiable, and what things need to change. Call it thinly-veiled, but I didn't even use my employer slide master - this was geared as a community discussion.

And I don’t represent a membership-based IXP.

An important distinction. Poring through http://www.netnod.se/about/documents , there is very little transparency into the actual operations of NetNod.

If you stop adding value to those networks peering at the IX, you will slowly become irrelevant.

And therein lies the rub, we (many of us, not just you and I) disagree about what "adding value" is defined as. I'm glad we can have this conversation.

While some think that a good technical solution would sell itself, I believe that is a fallacy (not only in the IXP world). Netnod started out as a very small IXPs with only a few local operators connected to it. And I strongly believe that if we hadn’t done as much outreach as we do, we would’ve stayed tiny until this day.

Outreach is fantastic!

We work in a similar way with our pricing. (You mention that there is a lot of negotiations on pricing with IXPs.) I would like to be 100% clear that for the Netnod IX, we don’t negotiate or give “sweet deals” to anyone. We publish our fee schedule and we stick to it. Whenever someone wants a special deal (which happens often, particularly with the larger customers), our response is that we treat everyone equally. If you want a cheaper deal, then another customer is basically funding your reduction. So we don’t do this. We believe this is more fair and transparent.

That's fantastic, and I agree with this approach. And that's why it's important to make this a community discussion, not a "Netflix and Netnod" discussion.

As for a general discussion about costs, service levels and IXPs, I think there is a very interesting discussion that could be had with a more focused discussion. How do “we” best serve today's very diverse set of operators? How does an IXP strike that balance? How do operators best solve their interconnection needs (through IXPs, private peering, transit etc) and is that changing? What type of interconnection environment do we believe best scales Internet growth in the future? What is the total cost of interconnection, where are the big costs, what are the different models and where is the whole industry moving? Now THOSE are discussions I personally would find very valuable!

We agree. I'm really glad that this has sprouted so many threads of discussion. This seems to have kicked off the discussion within the larger community beyond just the four examples, and I think that what we've seen thus far is healthy discourse.

Hi Dave,

On 16 juni 2016, at 16:40, Dave Temkin <dave@temk.in> wrote:

<snip>

Nothing in my presentation said "Netflix seeks to get better port fees". You'll find that I, not once, in my deck or oral presentation, mentioned Netflix. I spoke at length with LINX after the presentation and pointed out that I seek to help the entire market, not just my employer, better understand how IXPs price their services, what things are negotiable, and what things need to change. Call it thinly-veiled, but I didn't even use my employer slide master - this was geared as a community discussion.

Ok.

And I don’t represent a membership-based IXP.

An important distinction. Poring through http://www.netnod.se/about/documents , there is very little transparency into the actual operations of NetNod.

Well, we do describe our governance structure and we are always clear about being owned by a foundation on our information material. We even present financial figures at our Netnod meetings. But ok, maybe this could be better documented on our website. Fair enough. Our current website sucks somewhat and we’re in the process of reworking it, so I’ll take your point onboard and we’ll try to improve this.

If you stop adding value to those networks peering at the IX, you will slowly become irrelevant.

And therein lies the rub, we (many of us, not just you and I) disagree about what "adding value" is defined as. I'm glad we can have this conversation.

Yes. And we will never agree. You and I may of course agree in one point in time, but all the world’s operators will never agree. I think this thread has proven that. Some seem to argue that all IXPs should simply be a donated L2 switch sitting in free rack space, while others clearly need more than that. Having a discussion about that is useful, I agree. And it’s a discussion that will continue to evolve as the industry evolves. And it will maybe also reach different conclusions for LINX, as opposed to INEX, LONAP or Netnod (which was the point I was trying to make about diversity). Also, as we know, IXPs is not the only solution to interconnection. To me it was not clear that this is the conversation you wanted to have. If that’s the case, then great! <snip>

We work in a similar way with our pricing. (You mention that there is a lot of negotiations on pricing with IXPs.) I would like to be 100% clear that for the Netnod IX, we don’t negotiate or give “sweet deals” to anyone. We publish our fee schedule and we stick to it. Whenever someone wants a special deal (which happens often, particularly with the larger customers), our response is that we treat everyone equally. If you want a cheaper deal, then another customer is basically funding your reduction. So we don’t do this. We believe this is more fair and transparent.

That's fantastic, and I agree with this approach. And that's why it's important to make this a community discussion, not a "Netflix and Netnod" discussion.

This is slightly different (although somewhat related) to “what value do IXPs bring?”. This is about keeping the IXPs honest. Like Nick, I’m all for that.

As for a general discussion about costs, service levels and IXPs, I think there is a very interesting discussion that could be had with a more focused discussion. How do “we” best serve today's very diverse set of operators? How does an IXP strike that balance? How do operators best solve their interconnection needs (through IXPs, private peering, transit etc) and is that changing? What type of interconnection environment do we believe best scales Internet growth in the future? What is the total cost of interconnection, where are the big costs, what are the different models and where is the whole industry moving? Now THOSE are discussions I personally would find very valuable!

We agree. I'm really glad that this has sprouted so many threads of discussion. This seems to have kicked off the discussion within the larger community beyond just the four examples, and I think that what we've seen thus far is healthy discourse.

Sure. A broader discussion would be both useful and interesting. Nurani

Hi, I have studied Netnod extensively because we want to become members, but we can not simply because it is too expensive. I just signed a deal with he.net for a flatrate 10G transit for about the same as the 10G Comix port cost. The difference being that the he.net port provides much more value and besides also provides indirect one-step-away peering with everyone on Comix. So from my perspective it is clear that Netnod has a pricing problem here. Especially for the lower speeds (10G). There is also a value problem because the only Comix peer that moves a lot of traffic to us is Akamai, and they promised that we could peer directly skipping the middleman. We are based in Copenhagen. The Netnod IX in Stockholm would provide a lot more value, but to get there we have to add the cost for transport and after doing that, the comparison to the 10G he.net transit is just silly. Here is an opinion: If the IX pricing is comparable to transit, the service needs to be too. Netnod will need to connect the five (I think there are five) Netnod IX'es into one big domain. I am meeting with NL-IX next week and as I understand it, that is exactly what they did - we will probably buy NL-IX and skip Netnod for this single reason. I feel that smaller providers are being let down by the IX community at this point. The value of "smaller" is going to get larger if the IX community does not move with the transit providers. We want to take part but there is a limit of how much over price you can sign onto and keep your job. Regards, Baldur On 16 June 2016 at 15:52, Nurani Nimpuno <nurani@netnod.se> wrote:

Hi Dave,

So, I watched your presentation this week at NANOG remotely, sorry I couldn’t be there.

Ok, so while you make a lot of very different points in your presentations, I *think* the basic argument you are making is that IXPs are too expensive. Correct me if I’m wrong. Or more specifically, you are saying that Ams-IX, Linx, Netnod and DE-CIX are too expensive. You have not looked at US IXPs because they don’t publish their fees, and you have not looked at the whole IXP community.

I think you are then also questioning if these IXPs are using their funds wisely. You are also stating that you are talking about these IXPs from the perspective of a big US provider connecting into Europe (i.e not a small local ISP). You question some of the IXP expansions into the US. You question the membership model as a viable model for IXPs. You also say that those who sustain the IXPs growth should benefit from them. And you question why there are so many IXPs, and not only a handful of very big ones. I hope I have captured this correctly.

Ok, so firstly, I must say I’m a little disappointed that you or your staff have never approach us to discuss any of this. We have Netnod meetings twice a year, we have been present at many of the same events in the last year and we have always strived to be open, transparent and to listen to our entire customer base. I take your point about the Netnod fees (even though I would also like to point out that we have actually reduced our other port fees for 100mbps, 1G, remote peering). But I’m not sure why you haven’t brought it to us directly. Netflix has been at several Netnod meetings in the past, so we have had plenty of opportunity to discuss this.

But ok, let’s leave that aside. I will try to address some of your points.

Firstly as many have pointed out, these four IXPs are not representative of all IXPs, and the four of us are also very different from each other.

I can’t address the IXP expansion into the US. And I don’t represent a membership-based IXP.

The European IXP community is a very diverse one, serving different regions, markets and different types of customers. I personally believe that this rich diversity is one of the reasons the European interconnection scene has been flourishing as well as it has. There is a big difference between Europe and the rest of the world, particularly the US. And the European IXP community was held up as a model for the rest of the world by many. We have been cooperating for many years through the Euro-IX where our common goals have been to improve interconnection in the region, share information and experience and work to improve services for our customers. (I believe you have been trying to do the same through Open-IX.)

The diversity has also been seen as important to serve both the very large international providers like yourself, and the small local ISPs. Localising traffic and building a local operator community have been seen as an important ingredient in the value of the IXPs. Our challenge as IXPs is to find the best way to serve all these different needs and wishes from our very diverse customer base. Having only a handful of very large IXPs would in my view not serve these different needs as well. Personally I am a subscriber to both Netflix and HBO. I like diversity. :) But sure, it’s an interesting discussion to be had!

As others have pointed out, contrary to common belief, the technical part of an IXP is one of the simplest. There is a plethora of examples of IXPs in Africa, but also in the US, where IXPs simply are a single switch sitting in a closet somewhere, only serving a handful of ASes. One of the biggest challenges for an IXP is to gain customers and get enough gravitation and value to the exchange. A growing exchange point is not only a "nice-to-have" for those operating it, but vital to those networks who peer there. If you stop adding value to those networks peering at the IX, you will slowly become irrelevant.

While some think that a good technical solution would sell itself, I believe that is a fallacy (not only in the IXP world). Netnod started out as a very small IXPs with only a few local operators connected to it. And I strongly believe that if we hadn’t done as much outreach as we do, we would’ve stayed tiny until this day.

As for how we do this outreach and what events we go to, while I can’t speak for any other IXes, I seriously doubt that any professional IXP today would not carefully assess the business value for each event it attends. At Netnod, we evaluate each event we send people to, and assess and measure the value afterwards.

Then I thought I would write some words about Netnod specifically since you bring us up.

(As others have pointed out, the RIPE meeting social is covered partly by the RIPE NCC, partly by the sponsor, and partly by the participants themselves, so I’ll just leave that there.)

Firstly, yes we are a little strange. We are not just an IXP. We run i-root.servers.net and we provide DNS anycast service, among other things. We also have a funny governance structure for historical reasons (which was set up when Netnod was established and the IXP and I-root “moved” there) many years ago. We are owned by a foundation and we describe ourselves as non-profit. In Sweden there is actually no “non-profit” status as such, but we have always operated under that principle. We are not a membership organisation, but we have always strived to be transparent, and whenever we have wanted to make major changes to our services, we have consulted the customer base. That is how we have worked on both the IX and DNS side.

We work in a similar way with our pricing. (You mention that there is a lot of negotiations on pricing with IXPs.) I would like to be 100% clear that for the Netnod IX, we don’t negotiate or give “sweet deals” to anyone. We publish our fee schedule and we stick to it. Whenever someone wants a special deal (which happens often, particularly with the larger customers), our response is that we treat everyone equally. If you want a cheaper deal, then another customer is basically funding your reduction. So we don’t do this. We believe this is more fair and transparent.

Coming back to Netnod's broader scope, this also means that we do other things than sell peering. We go to, and sponsor events that might not make sense from a peering perspective. We support other “Good of the Internet” initiatives, we participate in standards development (particularly on the DNS side), we participate in TLD associations etc. Some of these activities may seem odd to some who are only involved with one part of our business, I understand that. We try to be open with this though.

As for a general discussion about costs, service levels and IXPs, I think there is a very interesting discussion that could be had with a more focused discussion. How do “we” best serve today's very diverse set of operators? How does an IXP strike that balance? How do operators best solve their interconnection needs (through IXPs, private peering, transit etc) and is that changing? What type of interconnection environment do we believe best scales Internet growth in the future? What is the total cost of interconnection, where are the big costs, what are the different models and where is the whole industry moving? Now THOSE are discussions I personally would find very valuable!

Cheers,

Nurani Netnod

...

On 15 juni 2016, at 13:21, Dave Temkin <dave@temk.in> wrote:

On Wed, Jun 15, 2016 at 3:43 AM, Aled Morris <aledm@qix.co.uk> wrote:

...

Me too and I was confused about what the point of it was.

I had always assumed the customers of those IXs he singled out were generally happy with the service they were getting and the money they

are

...

paying.

Is Dave trying to say they are being duped? Is he trying to identify a need for regulation?

I was pointing out facts about IXPs that many did not know, including the actual organizational structure.

I was also opining on how these IXPs could be better; mainly, how they choose to spend money.

...

Perhaps Dave was advocating the SIX model and suggesting the customers

of

...

the existing exchanges should be looking to organise an alternative in their localities.

Absolutely correct (which should answer Hank's question, as well).

...

Or perhaps this is a wakeup call for LoNAP and the smaller exchanges who "compete" with AMS-IX, DE-CIX and NetNod - stop trying to mimic their commercial models (big fees which pay for staff and marketing) and look instead at the lean SIX as the way of offering a service at a price competitive to transit.

Also absolutely correct. I don't want to see them falling into a trap of conflating marketing and outreach and/or offering an overly rich product set at the cost of price and operational simplicity.

...

Or was there a hidden message in Dave's presentation that I missed?

Seems like you got it.

...

Aled

On 17 June 2016 at 03:18, Dave Temkin <dave@temk.in> wrote:

A key point:

On Thu, Jun 16, 2016 at 6:09 PM, Baldur Norddahl < baldur.norddahl@gmail.com> wrote:

I have studied Netnod extensively because we want to become members

...

You meant a customer, but because of a lack of transparency (and great marketing) amongst some IXPs, it's very easy to conflate member-mutual IXPs, commercial, and non-profit IXPs. Being a "member" of NetNod provides you with a very different set of benefits vs. LINX and unless you read the letters of incorporation, you may not know that.

Ok, customer. Actually I don't care. I will evaluate the business value no matter who owns it and I will vote with my dollars/euros/kroner. Yes it would be nice if we had a true self owned entity that would help people peer the most cost effective way and only that. Apparently they have something like it in Seattle. But not in my part of the world, hence I view them all as purely commercial entities. I need a good balance of price and quality and will buy from whoever comes with the best business proposal. Regards, Baldur

On Fri, 17 Jun 2016, Mikael Abrahamsson wrote:

If you are, I'm very interested in hearing your motivation for doing that.

I realised I probably used the wrong english word here. The correct english word(s) would probably be "rationale/reason/facts", not "motivation". -- Mikael Abrahamsson email: swmike@swm.pp.se

On 17/06/16 01:09, Baldur Norddahl wrote:

Hi,

I have studied Netnod extensively because we want to become members, but we can not simply because it is too expensive. I just signed a deal with he.net for a flatrate 10G transit for about the same as the 10G Comix port cost. The difference being that the he.net port provides much more value and besides also provides indirect one-step-away peering with everyone on Comix.

So from my perspective it is clear that Netnod has a pricing problem here. Especially for the lower speeds (10G). There is also a value problem because the only Comix peer that moves a lot of traffic to us is Akamai, and they promised that we could peer directly skipping the middleman.

We are based in Copenhagen. The Netnod IX in Stockholm would provide a lot more value, but to get there we have to add the cost for transport and after doing that, the comparison to the 10G he.net transit is just silly.

Here is an opinion: If the IX pricing is comparable to transit, the service needs to be too. Netnod will need to connect the five (I think there are five) Netnod IX'es into one big domain. I am meeting with NL-IX next week and as I understand it, that is exactly what they did - we will probably buy NL-IX and skip Netnod for this single reason.

I feel that smaller providers are being let down by the IX community at this point. The value of "smaller" is going to get larger if the IX community does not move with the transit providers. We want to take part but there is a limit of how much over price you can sign onto and keep your job.

Regards,

Baldur

For me (and middle-sized non-profit ISP) this is something that will eventually solve itself, because of people tend to vote with the wallets. As of now (looking at job's excellent spreadsheet) we can see that regular transit is cheaper or atleast on-par with 6-8 of the more expensive IXPs in that spreadsheet. We can use hurricane-electrics longlasting google-ad that promises $0.26/Mbps on a 10G as a baseline, there is cheaper and much more expensive bits to be had naturally but for the sake of argument. While transit is almost always a better thing then connecting a IXP if cheap bits is of your prime concern, there is still quality to gain by being able to take care of some of the traffic yourself over an IXP. But this also has a tippingpoint in terms of price. The other battle IXPs needs to take into consideration is the price of PNI´s nowadays. It is MUCH easier to get really good value from private interconnects seeing as most operators has more traffic to less destinations then what we had a few years back, and $BIGCLOUD is usually very forthcoming into helping out with this since the bits is the product they sell, not the transport of the bits. A 10G port at DECIX Frankfurt is 22800EUR/year. For 22800/year you can go out in the market today, buy a jericho-switch which does internet-scale tables and BGP and has buffers and all the stuff you need for a peering-edge and will get 1T worth of faceplate connectors, support for that same switch for about three years and you still has some change left for icecream. Thats alot of possible PNI´s. Then we need to factor in costs of crossconnects - if you are in a $BIGNAMEDATACENTER will probably have to pay anything between 19 - 300 EURO/month for said crossconnect (or the other company has to, or you split it). If you are in $NOTSOBIGNAMEDATACENTER you can perhaps pull the patch yourself and only pay the CAPEX of 11EUR of buying the patch so this varies heavily. What i do when i think about these things is to produce a matrix where the cost of the IXP in that region is measured towards a full TCO of a PNI, factoring in what a 10G/100G port costs in terms of buying hardware and then all the OPEX surrounding it in terms of hardwaresupport/xconns/powerconsumtion/patching/documentation of patching etc etc. For me - PNIs make economical sense if i can shave of 313Mbit/s of my IXP (cheap DC / high IXP cost) in the lowside, and about 3.67Gbit/s on the highside (expensive DC / cheap IX). And this is with quite high port CAPEX (juniper mx), when and if peering-edge is being produced in something like a Jericho-box i expect these requirements will drop to the floor. Last year i added 0 new IXPs, upgraded 0 IXPs, but i added over 30 new PNI's. If IXPs wants more of those bits, adjusting prices much more aggresively is what can bring this back to their market, instead of the datacenter-crossconnect market. -- hugge

Hmm - as far as whether this was a good or bad NANOG presentation...this is some of the best discussion I've seen on list in a while. There is a frank exchange of views between many different parties. This may result in some follow-up presentations at future NANOGs by IXP operators (please!). Seems that, whether you agree with Dave or not, it was successful. It also seems that the IXP operators who came under the most criticism have reacted with a lot of professionalism and maturity. Other IXP operators have reacted pretty poorly, which is ironic. Dan

I have to agree with Rick here. Owen

On Jun 29, 2016, at 22:43 , Rick Astley <jnanog@gmail.com> wrote:

I have to agree with Dan in that even if you disagreed with the talk you have to agree that it probably spawned relevant discussion and reflection (both on and off list). I would hate to see a move to ideas and discussions that are chosen simply for offending the fewest people. Another sort of similar critique aimed at large routing vendors was "Help! My big expensive router is really expensive" at NANOG 60 in Atlanta. Perhaps the critiques were seen as more constructive and I don't remember the same backlash after the talk but I found both talks and various discussions that followed insightful.

On Fri, Jun 17, 2016 at 4:53 PM, Daniel Golding <dgolding@gmail.com> wrote:

...

Hmm - as far as whether this was a good or bad NANOG presentation...this is some of the best discussion I've seen on list in a while. There is a frank exchange of views between many different parties. This may result in some follow-up presentations at future NANOGs by IXP operators (please!).

Seems that, whether you agree with Dave or not, it was successful. It also seems that the IXP operators who came under the most criticism have reacted with a lot of professionalism and maturity. Other IXP operators have reacted pretty poorly, which is ironic.

Dan

Perhaps Dave was advocating the SIX model

that is where the big euro exchanges started. then they got equinix envy and colonialism. let's see (and help) the six avoid these diseases over the next years. randy

So here we are now... Where do we want to go?

I think IXPs have indeed become too much like ISPs, providing more services but also increasing complexity and cost. I prefer simple, scalable and cheap solutions! I want to go to an IXP being a nice simple ethernet switch. Add some nice graphs and a route server, and we're done. Redundancy is a separate switch :) Cheers, Sander

On Wed, 15 Jun 2016, Sander Steffann wrote:

I want to go to an IXP being a nice simple ethernet switch. Add some nice graphs and a route server, and we're done. Redundancy is a separate switch :)

So how should the larger distributed IXPs solve this? Provide optical DWDM transport? Dark fiber? What if there are no chassis on the market large enough to accomodate creating a single switch for all customers to connect to? Have multiple L2 domains and require people to connect to multiple switches? Even bumping people off of an existing switch when that is full, to move some traffic over to another new switch chassis? What about buffer requirements? If you want a buffered switch, it increases capex and lowers number of switch-models that can be used. Microbuffered switches may lose packets in microbursts when ports are being run (near) full. -- Mikael Abrahamsson email: swmike@swm.pp.se

On 15.06.2016 20:23, Sander Steffann wrote:

...

So here we are now... Where do we want to go?

I think IXPs have indeed become too much like ISPs, providing more services but also increasing complexity and cost. I prefer simple, scalable and cheap solutions!

You all know this saying: Fast, Good or Cheap. Pick any two!

I want to go to an IXP being a nice simple ethernet switch. Add some nice graphs and a route server, and we're done. Redundancy is a separate switch :)

Unfortunately the IXP world - at least for the really big IXPs - is not longer that simple. Paradise was yesterday ;-) btdt Arnold -- Arnold Nipper email: arnold@nipper.de phone: +49 6224 5593407 2 mobile: +49 172 2650958 fax: +49 6224 5593407 9

On Thu, Jun 16, 2016 at 12:58 PM Will Hargrave <will@harg.net> wrote: {snip}

Dan Golding disagreed with me but I can certainly speak for LONAP where I feel our mission of “promoting efficient interconnection in the UK” is hugely enhanced by our ability to provide services in any of our current seven datacentres, across four different operators. London would not be the great city of interconnection it is without the east London cluster of DCs from different operators.

Of course, that's not what I said, but only people who were present actually know that. You said that LONAP's distributed strategy "kept datacenters honest" to use your exact quote. That implied some sort of benefit for members in acting as some sort of counterweight to (rapacious?) data center providers. I made the point that distributed IX's don't really impact power or space costs in data centers. I can provide actual data on this, if you would like. Dan

{snip}

Will

In a message written on Thu, Jun 16, 2016 at 05:56:36PM +0100, Will Hargrave wrote:

Most of the major IXs in the European market operate in multiple datacentres. Why? Because it decreases the monopoly conferred upon one particular datacentre in a market which becomes the ‘go to’ location.

It moves the monopoly to the IXP operator! When everyone is in one facility (or at least building) it is typically easy to get low priced (although maybe not low enough, see other posts in this thread) cross connects. It's common to see a pair of public peers fill up a significant part of their port, and then move to a private peering model getting off the IXP and onto glass directly. When the IXP is distributed, this becomes glass between buildings, often requiring yet another supplier as well. The MRCs are higher making the justification to move off harder. What happens is rather than moving off to glass, they have to buy faster/more ports from the IXP and move the traffic over the IXP. The IXP becomes the go-to monopoly as a result. Now, perhaps IXP's are more benevolent than data center opertors, and this is a good trade. I think one thing the presentation was asking people to do was step back, look at the situation, and reevaluate that particular tradeoff. -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/

In a message written on Fri, Jun 17, 2016 at 02:58:12PM +0100, Marty Strong wrote:

Yes, if the IXP is distributed across more than one building then you have choice as to where you (and other people) put their equipment, so you may have to go to another building to connect to certain peers. Sadly nobody lives in a perfect world, so IMO having the IXP distributed across multiple buildings is better as you can connect to all those who are in your building directly, and peer with the rest over the distributed IXP.

I don't think there is an absolute right or wrong answer. The ISP who needs to connect to 100 ISP's at 50M each has a dramatically different need than the ISP that needs to connect to 20 ISP's at 6x100G each. Both exist in the world. The presenter clearly thought that a number of IXP's aren't serving their customers/members well. What we're finding out in this thread is how many folks agree or disagree! :) Personally I'm with another poster, the real problem here is colos that want to charge large MRC's for a cross connect. I know of at least one still trying to get $1000/mo for a fiber pair to another customer. For $1000/mo I can get GigE transit delivered _to my office_ by multiple carriers. To charge that for a cross connect is just so, so wrong. IMHO in building fiber should be NRC only, but if it has a MRC component (to pay for future troubleshooting or somesuch) it should be small, like $5/mo. That's $60 year to do nothing, and even if the $40 an hour fiber tech spends a hour troubleshooting _every fiber_ (which doesn't happen) the colo still makes money. Cross connects are our industry's $100 gold plated HDMI cables. -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/

Starting to see people like Telehouse move into the monthly XC market, so one might think we're at the precipice of the slippery slope. And with Equinix buying Telecity, how long until we see US-style XCs in Europe? On Fri, Jun 17, 2016 at 10:29 AM, Nick Hilliard <nick@foobar.org> wrote:

Leo Bicknell wrote:

...

Cross connects are our industry's $100 gold plated HDMI cables.

In the US maybe. Cross-connect prices are much more reasonable in Europe (€0 - €50/month).

Personally, I don't have a problem with MRCs when ordering cross-connects: data centres are expensive to build and run. But yeah, $300/m is nuts and that price point has zero relationship to the cost of delivery of the service.

Nick

On 20/Jun/16 09:59, Thomas Mangin wrote:

Telecity Manchester (UK), now Equinix Manchester, have charged MRC for internal cabling since forever (in my case, forever being 2001 when I first became customer). They normally run their cables through their switches but when the distance is short enough you can insist on a P2P.

Really? The x-connect is run through active equipment operated by the data centre? Is this a specific service you purchased, or is this the way they deliver x-connects? Mark.

On 20/Jun/16 10:17, Jürgen Jaritsch wrote:

Same drama in Chicago with Atlantic Metro ... you can purchase a SMF DF xcon for 800USD/month ... everything else is actively transported on an Ethernet platform with simple/stupid VLAN tagging. You're even receiving their STP packets ....

Well, now I know where I won't co-lo... Mark.

On 20/Jun/16 10:30, Thomas Mangin wrote:

It is how they provide x-connects.

Which is fine, but does not work for me. So I won't be using them. Thanks for the info, wasn't aware. Mark.

Hi, well, you an say one thing - the talk got a lot of conversation going - most of it useful and positive and informational.....isnt that the sign of a good talk? ;-) seriously, this thread has been very active/alive based on the initial trigger of his talk. as for the talk itself....everyone has their views....and people should feel free to provide their opinion when on the soapbox/presentation stage - as long as its within the law (in some doamins being offensive / testing boundaries is part of the territory - eg comedians - but I wouldnt accept that sort of boundary/officensiveness at an IT/networking presentation). theres an old adage about opinions and everyone having one....its a tru-ism for sure - but whilst he might not have had a full picture the resulting conversation on this mailing list has provided much information. Now, just need similar talk on the topic of BGP peering security ;-) alan

...

Cross connects are our industry's $100 gold plated HDMI cables.

In the US maybe. Cross-connect prices are much more reasonable in Europe (€0 - €50/month).

Personally, I don't have a problem with MRCs when ordering cross-connects: data centres are expensive to build and run. But yeah, $300/m is nuts and that price point has zero relationship to the cost of delivery of the service.

[ i know you know this, but i hope you do not mind being a soapbox ] back in the day, the seattle westin building (not the hotel, but a 33 story office building next door) had a smart manager. he had a riser shaft built out and set a very low nrc/mrc for copper and fiber. given the building's copper mmr (meet me room) and a fiber mmr, colos exploded, and now the building is mostly racks, vying for being the largest carrier hotel in the left coast. the six was built and expanded rapidly in this fertile environment. no, the westin is not making millions off of cross-connects, but they do not have rental vacancies. due to the shortage, and maybe price (dunno), of westin space, the six has been extended to a few other buildings; see diagram at bottom of https://www.seattleix.net/topology. this makes sense to me. extensions to distant cities make less sense to me; but i am an old fogey. randy

On Fri, 17 Jun 2016, Eric Kuhnke wrote:

What Randy just wrote is exactly the point I was trying to make in my last email. Some real estate facility owners/managers have got into the mistaken mindset that they can get the greatest value and the most monthly revenue from the square-footage of their building by charging additional MRC XC fees to the tenants of the building.

There are some VERY sucessful companies that would strongly disagree with you.

When in fact the opposite is true, and we need a concerted community effort to lobby every IX real estate owner with this fact: Your real estate will be MORE valuable and will attract a greater critical mass of carriers, eyeball networks, CDNs, huge hosting providers/colo/VM, etc if you make the crossconnects free.

But then why would we want to do that? If you are correct and doing so would raise the value of the real esatate, doesn't that mean that the building managers would be able to charge operators a whole lot more than they are able to today, in aggregate? Value based pricing is all the rage these days, which is why they charge you so much for cross connects. Why do you think they wouldn't take advantage of higher value real estate by charging you more for that, instead? After all, the free cross connect situation would be a great way for the owners to lock you into their real estate, then all they have to do is dramatically hike the rates when you can no longer leave. -- Brandon Ross Yahoo & AIM: BrandonNRoss +1-404-635-6667 ICQ: 2269442 Skype: brandonross Schedule a meeting: http://www.doodle.com/bross

On 6/17/16 07:59, Leo Bicknell wrote:

IMHO in building fiber should be NRC only, but if it has a MRC component (to pay for future troubleshooting or somesuch) it should be small, like $5/mo. That's $60 year to do nothing, and even if the $40 an hour fiber tech spends a hour troubleshooting _every fiber_ (which doesn't happen) the colo still makes money.

I would expect some kind of MRC if it has any SLA, service, or support attached. Or someone manages it and protects the infrastructure and enforces the rules of the facility. Or the facility uses that money to maintain the MMR. If it's a free for all where anyone can accidentally unplug it or cut it or rummage around in the overhead causing damage then free is fine. What I don't accept is variable pricing depending on what the xconnect is carrying or what it's for. I don't buy into the thought process that an xconnect is more expensive if it's carrying 10GbE vs. GigE when they're both SMF. ~Seth

On 17/Jun/16 19:31, Seth Mattinen wrote:

I would expect some kind of MRC if it has any SLA, service, or support attached. Or someone manages it and protects the infrastructure and enforces the rules of the facility. Or the facility uses that money to maintain the MMR. If it's a free for all where anyone can accidentally unplug it or cut it or rummage around in the overhead causing damage then free is fine. What I don't accept is variable pricing depending on what the xconnect is carrying or what it's for. I don't buy into the thought process that an xconnect is more expensive if it's carrying 10GbE vs. GigE when they're both SMF.

You won't see this is typical carrier-neutral facilities. You're more likely to see this in telco- or consortium-owned facilities, in specific regions that exploit telecommunications requirements to milk customers. Mark.

* zbynek@dialtelecom.cz (Zbyněk Pospíchal) [Thu 16 Jun 2016, 14:23 CEST]:

Dne 15.06.16 v 20:10 Mikael Abrahamsson napsal(a):

...

Well, the customers also wanted more functions and features. They wanted sFLOW statistics to show traffic, customer portals, better SLAs, distributed IXes, remote peering, more hand-holding when connecting etc.

Are you sure they still want them if they have to pay for these features separately?

Currently, such luxury functions are increasing costs also for networks who don't need/want it.

sFlow statistics isn't a luxury function. Neither is remote peering. The others Mikael mentioned are debatable at worst but you'd be telling the membership what they really want rather than listening to them saying what they want. This thread is full of people who have never run large L2 networks stating their opinions on running large L2 networks, and they invariably underestimate their complexity and the logistics required. -- Niels.

...

This thread is full of people who have never run large L2 networks stating their opinions on running large L2 networks, and they invariably underestimate their complexity and the logistics required.

* randy@psg.com (Randy Bush) [Thu 16 Jun 2016, 17:56 CEST]:

maybe the complexity and the logistics required are WHY they don't build large L2 networks. SMITH: Doctor, it hurts when I do this. DALE: Don't do that.

Wait. I thought vijay "your solution doesn't scale" gill was a hero of the NANOG community, but now you're telling me that actually scaling up is a sin?

...

sFlow statistics isn't a luxury function. Neither is remote peering. by 'remote peering' do you mean an exchange essentially selling transit?

I mean the common practice of connecting via a L2 pseudowire with a provider that has an arrangement with the IXP to do so, rather than putting a router in a datacenter the IXP is in and then connecting to that router with possibly the same provider to the rest of your network. Mikael Abrahamsson probably meant the same thing when he used the term two mails upthread. -- Niels.

Hello all, I wasn't able to attend NANOG this time around, but watched Dave Temkin's presentation on youtube. My comments are: 1) Over the past 5 years: My cost for switch/router ports have gone down a lot. My cost for transit has gone down a lot. My cost for exchange ports have gone down, but not quite as fast as my transit and switch/router ports, and this does lead to some value questions. Dave is right to ask them. However: exchange port fees are not my biggest enemy today. My cross connect fees have not gone down *at all*. On a proportion basis, cross connect fees have gone from "not mattering" to being an important part of any deployment cost calculation. Why aren't we raising hell about cross connect fees? 2) Exotic features -- Pvlan, L2VPN, L3VPN have absolutely no purpose on an exchange. If it could be done 'free' with commodity hardware, then fine -- but if it translates to requiring Big Expensive Routers instead of a cheaper but fast switch, this should translate to higher pricing for the customers requiring these exotic features -- not the customers who just want a big L2 vlan. 3) Remote peering -- This is mostly a question about distance for value. There is a clear benefit in providing multi-datacenter exchanges within a metro, and both FL-IX and SIX are doing this with a very good value proposition. Having the ability to join DECIX Frankfurt from NYC and vice versa -- again, this is a bizarre service to be offered, and regular users should not be expected to pay for this. If there is a market for these services at an unsubsidized price, then fine -- but regular members should not be subsidizing this service. 4) sFlow -- I'm not sure why this is even really a topic. Commodity hardware does have sFlow capability, and FLIX demonstrates this well. With that said, for us, it is of extremely limited value. We might check these graphs to validate measurements of our internal netflow/sflow graphing systems, but generally, I look at the graphs generated by my exchange vendors less than once per year per exchange. I am honestly not even sure if SIX offers this service, as I never had a reason to check. 5) Marketting vs Outreach: These things are honestly basically the same thing, mostly separated by the question of "is it good marketing or not". I like having more members at the exchanges I am a member of. If it translates to an additional 3% per year to have an additional 5% of traffic to new members, I am fine with this. If it translates to an extra 50% of cost for 5% of additional traffic, I am not fine with it. Finally -- there is nothing wrong with asking questions. If you are an exchange company and you can defend your prices for what you offer, then there is no problem. If you are an exchange and are mostly just hoping nobody asks the questions because you won't have any good answers -- well, I think this is exactly why Dave asked the question. Best Regards, -Phil Rosenthal

On Jun 16, 2016, at 1:58 PM, Adam Rothschild <asr@latency.net> wrote:

I think a fresh conversation is needed around what makes up a "minimally viable" feature set for an IXP:

The days of an IXP "needing" to engineer and support a multi-tenant sFlow portal, because the only other option is shelling out the big bucks for Arbor, have long passed -- overlooking the plethora of open sourced tools, folk like Kentik have broken into that market with rationally priced commercial alternatives. Likewise, one might argue that offering layer-2 and layer-3 (!) VPNs is at best non-essential, and a distraction that fuels purchasing very expensive hardware, and at worse competitive with customers.

On the other hand, building out a metro topology to cover all relevant carrier hotels, with reasonable path diversity, is absolutely table stakes. And outreach is a great function, *when* it nets unique new participants. To cite a recent example, the various R&E networks and smaller broadband and mobile providers showing up here in the US, due to excellent efforts by the NYIIX and DE-CIX teams.

At the end of the day, IXP peering must be significantly cheaper than transit alternatives, many of which are priced based on utilization (as opposed to port capacity). We can dance around this point all we want, but absent a change in trajectory, I worry some IXPs will ultimately price themselves out of the market, and all the gold-plated features in the world won't satiate those making purchasing decisions.

$0.02, -a

On Thu, Jun 16, 2016 at 11:17 AM, Niels Bakker <niels=nanog@bakker.net> wrote:

...

* zbynek@dialtelecom.cz (Zbyněk Pospíchal) [Thu 16 Jun 2016, 14:23 CEST]:

...

Dne 15.06.16 v 20:10 Mikael Abrahamsson napsal(a):

...

Well, the customers also wanted more functions and features. They wanted sFLOW statistics to show traffic, customer portals, better SLAs, distributed IXes, remote peering, more hand-holding when connecting etc.

Are you sure they still want them if they have to pay for these features separately?

Currently, such luxury functions are increasing costs also for networks who don't need/want it.

sFlow statistics isn't a luxury function. Neither is remote peering. The others Mikael mentioned are debatable at worst but you'd be telling the membership what they really want rather than listening to them saying what they want.

This thread is full of people who have never run large L2 networks stating their opinions on running large L2 networks, and they invariably underestimate their complexity and the logistics required.

-- Niels.

I think a similar point was made at NANOG. A distributed IX will let the market dictate that. Places that are better for people to operate in will see a rise in customers and places that aren't won't. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Eric Kuhnke" <eric.kuhnke@gmail.com> To: nanog@nanog.org Sent: Thursday, June 16, 2016 6:17:51 PM Subject: Re: NANOG67 - Tipping point of community and sponsor bashing?

However: exchange port fees are not my biggest enemy today. My cross connect fees have not gone down *at all*. On a proportion basis, cross connect fees have gone from "not mattering" to being an important part of any deployment cost calculation. Why aren't we raising hell about cross connect fees?

IMHO we should be, in the spirit of: https://en.wikipedia.org/wiki/Rent_Is_Too_Damn_High_Party Assuming the existence of overhead fiber trays throughput, when you consider the actual cost of a two strand XC between two cages in the same facility: 30 meter SC-SC duplex 9/125 G.657.A1 cable: $11 There should be a community effort to lobby facility managers and colo/IX real estate management that the value of their facility will be greater if XCs are free or nearly free, resulting in higher occupancy and a greater critical mass of carriers, rather than trying to extract revenue from the tenants by $300/mo MRC per fiber pair between two racks. On Thu, Jun 16, 2016 at 4:06 PM, Phil Rosenthal <pr@isprime.com> wrote:

Hello all,

I wasn't able to attend NANOG this time around, but watched Dave Temkin's presentation on youtube.

My comments are: 1) Over the past 5 years: My cost for switch/router ports have gone down a lot. My cost for transit has gone down a lot. My cost for exchange ports have gone down, but not quite as fast as my transit and switch/router ports, and this does lead to some value questions. Dave is right to ask them.

However: exchange port fees are not my biggest enemy today. My cross connect fees have not gone down *at all*. On a proportion basis, cross connect fees have gone from "not mattering" to being an important part of any deployment cost calculation. Why aren't we raising hell about cross connect fees?

2) Exotic features -- Pvlan, L2VPN, L3VPN have absolutely no purpose on an exchange. If it could be done 'free' with commodity hardware, then fine -- but if it translates to requiring Big Expensive Routers instead of a cheaper but fast switch, this should translate to higher pricing for the customers requiring these exotic features -- not the customers who just want a big L2 vlan.

3) Remote peering -- This is mostly a question about distance for value. There is a clear benefit in providing multi-datacenter exchanges within a metro, and both FL-IX and SIX are doing this with a very good value proposition. Having the ability to join DECIX Frankfurt from NYC and vice versa -- again, this is a bizarre service to be offered, and regular users should not be expected to pay for this. If there is a market for these services at an unsubsidized price, then fine -- but regular members should not be subsidizing this service.

4) sFlow -- I'm not sure why this is even really a topic. Commodity hardware does have sFlow capability, and FLIX demonstrates this well. With that said, for us, it is of extremely limited value. We might check these graphs to validate measurements of our internal netflow/sflow graphing systems, but generally, I look at the graphs generated by my exchange vendors less than once per year per exchange. I am honestly not even sure if SIX offers this service, as I never had a reason to check.

5) Marketting vs Outreach: These things are honestly basically the same thing, mostly separated by the question of "is it good marketing or not". I like having more members at the exchanges I am a member of. If it translates to an additional 3% per year to have an additional 5% of traffic to new members, I am fine with this. If it translates to an extra 50% of cost for 5% of additional traffic, I am not fine with it.

Finally -- there is nothing wrong with asking questions. If you are an exchange company and you can defend your prices for what you offer, then there is no problem. If you are an exchange and are mostly just hoping nobody asks the questions because you won't have any good answers -- well, I think this is exactly why Dave asked the question.

Best Regards, -Phil Rosenthal

...

On Jun 16, 2016, at 1:58 PM, Adam Rothschild <asr@latency.net> wrote:

I think a fresh conversation is needed around what makes up a "minimally viable" feature set for an IXP:

The days of an IXP "needing" to engineer and support a multi-tenant sFlow portal, because the only other option is shelling out the big bucks for Arbor, have long passed -- overlooking the plethora of open sourced tools, folk like Kentik have broken into that market with rationally priced commercial alternatives. Likewise, one might argue that offering layer-2 and layer-3 (!) VPNs is at best non-essential, and a distraction that fuels purchasing very expensive hardware, and at worse competitive with customers.

On the other hand, building out a metro topology to cover all relevant carrier hotels, with reasonable path diversity, is absolutely table stakes. And outreach is a great function, *when* it nets unique new participants. To cite a recent example, the various R&E networks and smaller broadband and mobile providers showing up here in the US, due to excellent efforts by the NYIIX and DE-CIX teams.

At the end of the day, IXP peering must be significantly cheaper than transit alternatives, many of which are priced based on utilization (as opposed to port capacity). We can dance around this point all we want, but absent a change in trajectory, I worry some IXPs will ultimately price themselves out of the market, and all the gold-plated features in the world won't satiate those making purchasing decisions.

$0.02, -a

On Thu, Jun 16, 2016 at 11:17 AM, Niels Bakker <niels=nanog@bakker.net> wrote:

...

* zbynek@dialtelecom.cz (Zbyněk Pospíchal) [Thu 16 Jun 2016, 14:23 CEST]:

...

Dne 15.06.16 v 20:10 Mikael Abrahamsson napsal(a):

...

Well, the customers also wanted more functions and features. They

wanted

...

...

sFLOW statistics to show traffic, customer portals, better SLAs, distributed IXes, remote peering, more hand-holding when connecting etc.

Are you sure they still want them if they have to pay for these features separately?

Currently, such luxury functions are increasing costs also for networks who don't need/want it.

sFlow statistics isn't a luxury function. Neither is remote peering. The others Mikael mentioned are debatable at worst but you'd be telling the membership what they really want rather than listening to them saying what they want.

This thread is full of people who have never run large L2 networks stating their opinions on running large L2 networks, and they invariably underestimate their complexity and the logistics required.

-- Niels.

Dne 16.06.16 v 17:17 Niels Bakker napsal(a):

* zbynek@dialtelecom.cz (Zbyněk Pospíchal) [Thu 16 Jun 2016, 14:23 CEST]:

...

Are you sure they still want them if they have to pay for these features separately?

Currently, such luxury functions are increasing costs also for networks who don't need/want it.

sFlow statistics isn't a luxury function.

Anything more than plain L2 in an IXP is a kind of luxury. An IXP member with it's own flow collection (or at least mac accounting) can feel they don't need sFlow statistics in an exchange. It's also proven it's possible to run an IXP, including a big one, without sFlow stats. We can say the same about route servers, SLA, customer portals etc. (ok, remote peering is a different case). If IXP members think they have to pay such functionality in their port fees, ok, it's their own decision, but member's opinion "we don't need it and we don't want to pay for it" is rational and plausible. Best Regards, Zbynek

On Tue, Jun 14, 2016 at 11:18 AM, Daniel Golding <dgolding@gmail.com> wrote:

I don't see any violation of the presentation guidelines. Also, the day we decide to censor ourselves to avoid offending vendors is the end of my involvement in NANOG - and I suspect that is the case for many others.

Censorship is a strong word and one I would also not be in favor of too (in the generic sense). What is concerning is when bashing is framed as personal attack. A possible PC revision could have been 1) add more flavor of dominate US IXP's (of all organization structures) - as that geographical focus makes more sense for NANOG 2) don't list specific organizations by name, but instead just list their organization structure and a random identifier.

Matt is being coy, for some reason. He didn't like Dave Temkin's talk about IXP costs. I listened very carefully and did not hear any specific members or people targeted - only organizations and companies.

As noted earlier in the thread, the specific presentation isn't my interest here - I actually enjoyed the talk and agree with many of the points stated. What made me uncomfortable was peer IXP's feeling uncomfortable and even a college immersion participant asking "is NANOG always such a threatening environment?". Organizations and companies are members of our greater community, even if they don't technically have a membership role. At this morning's membership meeting - it was restated that NANOG is highly dependent on sponsorships (rarely do we see such financial contributions from individuals that would be enough to support NANOG). It would be a shame to loose that income source when only minor content guidelines could be made.

NANOG is not and has never been a "safe space" for sponsors or organizations that exist in the network space. It never should be. If LINX or AMSIX or anyone else didn't like what was said, they should have rocked the mic (which they did!) and they should come to the next NANOG and present a counterpoint.

I'm sorry Dan, but this sort of "old boys network" attitude has gone on for way too long in NANOG. I've already received 13 off list responses "well said", "nicely done", "finally a reality check", etc. I'm not at all suggesting bashing should go away, as you note - that is a paramount feature of NANOG. Instead the question is when is it appropriate to shame members of the industry and how do we frame that in an professional manner (I realize you may have challenges in such a demonstration) . Clearly a disconnect exists between some members and some board / PC members. As a board member, it would be nice to see a commitment to improving this situation. Thank you.

A possible PC revision could have been 1) add more flavor of dominate US IXP's (of all organization structures) - as that geographical focus makes more sense for NANOG 2) don't list specific organizations by name, but instead just list their organization structure and a random identifier.

< rant > pablum nog. you are pandering to vendors to keep attendee costs down so you can have a high attendee count most of whom are sales folk. what can possibly go wrong? the pc be should have at least two talks including the unvarnished truth about specific named vendors, and at least one talk must be about a 'sponsor', whatever the hell that is and why it is needed. different ones every time, it is a target rich environment. the O in nanog is operator, not sponsor, panderer, suck up, ... we're spending millions for half debugged underperforming crap and we are cornered by infrastructure providers (e.g. ixps) who run us over time and again if it makes an extra penny. if you tell the vendors the truth, the real vendor engineers can go home and explain why they need management support to fix things. the truth makes us all free. randy

On 14/06/2016 20:49, Randy Bush wrote:

the O in nanog is operator, not sponsor, panderer, suck up, ... we're spending millions for half debugged underperforming crap and we are cornered by infrastructure providers (e.g. ixps) who run us over time and again if it makes an extra penny.

I am not at NANOG67 and am following this issue remotely. Excuse me if I am getting this all wrong. Dave shows a slide that LINX made $2.3M profit and AMS-IX made $4.1M last year and Randy states "that the IXPs run us over to make an extra penny"? Would we prefer that Level3 which reported profits of $122M operate the IXPs instead? Or perhaps NTT with $4.7B in profits in 2015? The Internet is no longer a volunteer operation run out of a few CS departments in major universities. If an IXP makes $4M of profit - good for them. I'd rather have each of these IXPs not only secure from cyber attacks but also secure financially. To all IXPs that keep the bits flowing - keep up the good work. Kudos! -Hank

In a message written on Wed, Jun 15, 2016 at 08:25:04AM +0300, Hank Nussbacher wrote:

I am not at NANOG67 and am following this issue remotely. Excuse me if I am getting this all wrong. Dave shows a slide that LINX made $2.3M profit and AMS-IX made $4.1M last year and Randy states "that the IXPs run us over to make an extra penny"?

When I vew the presentation from a raw money basis, it seems like just a "we hate our suppliers" whine. For instance it's quite normal for a "not for profit" to both pay salaries and marketing, and to even "make a profit" from a raw accounting perspective. There's also nothing in the non-profit rules that disallow marketing departments or spending money on socials. If those things further their non-profit mission, it's all good. However there is another perspective where I think a good point is raised, and perhaps a bit lost. Some of these IXP's are "community run". Or well, they say they are "community run". But when the curtian is pulled back, perhaps they look a lot less community run and a lot more like a business with a savvy marketing department leading people to believe they are community run. I do wonder how many people became a _member_ of these "community run" IXP's thinking that entited them to some say over how it was run, only to discover due to the bylaws and corporate structure they have in fact little to no say over anything? That is a form of bait-and-switch, and _may_ be a problem with some IXP's. Maybe the community wants a no-marketing cost-recovery co-op service, and this is a way of rallying and organizing for that. It's on that point that the presentation is classic NANOG, a bunch of operators getting together to discuss their common issues and figure out if there if there is a path forward to make things better. -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/

I don't see any violation of the presentation guidelines. Also, the day we decide to censor ourselves to avoid offending vendors is the end of my involvement in NANOG - and I suspect that is the case for many others.

thanks for speaking up with a clear voice randy, who generally does not like to meee tooo

"If a presentation will name a particular vendor, that vendor should receive an advance draft so that their reps are prepared to speak at the mic about their intentions. " One of the least savory aspects of the technical press and industry analyst worlds is something called pre-pub review. That's where big vendors pay you to see stuff before its printed, so they can attempt to censor it. It happens all the time, and you never know about it. Sharing people's decks with vendors before they are presented might be a nice thing for the presenter to do, but its not appropriate for the NANOG organization. Dan On Tue, Jun 14, 2016 at 11:27 AM William Herrin <bill@herrin.us> wrote:

On Tue, Jun 14, 2016 at 11:53 AM, Patrick W. Gilmore <patrick@ianai.net> wrote:

...

On Jun 14, 2016, at 11:50 AM, Hugo Slabbert <hugo@slabnet.com> wrote:

...

On Tue 2016-Jun-14 10:12:10 -0500, Matt Peterson <matt@peterson.org> wrote:

...

As a community, how do we provide constructive criticism to industry suppliers (that may also be fellow competitors, members, and/or suppliers)? For example, router vendors are routinely compared without specific names mentioned (say in the case of a unpublished vulnerability) - how is a service provider any different?

I understand the discretion involved in your question, but could we clarify exactly what presentation is being discussed so those of us who were not present at NANOG67 can also participate in an informed way?

I personally think the meta-question Matt asked is more important than opinions on a specific presentation. Plus I worry about devolving into a “that was a good preso” / “no it wasn’t!!” flamefest.

Hello,

A vague question can only yield a vague response. I have no clue what presentation you're talking about nor any idea why anyone should be upset about it.

IMO, their are four tiers of meritorious criticism:

1. Privately, directly with the vendor 2. On the mailing list naming no names 3. On the mailing list, name and shame 4. A call to carpet at a meeting

#1 is not always practical -- vendors make it increasingly hard to contact them as customers, let alone as non-customers. Tried to reach Google about a problem? Like, ever?

#2 should happen before #3. If #2 hasn't happened yet, #3 is rude.

#3 should happen before #4. If #3 hasn't happened yet, I think the program committee should encourage a presenter to open a discussion on the list first.

If #2 and #3 have happened, I think it's entirely appropriate to publicly present the vendor's misbehavior and encourage the audience to speak at the mic about how the vendor's error is harming them. It's information the vendor needs to know to stay in business, it's information the rest of us need to know when evaluating the vendor, and in some cases its information the regulatory authorities need to know when considering consumer protection.

That having been said, I see no reason why presentations naming a vendor should be allowed to surprise the vendor. If a presentation will name a particular vendor, that vendor should receive an advance draft so that their reps are prepared to speak at the mic about their intentions. Also, occurrences of #4 should be exactly as rare as persistent vendor misbehavior.

Anyway, not a fan of dancing on eggshells. If something deserves to be said, it should be said. If we can't take a little honesty, we're in the wrong line of work.

Regards, Bill Herrin

-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

On Jun 14, 2016, at 14:05 , William Herrin <bill@herrin.us> wrote:

On Tue, Jun 14, 2016 at 12:30 PM, Daniel Golding <dgolding@gmail.com> wrote:

...

...

"If a presentation will name a particular vendor, that vendor should receive an advance draft so that their reps are prepared to speak at the mic about their intentions. "

One of the least savory aspects of the technical press and industry analyst worlds is something called pre-pub review. That's where big vendors pay you to see stuff before its printed, so they can attempt to censor it. It happens all the time, and you never know about it.

Hi Daniel,

That's quite a bit further than anything I suggested. My notion was that after the presentation has been accepted, the mentioned vendors receive a copy... not so they can attempt to squelch it but so that they can respond when the presentation is given.

Here’s the problem with your theory, William… Once you hand it to the vendors, you lose any semblance of control over how they use it. Indeed, you’d like to hand it to them for that purpose, but once you do, nothing prevents them from the actions described by Daniel. Indeed, this tradition in the print press began with exactly the kind of fair-minded idea of which you speak. Nonetheless, commercial interests drove it off the rails into exactly the dark and twisted kind of thing that Daniel describes. Such is the nature of commerce.

Besides, which among you would dare try to squelch a NANOG presentation? Y'all are well familiar with the Streisand Effect.

Well… Some $LARGE_ORGANIZATIONS seem to either not know, don’t learn, or don’t care. Hard to tell which. For example, there’s a certain industry that has gone through the following progression: 1. The FCC put forth very light-handed almost voluntary regulations on net neutrality. 2. They sued the FCC claiming it had no such authority. 3. The FCC put forth a slightly heavier-handed regulation. 4. They sued the FCC claiming it had no such authority. 5. The FCC issued a ruling entirely within its authority which was quite a bit more heavy-handed. It was the only tool left in the tool box once the lighter approaches were ruled invalid. 6. They complained that the regulation was “too heavy-handed”. 7. They sued the FCC claiming it had no such authority. 8. A federal appeals court just upheld the FCC 2:1. 9. They say they want to take it to the supreme court. The bottom line, when there’s enough money on the line, some organizations with a vested interest will do anything and everything they can to protect their ability to obtain that money. Helping them is optional. Helping them by trying to be nice is naive. Owen

On Tue, Jun 14, 2016 at 01:40:20PM -0400, Peter Beckman wrote:

Negative feedback, respectfully and objectively delivered, should be embraced as opportunities to improve ourselves, our products and our services, not shunned and silenced because it points out a flaw.

1. This. A hundred times this. 2. This is why we have RFC 2142 (which specifies role addresses such as postmaster@, abuse@, and so on): so that we can easily and quickly tell each other when we're screwing up so that it can be fixed. This is why all professional and responsible operations maintain those addresses, pay attention to what shows up there, read it, analyze it, act on it, and respond to it. This is and has been an instrinic part of our operational culture for decades -- even though we all know that just about every message ever received via them will be negative. (Because nobody's going to drop a line to hostmaster@ noting that our DNS servers are all working perfectly.) A critical presentation is really no different than an email message to webmaster@ that points out a 404'd URL. It's an opportunity to fix something and to do better. ---rsk

I agree that the SIX is a fine organization, but the framework of the organization has little to do with the members getting screwed over. A non-profit donation-based IX that doesn't produce results could be screwing its "customers" over more than a MRC-based for-profit IX that does produce. I also think that the individual merits of an organization or business model is pretty astray from the OP's original point (correct or not) about using the NANOG presentation platform for thinly veiled personal agenda. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Eric Kuhnke" <eric.kuhnke@gmail.com> To: nanog@nanog.org Sent: Wednesday, June 15, 2016 12:43:13 AM Subject: Re: NANOG67 - Tipping point of community and sponsor bashing? Re: Item #3 there, the Google Docs spreadsheet with the IX costs... Scroll all the way down to the bottom in $/Mbps and you will find the SIX. Everyone in the Pacific NW should appreciate the excellent work that the SIX does. It's a nonprofit with transparency in its finances, a health cash reserve for emergencies and new equipment and meets very stringent uptime and reliability requirements. ISP entities and enterprise end users 1000 km away from the SIX in random locations in British Columbia, Montana, Utah and other western US states benefit from it. People who have no idea what an IX is or how it functions have better, faster and lower cost last mile Internet access thanks to their local small ISP that has had the foresight to purchased a transport circuit to Seattle to reach the SIX. It is worth mentioning that the fine people at the NWAX in Portland are working to build on the example set by the SIX, and are a 501(c)6 nonprofit: http://www.nwax.net/ On Tue, Jun 14, 2016 at 1:20 PM, Jared Mauch <jared@puck.nether.net> wrote:

...

On Jun 14, 2016, at 11:12 AM, Matt Peterson <matt@peterson.org> wrote:

This week at NANOG67, a presentation was given early on that did not reflect well for our community at large.

I think that the data presented was interesting but the style of the presenter and tone could have been different. It seemed to be a variant of “The Rent is Too Damn High”[1] while it can be interesting, there wasn’t a complete talk there IMHO.

The feedback mechanism for this is honestly the survey[2]. I’m confident that the PC will take this input seriously and work with presenters in this regard.

The IXP cost sheet[3] that is being maintained by Job I think gives an idea of the peering vs transit costs assuming various bitrates and list prices.

The fates of IXPs and their roles will naturally resolve itself through market economics I suspect.

- Jared

- snip - links - snip - 1 - https://en.wikipedia.org/wiki/Rent_Is_Too_Damn_High_Party 2 - https://www.nanog.org/meetings/nanog67/survey 3 - https://docs.google.com/spreadsheets/d/18ztPX_ysWYqEhJlf2SKQQsTNRbkwoxPSfaC6...

Getting people to show up can be a challenge. I've been asked by members of two midwestern IXes to come to their markets because their existing donation-supported loose and easy IX isn't really doing anything for them. Not arguing models, arguing that what should matter is results. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Seth Mattinen" <sethm@rollernet.us> To: nanog@nanog.org Sent: Wednesday, June 15, 2016 2:14:21 PM Subject: Re: NANOG67 - Tipping point of community and sponsor bashing? On 6/15/16 05:37, Mike Hammett wrote:

I agree that the SIX is a fine organization, but the framework of the organization has little to do with the members getting screwed over. A non-profit donation-based IX that doesn't produce results could be screwing its "customers" over more than a MRC-based for-profit IX that does produce.

An IX just needs to "produce" a layer 2 peering fabric. That's not a tall order to get results from. Anything beyond that is extra fluff. Some people want to pay more for the fluff, some don't. ~Seth

...

...

On 6/15/16 05:37, Mike Hammett wrote: A non-profit donation-based IX that doesn't produce results could be screwing its "customers" over more than a MRC-based for-profit IX that does produce.

On 15.06.2016 21:14, Seth Mattinen wrote: An IX just needs to "produce" a layer 2 peering fabric. That's not a tall order to get results from. Anything beyond that is extra fluff. Some people want to pay more for the fluff, some don't.

On Jun 15, 2016, at 6:36 PM, Arnold Nipper <arnold@nipper.de> wrote: This is a *common* misunderstanding. The by far easiest part of running a successful IXP is the technical part. The more challenging is to build a community around it. And that's purely non technical and involves a lot of *social* networking and bringing people together.

There’s a difference between the cost and the product. As regards the cost, Arnold is exactly right. Across the many hundreds of exchanges that we’ve worked with over the past 22 years, our observation has been that, at a rough average, most IXPs spend 45% of their first-year effort on location selection, 45% on governance definition and establishment, and 10% on technical decisions and implementation. But the total effort and the governance portion both increase drastically for those that choose to handle money; at a very, very rough average, about four-fold. In subsequent years, location selection generally drops away to near zero, except in cases like the JINX, and technical work dips for the first couple of years, and then spikes once every three years or so as switches are replaced and new configs are needed. Many exchanges have an annual in-person meeting where elections are conducted and policy changes ratified, so that typically becomes the largest ongoing expense, as Arnold implies. As regards the product, no, Seth, the layer 2 peering fabric is merely a necessary precondition for producing bandwidth. The actual bandwidth production has other preconditions as well: peers physically connected to the peering switch fabric, BGP sessions established between the peers, routes advertised across those sessions, a reasonable matching of potential traffic sources and sinks available through those routes, and a set of customer behaviors that prefer those source/sink matchings. Only then does an IXP produce bandwidth. So, the role of a salesperson or advocate or evangelist or tout can be a net beneficial one, if they do a good job of recruiting participants, making sure they follow through with peering, and encouraging the preference of locally-available content. WAIX was among the first IXPs to do this well, in my opinion. -Bill

On Wednesday, June 15, 2016, Seth Mattinen <sethm@rollernet.us> wrote:

On 6/15/16 4:03 PM, Bill Woodcock wrote:

...

There’s a difference between the cost and the product. As regards the cost, Arnold is exactly right. Across the many hundreds of exchanges that we’ve worked with over the past 22 years, our observation has been that, at a rough average, most IXPs spend 45% of their first-year effort on location selection, 45% on governance definition and establishment, and 10% on technical decisions and implementation. But the total effort and the governance portion both increase drastically for those that choose to handle money; at a very, very rough average, about four-fold. In subsequent years, location selection generally drops away to near zero, except in cases like the JINX, and technical work dips for the first couple of years, and then spikes once every three years or so as switches are replaced and new configs are needed. Many exchanges have an annual in-person meeting where elections are conducted and policy changes ratified, so that typically becomes the largest ongoing expense, as Arnold implies.

Why do IXes seek to create a bureaucracy that needs to be fed increasing sums of money?

~Seth

Cough cough ARIN cough. I don't know why they need to meet face to face 2 or 3 times a year. But, i am sure ppml will tell you it is a ground up process and these people on ppml like traveling and talking about policy.... And they do what members want. Much of our industry can be gleened by googling pictures of "peering cruise". At my office, we joke about that a lot. Peering cruise ...jeeshhh. I can't recommend SIX enough. Epic IX, amazing support. I get much better support from SIX than any of my transit providers. Amazing support. I also like sfmix and fl-ix.

On Jun 15, 2016, at 22:24, Ca By <cb.list6@gmail.com> wrote:

...

On Wednesday, June 15, 2016, Seth Mattinen <sethm@rollernet.us> wrote:

...

On 6/15/16 4:03 PM, Bill Woodcock wrote:

[ clip ]

I also like sfmix and fl-ix.

FL-IX is great. It created real competition in the American South and for the Americas peering. I kept my interconnections to VZ while we see what happens when Zayo needs to recover their costs for the panel they're using to drain NOTA. There will be a day of reckoning, free is never free. Its a good hedge. SFMIX is great. But poorly distributed. We should support their efforts, but how many IXPs do we need in the Bay area? AMS-IX Bay Area is creating a market along with SFMIX. There is still copious amounts of traffic at VZ NOTA MIA. Two in a market works well IMHO. Best, -M<

On Jun 16, 2016, at 01:12, Leslie <geekgirl@gmail.com> wrote:

...

On Wed, Jun 15, 2016 at 8:41 PM, Martin Hannigan <hannigan@gmail.com> wrote: <snip> SFMIX is great. But poorly distributed. We should support their efforts, but how many IXPs do we need in the Bay area? AMS-IX Bay Area is creating a market along with SFMIX.

SFMIX is in 5 physical locations( https://www.sfmix.org/connect/locations ) and is always open to talking to other providers about extending into their datacenter. So I'd say we're in a variety of locations! We've also just celebrated our 10 year anniversary :)

Leslie

Agreed. Extended XC? Best, -M<

On Thursday, June 16, 2016, Owen DeLong <owen@delong.com> wrote:

...

Cough cough ARIN cough. I don't know why they need to meet face to face 2 or 3 times a year. But, i am sure ppml will tell you it is a ground up process and these people on ppml like traveling and talking about policy.... And they do what members want.

I don’t speak for the organization or even for the Advisory Council, but I can tell you that we get more focused community input and participation during those face to face meetings than we do during the rest of the year.

Like it or not, when trying to get interactive participation from more than 5 or so people, there’s really no useful substitute for the face-to-face meeting. Every existing alternative is less useful and comes with significant drawbacks.

So, yes, I believe ARIN does what the members want, but more importantly, I believe that the face to face meetings are a useful mechanism to preserve the bottom up nature of the policy development process.

...

Much of our industry can be gleened by googling pictures of "peering cruise". At my office, we joke about that a lot. Peering cruise …jeeshhh.

I suppose you can joke about anything you want. Personally, I’ve never been on a peering cruise, but I will say that it’s a pretty classic fallacy to discount the value of the social times at conferences. In fact, I find those times to often be when most of the real work gets done and when most of the benefit of getting everyone together in the same place is realized.

While NANOG puts on a good technical program, my company gets far more benefit from the time I spend meeting with network partners, suppliers, and potential customers during the conference than they will ever see from my time in the sessions. So much so that I generally attend the sessions on an as-available basis when I’m not able to schedule a more useful meeting. There are, of course exceptions. Some of the sessions (maybe 3-4 per conference) are worth holding my time open to attend, but most are not.

This does not mean that I don’t consider NANOG valuable, just that the primary value is in the ability to meet with other attendees rather than directly in the technical program itself.

I don’t mean this to be insulting to NANOG. I think the PC generally does a fine job of putting on a good conference with great content. Most importantly, it’s good enough that it draws in a large selection of people I need/want to meet with in a concentrated time frame in a single location.

Peering fora, peering cruises, and the like have a similar effect.

So scoff all you want, but if you imagine that these events are silly junkets where nothing gets accomplished, then you are seriously underestimating this community, IMHO.

Owen

Owen, I agree with most of what you are saying. I'll digress on if arin needs to meet or exist. Perhaps it is me and my sensibilities, perhaps it is my miser corp culture, but i could not even dream of asking to go to Jamaica (arin area) for the last ARIN meeting. I am not alone. Have a look at Ren's comments from http://research.dyn.com/2006/02/lovely-peering-cruise-on-lake/ Posted here: " The Peering Forum is more for peer & IX information distribution and contact refresh across a multi-continent body of participants than it is for initial trial concerns. The invite only nature implies the attendees are actively peering at one or more of the IXs sponsoring portions of the Forum. Many of us, hand raised here, are taking vacation time, covering flights, upgraded cabins, etc. to help remove the conflict of interest concerns. It is unfortunate there is not a link to the agenda as it is jam packed with useful peering focused presentations and more than qualifies this as a business need. Last year dozens of participants interacted for the first time and I’m looking forward to similar introductions later this month. Putting a face to the name helps significantly in this very relationship based role which has more to do with international relations than enable." I am not willing to fork over my pto or personal cash to "remove the conflict of interest" . I'd rather buy transit. And, i am also not willing to spend my corp money with dec-ix or others to send my competitors on a cruise. Unless ... why cant this just be business? That said, i have never been on a peering cruise and all my peering needs are met with peeringdb and email. So it is just business for me, and no i am not going to spend money at an IX that does not see things the way i do. CB

Owen, On Jun 17, 2016, at 1:20 AM, Owen DeLong <owen@delong.com> wrote:

...

On Jun 16, 2016, at 06:03 , Ca By <cb.list6@gmail.com> wrote:

Perhaps it is me and my sensibilities, perhaps it is my miser corp culture, but i could not even dream of asking to go to Jamaica (arin area) for the last ARIN meeting.

You are entitled to your opinion.

If ARIN didn’t exist, how would you go about guaranteeing unique registered GUA blocks and ASNs? Who would operate whois and in-addr.arpa, ip6.arpa?

ICANN operates in-addr.arpa and ip6.arpa. Regards, -drc (speaking only for myself)

Owen

On Jun 20, 2016, at 6:03 PM, Owen DeLong <owen@delong.com> wrote:

...

...

If ARIN didn’t exist, how would you go about guaranteeing unique registered GUA blocks and ASNs? Who would operate whois and in-addr.arpa, ip6.arpa? ICANN operates in-addr.arpa and ip6.arpa. ICANN takes the data supplied by the RIRs and compiles it into zone files which are then distributed to servers.

Among others, yes (hint: not all the IPv4 and IPv6 address space is managed by the RIRs).

AIUI, most of the servers are hosted and maintained by the RIRs. Most if not all of the zone file information is supplied to ICANN by the RIRs.

Perhaps you should review how the DNS works. Hint: who operates the master for in-addr.arpa and ip6.arpa and what is the role of secondaries?

I stand by my statement to the extent that it is close enough for the purposes for which it was made.

Your statement posited the nonexistence of ARIN. ARIN is a secondary for in-addr.arpa and ip6.arpa (like the other RIRs) and maintains a registry for the address blocks allocated to them by ICANN as the IANA Numbering Function operator. If ARIN did not exist, then the reverse delegations for which ARIN is authoritative could easily be managed by the other RIRs, the entities to which ARIN currently delegates, or the myriad of other DNS registries. This really isn't rocket science.

Without the RIR, ICANN’s idea of what should go into ip6.arpa and in-addr.arp would get pretty stale pretty fast.

Or not, as long as the entities to which ARIN delegated were aware of the registry they should update. This really isn't rocket science.

Of course, that wouldn’t matter because AIUI, without the servers being supplied, hosted, maintained by the RIRs, it would also be fairly invisible as well.

Again, perhaps you should review how the DNS works. Hint: where do referrals to the sub-trees of .ARPA come from?

But keep those ICANN delusions of grandeur coming.

You asked a question. I answered that question accurately. I am unsure why that would cause you to assert delusions of grandeur. Regards, -drc (speaking only for myself)

On Wed, Jun 15, 2016 at 4:03 PM, Bill Woodcock <woody@pch.net> wrote:

[...] Only then does an IXP produce bandwidth.

Minor nitpick--an IXP never 'produces' bandwidth; it facilitates movement of data between entities, but the IXP itself shouldn't be producing bandwidth. It's the allocation of ports and cross connects from members into the IXP that produce the bandwidth, and that would be the case even if the IXP were removed from the picture and the ports were cross-connected back-to-back. (I suppose if an IXP switch fabric were compromised, someone could use it to generate traffic that did not originate from any member port, but that would be a very unusual circumstance indeed...) Thanks! Matt