As mentioned before: even the new SUP2T from Cisco is limited to 1Mio routes ... There are MANY other vendors with the same limitations: Juniper, Brocade, etc And the solt equipment is not the 99USD trash from the super market at the corner ... Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Original Message----- From: Max Tulyev [maxtul@netassist.ua] Received: Samstag, 03 Okt. 2015, 9:11 To: nanog@nanog.org [nanog@nanog.org] Subject: Re: AW: /27 the new /24 Which routers? DIR-300 with OpenWRT/Quagga? :) I think all above-the-trash level routers supports >1M routes, isn't it? On 02.10.15 17:45, Jürgen Jaritsch wrote:

Hi,

this would at least help to get rid of many old routing engines around the world :) ... or people would keep their "learn nothing smaller than /24" filters in place. Also an option - but not for companies who act as an IP transit provider.

best regards

Jürgen Jaritsch Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300 Telefax: +43-5-0556-500

E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Justin Wilson - MTIN Gesendet: Freitag, 02. Oktober 2015 16:32 An: NANOG Betreff: /27 the new /24

I was in a discussion the other day and several Tier2 providers were talking about the idea of adjusting their BGP filters to accept prefixes smaller than a /24. A few were saying they thought about going down to as small as a /27. This was mainly due to more networks coming online and not having even a /24 of IPv4 space. The first argument is against this is the potential bloat the global routing table could have. Many folks have worked hard for years to summarize and such. others were saying they would do a /26 or bigger.

However, what do we do about the new networks which want to do BGP but only can get small allocations from someone (either a RIR or one of their upstreams)?

Just throwing that out there. Seems like an interesting discussion.

Justin Wilson j2sw@mtin.net

--- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth

http://www.midwest-ix.com COO/Chairman Internet Exchange - Peering - Distributed Fabric

Hi, yeah, of course there are newer models ... I mentioned the older ones (from the past 3-5 years). There are also Cisco routers available that are able to handle more than 1 Mio routes - of course also from Juniper. Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: Youssef Bengelloun-Zahr [mailto:youssef@720.fr] Gesendet: Samstag, 03. Oktober 2015 11:03 An: Jürgen Jaritsch <jj@anexia.at> Cc: nanog@nanog.org; maxtul@netassist.ua Betreff: Re: AW: /27 the new /24 Hi, FYI, newer linecard models from BROCADE can hold 2 million routes. Probably others can do that now too. Disclaimer : I'm not working for them or defending them, just setting an information straight. My 2 cents.

Le 3 oct. 2015 à 10:33, Jürgen Jaritsch <jj@anexia.at> a écrit :

As mentioned before: even the new SUP2T from Cisco is limited to 1Mio routes ...

There are MANY other vendors with the same limitations: Juniper, Brocade, etc

And the solt equipment is not the 99USD trash from the super market at the corner ...

Jürgen Jaritsch Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300 Telefax: +43-5-0556-500

E-Mail: jj@anexia.at Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-----Original Message----- From: Max Tulyev [maxtul@netassist.ua] Received: Samstag, 03 Okt. 2015, 9:11 To: nanog@nanog.org [nanog@nanog.org] Subject: Re: AW: /27 the new /24

Which routers? DIR-300 with OpenWRT/Quagga? :)

I think all above-the-trash level routers supports >1M routes, isn't it?

...

On 02.10.15 17:45, Jürgen Jaritsch wrote: Hi,

this would at least help to get rid of many old routing engines around the world :) ... or people would keep their "learn nothing smaller than /24" filters in place. Also an option - but not for companies who act as an IP transit provider.

best regards

Jürgen Jaritsch Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300 Telefax: +43-5-0556-500

E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Justin Wilson - MTIN Gesendet: Freitag, 02. Oktober 2015 16:32 An: NANOG Betreff: /27 the new /24

I was in a discussion the other day and several Tier2 providers were talking about the idea of adjusting their BGP filters to accept prefixes smaller than a /24. A few were saying they thought about going down to as small as a /27. This was mainly due to more networks coming online and not having even a /24 of IPv4 space. The first argument is against this is the potential bloat the global routing table could have. Many folks have worked hard for years to summarize and such. others were saying they would do a /26 or bigger.

However, what do we do about the new networks which want to do BGP but only can get small allocations from someone (either a RIR or one of their upstreams)?

Just throwing that out there. Seems like an interesting discussion.

Justin Wilson j2sw@mtin.net

--- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth

http://www.midwest-ix.com COO/Chairman Internet Exchange - Peering - Distributed Fabric

On Sat, 3 Oct 2015 12:42:01 +0200, Baldur Norddahl <baldur.norddahl@gmail.com> said: > 2 million routes will not be enough if we go full /27. This is > not a scalable solution. Something else is needed to provide > multihoming for small networks (LISP?). It's not too far off though. One way of looking at it is, for each extra bit we allow, we potentially double the table size. So with 500k routes and a /24 limit now, we might expect 4 million with /27. Not exactly because it depends strongly on the distribution of prefix lengths, but probably not a bad guess. Also there are optimisations that I wonder if the vendors are doing to preserve TCAM such as aggregating adjacent networks with the same next hop into the supernet. That would mitigate the impact of wanton deaggregation at least and the algorithm doesn't look too hard. Do the big iron vendors do this? -w -- William Waites <wwaites@tardis.ed.ac.uk> | School of Informatics http://tardis.ed.ac.uk/~wwaites/ | University of Edinburgh https://hubs.net.uk/ | HUBS AS60241 The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

The race is on… One or more of these things will be the death of IPv4: 1. Not enough addresses 2. Routing Table Bloat due to one or more of: A. Traffic Engineering B. Stupid configuration C. Address Fragmentation through transfers and microallocations of “Transition space” D. Stupid network designs (where the physical deployment forces something like B rather than just software stupidity) 3. Some highly desirable feature or application which simply can’t be implemented effectively on IPv4. As such, I say go ahead and route whatever. ARIN provides a nice constrained prefix for these transitional allocations so you can at least limit the bloat from that factor, but even at the /24 level, we’re faced with upwards of 16 million routes that we are nowhere near ready to handle. In case you haven’t been paying attention for the last 25 years, there are a number of ways in which IPv4 DOES NOT SCALE. We’ve been keeping it alive on life support for a VERY LONG time. As with all patients on life support, the longer it continues, the more it becomes a losing battle and the harder (and more resource intensive and more expensive) it becomes to keep the patient alive. Fortunately, in this case, we have a nice new body that we can transplant the internet into that has many fruitful years ahead of it. So… Do whatever you have to to survive in the meantime, but focus on getting your stuff onto the IPv6 internet so that we can all let IPv4 go gently into that good night and have it’s well deserved final rest. Owen

On Oct 3, 2015, at 06:18 , Baldur Norddahl <baldur.norddahl@gmail.com> wrote:

Except we might very well reach 1+ million routes soon without accepting longer prefixes than /24. Also route updates is a concern - do I really need to be informed every time someone on the other end of the world resets a link?

On 3 October 2015 at 12:57, William Waites <wwaites@tardis.ed.ac.uk> wrote:

...

On Sat, 3 Oct 2015 12:42:01 +0200, Baldur Norddahl < baldur.norddahl@gmail.com> said:

...

2 million routes will not be enough if we go full /27. This is not a scalable solution. Something else is needed to provide multihoming for small networks (LISP?).

It's not too far off though. One way of looking at it is, for each extra bit we allow, we potentially double the table size. So with 500k routes and a /24 limit now, we might expect 4 million with /27. Not exactly because it depends strongly on the distribution of prefix lengths, but probably not a bad guess.

Also there are optimisations that I wonder if the vendors are doing to preserve TCAM such as aggregating adjacent networks with the same next hop into the supernet. That would mitigate the impact of wanton deaggregation at least and the algorithm doesn't look too hard. Do the big iron vendors do this?

-w

-- William Waites <wwaites@tardis.ed.ac.uk> | School of Informatics http://tardis.ed.ac.uk/~wwaites/ | University of Edinburgh https://hubs.net.uk/ | HUBS AS60241

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

Keep in mind that IPv6 has IPSec VPN built into the protocol. It doesn't need to be in the router. Unlike IPv4, where the IPSec VPN protocol is an add-on, optional service, with IPv6 it's built into every device, because IPsec is a mandatory component for IPv6, and therefore, the IPsec security model is required to be supported for all IPv6 implementations. Thus it is a true end-to-end secure transport between two nodes -- even when those nodes are behind a firewall. You can still created IPv6 VPNs from site-to-site (called "tunnel mode"), but the idea with IPv6 is that since you can directly encrypt every TCP session, eventually the need for tunnels will diminish, if not go away completely. Interestingly, IPsec came out of funding from Clinton administration for securely hosting the whitehouse.gov email server. Trusted Information Systems software engineer Wei Xu started researching IP security methods in July 1994, and ultimately developed the first rendition of IPSec. He ported it to several server OSes of the time. -mel beckman

On Oct 4, 2015, at 6:41 AM, Matthias Leisi <matthias@leisi.net> wrote:

The built-in VPN which only supports IPv4 (that one specifically on an Asus router).

If it doesn't support IPSec, it's not really IPv6. Just as if it failed to support any other mandatory IPv6 specification, such as RA. There's really no excuse for not supporting IPSec, as it's a widely available open source component that costs nothing to incorporate into an IPv6 stack. Your observation simply means that users must be informed when buying IPv6 devices, just as they must with any product. You can buy either genuine IPv6 or half-baked IPv6 products. When I speak of IPv6, I speak only of the genuine article. -mel beckman On Oct 4, 2015, at 7:41 AM, "sthaug@nethelp.no" <sthaug@nethelp.no> wrote:

...

Keep in mind that IPv6 has IPSec VPN built into the protocol. It doesn't need to be in the router.

Unlike IPv4, where the IPSec VPN protocol is an add-on, optional service, with IPv6 it's built into every device, because IPsec is a mandatory component for IPv6, and therefore, the IPsec security model is required to be supported for all IPv6 implementations.

If you really believe all IPv6 devices support IPsec, I can only conclude that your IPv6 experience is limited...

Steinar Haug, Nethelp consulting, sthaug@nethelp.no

Randy, Your claim is a red herring. IPSec has nothing to do with IPv6 deployment. Deployment doesn't require global IPSec, which need only reside in endpoint nodes. It's not needed at all in the routjg and distribution infrastructure, which is where deployment happens The vast majority of IPv6 nodes -- which is where the IPSec requirement exists -- have IPSec built in: Linux, Mac OSX, and Windows. Devices that sometimes act as nodes, such as firewalls terminating IPSec tunnels, also obviously need IPSec. Devices that are simply IPv6 pass-through, such as consumer-grade routers, don't. Users can buy whatever level of functionality they need at the edges. If you don't need IPSec tunnel support in your firewall, you can buy one without it. Deployment cares nothing about IPSec. -mel beckman On Oct 4, 2015, at 8:05 AM, Randy Bush <randy@psg.com<mailto:randy@psg.com>> wrote: If it doesn't support IPSec, it's not really IPv6. by that criterion, ipv6 deployment is effectively zero

What Cisco routers, and what vintage IOS, are you finding have no IPSec support? I've not run into that problem. -mel beckman

On Oct 4, 2015, at 8:33 AM, Jon Lewis <jlewis@lewis.org> wrote:

...

On Sun, 4 Oct 2015, Mel Beckman wrote:

If it doesn't support IPSec, it's not really IPv6. Just as if it failed to support any other mandatory IPv6 specification, such as RA.

Go tell cisco that. IIRC, the first network I dual-stacked, I was kind of surprised when I found I could not use authentication in OSPFv3, because OSPFv3 assumes IPv6 will supply the IPSec to do auth...but these routers didn't support IPSec. They still managed to route IPv6 and support IPv6 customers...so it really was IPv6...just not the full suite of everything you'd expect from IPv6.

...

Your observation simply means that users must be informed when buying IPv6 devices, just as they must with any product. You can buy either genuine IPv6 or half-baked IPv6 products. When I speak of IPv6, I speak only of the genuine article.

Does anyone buy "IPv6 devices"?

The biggest hurdle I've seen with IPv6 adoption (i.e. going dual-stack, with the idea that we'll gradually transition most things / most traffic from v4 to v6) is the number of end-user network providers that don't offer v6 at all. My home cable internet provider still doesn't offer IPv6. When I asked one of their support people about it recently, I was told not to worry, they have plenty of v4 addresses left, but it was implied that they do plan to start offering v6 sometime soon. They should have started rolling out IPv6 to any customers that wanted it years ago, so that by today, it would be standard for all their installations to be dual-stack. But here we are, nearly 2016, and they don't have a single IPv6 customer (AFAIK) yet.

---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

The times for tonight's event differ from the guidebook to the online version @ nanog nanog 6-8PM guidebook 7-11PM Which is correct?

sup720-3bxl, but this was a number of years ago. I don't recall the exact version. It was probably 12.2SXI-something. On Sun, 4 Oct 2015, Mel Beckman wrote:

What Cisco routers, and what vintage IOS, are you finding have no IPSec support? I've not run into that problem.

-mel beckman

---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

On Oct 4, 2015, at 8:33 AM, Jon Lewis <jlewis@lewis.org> wrote:

On Sun, 4 Oct 2015, Mel Beckman wrote:

...

If it doesn't support IPSec, it's not really IPv6. Just as if it failed to support any other mandatory IPv6 specification, such as RA.

Go tell cisco that. IIRC, the first network I dual-stacked, I was kind of surprised when I found I could not use authentication in OSPFv3, because OSPFv3 assumes IPv6 will supply the IPSec to do auth...but these routers didn't support IPSec. They still managed to route IPv6 and support IPv6 customers...so it really was IPv6...just not the full suite of everything you'd expect from IPv6.

A router with OSPFv3 and no IPSec for securing the OSPFv3 sessions really is an incomplete implementation. This is one case where IPSec really should be considered mandatory rather than recommended.

...

Your observation simply means that users must be informed when buying IPv6 devices, just as they must with any product. You can buy either genuine IPv6 or half-baked IPv6 products. When I speak of IPv6, I speak only of the genuine article.

Does anyone buy "IPv6 devices”?

Yes… For some definitions of that term.

The biggest hurdle I've seen with IPv6 adoption (i.e. going dual-stack, with the idea that we'll gradually transition most things / most traffic from v4 to v6) is the number of end-user network providers that don't offer v6 at all. My home cable internet provider still doesn't offer IPv6. When I asked one of their support people about it recently, I was told not to worry, they have plenty of v4 addresses left, but it was implied that they do plan to start offering v6 sometime soon. They should have started rolling out IPv6 to any customers that wanted it years ago, so that by today, it would be standard for all their installations to be dual-stack. But here we are, nearly 2016, and they don't have a single IPv6 customer (AFAIK) yet.

Yeah, lots of providers still don’t get it like that. The problem is we’ve also done a poor job training people who call them up asking for IPv6. Many accept “We have plenty of IPv4 addresses” as an answer. Instead, the followup question is needed… “That’s great, but how does that help me reach a web site that doesn’t have and can’t get an IPv4 address?” Owen

On Oct 7, 2015, at 6:29 AM, Matthew Kaufman <matthew@matthew.at> wrote:

...

On Oct 7, 2015, at 5:01 AM, Owen DeLong <owen@delong.com> wrote:

Instead, the followup question is needed… “That’s great, but how does that help me reach a web site that doesn’t have and can’t get an IPv4 address?”

Owen

At the present time, a web site that doesn't have and can't get an IPv4 address isn't "on the Internet".

That may change in the future, but right now this is the web site's fault, not your ISP's.

Wishing that the IPv6 transition had gone differently does not change reality.

Matthew Kaufman

(Sent from my iPhone)

Swift boating aside, repeating the same assertion doesn’t make it true. Owen

On Oct 7, 2015, at 7:00 AM, Mark Andrews <marka@isc.org> wrote:

In message <A35FA880-B612-4458-BD22-323BEF66A5BC@matthew.at>, Matthew Kaufman w rites:

...

...

On Oct 7, 2015, at 5:01 AM, Owen DeLong <owen@delong.com> wrote: =20 =20 =20 Instead, the followup question is needed=E2=80=A6 =E2=80=9CThat=E2=80=99s g = reat, but how does that help me reach a web site that doesn=E2=80=99t have a= nd can=E2=80=99t get an IPv4 address?=E2=80=9D =20 Owen =20

At the present time, a web site that doesn't have and can't get an IPv4 addr= ess isn't "on the Internet".

It's on the Internet. ISP's that fail to supply IPv6 at this point in time are committing fraud if they claim to supply Internet connection.

Good luck prosecuting them for that. Along with all the internal IT departments that are failing to deliver v6 to wifi and desktops.

...

That may change in the future, but right now this is the web site's fault, n= ot your ISP's.

No, it isn't the site's fault. The internet ran out of IPv4 addresses years ago. Not everyone can get a public adddress.

Right. Now it is only people who can afford about $8 one time. (The going rate for IPv4 on the transfer market at modest block sizes)

There are millions of customers without a public IPv4 address that can host a site because they are behind a CGN which is only needed because of the short sightedness of lots of ISPs failing to deliver IPv6 to their customers.

I think you meant cannot. Most consumer ISPs also prevent this as a matter of policy. Good luck getting those policies changed.

...

Wishing that the IPv6 transition had gone differently does not change reality.

I don't see anyone wishing it went differnetly. I see someone pointing out the reality that lots of ISP's are way too late to delivering IPv6.

Sure, they're too late. Which is why, until there's more progress, a website not reachable over IPv4 is fairly useless if the goal is to serve "most of the users on the Internet"

*Every* ISP should have been planning to deliver IPv6 by the time the first RIR ran out of IPv4 addresses. That would have been just-in-time engineering. It's not like they didn't have over a decade to plan to do it, It's not like there wern't reasonable accurate forcecasts for when that would happen.

Yeah, totally agree. Didn't happen. Still hasn't happened. Won't happen tomorrow.

It was not hard to see what would happen if you didn't deliver IPv6 before the first RIR ran out.

No instead most of then stuck their heads in the sand and said "we have enough IPv4 addresses" without looking at whom they need to connect with.

Last I checked, things are still working out just fine for all of them. Despite the obvious concerns about the future. Matthew Kaufman (Sent from my iPhone)

On Oct 7, 2015, at 4:15 PM, Mark Andrews <marka@isc.org> wrote:

I don't have to. I'm sure some AG will do so soon enough.

There's always an optimist around. Good luck with that. Matthew Kaufman (Sent from my iPhone)

On 10/7/15 7:00 AM, Mark Andrews wrote:

I don't see anyone wishing it went differnetly. I see someone pointing out the reality that lots of ISP's are way too late to delivering IPv6. *Every* ISP should have been planning to deliver IPv6 by the time the first RIR ran out of IPv4 addresses.

Look, I'm as much a supporter of delivering IPv6 as anyone. I've had IPv6 enabled on my home network (and the small data center I run in my garage) for over a decade now. In 2004, I made sure that IPv6 was fully supported in the peer-to-peer stack I developed and that eventually became RFC 7016. And for the last 5 years I've been pushing for IPv6 support in the product I work on for my employer. But the reality is that there's a whole lot of small and medium-sized ISPs run by fine, upstanding individuals serving their communities -- even in and around the San Francisco Bay Area -- that have either no or very limited (tunnels only) support for IPv6. That's the reality of the transition. And threatening these folks with the attorney general isn't the way to get them to adopt IPv6, nor is shaming them. They will add IPv6 support when it is easy to do, when their staff has the time, and when the economics make sense. Meanwhile we have app developers trying to use cloud platforms that don't support IPv6 well (or at all), writing code while sitting in offices that don't have IPv6 service due either to their ISP or their internal IT department... and so there's another reason ISPs need to keep concentrating on IPv4 as their first priority. And so, in the current actual Internet, not some hypothetical one, if you want your website to be seen, you get it an IPv4 address. And with IPv4 going for $6-$8 each and it being possible to support hundreds or thousands of websites on a single IPv4 address, there's really no excuse. Will this be different in the future? I sure hope so. But we're not there yet. Matthew Kaufman

-------------------------------------------- On Thu, 10/8/15, Mark Andrews <marka@isc.org> wrote:

This is today's reality and ISP's are not meeting today's needs. It's not just about having enough IPv4 addresses. It's about providing the infrastructure to allow your customers to connect to everyone.

I think you should s/everyone/everyone they care about/ That roughly explains why there is no particular consumer outcry (which isn't about speed/bandwidth or mobile coverage, anyway). David Barak

On 10/08/2015 06:14 AM, Matthew Kaufman wrote:

On 10/7/15 7:00 AM, Mark Andrews wrote:

...

I don't see anyone wishing it went differnetly. I see someone pointing out the reality that lots of ISP's are way too late to delivering IPv6. *Every* ISP should have been planning to deliver IPv6 by the time the first RIR ran out of IPv4 addresses.

Look, I'm as much a supporter of delivering IPv6 as anyone. I've had IPv6 enabled on my home network (and the small data center I run in my garage) for over a decade now. In 2004, I made sure that IPv6 was fully supported in the peer-to-peer stack I developed and that eventually became RFC 7016. And for the last 5 years I've been pushing for IPv6 support in the product I work on for my employer.

But the reality is that there's a whole lot of small and medium-sized ISPs run by fine, upstanding individuals serving their communities -- even in and around the San Francisco Bay Area -- that have either no or very limited (tunnels only) support for IPv6. That's the reality of the transition. And threatening these folks with the attorney general isn't the way to get them to adopt IPv6, nor is shaming them. They will add IPv6 support when it is easy to do, when their staff has the time, and when the economics make sense.

Plus one to that. We are such a provider, and IPv6 is on my list of things to implement, but the barriers are still plenty high. Firstly, I do have an Ipv6 assignmnt and bgp (v4) and an asn, but until I can get IPv6 transit, there is not much point in my putting a lot of effort into enabling IPv6 for my subscribers. Yes I have a HE tunnel and yes it's working, but it's not the same as running native v6 and with my own address space. Second, on the group of servers that have v6 thru the HE tunnel, I still run into problems all the time where some operations over v6 simply fail inexplictly, requireing me to turn off v6 on that host so whatever it is I'm doing can proceed over v4. Stuff like OS updates for example. Damm maddening. Can't imagine the screaming I'll hear if a home user ever ran into similar so I am quite gun shy about the prospect. Secondly, the the dodgy nature of the CPE connected to our network and the terminally buggy fw they all run is sure to be a never ending source of stupidity. Thirdly, some parts of my network are wireless, and multicast is a huge, huge problem on wireless (the 802.11 varities anyways). The forwarding rates for multicast are sickeningly low for many brand of gear - yes, it's at the bottom of the barrel no matter how good or hot your signal is - and I honestly expect v6 to experience enough disruption over wireless as to render it unusable for exactly this reason alone. The wired portion of my subscriber network is only slightly better, im pretty sure it can deal with v6 in the middle, but the question is still wether specfic CPE models can and which set of bugs I'll hit on my access concentrators passing our v6 over PPPoE. I just read about a cisco bug where enabling rp-filtering on v6 causes a router reload, which I would hit immediately since rp-filtering is a standard subscriber profile option here (trying to be a good netizen). How many other network destroying bugs await? The longer I wait on v6, the less work I will have to do dealing with bugs. So, as the original posted said, we'll do v6 when it's easy, when we have time, and when the economics make sense. Mike-

On 10/08/2015 02:41 PM, Mark Andrews wrote:

Plus one to that. We are such a provider, and IPv6 is on my list of things to implement, but the barriers are still plenty high. Firstly, I do have an Ipv6 assignmnt and bgp (v4) and an asn, but until I can get IPv6 transit,

There are lots of transit providers that provide IPv6. It really is time to name and shame transit providers that don't provide IPv6.

NO, THERE IS NOT. We operate in rural and underserved areas and WE DO NOT HAVE realistic choices. Can you see me from your ivory tower?

...

there is not much point in my putting a lot of effort into enabling IPv6 for my subscribers. Yes I have a HE tunnel and yes it's working, but it's not the same as running native v6 and with my own address space. Second, on the group of servers that have v6 thru the HE tunnel, I still run into problems all the time where some operations over v6 simply fail inexplictly, requireing me to turn off v6 on that host so whatever it is I'm doing can proceed over v4. Stuff like OS updates for example. Then complain to the OS vendor. It is most probably someone breaking PMTU discover by filtering PTB. Going native will hide these problems until the MTU between the DC and the rest of the net increases. You could also just lower the advertised MTU internally to match the tunnel MTU which would let you simulate better what a native experience would be. Not my job. v4 works, v6 does not, end of story.

I can't remember the last time I saw a site stall due to reaching it over IPv6 it is that long ago.

It happens every day for me, which only amplifies my perception that v6 IS NOT READY FOR PRIME TIME.

...

Damm maddening. Can't imagine the screaming I'll hear if a home user ever ran into similar so I am quite gun shy about the prospect. Secondly, the the dodgy nature of the CPE connected to our network and the terminally buggy fw they all run is sure to be a never ending source of stupidity. CPE devices are buggy for IPv4 as well. Bugs in CPE devices are only found and fixed if the code paths are exercised.

That said IPv6 worked fine for me with the shipped image (old version of OpenWRT) using 6to4 before I reflashed it to a modern version of OpenWRT as I wanted to use the HE tunnel rather than 6to4. I know that is only one CPE device. And will you be providing all of my end users with replacement CPE

Not my job. v4 works, v6 does not. I am a provider not a developer. that meets all of the other requirements too? No? Because no such devices exist yet? OHHH yeah thats right, I'm a provider not a developer, so again, not a solution for my business.

...

Thirdly, some parts of my network are wireless, and multicast is a huge, huge problem on wireless (the 802.11 varities anyways). The forwarding rates for multicast are sickeningly low for many brand of gear - yes, it's at the bottom of the barrel no matter how good or hot your signal is - and I honestly expect v6 to experience enough disruption over wireless as to render it unusable for exactly this reason alone. You expect but haven't tested.

Based on observation and experience, I think v6 will wipe out the 802.11 portion of my network and no, Im not going to 'test' it, recovery would be near impossible and in any event I don't experiment with paying customers. I won't move until the underlaying issues are resolved, and that means fixing multicast in wireless, which won't be done by me again because, you guessed it, I am a provider and not a developer.

...

The wired portion of my subscriber network is only slightly better, im pretty sure it can deal with v6 in the middle, but the question is still wether specfic CPE models can and which set of bugs I'll hit on my access concentrators passing our v6 over PPPoE. I just read about a cisco bug where enabling rp-filtering on v6 causes a router reload, which I would hit immediately since rp-filtering is a standard subscriber profile option here (trying to be a good netizen). How many other network destroying bugs await? The longer I wait on v6, the less work I will have to do dealing with bugs. So, as the original posted said, we'll do v6 when it's easy, when we have time, and when the economics make sense. And is there a fix available yet? All code has bugs in it. They exist in both the IPv4 code paths and the IPv6 code paths. There are lots of places that are going IPv6 only internally and only having IPv4 at the fringe. You can't do that if routers are flakey when pushing IPv6 packets. This is basically just fear overriding rational decisions.

I am a provider and not a developer, and I am likely only going to use what I know works and what is within my sphere of control and influence. The flakey crappy state of v6 today means I am not putting it out anywhere a customer would have any exposure to it. I don't play games with my customers that way.

This thread, while originally interesting and helpful, seems to have degraded to a contest to see who can be the most arrogant, condescending and insulting. Congrats. On Oct 8, 2015 6:25 PM, "James Jun" <james@towardex.com> wrote:

On Thu, Oct 08, 2015 at 03:45:38PM -0700, Mike wrote:

...

NO, THERE IS NOT. We operate in rural and underserved areas and WE DO NOT HAVE realistic choices. Can you see me from your ivory tower?

Who is your upstream provider?

I think you're confused on how the IP transit industry works.

If you want choices in your transit providers, you should get a transport circuit (dark, wave or EPL) to a nearby carrier hotel/data center. Once you do that, you will suddenly find that virtually almost everyone in the competitive IP transit market will provide you with dual-stacked IPv4/IPv6 service.

If you are buying DIA circuit from some $isp to your rural location that you call "head-end" and are expecting to receive a competitive service, and support for IPv6, well, then your expectations are either unreasonable, ignorant or both.

Best, James

I have been reviewing the proposed submission to the FCC ET Docket No. 15-170, regarding the requirement that vendors of wireless equipment "lock down" updates, and find this quote in that submission particularly apropos to the ongoing IPv6-on-wireless discussion: "Most Wi-Fi routers, even the newest ones, do not or only barely support IPv6, with poor implementations of IPv6 leading to problems with interoperability" Quoting this thread, they are, in their submission! Does it make the assertion that "existing CPE isn't ready for prime time" accurate? That's still open for debate. What is "settled science" is that, after 17 years, we are still learning just how to deal with IPv6's quirks and foibles, just as the community did with IPv4 in the last quarter of the previous century (and continuing into this one). After 17 years, the community at large has a huge hurdle educating the vendors, the providers, and the consumers to the point that everyone (not just a few geeks) has a satisfactory comfort level with the new protocol. The heat and flame in this thread shows the range of comfort levels. Most of the conflict exposed in this thread concerns the differences in views by providers of general transit, versus providers of "last mile" and enterprise connectivity. The positions taken by the two camps are equally valid FOR THEIR MARKETS. As one of the participants in the latter market category, I have learned a great deal about IPv6. Moreover, I have been given pointers by people in this forum to sources of additional information, particularly sources that are more up to date. Amazon will be delivering a big box around the 17th. As for my own network, I have discovered that my upstream, Charter Communications, has put together some support pages for their deployment of IPv6. Once I have the appropriate firewall in place to mirror my IPv4 firewall I can start getting dual-stack IPv4/IPv6 up and going.

On Oct 8, 2015, at 11:24 PM, Jeremy Austin <jhaustin@gmail.com> wrote:

On Thu, Oct 8, 2015 at 3:25 PM, James Jun <james@towardex.com> wrote:

...

If you want choices in your transit providers, you should get a transport circuit (dark, wave or EPL) to a nearby carrier hotel/data center. Once you do that, you will suddenly find that virtually almost everyone in the competitive IP transit market will provide you with dual-stacked IPv4/IPv6 service.

The future is here, but it isn't evenly distributed yet. I'm in North America, but there are no IXPs in my *state*, let alone in my *continent* -- from an undersea fiber perspective. There is no truly competitive IP transit market within Alaska that I am aware of. Would love to be proved wrong. Heck, GCI and ACS (the two providers with such fiber) only directly peered a handful of years ago.

Alaska is in the same continent as Canda and the Contiguous US. VANIX (Vancouver), CIX (Calgary), Manitoba-IX (Winnipeg), WPGIX (WInnipeg), TORIX (Toronto), and an exchange in Montreal (I forget the name) exist as well as a few others in Canada (I think there’s even one out in the maritimes). There are tons of exchanges all over the contiguous US. I’m surprised that there isn’t yet an exchange point in Juneau or Anchorage, but that does, indeed, appear to be the case. Perhaps you should work with some other ISPs in your state to form one. According to this: http://www.alaskaunited.com <http://www.alaskaunited.com/> There is subsea fiber to several points in AK from Seattle and beyond. And on a continental basis, quite a bit of undersea fiber in other landing stations around the coastal areas of the contiguous 48.

...

If you are buying DIA circuit from some $isp to your rural location that you call "head-end" and are expecting to receive a competitive service, and support for IPv6, well, then your expectations are either unreasonable, ignorant or both.

Interestingly both statewide providers *do* provide both IPv4 and IPv6 peering. The trick is to find a spot where there's true price competition. The 3 largest statewide ISPs have fiber that meets a mere three city blocks from one of my POPs, but there's no allowable IX. I'm looking at you, AT&T.

I’m not sure what you mean by “allowable IX”, to the best of my knowledge, anyone can build an IX anywhere. Owen

As I "know" Jeremy from elsewhere, I reached out to him this morning about this. I would like to speak with any other Alaskers? Alaskites? people from Alaska. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Jeremy Austin" <jhaustin@gmail.com> To: "Owen DeLong" <owen@delong.com> Cc: nanog@nanog.org, "James Jun" <james@towardex.com> Sent: Friday, October 9, 2015 3:51:12 PM Subject: Re: /27 the new /24 On Fri, Oct 9, 2015 at 12:04 PM, Owen DeLong <owen@delong.com> wrote:

The future is here, but it isn't evenly distributed yet. I'm in North America, but there are no IXPs in my *state*, let alone in my *continent* -- from an undersea fiber perspective. There is no truly competitive IP transit market within Alaska that I am aware of. Would love to be proved wrong. Heck, GCI and ACS (the two providers with such fiber) only directly peered a handful of years ago.

Alaska is in the same continent as Canda and the Contiguous US.

Geographically yes, but not IP-topologically. It may strictly speaking be an exaggeration to speak of continental latencies, but we do feel a bit cut off up here. From me to Ohio is just about twice as far as from me to CA. The distance from the eastern US to Portugal is only about twice as long as the Anchorage to Seattle route.

VANIX (Vancouver), CIX (Calgary), Manitoba-IX (Winnipeg), WPGIX (WInnipeg), TORIX (Toronto), and an exchange in Montreal (I forget the name) exist as well as a few others in Canada (I think there’s even one out in the maritimes).

If there were ever an Alaska-to-Canada pipeline or gas line built, no doubt there could be fiber. To my knowledge no non-Arctic Alaska to Yukon route exists or is in public planning. I think AT&T may have some microwave. The Yukon has less overall population than the city of Fairbanks, AK, and it would be difficult to justify a fiber build, say, from Tok to Whitehorse, without other reasons. I'm not looking at great circle routes at the moment, but an overland route would probably be *longer* from Anchorage to Vancouver than the current undersea routes.

There are tons of exchanges all over the contiguous US.

Exactly. Now imagine an area — Alaska not including Anchorage — twice the size of Texas, with the population of Pittsburgh, in tiny clumps far apart. It is *possible* that the lack of IX in Alaska is due solely to geography and not, say, to an inadequately competitive ISP environment. I’m surprised that there isn’t yet an exchange point in Juneau or

Anchorage, but that does, indeed, appear to be the case. Perhaps you should work with some other ISPs in your state to form one.

Juneau, I'm not so surprised; how many other cities that small and isolated have IXes? I'm curious. It's an interesting prospect, at least for some value of $location. Anyone interested, hit me up. According to this:

http://www.alaskaunited.com

There is subsea fiber to several points in AK from Seattle and beyond.

Said undersea fiber is owned by GCI and ACS. There are some pending routes west and north, I believe.

And on a continental basis, quite a bit of undersea fiber in other landing stations around the coastal areas of the contiguous 48.

If you are buying DIA circuit from some $isp to your rural location that you call "head-end" and are expecting to receive a competitive service, and support for IPv6, well, then your expectations are either unreasonable, ignorant or both.

Interestingly both statewide providers *do* provide both IPv4 and IPv6 peering. The trick is to find a spot where there's true price competition. The 3 largest statewide ISPs have fiber that meets a mere three city blocks from one of my POPs, but there's no allowable IX. I'm looking at you, AT&T.

I’m not sure what you mean by “allowable IX”, to the best of my knowledge, anyone can build an IX anywhere.

I should have been more clear. No allowable IX *at the nearest fiber meetup to me*. It would be illuminating to see what minimum peak hour per-capita bw is necessary to make rural IX pay, and for what value of $rural. "Alaska suffers from… an abject lack of density." —Joe Freddoso, Mighty River/USAC

On 10/12/15 1:57 AM, Henrik Thostrup Jensen wrote:

On Fri, 9 Oct 2015, Jeremy Austin wrote:

...

Juneau, I'm not so surprised; how many other cities that small and isolated have IXes? I'm curious. It's an interesting prospect, at least for some value of $location.

Several small cities in Sweden have IXes. Not sure than any of them are quite as small as Juneau, but some (Borås, Luleå, Sundsvall) are sub 100k people, and other cities (Umeå, Uppsala) are just over 100k inhabitants. Umeå and Luleå are releativly isolated - at least by European standards.

Most of these are probably just a switch or two, and are probably there to provide better quality of service, and not because it makes for a good business.

Sweden's IX infrastructure is not entirely unique but are certainly borne out of a particular set of circumstances and public private partnerships that don't generally exist elsewhere. https://en.wikipedia.org/wiki/Netnod

Best regards, Henrik

Henrik Thostrup Jensen <htj at nordu.net> Software Developer, NORDUnet

On 12 Oct 2015, at 11:23, Todd Underwood wrote:

it's also not entirely obvious what the point of having local IXes that serve these kinds of collections of people.

I think that's true. But I don't think it's always the case this means there is no point. When Citylink (incubated by the City Council) started stringing fibre around central Wellington, New Zealand and building fast ethernet access into buildings in the mid-90s it wasn't obvious what benefit that would give anybody at a time when most of the country was hanging off a collection of E1 half-circuit leases to the west-coast US. But it led to a cottage industry in teleworking, off-site data storage, digital print shops and video conferencing in the city that would not have been possible otherwise. What Citylink built was part access network, part exchange, but to their credit they didn't spend too much time worrying about what to call what they were building: they just built it. Part of keeping the network stupid is giving people at the edge room to innovate. Sometimes when you build it, they come. Joe

On Mon, Oct 12, 2015 at 11:23 AM, Todd Underwood <toddunder@gmail.com> wrote:

it's also not entirely obvious what the point of having local IXes that serve these kinds of collections of people.

one might consider that localized services or peer-to-peer traffic might not want to burden the long-haul links, for the cost of a 1g or 10g port on a local switch/router. this conversation is sort of like the ipv6 part earlier though... 'if people want to do this, cool! if they don't or can't for $REASONS also cool.'

how much inter-ASN traffic is there generally for a city of 100k people, even if they all have 1Gb/s connections? are they all torrenting, accessing local business web pages that are hosted locally, streaming video from local streaming caches? if a local IX is a good place for a llnw, akamai, ggc, netflix cache node, i can see it, but that's about it.

it's not clear that a flix cache would rate showing up at an IX in (for example) juneau... though it'd sure be interesting to know the share of traffic your 4 exemplars contribute to the overall longhaul traffic mix.

t

On Mon, Oct 12, 2015 at 10:33 AM, joel jaeggli <joelja@bogus.com> wrote:

...

On 10/12/15 1:57 AM, Henrik Thostrup Jensen wrote:

...

On Fri, 9 Oct 2015, Jeremy Austin wrote:

...

Juneau, I'm not so surprised; how many other cities that small and isolated have IXes? I'm curious. It's an interesting prospect, at least for some value of $location.

Several small cities in Sweden have IXes. Not sure than any of them are quite as small as Juneau, but some (Borås, Luleå, Sundsvall) are sub 100k people, and other cities (Umeå, Uppsala) are just over 100k inhabitants. Umeå and Luleå are releativly isolated - at least by European standards.

Most of these are probably just a switch or two, and are probably there to provide better quality of service, and not because it makes for a good business.

Sweden's IX infrastructure is not entirely unique but are certainly borne out of a particular set of circumstances and public private partnerships that don't generally exist elsewhere.

https://en.wikipedia.org/wiki/Netnod

...

Best regards, Henrik

Henrik Thostrup Jensen <htj at nordu.net> Software Developer, NORDUnet

On Mon, Oct 12, 2015 at 1:19 PM, Todd Underwood <toddunder@gmail.com> wrote:

all,

On Mon, Oct 12, 2015 at 1:15 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:

...

On Mon, Oct 12, 2015 at 11:23 AM, Todd Underwood <toddunder@gmail.com> wrote:

...

it's also not entirely obvious what the point of having local IXes that serve these kinds of collections of people.

this conversation is sort of like the ipv6 part earlier though... 'if people want to do this, cool! if they don't or can't for $REASONS also cool.'

oh, for sure. anyone who wants to should, of course.

i'm just pointing out (in opposition to the drumbeat of "MOAR IXes EVERYWHERE!!!" message) that IXes are often not that useful and people should critically evaluate whether they need one and would benefit from the cost.

sure... folk in a position to do so might want to look at their netflow/etc data and decide to where they send/receive the most traffic, if it's their neighbor consider saying: "Howdy neighbor! how about we uncongest our longhaul and send these bits over a local ethernet? Oh! jane's also in the mix, let's get together on a hp switch and win!"

so far, the "coolness", "psychological", "possible future industry" benefits are all cited. that's fine. but there's often zero business case for an IX outside of major fibre confluences.

'major' perhaps depends on your perspective here, right? Sure, in Chicago where boatloads of east/west (and some North/South) fiber shares conduit it sure seems clearly a win to have an IX there... but I bet if you have 1g to SEA from ANK... losing 200mbps to crappy-gammer-uturn traffic would be nice to avoid too, eh? Or hell email even... anyway, sure more numbers and metrics and thought seems like a good plan, just like in the v6 discussion earlier.

In general, most of NANOG recipients live in the populated metros and know very little about what it's like to try to provide internet access in the hinterlands. do not pay attention to there magical claims of 'just connect to some IX and everything will be fine'. you already know that that's not how the internet in the rural west works. it's fine. smile and nod and pretend that they are making sensible claims and move back to trying to figure out how to make things work on your own network. cheers, t On Oct 10, 2015 2:43 PM, "Eric Kuhnke" <eric.kuhnke@gmail.com> wrote:

As Jeremy has described in detail, the problem is at OSI layer 1. Not a lack of peering exchanges such as the VANIX. There is no dark fiber route from Alaska via the Yukon to Vancouver.

I know where most of the Telus (ILEC) and Northwestel (Bell) fiber is in northern BC and none of interconnects with Alaska.

Network topologically all locations in Alaska which are fiber fed via the existing submarine cable routes (not on geostationary C/Ku-band satellite) are a suburb of Seattle. Imagine an island with a population of about 600,000 people located somewhere in Puget Sound with various DWDM circuits that have their other ends in the Westin Building. Various IP transit, peering, transport and IX connections at that location.

Other satellite fed singlehomed locations in Alaska can be logically just about anywhere thanks to the way bent-pipe relay via geostationary transponders work. There's at least a couple of dozen large teleports in the US 48 states with 7.3m and larger C-band dishes that support two way TDMA and SCPC services into Alaska. In such case the sites are indistinguishable from very low bandwidth singlehomed FDD microwave sites which happen to have at minimum 495ms latency.

On Fri, Oct 9, 2015 at 1:04 PM, Owen DeLong <owen@delong.com> wrote:

...

...

On Oct 8, 2015, at 11:24 PM, Jeremy Austin <jhaustin@gmail.com> wrote:

On Thu, Oct 8, 2015 at 3:25 PM, James Jun <james@towardex.com> wrote:

...

If you want choices in your transit providers, you should get a

transport

...

circuit (dark, wave or EPL) to a nearby carrier hotel/data center.

...

...

...

you do that, you will suddenly find that virtually almost everyone in the competitive IP transit market will provide you with dual-stacked IPv4/IPv6 service.

The future is here, but it isn't evenly distributed yet. I'm in North America, but there are no IXPs in my *state*, let alone in my *continent* -- from an undersea fiber perspective. There is no truly competitive IP transit market within Alaska that I am aware of. Would love to be

...

...

wrong. Heck, GCI and ACS (the two providers with such fiber) only directly peered a handful of years ago.

Alaska is in the same continent as Canda and the Contiguous US.

VANIX (Vancouver), CIX (Calgary), Manitoba-IX (Winnipeg), WPGIX (WInnipeg), TORIX (Toronto), and an exchange in Montreal (I forget the name) exist as well as a few others in Canada (I think there’s even one out in the maritimes).

There are tons of exchanges all over the contiguous US.

I’m surprised that there isn’t yet an exchange point in Juneau or Anchorage, but that does, indeed, appear to be the case. Perhaps you should work with some other ISPs in your state to form one.

According to this: http://www.alaskaunited.com <http://www.alaskaunited.com/>

There is subsea fiber to several points in AK from Seattle and beyond.

And on a continental basis, quite a bit of undersea fiber in other landing stations around the coastal areas of the contiguous 48.

...

...

If you are buying DIA circuit from some $isp to your rural location

Once proved that

...

...

...

you call "head-end" and are expecting to receive a competitive service, and support for IPv6, well, then your expectations are either unreasonable, ignorant or both.

Interestingly both statewide providers *do* provide both IPv4 and IPv6 peering. The trick is to find a spot where there's true price competition. The 3 largest statewide ISPs have fiber that meets a mere three city blocks from one of my POPs, but there's no allowable IX. I'm looking at you, AT&T.

I’m not sure what you mean by “allowable IX”, to the best of my knowledge, anyone can build an IX anywhere.

Owen

On Thu, 08 Oct 2015 18:45:38 -0400, Mike <mike-nanog@tiedyenetworks.com> wrote:

WE DO NOT HAVE realistic choices.

Or, apparently, realistic expectations. You, do, indeed, deserve public shaming for your complete lack of willingness to support IPv6. Your customers have no "realistic choices" either. How many other ISPs do they have to choice from? If you cannot be bothered to support IPv6, how are they supposed to? ("use a tunnel broker" is the *WRONG* answer) You are an ISP. You don't get to say "NO!" to IPv6. It is what the global internet is moving towards. You _WILL_ support it, or you will be left behind, and your customers who have little or no other options will suffer for it. https://youtu.be/g1GF4Gnb-D0

...

I can't remember the last time I saw a site stall due to reaching it over IPv6 it is that long ago.

It happens every day for me, which only amplifies my perception that v6 IS NOT READY FOR PRIME TIME.

This is just *your* flawed perception. Have you bothered to be an engineer and figure out _WHY_ it doesn't work? Or do you like keeping your head in the sand mumbling "it's not my job"?

...

...

Thirdly, some parts of my network are wireless, and multicast ... Based on observation and experience, I think v6 will wipe out the 802.11

If you are providing customer access via 802.11 technology, then yes, you do have a serious problem. But it's a problem you already have with v4 as well... or do you block _ALL_ broadcast traffic from your 802.11 network? Have you checked those networks, because I'm pretty sure there's multicast on them already. (windows and mac generate multicast by default) My experience shows multicast is a problem for 99% of WiFi gear. It's handled like all other broadcast and sent at "basic rate" to all stations, which is going to be slow as crap. However, IPv6 ND (and RAs) do not amount to a volume that will, under normal conditions, kill a WiFi network. I run IPv6 over my 802.11a/b/g/n networks; no one has even noticed! (even with Truly Ancient Hardware(tm)) --Ricky

In message <56172237.5030501@satchell.net>, Stephen Satchell writes:

On 10/08/2015 05:50 PM, Ricky Beam wrote:

...

You are an ISP. You don't get to say "NO!" to IPv6. It is what the global internet is moving towards. You _WILL_ support it, or you will be left behind, and your customers who have little or no other options will suffer for it.

ISP == "Internet Service Provider". The key word here is "service". tiedyenetworks.com is a provider of services to customers, and I suspect those are retail customers. What he just told you is that the service he provides, in his experience, does not play well with IPv6 AS CURRENTLY IMPLEMENTED IN AVAILABLE EQUIPMENT. On the one hand, IPv6 is "the future" (I just invested a fair amount of cred to get the books recommended to me here on NANOG to get up to speed) but like early versions of just about every thing and every product, there are still a few potholes.

tiedyenetworks.com, from my reading of this thread, has elected to limit his service offerings to his customers that he can reasonably support. That's good, solid business sense. Nothing is worse than providing a product that does not work as expected or advertised. VW, anyone?

...

(windows and mac generate multicast by default)

And unless there is a damn good need for that multicast traffic, it gets blocked. From my edge network, I block multicasts and broadcasts both inbound and outbound. When I was network admin for the web hosting company I worked for, I also blocked a number of ports at my edge, ports that had no business being used in the general case. I had *one* customer that needed to come in using 3309; I punched a hole in the ACLs for that one customer, and damn carefully.

...

This is just *your* flawed perception. Have you bothered to be an engineer and figure out _WHY_ it doesn't work?

Maybe you missed his earlier declaration: "I'm a provider, not a developer." He expects the equipment to work. It doesn't. Did he ask his vendor? I don't know, but my personal experience with wireless-equipment vendors is not encouraging. Some people don't have the money, resources, or time to winkle out all the wrinkles, so they go with what works in their situation. Consider the rural market: damn few customers, so $150K engineers are out of the question.

I also saw that he was using a tunnel yet was unwilling to configure the local network to account for this when testing yet was willing to bag IPv6 due to the side effects of being behind a tunnel. IPv4 also works poorly when you introduce a tunnel and the people you connect to are idiots that block / don't handle PTB messages. Do like for like testing before bagging the protocol. 20% of the US eyeballs have working native IPv6 without lots of complaints. If you are have problems over a tunnel and they aren't you may want to re-evalute your opinion of IPv6 and look to getting native connections. IPv6 really does work as well as IPv4 give like for like connections. Mark

...

I run IPv6 over my 802.11a/b/g/n networks; no one has even noticed! (even with Truly Ancient Hardware(tm))

That's your experience. He has a different experience. I suspect your customer base is considerably more dense than tiedyenetwork.com's base. Did you say you are primarily a rural provider? Mike did. Your earlier traffic suggests your base of operations is more in a city or suburban environment. Apples and oranges, if true. -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

On 10/08/2015 07:58 PM, Owen DeLong wrote:

I can't remember the last time I saw a site stall due to reaching it over IPv6 it is that long ago.

...

It happens every day for me, which only amplifies my perception that v6 IS NOT READY FOR PRIME TIME. Yet you refuse to troubleshoot your issues with it that are not shared by others and blame the protocol for whatever is probably wrong with your own network. Interesting tactic.

Thats invalid. It matters not that you claim these isses are not 'shared by others' - they are experienced routinely by others, and it's growing worse as more 'services' are transitioned to 'v6' but then the attendant support such as monitoring and operational knowledge/experience hasn't caught up and those transitioned services fail on v6 silently for long periods of time. That is the majority of the v6 world today and it's useless to claim otherwise. Mike-

On 10/09/2015 08:18 AM, Christopher Morrow wrote:

(I'm going to regret this but...)

No good deed ever goes unpunished.

(I'm sure there's a Dune quote to be used here somewhere as well...)

Indeed: "A beginning is the time for taking the most delicate care that the balances are correct." -- _Dune_ (1965) Frank Herbert

On 10/8/15, 6:45 PM, "NANOG on behalf of Mike" <nanog-bounces@nanog.org on behalf of mike-nanog@tiedyenetworks.com> wrote:

On 10/08/2015 02:41 PM, Mark Andrews wrote:

...

Plus one to that. We are such a provider, and IPv6 is on my list of things to implement, but the barriers are still plenty high. Firstly, I do have an Ipv6 assignmnt and bgp (v4) and an asn, but until I can get IPv6 transit,

There are lots of transit providers that provide IPv6. It really is time to name and shame transit providers that don't provide IPv6.

NO, THERE IS NOT. We operate in rural and underserved areas and WE DO NOT HAVE realistic choices. Can you see me from your ivory tower?

I looked up tiedyenetworks.com, and I think he¹s 100 miles from Sacramento. I hope some sales person from a transit provider is giving him a call right now, but it¹s entirely possible that there are no big providers in his neighborhood. Sorry, Mike, wish I could help you there. However, you can still mock your upstreams here. Then we can offer them help to support IPv6.

...

...

there is not much point in my putting a lot of effort into enabling IPv6 for my subscribers. Yes I have a HE tunnel and yes it's working, but it's not the same as running native v6 and with my own address space. Second, on the group of servers that have v6 thru the HE tunnel, I still run into problems all the time where some operations over v6 simply fail inexplictly, requireing me to turn off v6 on that host so whatever it is I'm doing can proceed over v4. Stuff like OS updates for example. Then complain to the OS vendor. It is most probably someone breaking PMTU discover by filtering PTB. Going native will hide these problems until the MTU between the DC and the rest of the net increases. You could also just lower the advertised MTU internally to match the tunnel MTU which would let you simulate better what a native experience would be. Not my job. v4 works, v6 does not, end of story.

It sounds like you do have some concern about the transition, and you know there¹s a bug, at least with OS downloads. Please do report those issues you know about. Usually, Happy Eyeballs masks problems in dual stack, whether that¹s good or bad. If we can get your upstream(s) to support IPv6, then maybe we can leverage them to help troubleshoot MTU problems, so you don¹t have to spend a lot of time on them. Or maybe they go away when you¹re no longer tunnelling.

...

I can't remember the last time I saw a site stall due to reaching it over IPv6 it is that long ago.

It happens every day for me, which only amplifies my perception that v6 IS NOT READY FOR PRIME TIME.

Would you at least keep a list of places you have these problems, even if you never follow up on it? Again, I¹m wondering if tunnelling is the problem, and once you have native dual-stack, you could refer to the list and see if problems just dry up.

...

...

Damm maddening. Can't imagine the screaming I'll hear if a home user ever ran into similar so I am quite gun shy about the prospect. Secondly, the the dodgy nature of the CPE connected to our network and the terminally buggy fw they all run is sure to be a never ending source of stupidity. CPE devices are buggy for IPv4 as well. Bugs in CPE devices are only found and fixed if the code paths are exercised.

Not my job. v4 works, v6 does not. I am a provider not a developer.

I would guess it is your job in IPv4. I would also guess, based on gateways I¹ve seen, than 10% of CPE has critical IPv4 bugs, and 25% of CPE has critical Ipv6 bugs. I agree with you that the difference is too high, and maybe waiting a year helps get those ratios aligned. CPE vendors: step it up!

...

That said IPv6 worked fine for me with the shipped image (old version of OpenWRT) using 6to4 before I reflashed it to a modern version of OpenWRT as I wanted to use the HE tunnel rather than 6to4. I know that is only one CPE device. And will you be providing all of my end users with replacement CPE that meets all of the other requirements too? No? Because no such devices exist yet? OHHH yeah thats right, I'm a provider not a developer, so again, not a solution for my business.

Ugh, 6to4 is a bad idea anyway. Deprecated, even. There are loads of gateways that support native dual-stack and several transition mechanisms (DS-lite, 6rd, and MAP pretty soon), but native dual-stack is the way to go if you possibly can. If you provide gateways as part of your service, you should at least make sure you¹re providing devices that at least *claim* IPv6 support now (and the IPv6 CPE Ready logo is meaningful here), so that as old equipment ages out, you¹re not stuck replacing newer boxes. Figure out how long until you think you really need all of your customers to have IPv6. Subtract your CPE replacement time. Start replacing CPE then. e.g., if users need IPv6 in 2018, and you replace all CPE on a 5 year schedule, you should begin providing IPv6-capable CPE in 2013.

...

...

Thirdly, some parts of my network are wireless, and multicast is a huge, huge problem on wireless (the 802.11 varities anyways). The forwarding rates for multicast are sickeningly low for many brand of gear - yes, it's at the bottom of the barrel no matter how good or hot your signal is - and I honestly expect v6 to experience enough disruption over wireless as to render it unusable for exactly this reason alone. You expect but haven't tested.

Based on observation and experience, I think v6 will wipe out the 802.11 portion of my network and no, Im not going to 'test' it, recovery would be near impossible and in any event I don't experiment with paying customers. I won't move until the underlaying issues are resolved, and that means fixing multicast in wireless, which won't be done by me again because, you guessed it, I am a provider and not a developer.

This might help: https://tools.ietf.org/html/draft-ietf-v6ops-reducing-ra-energy-consumption -02 Cisco has done some presentations on their use of IPv6 over WiFi at Cisco Live and other venues. For instance, http://www.rmv6tf.org/wp-content/uploads/2013/04/5-2013-04-17-RMv6TF-kreddy -02.pdf Might be able to mitigate with configuration.

...

...

The wired portion of my subscriber network is only slightly better, im pretty sure it can deal with v6 in the middle, but the question is still wether specfic CPE models can and which set of bugs I'll hit on my access concentrators passing our v6 over PPPoE. I just read about a cisco bug where enabling rp-filtering on v6 causes a router reload, which I would hit immediately since rp-filtering is a standard subscriber profile option here (trying to be a good netizen). How many other network destroying bugs await? The longer I wait on v6, the less work I will have to do dealing with bugs. So, as the original posted said, we'll do v6 when it's easy, when we have time, and when the economics make sense. And is there a fix available yet? All code has bugs in it. They exist in both the IPv4 code paths and the IPv6 code paths. There are lots of places that are going IPv6 only internally and only having IPv4 at the fringe. You can't do that if routers are flakey when pushing IPv6 packets. This is basically just fear overriding rational decisions.

I am a provider and not a developer, and I am likely only going to use what I know works and what is within my sphere of control and influence. The flakey crappy state of v6 today means I am not putting it out anywhere a customer would have any exposure to it. I don't play games with my customers that way.

It makes sense to me that you would want to wait another year. An ISP of your size doesn¹t have the support staff to troubleshoot new problems. I do hope you¹ll keep an eye on deployments, and at least be thinking in terms of deploying next year (e.g., by buying gear that at least claims IPv6 support, thinking about what monitoring you need to keep an eye on it as you deploy), so that when you do start, you have an easier time of it. Lee

On 12 October 2015 at 19:49, Mike <mike-nanog@tiedyenetworks.com> wrote:

No, it's not going to help. v6 over current wireless doesn't work for the reasons that multicast is a gaping hole.

Why is IPv6 multicast any different than IPv4 broadcast (required for ARP and many other things)? If you run without MLD snooping IPv6 multicast is implemented as broadcast at the layer 2 level, so it should behave exactly the same. You also have the option of using something like 6RD so you avoid IPv6 in your backbone entirely. Or use MPLS for the same effect. Regards, Baldur

On Fri, 9 Oct 2015, Mark Andrews wrote:

...

Plus one to that. We are such a provider, and IPv6 is on my list of things to implement, but the barriers are still plenty high. Firstly, I do have an Ipv6 assignmnt and bgp (v4) and an asn, but until I can get IPv6 transit,

There are lots of transit providers that provide IPv6. It really is time to name and shame transit providers that don't provide IPv6.

Unless he's buying from Bob's Bait, Tackle, and Internet (who's reselling service off his Brighthouse cable modem connection), I find it hard to believe there are "transit providers" in the NANOG region who still cannot provide dual-stack addressing and BGP for DIA.

...

there is not much point in my putting a lot of effort into enabling IPv6 for my subscribers. Yes I have a HE tunnel and yes it's

With some OS's (Apple) preferring v6 if it's there, it would actually be a bad idea to enable IPv6 for your subscribers before you have stable/reliable v6 connectivity hooked up for the network.

...

address space. Second, on the group of servers that have v6 thru the HE tunnel, I still run into problems all the time where some operations over v6 simply fail inexplictly, requireing me to turn off v6 on that host so whatever it is I'm doing can proceed over v4.

v6 routing doesn't always get the same level of scrutiny as v4. i.e. Suboptimal v6 paths might get used for some time before someone with enough clue to notice speaks up. Presumably, that will change as v6 adoption gets more widespread. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

On 7 Oct 2015, at 9:29, Matthew Kaufman wrote:

...

On Oct 7, 2015, at 5:01 AM, Owen DeLong <owen@delong.com> wrote:

Instead, the followup question is needed… “That’s great, but how does that help me reach a web site that doesn’t have and can’t get an IPv4 address?”

At the present time, a web site that doesn't have and can't get an IPv4 address isn't "on the Internet".

By the only definition of the Internet that matters, the function "is on the Internet" is very much in the eye of the beholder. Using this definition, v6-only services are most certainly "on the Internet" for people who have v6 connectivity. Similarly, various instances of the Pirate Bay that have v4 reachability are not "on the Internet" for end-users in draconian jurisdictions like the UK. Trying to make assertions about what "on the Internet" means in a general, global sense is an effort doomed to failure. I realise you're talking in pragmatic terms about services that have a general, dispersed, global end-user population, but not all services are like that. Joe [1] Internet, n: “the largest equivalence class in the reflexive transitive symmetric closure of the relationship ‘can be reached by an IP packet from’” (Seth Breidbart)

Hi,

Op 4 okt. 2015, om 16:52 heeft Mel Beckman <mel@beckman.org> het volgende geschreven:

If it doesn't support IPSec, it's not really IPv6. Just as if it failed to support any other mandatory IPv6 specification, such as RA.

I think you're still looking at an old version of the IPv6 Node Requirements. Check https://tools.ietf.org/html/rfc6434#section-11, specifically this bit: """ Previously, IPv6 mandated implementation of IPsec and recommended the key management approach of IKE. This document updates that recommendation by making support of the IPsec Architecture a SHOULD for all IPv6 nodes. """ This was published in December 2011. Cheers, Sander

Memory footprint is still an issue in lots of things like ESP8266 (which doesn’t yet support IPv6, but hopefully will soon). Not everything is a cell phone or larger. There are lots of cool new things coming out in the SoC world where you’ve got a micro controller, GPIOs, CAN, SPI, WiFi, and more all in a single chip or module. Another example (also currently IPv4 only, but hopefully that will get fixed) is particle.io. These are $10-$20 (and sometimes even less) complete systems with very small memories and very low power consumption which are great for deploying things like remote sensors and the like. Owen

On Oct 4, 2015, at 11:15 AM, Mel Beckman <mel@beckman.org> wrote:

Stefann,

You're right. I remember hearing rumblings of vendors requesting this change, mostly because embedded processors of the time had difficulty performing well with IPv6. I see that in 2011 rfc6434 lowered IPSec from "must" to "should". Nevertheless, plenty of products produced before 2011 included IPSec and the vast majority of IPv6-capable nodes on the Internet have it today. Performance is no longer an issue.

-mel beckman

...

On Oct 4, 2015, at 8:58 AM, Sander Steffann <sander@steffann.nl> wrote:

Hi,

...

Op 4 okt. 2015, om 16:52 heeft Mel Beckman <mel@beckman.org> het volgende geschreven:

If it doesn't support IPSec, it's not really IPv6. Just as if it failed to support any other mandatory IPv6 specification, such as RA.

I think you're still looking at an old version of the IPv6 Node Requirements. Check https://tools.ietf.org/html/rfc6434#section-11, specifically this bit:

""" Previously, IPv6 mandated implementation of IPsec and recommended the key management approach of IKE. This document updates that recommendation by making support of the IPsec Architecture a SHOULD for all IPv6 nodes. """

This was published in December 2011.

Cheers, Sander

We know. I recommend you read the whole thread before reacting. -mel beckman

On Oct 7, 2015, at 4:56 AM, Owen DeLong <owen@delong.com> wrote:

...

On Oct 4, 2015, at 7:52 AM, Mel Beckman <mel@beckman.org> wrote:

If it doesn't support IPSec, it's not really IPv6. Just as if it failed to support any other mandatory IPv6 specification, such as RA.

Not true. IPSec is recommended, not mandatory.

This change was made in favor of resource constrained nodes (think micro controllers with very small memories).

...

There's really no excuse for not supporting IPSec, as it's a widely available open source component that costs nothing to incorporate into an IPv6 stack.

Simply not true. There are nodes which have no need for it and are resource constrained.

...

Your observation simply means that users must be informed when buying IPv6 devices, just as they must with any product. You can buy either genuine IPv6 or half-baked IPv6 products. When I speak of IPv6, I speak only of the genuine article.

This is true. If you need the device to support IPv6, you should definitely make sure that it does, but that is ordinary reality with any feature of any product rather than anything specific to IPv6.

Owen

On 04/10/2015 16:03, Randy Bush wrote:

yet another ipv6 fantasy. it may be in the powerpoint but it is not in the implementations.

the ipsec tickbox was removed from ipv6 in rfc6434 (2011). Nick

I recommend any of a number of online courses for a quick understanding of IPv6. But nothing beats making your own IPv6 lab and getting hands-on experience. Here's a course I built walking you through that process: http://windowsitpro.com/build-your-own-ipv6-lab-and-become-ipv6-guru-demand -mel beckman

On Oct 4, 2015, at 7:49 AM, Stephen Satchell <list@satchell.net> wrote:

...

On 10/04/2015 06:40 AM, Matthias Leisi wrote: Fully agree. But the current state of IPv6 outside "professional“ networks/devices is sincerely limited by a lot of poor CPE and consumer device implementations.

I have to ask: where is the book _IPv6 for Dummies_ or equivalent?

Specifically, is http://www.xnetworks.es/contents/Infoblox/IPv6forDummies.pdf any good? (I just downloaded it to inspect.)

My bookshelf is full of books describing IPv4. Saying "IPv6 just works" ignores the issues of configuring intelligent firewalls to block the ne-er-do-wells using the new IP-level protocol.

In Robert L. Ziegler's book _Linux Firewalls_ Second Edition (2002, ISBN 0-7357-1099-6), the *only* mention of IPv6 is in the discussion of NAT, and that discussion is limited to "NAT is a stopgap until IPv6 achieves wide implementation. A preview of the Third Edition fails to mention ip6tables at all, the same lack that the Second Edition has.

I use CentOS, the community version of Red Hat Enterprise. I looked around for useful books on building IPv6 firewalls with the same granularity as the above-mentioned book for IPv4, and haven't found anything useful as yet. In particular, I'm looking for material that lays out how to build a mostly-closed firewall and DMZ in IPv6. The lack of IPv6 support goes further: I didn't find anything useful in Red Hat (CentOS) firewall tools that provides IPv6 support...but that's probably because I don't know what I'm looking for. (Also, that GUI software is intended for use on individual servers/computers, not in a edge-firewall with forwarding and DMZ responsibilities.)

Building a secure firewall takes more than just knowing how to issue ip6table commands; one also needs to know exactly what goes into those commands. NANOG concentrates on network operators who need to provide a good Internet experience to all their downstream customers, which is why I see the bias toward openness...as it should be. Those of us who run edge networks have different problems to solve.

I'm not asking NANOG to go past its charter, but I am asking the IPv6 fanatics on this mailing list to recognize that, even though the net itself may be running IPv6, the support and education infrastructure is still behind the curve. Reading RFCs is good, reading man pages is good, but there is no guidance about how to implement end-network policies in the wild yet...at least not that I've been able to find.

"ipv6.disable" will be changed to zero when I know how to set the firewall to implement the policies I need to keep other edge networks from disrupting mine.

On Oct 4, 2015, at 7:49 AM, Stephen Satchell <list@satchell.net> wrote:

On 10/04/2015 06:40 AM, Matthias Leisi wrote:

...

Fully agree. But the current state of IPv6 outside "professional“ networks/devices is sincerely limited by a lot of poor CPE and consumer device implementations.

I have to ask: where is the book _IPv6 for Dummies_ or equivalent?

Specifically, is http://www.xnetworks.es/contents/Infoblox/IPv6forDummies.pdf any good? (I just downloaded it to inspect.)

My bookshelf is full of books describing IPv4. Saying "IPv6 just works" ignores the issues of configuring intelligent firewalls to block the ne-er-do-wells using the new IP-level protocol.

You will need most of the same blockages in IPv6 that you needed in IPv4, actually. There are some important differences for ICMP (don’t break PMTU-D or ND), but otherwise, really not much difference between your IPv4 security policy and your IPv6 security policy. In fact, on my linux box, I generate my IPv4 iptables file using little more than a global search and replace on the IPv6 iptables configuration which replaces the IPv6 prefixes/addresses with the corresponding IPv4 prefixes/addresses. (My IPv6 addresses for things that take incoming connections have an algorithmic map to IPv4 addresses for things that have them.)

I use CentOS, the community version of Red Hat Enterprise. I looked around for useful books on building IPv6 firewalls with the same granularity as the above-mentioned book for IPv4, and haven't found anything useful as yet. In particular, I'm looking for material that lays out how to build a mostly-closed firewall and DMZ in IPv6. The lack of IPv6 support goes further: I didn't find anything useful in Red Hat (CentOS) firewall tools that provides IPv6 support...but that's probably because I don't know what I'm looking for. (Also, that GUI software is intended for use on individual servers/computers, not in a edge-firewall with forwarding and DMZ responsibilities.)

Where you have an iptables file, you add an ip6tables file and change the prefixes and addresses. Otherwise, it’s really pretty much the same. There is limited IPv6 support in many of the GUIs still, unfortunately, but the command line tools are all there and for the most part work pretty much identically for v4 and v6, the difference often being as little as ping vs ping6 or <command> <args> vs. <command> -6 <args>.

Building a secure firewall takes more than just knowing how to issue ip6table commands; one also needs to know exactly what goes into those commands. NANOG concentrates on network operators who need to provide a good Internet experience to all their downstream customers, which is why I see the bias toward openness...as it should be. Those of us who run edge networks have different problems to solve.

If you know what goes into the iptables commands, then there’s very little difference for the ip6tables commands. Primarily it involves changing the IPv4 addresses and/or prefixes into IPv6 addresses and/or prefixes. The rest of the commands are very much literally the same… An example from my actual configurations: iptables: -A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A RH-Firewall-1-INPUT -s 192.159.10.0/24 -m state --state NEW -m tcp -p tcp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 53 -m limit --limit 30/minute --limit-burst 90 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 53 -m limit --limit 30/minute --limit-burst 90 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5900 -j ACCEPT ip6tables: -A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT -A RH-Firewall-1-INPUT -s 2001:470:1f00:3142::/64 -m state --state NEW -m udp -p udp --dport 53 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 53 -m limit --limit 30/minute --limit-burst 90 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 53 -m limit --limit 30/minute --limit-burst 90 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5900 -j ACCEPT This is not my entire configuration (which is somewhat complex and in need of some cruft removal due to organic growth over time), but these 6 lines do provide a reasonably representative sample of things and include rate-limiting DNS queries from outsiders.

I'm not asking NANOG to go past its charter, but I am asking the IPv6 fanatics on this mailing list to recognize that, even though the net itself may be running IPv6, the support and education infrastructure is still behind the curve. Reading RFCs is good, reading man pages is good, but there is no guidance about how to implement end-network policies in the wild yet...at least not that I've been able to find.

There is actually quite a bit of information out there. Sylvia Hagen’s IPv6 book covers a lot of this (O’Reilly publishes it). There are also several other good IPv6 books.

"ipv6.disable" will be changed to zero when I know how to set the firewall to implement the policies I need to keep other edge networks from disrupting mine.

You do. You just don’t realize that you do. See above. Owen

Here is a quick starting point for filtering IPv6 on a Linux host system if you don't feel comfortable opening up all ICMPv6 traffic: http://soucy.org/tmp/v6firewall/ip6tables.txt I haven't really re-visited it in a while, so if I'm forgetting something let me know. On Wed, Oct 7, 2015 at 9:13 AM, Stephen Satchell <list@satchell.net> wrote:

This is excellent feedback, thank you.

On 10/07/2015 04:54 AM, Owen DeLong wrote:

...

On Oct 4, 2015, at 7:49 AM, Stephen Satchell <list@satchell.net> wrote:

...

My bookshelf is full of books describing IPv4. Saying "IPv6 just works" ignores the issues of configuring intelligent firewalls to block the ne-er-do-wells using the new IP-level protocol.

You will need most of the same blockages in IPv6 that you needed in IPv4, actually.

There are some important differences for ICMP (don’t break PMTU-D or ND), but otherwise, really not much difference between your IPv4 security policy and your IPv6 security policy.

In fact, on my linux box, I generate my IPv4 iptables file using little more than a global search and replace on the IPv6 iptables configuration which replaces the IPv6 prefixes/addresses with the corresponding IPv4 prefixes/addresses. (My IPv6 addresses for things that take incoming connections have an algorithmic map to IPv4 addresses for things that have them.)

On my box, I have a librry of shell functions that do the generation, driven by parameter tables. If I'm reading you correctly, I can just augment the parameter tables and those functions to generate the appropriate corresponding ip6table commands in parallel with the iptable commands.

Question: should I still rate-limit ICMP packets in IPv6? Also, someone on this list pointed me to NIST SP800-119, "Guidelines for the Secure Deployment of IPv6", the contents of which which I will incorporate.

There is limited IPv6 support in many of the GUIs still,

...

unfortunately, but the command line tools are all there and for the most part work pretty much identically for v4 and v6, the difference often being as little as ping vs ping6 or <command> <args> vs. <command> -6 <args>.

I've not been happy with the GUIs, because getting them to do what I want is a royal pain. For example, I'm forced to use port-based redirection in one edge firewall application -- I blew a whole weekend figuring out how to do that with the CentOS 7 firewalld corkscrew, for a customer who outgrew the RV-220 he used for the application. At least that didn't need IPv6!

Primarily it involves changing the IPv4 addresses and/or prefixes

...

into IPv6 addresses and/or prefixes.

What about fragmented packets? And adjusting the parameters in ip6table filters to detect the DNS "ANY" requests used in the DDoS amplification attacks?

I'm not asking NANOG to go past its charter, but I am asking the

...

...

IPv6fanatics on this mailing list to recognize that, even though the net itself may be running IPv6, the support and education infrastructure is still behind the curve. Reading RFCs is good, reading man pages is good, but there is no guidance about how to implement end-network policies in the wild yet...at least not that I've been able to find.

There is actually quite a bit of information out there. Sylvia Hagen’sIPv6 book covers a lot of this (O’Reilly publishes it).

Um, that would be "books". Which one do you recommend I start with?

* IPv6 Essentials (3rd Edition), 2014, ASIN: B00RWSNEKG * Planning for IPv6 (1st Edition), 2011, ISBN-10: 1449305393

(I would assume the first, as the NIST document probably covers the contents of the second)

There are also several other good IPv6 books.

...

Recommendations?

"ipv6.disable" will be changed to zero when I know how to set the

...

...

firewall to implement the policies I need to keep other edge networks from disrupting mine.

You do. You just don’t realize that you do. See above.

That's encouraging. Being able to leverage the knowledge from IPv4 to project the same policies into IPv6 makes it easier for me, as I'm already using programmatic methods of generating the firewalls. I expected that the implementation of existing applications-level policies would be parallel; it's the policies further down the stack that was my concern.

Also, I have a lot of IP level blocks (like simpler Cisco access control lists) to shut out those people who like to bang on my SSH front door. I believe that people who are so rude as to try to break through dozens or hundreds of time a day will have other bad habits, and don't deserve to be allowed for anything. (I have similar blocks for rabid spammers not in the DNSBLs, but that's a different story.) I would expect to maintain a separate list of IPv6 subnets, based on experience.

Which brings up another question: should I block IPv6 access to port 25 on my mail servers, and not announce a AAAA record for it? Postfix handles IPv6, but I've seen discussion that e-mail service is going to be IPv4 only for quite a while. Should I even enable IPv6 on my mail server at this time? Or is that a question I should post elsewhere?

As an aside, my day job is converting to Python programming, so my first Python project may well be the conversion of my existing firewall generator into that language.

-- *Ray Patrick Soucy* Network Engineer I Networkmaine, University of Maine System US:IT 207-561-3526

It's not too far off though. One way of looking at it is, for each extra bit we allow, we potentially double the table size.

that is math. in reality table size is proportional to multihoming + traffic engineering randy

Much m ore than I'm willing to spend. ;-) ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Matthew Kaufman" <matthew@matthew.at> To: "Justin Wilson - MTIN" <lists@mtin.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 9:48:33 AM Subject: Re: /27 the new /24 A /24 isn't that expensive yet... Matthew Kaufman (Sent from my iPhone)

On Oct 2, 2015, at 7:32 AM, Justin Wilson - MTIN <lists@mtin.net> wrote:

I was in a discussion the other day and several Tier2 providers were talking about the idea of adjusting their BGP filters to accept prefixes smaller than a /24. A few were saying they thought about going down to as small as a /27. This was mainly due to more networks coming online and not having even a /24 of IPv4 space. The first argument is against this is the potential bloat the global routing table could have. Many folks have worked hard for years to summarize and such. others were saying they would do a /26 or bigger.

However, what do we do about the new networks which want to do BGP but only can get small allocations from someone (either a RIR or one of their upstreams)?

Just throwing that out there. Seems like an interesting discussion.

Justin Wilson j2sw@mtin.net

--- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth

http://www.midwest-ix.com COO/Chairman Internet Exchange - Peering - Distributed Fabric

How many routers out there have this limitation? A $100 router I bought ten years ago could manage many full tables. If someone's network can't match that today, should I really have any pity for them? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Matthew Kaufman" <matthew@matthew.at> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 10:48:29 AM Subject: Re: /27 the new /24 Cheaper than buying everyone TCAM Matthew Kaufman (Sent from my iPhone)

On Oct 2, 2015, at 8:32 AM, Mike Hammett <nanog@ics-il.net> wrote:

Much m ore than I'm willing to spend. ;-)

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com

Midwest Internet Exchange http://www.midwest-ix.com

----- Original Message -----

From: "Matthew Kaufman" <matthew@matthew.at> To: "Justin Wilson - MTIN" <lists@mtin.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 9:48:33 AM Subject: Re: /27 the new /24

A /24 isn't that expensive yet...

Matthew Kaufman

(Sent from my iPhone)

...

On Oct 2, 2015, at 7:32 AM, Justin Wilson - MTIN <lists@mtin.net> wrote:

I was in a discussion the other day and several Tier2 providers were talking about the idea of adjusting their BGP filters to accept prefixes smaller than a /24. A few were saying they thought about going down to as small as a /27. This was mainly due to more networks coming online and not having even a /24 of IPv4 space. The first argument is against this is the potential bloat the global routing table could have. Many folks have worked hard for years to summarize and such. others were saying they would do a /26 or bigger.

However, what do we do about the new networks which want to do BGP but only can get small allocations from someone (either a RIR or one of their upstreams)?

Just throwing that out there. Seems like an interesting discussion.

Justin Wilson j2sw@mtin.net

--- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth

http://www.midwest-ix.com COO/Chairman Internet Exchange - Peering - Distributed Fabric

Welcome to the real world ... Cisco SUP720-3BXL Cisco RSP720-3BXL and even the new and shiny SUP2T only supports 1 Mio routes (dicvided to IPv4 MPLS, IPv4 VRF, IPv4 global routes, etc). I guess this is still the truth: there are at least a few ten thousand of these devices running big parts of the internet. Take a look at some big players network - e.g. Level3. Their customer access routers in Slovakia, Austria and Germany are still based on the Cisco 6500/7600 platform. Of course there are many other vendors and platforms available which do NOT have this limitations. But there are also at least a ton of vendors on the market with exactly the same limitation :(. best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Freitag, 02. Oktober 2015 17:51 Cc: NANOG Betreff: Re: /27 the new /24 How many routers out there have this limitation? A $100 router I bought ten years ago could manage many full tables. If someone's network can't match that today, should I really have any pity for them? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Matthew Kaufman" <matthew@matthew.at> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 10:48:29 AM Subject: Re: /27 the new /24 Cheaper than buying everyone TCAM Matthew Kaufman (Sent from my iPhone)

On Oct 2, 2015, at 8:32 AM, Mike Hammett <nanog@ics-il.net> wrote:

Much m ore than I'm willing to spend. ;-)

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com

Midwest Internet Exchange http://www.midwest-ix.com

----- Original Message -----

From: "Matthew Kaufman" <matthew@matthew.at> To: "Justin Wilson - MTIN" <lists@mtin.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 9:48:33 AM Subject: Re: /27 the new /24

A /24 isn't that expensive yet...

Matthew Kaufman

(Sent from my iPhone)

...

On Oct 2, 2015, at 7:32 AM, Justin Wilson - MTIN <lists@mtin.net> wrote:

I was in a discussion the other day and several Tier2 providers were talking about the idea of adjusting their BGP filters to accept prefixes smaller than a /24. A few were saying they thought about going down to as small as a /27. This was mainly due to more networks coming online and not having even a /24 of IPv4 space. The first argument is against this is the potential bloat the global routing table could have. Many folks have worked hard for years to summarize and such. others were saying they would do a /26 or bigger.

However, what do we do about the new networks which want to do BGP but only can get small allocations from someone (either a RIR or one of their upstreams)?

Just throwing that out there. Seems like an interesting discussion.

Justin Wilson j2sw@mtin.net

--- http://www.mtin.net Owner/CEO xISP Solutions- Consulting – Data Centers - Bandwidth

http://www.midwest-ix.com COO/Chairman Internet Exchange - Peering - Distributed Fabric

Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router. I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient. I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse. Stop using old shit. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "William Herrin" <bill@herrin.us> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 1:09:16 PM Subject: Re: /27 the new /24 On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote:

How many routers out there have this limitation? A $100 router I bought ten years ago could manage many full tables. If someone's network can't match that today, should I really have any pity for them?

Hi Mike, The technology doesn't work the way you think it does. Or more precisely, it only works the way you think it does on small (cheap) end-user routers. Those routers do everything in software on a general-purpose CPU using radix tries for the forwarding table (FIB). They don't have to (and can't) handle both high data rates and large routing tables at the same time. For a better understanding how the big iron works, check out https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see folks here talk about TCAM. This stands for Ternary Content Addressable Memory. It's a special circuit, different from DRAM and SRAM, used by most (but not all) big iron routers. The TCAM permits an O(1) route lookup instead of an O(log n) lookup. The architectural differences which balloon from there move the router cost from your $100 router into the hundreds of thousands of dollars. Your BGP advertisement doesn't just have to be carried on your $100 router. It also has to be carried on the half-million-dollar routers. That makes it expensive. Though out of date, this paper should help you better understand the systemic cost of a BGP route advertisement: http://bill.herrin.us/network/bgpcost.html Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

Can I suggest you not use a $1000 software driven device to do the job of a 500 watt device on a 10 Gbps network? Mikrotik has its faults, yes, but it certainly has a place as well. That just happens to not be where the $4,000 Cisco is. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Oct 2, 2015 at 3:22 PM, Mel Beckman <mel@beckman.org> wrote:

Often I find that used Cisco gear is more reliable and just as affordable than newer gear with that tasty, flakey crust. I've had a terrible time with CCRs falling over with 1GB traffic while Cisco L3 3750s don't even breathe hard at 10Gbps. I see no reason to use anything like 2500w even with Cisco gear. A dual Cisco 3750 stack consumes maybe 500W. Cisco firmware, for all its faults, seems to be much better tested than Mikrotik's.

I once asked Mikrotik's support engineers how they performed regression testing, and they said "because we are a small, agile, disruptive innovator we don't follow old-school testing regimens. We're more interested in shipping affordable product." That's also their excuse for poor documentation.

From what I can see, "small, agile, disruptive innovator" is an excuse newer networking companies often give for "sloppy, poorly tested, ill-conceived" product development.

-mel beckman

...

On Oct 2, 2015, at 11:44 AM, Mike Hammett <nanog@ics-il.net> wrote:

Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router.

I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient.

I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse.

Stop using old shit.

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com

Midwest Internet Exchange http://www.midwest-ix.com

----- Original Message -----

From: "William Herrin" <bill@herrin.us> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 1:09:16 PM Subject: Re: /27 the new /24

...

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote: How many routers out there have this limitation? A $100 router I bought ten years ago could manage many full tables. If someone's network can't match that today, should I really have any pity for them?

Hi Mike,

The technology doesn't work the way you think it does. Or more precisely, it only works the way you think it does on small (cheap) end-user routers. Those routers do everything in software on a general-purpose CPU using radix tries for the forwarding table (FIB). They don't have to (and can't) handle both high data rates and large routing tables at the same time.

For a better understanding how the big iron works, check out https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see folks here talk about TCAM. This stands for Ternary Content Addressable Memory. It's a special circuit, different from DRAM and SRAM, used by most (but not all) big iron routers. The TCAM permits an O(1) route lookup instead of an O(log n) lookup. The architectural differences which balloon from there move the router cost from your $100 router into the hundreds of thousands of dollars.

Your BGP advertisement doesn't just have to be carried on your $100 router. It also has to be carried on the half-million-dollar routers. That makes it expensive.

Though out of date, this paper should help you better understand the systemic cost of a BGP route advertisement: http://bill.herrin.us/network/bgpcost.html

Regards, Bill Herrin

-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

Stop using old shit.

Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about carrier grade networks. Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Freitag, 02. Oktober 2015 20:38 An: NANOG <nanog@nanog.org> Betreff: Re: /27 the new /24 Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router. I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient. I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse. Stop using old shit. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "William Herrin" <bill@herrin.us> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 1:09:16 PM Subject: Re: /27 the new /24 On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote:

How many routers out there have this limitation? A $100 router I bought ten years ago could manage many full tables. If someone's network can't match that today, should I really have any pity for them?

Hi Mike, The technology doesn't work the way you think it does. Or more precisely, it only works the way you think it does on small (cheap) end-user routers. Those routers do everything in software on a general-purpose CPU using radix tries for the forwarding table (FIB). They don't have to (and can't) handle both high data rates and large routing tables at the same time. For a better understanding how the big iron works, check out https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see folks here talk about TCAM. This stands for Ternary Content Addressable Memory. It's a special circuit, different from DRAM and SRAM, used by most (but not all) big iron routers. The TCAM permits an O(1) route lookup instead of an O(log n) lookup. The architectural differences which balloon from there move the router cost from your $100 router into the hundreds of thousands of dollars. Your BGP advertisement doesn't just have to be carried on your $100 router. It also has to be carried on the half-million-dollar routers. That makes it expensive. Though out of date, this paper should help you better understand the systemic cost of a BGP route advertisement: http://bill.herrin.us/network/bgpcost.html Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

Hrm. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Jürgen Jaritsch" <jj@anexia.at> To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 2:25:10 PM Subject: AW: /27 the new /24

Stop using old shit.

Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about carrier grade networks. Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Freitag, 02. Oktober 2015 20:38 An: NANOG <nanog@nanog.org> Betreff: Re: /27 the new /24 Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router. I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient. I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse. Stop using old shit. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "William Herrin" <bill@herrin.us> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 1:09:16 PM Subject: Re: /27 the new /24 On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote:

How many routers out there have this limitation? A $100 router I bought ten years ago could manage many full tables. If someone's network can't match that today, should I really have any pity for them?

Hi Mike, The technology doesn't work the way you think it does. Or more precisely, it only works the way you think it does on small (cheap) end-user routers. Those routers do everything in software on a general-purpose CPU using radix tries for the forwarding table (FIB). They don't have to (and can't) handle both high data rates and large routing tables at the same time. For a better understanding how the big iron works, check out https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see folks here talk about TCAM. This stands for Ternary Content Addressable Memory. It's a special circuit, different from DRAM and SRAM, used by most (but not all) big iron routers. The TCAM permits an O(1) route lookup instead of an O(log n) lookup. The architectural differences which balloon from there move the router cost from your $100 router into the hundreds of thousands of dollars. Your BGP advertisement doesn't just have to be carried on your $100 router. It also has to be carried on the half-million-dollar routers. That makes it expensive. Though out of date, this paper should help you better understand the systemic cost of a BGP route advertisement: http://bill.herrin.us/network/bgpcost.html Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

Well said, Jürgen! -mel via cell

On Oct 2, 2015, at 4:13 PM, Jürgen Jaritsch <jj@anexia.at> wrote:

Hi Mike,

sorry, this was probably sent to quick ... let me please explain my POV of your statement:

I want to concentrate my detailed answer only to the backbone situation which is often handled by the 6500/7600 - I guess all of us know that the 6500/7600 has a ton of additional features ...

6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone capacities built with platforms that only had n*100M interfaces another 3-5 years before. Their only invest in these 3-5 years was to add the Gig line cards, install some software updates and add new fibre optics (GBICs). Chassis, cabling, management interfaces etc could be remain in the cabinet - they only had to replace ONE line card (let's say for a few thousand bucks) and with this invest they were able to scale up their capacities. Of course: at some point they also had to replace the SUPs, PSUs, FANs, etc. But the invest in the surrounding stuff is nothing compared with completely new machines.

So what all these companies did was buying a machine with an basic configuration and since 10(!) years they are able to expand this machines with (more or less) small and cheap upgrades.

In backbone situations the 6500/7600 are definitely at the end of the resources the platform can provide. Most of the carriers (and of course also the bigger ISPs) had a real chance to evaluate a new model/vendor to ran future networks (with possibly also a very good scale-up path and scaling- and upgrade-options). Most of the before mentioned are already in an migration process (let's take a look at Seabone ... they are migration from Cisco to a mix of Juniper and Huawei).

Summary: there are strict limitations within the Cisco 6500/7600 platform and these limitations forces the big players to move this boxes out (or move them into other parts of their network). The limitation with 1Mio routes is not a secret and the admins of these boxes decide what they want to use (e.g. 768k routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If the global routing table reaches the 768k mark (I guess this will happen in the next 12-18months) most of the boxes will crash again (as it happened in Aug 2014).

Regarding the words "I have a small router which handles multiple full tables ...": push and pull a few full tables at the same time and you'll see what's happening: the CCRs are SLOW. And why? Because the software is not as good as it could be: the BGP daemon uses only one core of a 36(?) core CPU. Same problem in the past with the EoIP daemon (not sure if they fixed it on the CCRs - they fixed it on x86).

Routerboards are nice and cool and to be honest: I'm a big fan of this stuff (also Ubiquiti). But with this boxes you're not able to ran a stable enterprise class carrier network with >99,5% uptime. And that’s thei MAIN reason why "the old shit" is still online :).

Hopefully my words explained my hard "you know nothing" blabla ?

Best regards

Jürgen Jaritsch Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300 Telefax: +43-5-0556-500

E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Freitag, 02. Oktober 2015 21:33 Cc: NANOG <nanog@nanog.org> Betreff: Re: AW: /27 the new /24

Hrm.

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com

Midwest Internet Exchange http://www.midwest-ix.com

----- Original Message -----

From: "Jürgen Jaritsch" <jj@anexia.at> To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 2:25:10 PM Subject: AW: /27 the new /24

...

Stop using old shit.

Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about carrier grade networks.

Jürgen Jaritsch Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300 Telefax: +43-5-0556-500

E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Freitag, 02. Oktober 2015 20:38 An: NANOG <nanog@nanog.org> Betreff: Re: /27 the new /24

Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router.

I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient.

I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse.

Stop using old shit.

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com

Midwest Internet Exchange http://www.midwest-ix.com

----- Original Message -----

From: "William Herrin" <bill@herrin.us> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 1:09:16 PM Subject: Re: /27 the new /24

...

On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote: How many routers out there have this limitation? A $100 router I bought ten years ago could manage many full tables. If someone's network can't match that today, should I really have any pity for them?

Hi Mike,

The technology doesn't work the way you think it does. Or more precisely, it only works the way you think it does on small (cheap) end-user routers. Those routers do everything in software on a general-purpose CPU using radix tries for the forwarding table (FIB). They don't have to (and can't) handle both high data rates and large routing tables at the same time.

For a better understanding how the big iron works, check out https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see folks here talk about TCAM. This stands for Ternary Content Addressable Memory. It's a special circuit, different from DRAM and SRAM, used by most (but not all) big iron routers. The TCAM permits an O(1) route lookup instead of an O(log n) lookup. The architectural differences which balloon from there move the router cost from your $100 router into the hundreds of thousands of dollars.

Your BGP advertisement doesn't just have to be carried on your $100 router. It also has to be carried on the half-million-dollar routers. That makes it expensive.

Though out of date, this paper should help you better understand the systemic cost of a BGP route advertisement: http://bill.herrin.us/network/bgpcost.html

Regards, Bill Herrin

-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

Hi Mike,

but the boxes that have been there for 10 years have more than paid for themselves (unless they're a shitty business).

No question about that! But why should they throw them away if they can still print $$$ with these boxes? They have to change nothing till the global routing table reaches at least 768k ... so let's say this will happen in 12-18 months. They have enough time to prepare, migrate, etc ... and while all the side stories are happening they are still able to print $$$ with the "old shit".

What I was saying is that my little business with meager means (and revenues) can afford a box to do it.

This is definitely a question about sizing. Replacing a box with ~200 connected customers (only at this box!) is way more complex and this is nothing unrealistic.

If their business hasn't boomed, maybe it's time to replace that old 6500 with a 4500x or a QFX-5100 or an x670 or whatever.

4500x => no MPLS features QFX-5100 => very nice box (I'm a big fan) but complicate (and expensive!) licensing. Extreme x670 => nice box too - we also use this. But it's simply too small and the BGP configuration on these boxes is horrible. It's also not possible to provide Ethernet over MPLS with LACP BPDU forwarding ... too less features. Nice for aggregation and POP interconnect. All three models are new and shiny but they can't replace a 6500/7600. Too less port density and too less features (people are still using SDH. You need SDH in an 6500/7600? Simply install the required line card ...). If you really plan to replace a 6509 or even a 6513 you have to go with something like Juniper MX480/960 (I'm in love ... :D) or Cisco Nexus 7k/9k. One thing that will more and more happen: physical separation. There will be boxes with 10G/40G/100G only and boxes with 100M/1G only. Why? It's easier for vendors to remove old compatibility requirements (like electrical interfaces). So what we did in the past 3 years (replacing old boxes with new boxes with 1G/10G interfaces) was useless - we'll get our "old shit" back in place and bring them up and running. Of course: the "old shit" will be reduced to do aggregation layer or to something like "multihop instance" to transport the customers access port to the "real big and powerful router". Solving this with Layer2 extensions (like VLANs) is not practicable because you'll ran into other problems (like STP instances, etc). Probably it makes sense to solve it with Layer2VPN (Ethernet over MPLS, etc) to transport the physical interface to a virtual interface. Lots of things to think about :(.

Your decreased power bill alone will pay it off. If it has boomed, then ten years of revenues should get you whatever the bigger Ciscos are or an MX or whatever the bigger Extremes are.

Power is no argument. You get power starting at 0,10 Eur /kWh. Another 0,10 Eur / kWh for cooling and we talk about 0,20 Eur / kWh => Cisco 6513 (configured with 11 line cards + 2x SUP) with 2x 6kW PSU uses 3,8kW. 3,8kW * 24 hours * 30 days = 2.736 kWh per month. 2.736 * 0,20 Eur = 547,2 Eur per month for power consumption + cooling. If you have a good sales engineer you earn the revenue for this "side cost" with 1 customer :). Realistic calculation is: 10 customers are required to earn the money for the footprint.

Don't whine about my choices in gear I mentioned. I was just throwing things out there. Old big, new small if no money or old big new big if money.

Think the other way around: companies are earning Mio (or even Bil??) with the old equipment and everything is up and running. Only sometimes there is a small hick up because (of course!) also the "old shit" gets stuck from time to time and crashes. They did everything the right way (especially Level3 ...) from the commercial POV.

BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle full table imports in a significantly reduced time. Oh, and I'm not sure that you couldn't do at least three nines with MT\UBNT. Well, no experience with the EdgeRouters yet.

Never tried the earlier versions - my last tests happened in the end of 2014. I think we're talking a little bit about different sizes: you're talking about the CCRs and EdgeRouters (which are nice of course - no question about that!) and I'm talking about customer access devices (not CEP!) at carrier grade networks. Boxes I'm talking about have at least a few hundred ports. I think it's very important what UBNT and MT does: they bring fresh wind at the customer/semi-pro market and they show up that you (as a vendor) could get in touch with customers and optimize your equipment with customers feedback. best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Samstag, 03. Oktober 2015 02:52 Cc: NANOG <nanog@nanog.org> Betreff: Re: AW: AW: /27 the new /24 I don't expect carriers to be running UBNT\Mikrotik, but the boxes that have been there for 10 years have more than paid for themselves (unless they're a shitty business). It's time to rip and replace with whatever is appropriate for that site. No, I obviously don't think I'm going to change anyone's opinion on the matter (at least not anyone that matters in one of these networks). What I was saying is that my little business with meager means (and revenues) can afford a box to do it. They can too. I don't doubt their situation sucks... but either you fix it or you don't. Time and the rest of the Internet won't wait for them. If their business hasn't boomed, maybe it's time to replace that old 6500 with a 4500x or a QFX-5100 or an x670 or whatever. Your decreased power bill alone will pay it off. If it has boomed, then ten years of revenues should get you whatever the bigger Ciscos are or an MX or whatever the bigger Extremes are. Don't whine about my choices in gear I mentioned. I was just throwing things out there. Old big, new small if no money or old big new big if money. BTW: ROS 7 won't have multi-threaded BGP, but will be optimized to handle full table imports in a significantly reduced time. Oh, and I'm not sure that you couldn't do at least three nines with MT\UBNT. Well, no experience with the EdgeRouters yet. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Jürgen Jaritsch" <jj@anexia.at> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 6:11:47 PM Subject: AW: AW: /27 the new /24 Hi Mike, sorry, this was probably sent to quick ... let me please explain my POV of your statement: I want to concentrate my detailed answer only to the backbone situation which is often handled by the 6500/7600 - I guess all of us know that the 6500/7600 has a ton of additional features ... 6-7 years in the past carriers (and/or big ISPs) had only n*1G backbone capacities built with platforms that only had n*100M interfaces another 3-5 years before. Their only invest in these 3-5 years was to add the Gig line cards, install some software updates and add new fibre optics (GBICs). Chassis, cabling, management interfaces etc could be remain in the cabinet - they only had to replace ONE line card (let's say for a few thousand bucks) and with this invest they were able to scale up their capacities. Of course: at some point they also had to replace the SUPs, PSUs, FANs, etc. But the invest in the surrounding stuff is nothing compared with completely new machines. So what all these companies did was buying a machine with an basic configuration and since 10(!) years they are able to expand this machines with (more or less) small and cheap upgrades. In backbone situations the 6500/7600 are definitely at the end of the resources the platform can provide. Most of the carriers (and of course also the bigger ISPs) had a real chance to evaluate a new model/vendor to ran future networks (with possibly also a very good scale-up path and scaling- and upgrade-options). Most of the before mentioned are already in an migration process (let's take a look at Seabone ... they are migration from Cisco to a mix of Juniper and Huawei). Summary: there are strict limitations within the Cisco 6500/7600 platform and these limitations forces the big players to move this boxes out (or move them into other parts of their network). The limitation with 1Mio routes is not a secret and the admins of these boxes decide what they want to use (e.g. 768k routes for IPv4 unicast and 256k routes for MPLS+VRF, etc). If the global routing table reaches the 768k mark (I guess this will happen in the next 12-18months) most of the boxes will crash again (as it happened in Aug 2014). Regarding the words "I have a small router which handles multiple full tables ...": push and pull a few full tables at the same time and you'll see what's happening: the CCRs are SLOW. And why? Because the software is not as good as it could be: the BGP daemon uses only one core of a 36(?) core CPU. Same problem in the past with the EoIP daemon (not sure if they fixed it on the CCRs - they fixed it on x86). Routerboards are nice and cool and to be honest: I'm a big fan of this stuff (also Ubiquiti). But with this boxes you're not able to ran a stable enterprise class carrier network with >99,5% uptime. And that’s thei MAIN reason why "the old shit" is still online :). Hopefully my words explained my hard "you know nothing" blabla ? Best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Freitag, 02. Oktober 2015 21:33 Cc: NANOG <nanog@nanog.org> Betreff: Re: AW: /27 the new /24 Hrm. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Jürgen Jaritsch" <jj@anexia.at> To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 2:25:10 PM Subject: AW: /27 the new /24

Stop using old shit.

Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about carrier grade networks. Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Freitag, 02. Oktober 2015 20:38 An: NANOG <nanog@nanog.org> Betreff: Re: /27 the new /24 Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router. I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient. I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse. Stop using old shit. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "William Herrin" <bill@herrin.us> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 1:09:16 PM Subject: Re: /27 the new /24 On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote:

How many routers out there have this limitation? A $100 router I bought ten years ago could manage many full tables. If someone's network can't match that today, should I really have any pity for them?

Hi Mike, The technology doesn't work the way you think it does. Or more precisely, it only works the way you think it does on small (cheap) end-user routers. Those routers do everything in software on a general-purpose CPU using radix tries for the forwarding table (FIB). They don't have to (and can't) handle both high data rates and large routing tables at the same time. For a better understanding how the big iron works, check out https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see folks here talk about TCAM. This stands for Ternary Content Addressable Memory. It's a special circuit, different from DRAM and SRAM, used by most (but not all) big iron routers. The TCAM permits an O(1) route lookup instead of an O(log n) lookup. The architectural differences which balloon from there move the router cost from your $100 router into the hundreds of thousands of dollars. Your BGP advertisement doesn't just have to be carried on your $100 router. It also has to be carried on the half-million-dollar routers. That makes it expensive. Though out of date, this paper should help you better understand the systemic cost of a BGP route advertisement: http://bill.herrin.us/network/bgpcost.html Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

On Sat, Oct 03, 2015 at 08:10:36AM -0500, Mike Hammett wrote:

People keep thinking I want Level 3 to replace a loaded 6500 with a CCR and that's simply not what I'm saying at all. The point of rattling off the newer\smaller hardware was to say that if the site doesn't require 40G\100G, doesn't have the revenue to support an MX480, etc. you should put in a smaller\cheaper box. Cost is a non-issue at that point because the smaller gear that's all you need will have far less operational cost. Someone thought a particular POP was going to be a big hit... and wasn't.

In an SP environment, there is an escalating operating cost and network complexity to having small full-featured routers (ie. MX80, ASR9001, CER2k, etc) at every data center, POP or anywhere you need to terminate customers. The reality is that small routers (even if you were to use ghetto routers) have poor economics in port density. It's feasible for a startup ISP to spam MX80 or equivalent anytime they need more ports, but there comes a point where plopping a big chassis is the way to go. At my place, we started with MX80s to cheap out on router ports anytime we had to light a data center. That only got us far and we ended up having to migrate to ASR 9010s and start phasing out small routers. The increasing complexity of having dozens of small routers and managing LSP mesh to remainder of the network is ugly. Moreover, full-table BGP routers are also the places where you exercise edge policy with complex routing policies. Even with automation, managing dozens of those in a region that could have been served by only 2 routers is annoying. It's easier to haul IP customers to fewer, but more reliable large-chassis platforms and use packet-optical network to get to the customer premise. Between the above and the lack of control-plane redundancy on most small routers, there are operational complexities & economic realities to keep in mind; it's not strictly about whether a site requires 40G/100G.

On the flip side, if there are 200 ports of customers chances are you need the big interfaces that aren't on the old boxes. You have the bigger revenue. Heck, the new big boxes probably still use less power than the old big boxes anyway.

The idea has its merits, however in practice, it isn't feasible. People don't put in line cards into their router with expectation that they need to be replaced 2 years down the road because FIB TCAM ran out. Even if you have the revenue to justify new line cards, constant migration of customer interfaces means disruptive maintenance for that customer. We'd prefer IP network to be as reliable as dial-tone, if possible. The global routing table is approaching 600k today. Lot of line cards in installed base today only handle ~1.0/1.3 to ~1.8 million IPv4. When you start replacing those line cards (and mind you, a 24x10GE line card has a list price running into $300k range), the next logical level is 4 million IPv4. With all the deaggs with /24s, just how long of time are we going to have with /27 explosion before 4 million FIB runs out of space? I can see /25 being contemplated, but the cost of moving to /27 just isn't worth it at the moment.

What I learned from this thread: Once you mention MT\UBNT routers, people assume you're using a MT\UBNT hammer everywhere.

I'm not aware of any carrier-grade network that operates on these things. Best, james

I know of literally hundreds of ISPs using them in the US and I'm sure that number is in the thousands. After hearing complaints from larger networks of their larger gear... it's the same shit everyone else deals with. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Jérôme Nicolle" <jerome@ceriz.fr> To: nanog@nanog.org Sent: Friday, October 9, 2015 7:10:31 AM Subject: Re: AW: AW: AW: /27 the new /24 Hello James, Le 05/10/2015 06:45, James Jun a écrit :

I'm not aware of any carrier-grade network that operates on these things.

With the availability of a 80Gbps model and upcoming updates to the routing process (in RouterOS 7), chances are these boxes will drag much more attention in the next 2-3 years. Still, it looks like their products are widely used in developping countries where cheap hardware and flexible / low power requirements (you'd run a CCR1009 off a car battery for a solid 2 weeks - no regulator needed) makes them the only viable choice. I know of at least a dozen ISP running these as well, here in western Europe. It solved many space and power issues in dense carrier hotels, and is a cheap and efficient way out of a 6500/7600 (in sub 20Gbps scenarios) based network. I wouldn't sell transit or provide criticial services with a mikrotik-based network just yet, mostly for the lack of enough personnal confidence and experience with them, but havin endured nights of debugging with poor quality code in recent major player's routers, I doubt they're as misfits as you suggest. Best regards, -- Jérôme Nicolle

Sure MT has issues, but so does everyone. As someone that has used them for 10+ years, the past six months has seen a bit of a re-awakening over there. You can see this in the time to completion of many feature requests, bug fixes, new features, etc. I'm not sure they're going to do everything everyone is after, but they certainly have shown a huge increase in willingness to go the right direction. Of course it's easy for someone running big iron to scoff at the lack of feature X or feature Y. To that I say, what are the capabilities of your $200 router? Your $2k router? I haven't priced out new low-end gear from the big iron vendors, but I can't imagine at what price point you need to be at to have a multi-gig capable VPLS router. For Mikrotik you're in the $200 - $1k range, depending on what you mean by "multi-gig". One thing I miss as I start to use more non-Mikrotik hardware... Torch. I wish everything had Torch. Put Packet Sniffer in the list of things I'd like to see everywhere. I don't want port mirror as who's to say I have something to mirror to everywhere that can also capture? Put a few basic filters and drop the PCAP right on the damn box. Now obviously with something running BSD you could code up whatever you'd like or have an array of open-source packages to work with , but that wouldn't have the nice feature integration of a router OS. I have no problem running Mikrotik in the DFZ. Mine pull down full tables in 30 - 35 seconds, can handle somewhere in the 30 - 60 gb range when firewall rules are applied and so on. They'd cost under $1,500 new, but I got mine put together for a fraction of that. They're so cheap you can run two. Run two and now you have the advantage of being able to do maintenance without downtime. It's a little kludgey, but can get get the job done at a price point the others can't. Maybe with newer CCRs and ROS7 I could drop the need for the x86 boxes. We'll see. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "William Waites" <wwaites@tardis.ed.ac.uk> To: jj@anexia.at Cc: nanog@ics-il.net, nanog@nanog.org Sent: Saturday, October 3, 2015 3:23:49 AM Subject: Mikrotik in the DFZ (Was Re: AW: AW: /27 the new /24) On Fri, 2 Oct 2015 23:11:47 +0000, Jürgen Jaritsch <jj@anexia.at> said:

Regarding the words "I have a small router which handles multiple full tables ...": push and pull a few full tables at the same time and you'll see what's happening: the CCRs are SLOW. And why? Because the software is not as good as it could be: the BGP daemon uses only one core of a 36(?) core CPU.

To expand on this, the problem is worse than being single-threaded. I had one of these in the lab and fed it 2x full tables. Sure it wasn't the fastest at accepting them but then I noticed that even in steady state one of the CPUs was pegged. What was happening -- and this was confirmed by Mikrotik -- was that it was recalculating the *entire* FIB for each update. The general background noise of announce / withdraw messages means it is doing this all the time. Any churn and it would have a very hard time. There are other serious bugs such as not doing recursive next hop lookup for IPv6 (it does for IPv4). This makes them unuseable as BGP routers even for partial tables with most non-trivial iBGP topologies. All of which may be fixed one day in version 7 of their operating system, which will inevitably have many bugs as any software project .0 release will, so we'll have to wait for 7.x for it to be reasonably safe to use. That said, we use a lot of Mikrotik kit for our rural networks. They're weird and quirky but you can't beat them on price, port density and power consumption. With 16 ports and 36 cores surely they should be capable of pushing several Gbps of traffic with a few full tables. I wish it were possible today to run different software on their larger boxes. If some like-minded small providers wanted to get together with us to fund a FreeBSD port to the CCR routers that would be great. Please contact me off-list if you are interested in this, I'll coordinate. As it is we don't let them anywhere near the DFZ, that's done with PCs running FreeBSD and BIRD which can easily do the job but is still an order of magnitude more expensive (and an order of magnitude less expensive than what you need if you want 10s of Gbps). -w -- William Waites <wwaites@tardis.ed.ac.uk> | School of Informatics http://tardis.ed.ac.uk/~wwaites/ | University of Edinburgh https://hubs.net.uk/ | HUBS AS60241 The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

From: "Jürgen Jaritsch" <jj@anexia.at> To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org>

...

Stop using old shit. Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about carrier grade networks.

bingo!

Hi Mike, it's not a bureaucracy problem ... if you're a big player and you have to decide about a 2-3 Mio invest to upgrade only a few of your POPs (and let's say you have hundreds of POPs) it will be hard to find the "right" decision. Some questions these decision makers have to think about: #) What are the future plans for this POP? #) How upgradeable / expandable is the new equipment? #) Does our engineers know everything they need to run & debug & fix this new equipment? #) TOC incl support contract over the complete lifetime? #) Product life cycle? (Is it outdated in two years??) #) Will we keep spare parts onsite or nearby? #) How long needs the vendor to deliver everything I need? #) Is it compatible with all the already installed equipment? #) Migration plan to move existing customers to the new equipment? There are a ton of additional questions ... but I guess I pointed out some of the most important. Big players can't only calculate the price of the equipment - most of the time all the surrounding stuff (installation, new cabinets, migrations, training of engineers, etc) is producing 0,5x to 1x of the equipment costs. To get some easy numbers: take the discounted price (no one pays list prices ...) of an equipment and take this price x2 => that will be a realistic number to get the box onsite, up and running. It's not all the time something simple like a router with 20 patch cords :(. Best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Samstag, 03. Oktober 2015 04:53 Cc: NANOG <nanog@nanog.org> Betreff: Re: AW: /27 the new /24 A better truth may be that I have no idea about bureaucracies... which I'll happily admit to. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Jürgen Jaritsch" <jj@anexia.at> To: "Mike Hammett" <nanog@ics-il.net>, "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 2:25:10 PM Subject: AW: /27 the new /24

Stop using old shit.

Sorry, but the truth is: you have no idea about how earning revenue works and you obviously also have no idea about carrier grade networks. Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: JJaritsch@anexia-it.com Web: http://www.anexia-it.com Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Mike Hammett Gesendet: Freitag, 02. Oktober 2015 20:38 An: NANOG <nanog@nanog.org> Betreff: Re: /27 the new /24 Chances are the revenue passing scales to some degree as well. Small business with small bandwidth needs buys small and has small revenue. Big business with big bandwidth needs buys big and has big revenue to support big router. I can think of no reason why ten years goes by and you haven't had a need to throw out the old network for new. If your business hasn't scaled with the times, then you need to get rid of your Cat 6500 and get something more power, space, heat, etc. efficient. I saw someone replace a stack of Mikrotik CCRs with a pair of old Cisco routers. I don't know what they were at the moment, but they had GBICs, so they weren't exactly new. Each router had two 2500w power supplies. They'll be worse in every way (other than *possibly* BGP convergence). The old setup consumed at most 300 watts. The new setup requires $500/month in power... and is worse. Stop using old shit. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "William Herrin" <bill@herrin.us> To: "Mike Hammett" <nanog@ics-il.net> Cc: "NANOG" <nanog@nanog.org> Sent: Friday, October 2, 2015 1:09:16 PM Subject: Re: /27 the new /24 On Fri, Oct 2, 2015 at 11:50 AM, Mike Hammett <nanog@ics-il.net> wrote:

How many routers out there have this limitation? A $100 router I bought ten years ago could manage many full tables. If someone's network can't match that today, should I really have any pity for them?

Hi Mike, The technology doesn't work the way you think it does. Or more precisely, it only works the way you think it does on small (cheap) end-user routers. Those routers do everything in software on a general-purpose CPU using radix tries for the forwarding table (FIB). They don't have to (and can't) handle both high data rates and large routing tables at the same time. For a better understanding how the big iron works, check out https://www.pagiamtzis.com/cam/camintro/ . You'll occasionally see folks here talk about TCAM. This stands for Ternary Content Addressable Memory. It's a special circuit, different from DRAM and SRAM, used by most (but not all) big iron routers. The TCAM permits an O(1) route lookup instead of an O(log n) lookup. The architectural differences which balloon from there move the router cost from your $100 router into the hundreds of thousands of dollars. Your BGP advertisement doesn't just have to be carried on your $100 router. It also has to be carried on the half-million-dollar routers. That makes it expensive. Though out of date, this paper should help you better understand the systemic cost of a BGP route advertisement: http://bill.herrin.us/network/bgpcost.html Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

On Fri, Oct 2, 2015 at 11:55 AM, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:

Besides which more than one provider filters by a minimum prefix length per /8 - wasn't Swisscom or someone similar doing that? So multi homing with even a /24 is somewhat patchy in terms of effectiveness

Hi Suresh, That hasn't been true for something like a decade. Anybody who filters anything shorter than /24 without also taking a default route (or the equivalent) is not fully connected to the Internet. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

On Oct 2, 2015 12:47 PM, "Jason Baugher" <jason@thebaughers.com> wrote:

Are you suggesting that the Tier 1 and 2's that I connect to are not

filtering out anything shorter than /24? My expectation is that they are dropping shorter than /24, just like I am. You mean longer. A /16 is shorter than /24. A /28 is longer. More 1 bits in a row. -Bill

Bill, I see where I went wrong now that I went back and re-read your comment. I was conflating "longer" and "shorter". Thanks for your patience on this trying Friday. On Fri, Oct 2, 2015 at 12:06 PM, William Herrin <bill@herrin.us> wrote:

On Oct 2, 2015 12:47 PM, "Jason Baugher" <jason@thebaughers.com> wrote:

...

Are you suggesting that the Tier 1 and 2's that I connect to are not

filtering out anything shorter than /24? My expectation is that they are dropping shorter than /24, just like I am.

You mean longer. A /16 is shorter than /24. A /28 is longer. More 1 bits in a row.

-Bill

There are lots of transits that will take le 32 on their customers inbound but filter le 24 on egress announcements. On Fri, Oct 2, 2015 at 12:47 PM, Jason Baugher <jason@thebaughers.com> wrote:

Are you suggesting that the Tier 1 and 2's that I connect to are not filtering out anything shorter than /24? My expectation is that they are dropping shorter than /24, just like I am.

Correct me if I'm wrong, but every *NOG BGP best practices document I've read has advocated dropping all prefixes shorter than /24 at ingress and egress.

On Fri, Oct 2, 2015 at 11:34 AM, William Herrin <bill@herrin.us> wrote:

...

On Fri, Oct 2, 2015 at 11:55 AM, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:

...

Besides which more than one provider filters by a minimum prefix length per /8 - wasn't Swisscom or someone similar doing that? So multi homing with even a /24 is somewhat patchy in terms of effectiveness

Hi Suresh,

That hasn't been true for something like a decade. Anybody who filters anything shorter than /24 without also taking a default route (or the equivalent) is not fully connected to the Internet.

Regards, Bill Herrin

-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

-- [stillwaxin@gmail.com ~]$ cat .signature cat: .signature: No such file or directory [stillwaxin@gmail.com ~]$

There would be a default route sure - but the filter simply means that if your packets from say a src IP in a level 3 /24 (where the minimum alloc size was what, /20) wouldn't go through if you sent them though say a cogent interface --srs

On 02-Oct-2015, at 10:04 PM, William Herrin <bill@herrin.us> wrote:

On Fri, Oct 2, 2015 at 11:55 AM, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:

...

Besides which more than one provider filters by a minimum prefix length per /8 - wasn't Swisscom or someone similar doing that? So multi homing with even a /24 is somewhat patchy in terms of effectiveness

Hi Suresh,

That hasn't been true for something like a decade. Anybody who filters anything shorter than /24 without also taking a default route (or the equivalent) is not fully connected to the Internet.

Regards, Bill Herrin

-- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>

* tom@ninjabadger.net (Tom Hill) [Fri 02 Oct 2015, 18:34 CEST]:

Any RIR - or LIR - that considers allocating space in sizes smaller than a /24 (for the purpose of announcing to the DFZ) would do well to read this report from RIPE Labs:

https://labs.ripe.net/Members/emileaben/has-the-routability-of-longer-than-2...

tl;dr: it's still a bad idea to allocate smaller than a /24.

RIPE has long allocated up to /29. Not everybody needs addresses for the Internet; some just need a guarantee of global uniqueness. -- Niels.

You guys really don't need to argue on list. There are a lot of people subscribed here and I don't feel as if anything constructive is being accomplished. On Oct 2, 2015 4:07 PM, "Justin M. Streiner" <streiner@cluebyfour.org> wrote:

On Fri, 2 Oct 2015, Niels Bakker wrote:

* tom@ninjabadger.net (Tom Hill) [Fri 02 Oct 2015, 18:34 CEST]:

...

...

Any RIR - or LIR - that considers allocating space in sizes smaller than a /24 (for the purpose of announcing to the DFZ) would do well to read this report from RIPE Labs:

https://labs.ripe.net/Members/emileaben/has-the-routability-of-longer-than-2...

tl;dr: it's still a bad idea to allocate smaller than a /24.

RIPE has long allocated up to /29. Not everybody needs addresses for the Internet; some just need a guarantee of global uniqueness.

Right, but the OP's question seems to be pointed much more toward global reachability, not just global uniqueness.

jms