Afternoon, I run a small ISP in Tennessee. COVID has forced a lot of people to work from home. I am starting to run low on IP's and need to consider CGNAT. I do have IPV6 space, but we all know that until we force everyone to move to IPV6, we need to keep IPV4 up and running. I could buy more space, but I am really wondering if that is the best option. It is expensive. I know CGNAT devices are expensive as well, but it looks like I could stretch it out a bit. My thinking is to convert about 50% of my subscribers to CGNAT. I am interested in vendors or devices you have used in the past. I already know about the pitfalls many of my subscribers will have with CGNAT such as VPN's, Gamers, etc. What are your thoughts on CGNAT vendors? A10Networks F5Networks Others?
Just go with Linux and iptables. It is by far the cheapest option and it just works. tir. 28. apr. 2020 21.13 skrev John Alcock <john@alcock.org>:
Afternoon,
I run a small ISP in Tennessee. COVID has forced a lot of people to work from home. I am starting to run low on IP's and need to consider CGNAT.
I do have IPV6 space, but we all know that until we force everyone to move to IPV6, we need to keep IPV4 up and running.
I could buy more space, but I am really wondering if that is the best option. It is expensive. I know CGNAT devices are expensive as well, but it looks like I could stretch it out a bit.
My thinking is to convert about 50% of my subscribers to CGNAT.
I am interested in vendors or devices you have used in the past. I already know about the pitfalls many of my subscribers will have with CGNAT such as VPN's, Gamers, etc.
What are your thoughts on CGNAT vendors?
A10Networks F5Networks Others?
Hi John, I run a small/medium ISP in Texas. A few years ago, needing to do the same thing you are speaking of, I lab evaluated the Cisco ASR9k VSM-500 and Juniper MX104 MS-MIC-16G… in the end I went with Juniper. No regrets, been good and holding strong. I’ve scaled it way beyond what I originally envisioned. (but bought more as well) I slow started my CGNat deployment, like with most things, baby-steps when doing something as extreme as taking away the public ip address from my isp residential customers… so yeah, slow-start… DSL was my first target. One DSLAM at a time, waiting for issues to arise and dealing with them along the way, the best I could. …until we had 6,000 dsl customers behind a pair of Juniper MX104’s with MS-MIC-16G cards, running fine. (all done via mpls l3vpn for virtual L3 routing into and out of the nat boundary… so one vrf for inside, and one vrf for outside)…peak load as I recall was about 3 gbps on each MX104, so 6 gbps total. Next, about a year or so later, we went after Cable Modem CMTS communities. But, added MS-MPC-128G modules to a pair of our mpls 100 gig ring MX960 nodes. This was another 5,000 subs or so. (this was about 2 or 3 years ago). Learned a lot during that one. A lot about ecmp, inet.3 mp-ibgp route choices, (set protocols ldp track-igp-metric… is your friend), app, eim, eif, ams/mams interfaces and load-balancing on the source-ip…. Let that ride for a year or so…then… …went after our FTTH communities. Probably about 30 or 40 thousand ip’s were recoup’d here. FTTH was nat’d behind (4) additional MS-MPC-128G modules in (4) other 100 gig mpls ring mx960 nodes. There have been recent concerns about uPNP not working behind the cgnat’s. All in all, we are getting lots of use out of our Juniper CGNat solution. All told, it’s about 50,000 customers behind the (2) MX104’s and (6) MX960’s getting nat’d. -Aaron From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of John Alcock Sent: Tuesday, April 28, 2020 2:12 PM To: nanog@nanog.org Subject: CGNAT Solutions Afternoon, I run a small ISP in Tennessee. COVID has forced a lot of people to work from home. I am starting to run low on IP's and need to consider CGNAT. I do have IPV6 space, but we all know that until we force everyone to move to IPV6, we need to keep IPV4 up and running. I could buy more space, but I am really wondering if that is the best option. It is expensive. I know CGNAT devices are expensive as well, but it looks like I could stretch it out a bit. My thinking is to convert about 50% of my subscribers to CGNAT. I am interested in vendors or devices you have used in the past. I already know about the pitfalls many of my subscribers will have with CGNAT such as VPN's, Gamers, etc. What are your thoughts on CGNAT vendors? A10Networks F5Networks Others?
On Tue, Apr 28, 2020 at 12:12 PM John Alcock <john@alcock.org> wrote:
I run a small ISP in Tennessee. I am starting to run low on IP's and need to consider CGNAT.
Hi John, How small is small? Up to a certain size regular NAT with enough logging to trace back abusers will tend to work fine. if we're talking single-digit gbps, it may not be worth the effort to consider the wonderful world of CGNAT. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
On 4/28/20 4:53 PM, William Herrin wrote:
How small is small? Up to a certain size regular NAT with enough logging to trace back abusers will tend to work fine. if we're talking single-digit gbps, it may not be worth the effort to consider the wonderful world of CGNAT.
Depending on how many IPs you need to reclaim and what your target IP:subscriber ratio is, you may be able to eliminate the need for a lot of logging by assigning a range of TCP/UDP ports to a single inside IP so that the TCP/UDP port number implies a specific subscriber. You can't get rid of all the state tracking without also having the CPE know which ports to use (in which case you might as well use LW4o6 or MAP), but at least you can get it down to where you really only need to log (or block and dole out public IPs as needed) port-less protocols. -- Brandon Martin
On 4/29/20 2:35 AM, Masataka Ohta wrote:
If you mean getting rid of logging, not necessarily. It is enough if CPEs are statically allocated ranges of external port numbers.
Yes, you can get rid of the logging by statically allocating ranges of port numbers to a particular customer. What I was referring to, though, was the programmatic state tracking of the {external IP, external port}-{internal IP, internal port} mappings. You can't eliminate that unless the CPE also knows what internal port range it's mapped to so that it restricts what range it uses. If you can do that, you can get rid of the programmatic state tracking entirely and just use static translations for TCP and UDP which, while nice, is impractical. You're about 95% of the way to LW4o6 or MAP at that point. -- Brandon Martin
Brandon Martin wrote:
If you mean getting rid of logging, not necessarily. It is enough if CPEs are statically allocated ranges of external port numbers.
Yes, you can get rid of the logging by statically allocating ranges of port numbers to a particular customer.
And, that was the original concern.
What I was referring to, though, was the programmatic state tracking of the {external IP, external port}-{internal IP, internal port} mappings.
OK.
You can't eliminate that unless the CPE also knows what internal port range it's mapped to so that it restricts what range it uses. If you can do that, you can get rid of the programmatic state tracking entirely and just use static translations for TCP and UDP which, while nice, is impractical. You're about 95% of the way to LW4o6 or MAP at that point.
Interesting. Then, if you can LW4o6 or MAP, you are about 95% of the way to E2ENAT with complete end to end transparency using IPv4 only, which means we don't need IPv6 with 4to6 NAT lacking the transparency. https://tools.ietf.org/html/draft-ohta-e2e-nat-00 Masataka Ohta
On Wed, Apr 29, 2020 at 1:06 AM Masataka Ohta < mohta@necom830.hpcl.titech.ac.jp> wrote:
Brandon Martin wrote:
If you mean getting rid of logging, not necessarily. It is enough if CPEs are statically allocated ranges of external port numbers.
Yes, you can get rid of the logging by statically allocating ranges of port numbers to a particular customer.
And, that was the original concern.
What I was referring to, though, was the programmatic state tracking of the {external IP, external port}-{internal IP, internal port} mappings.
OK.
You can't eliminate that unless the CPE also knows what internal port range it's mapped to so that it restricts what range it uses. If you can do that, you can get rid of the programmatic state tracking entirely and just use static translations for TCP and UDP which, while nice, is impractical. You're about 95% of the way to LW4o6 or MAP at that point.
Interesting. Then, if you can LW4o6 or MAP, you are about 95% of the way to E2ENAT with complete end to end transparency using IPv4 only, which means we don't need IPv6 with 4to6 NAT lacking the transparency.
https://tools.ietf.org/html/draft-ohta-e2e-nat-00
Masataka Ohta
Since we are talking numbers ans hard facts 42% of usa accesses google on ipv6 https://www.google.com/intl/en/ipv6/statistics.html
On Wed, Apr 29, 2020 at 7:19 AM Ca By <cb.list6@gmail.com> wrote:
Since we are talking numbers ans hard facts
42% of usa accesses google on ipv6
Be careful with those stats; they might not be telling you what you think they are. For example, phone clients are characteristically different than classic ISP clients. A substantial portion, perhaps majority of Google's use comes from phone clients but the numbers aren't broken out in those roll-up stats. Among others, T-Mobile has demonstrated that a v6-only infrastructure (with some v4 and NAT at the border) is credible and achievable in a cell-phone environment. A classic small ISP fills a different niche. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
Ca By wrote:
You can't eliminate that unless the CPE also knows what internal port range it's mapped to so that it restricts what range it uses. If you can do that, you can get rid of the programmatic state tracking entirely and just use static translations for TCP and UDP which, while nice, is impractical. You're about 95% of the way to LW4o6 or MAP at that point.
Interesting. Then, if you can LW4o6 or MAP, you are about 95% of the way to E2ENAT with complete end to end transparency using IPv4 only, which means we don't need IPv6 with 4to6 NAT lacking the transparency.
https://tools.ietf.org/html/draft-ohta-e2e-nat-00
Masataka Ohta
Since we are talking numbers ans hard facts
I'm rather interested in not numbers but facts on the E2E transparency, because, without the transparency, legacy NAT44 should be enough. But, as you insist on numbers:
42% of usa accesses google on ipv6
The proper number to be considered should be percentage of IPv6 hosts which can not communicate with IPv4 only hosts. Isn't it 0%? Masataka Ohta
On Wed, Apr 29, 2020 at 7:46 PM Masataka Ohta < mohta@necom830.hpcl.titech.ac.jp> wrote:
Ca By wrote:
You can't eliminate that unless the CPE also knows what internal port range it's mapped to so that it restricts what range it uses. If you can do that, you can get rid of the programmatic state tracking entirely and just use static translations for TCP and UDP which, while nice, is impractical. You're about 95% of the way to LW4o6 or MAP at that point.
Interesting. Then, if you can LW4o6 or MAP, you are about 95% of the way to E2ENAT with complete end to end transparency using IPv4 only, which means we don't need IPv6 with 4to6 NAT lacking the transparency.
https://tools.ietf.org/html/draft-ohta-e2e-nat-00
Masataka Ohta
Since we are talking numbers ans hard facts
I'm rather interested in not numbers but facts on the E2E transparency, because, without the transparency, legacy NAT44 should be enough.
But, as you insist on numbers:
42% of usa accesses google on ipv6
The proper number to be considered should be percentage of IPv6 hosts which can not communicate with IPv4 only hosts.
Isn't it 0%?
For those of us running networks, especially growing networks, uniquely numbering hosts is our goal and ipv6 fits that task. For many networks, rfc1918 space is not sufficiently large to number end-points. Around the world, there are many networks that fit this. For those same network, nat44 scale is also a painful and costly effort. To that end, ipv6 / 464xlat provides the one-two punch of uniquely numbering nodes and by-passing NAT44 or NAT64 for the majority of traffic we see (google, fb, netflix ...) Being able to offer a product that disallows access to ipv4 is a non-goal So far, i just talked about why eyeball networks deploy ipv6 — which is basic and sensible engineering and economics. A similar set of forces are at work on the content / cloud / iot side.
Masataka Ohta
Ca By wrote:
The proper number to be considered should be percentage of IPv6 hosts which can not communicate with IPv4 only hosts.
Isn't it 0%?
I think you agree with me, here.
For those of us running networks, especially growing networks, uniquely numbering hosts is our goal and ipv6 fits that task.
Then, you should be running some isolated network. In this thread, we, except you, are discussing how to uniquely identify customers, not hosts, without (much) logging.
For many networks, rfc1918 space is not sufficiently large to number end-points. Around the world, there are many networks that fit this.
The global address space of IPv4 with NAT is combination of IPv4 address and part of port number spaces, which should be enough to identify customers and, maybe, hosts. and is much larger than private space of rfc1918.
So far, i just talked about why eyeball networks deploy ipv6 — which is basic and sensible engineering and economics. A similar set of forces are at work on the content / cloud / iot side.
Perfect argument for OSI. Masataka Ohta
On 4/28/20 11:01 PM, Brandon Martin wrote:
Depending on how many IPs you need to reclaim and what your target IP:subscriber ratio is, you may be able to eliminate the need for a lot of logging by assigning a range of TCP/UDP ports to a single inside IP so that the TCP/UDP port number implies a specific subscriber.
You can't get rid of all the state tracking without also having the CPE know which ports to use (in which case you might as well use LW4o6 or MAP), but at least you can get it down to where you really only need to log (or block and dole out public IPs as needed) port-less protocols.
I'm wondering if there are any real world examples of this, namely in the realm of subscriber to IP and range of ports required, etc. ie: Is is a range of 1000 ports enough for one residential subscriber? How about SMB where no global IP is required. One would think a 1000 ports would be enough, but if you have a dozen devices at home all browsing and doing various things, and with IOT, etc, maybe not? -- inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP: https://pgp.inoc.net/rblayzor/
hey,
I'm wondering if there are any real world examples of this, namely in the realm of subscriber to IP and range of ports required, etc. ie: Is is a range of 1000 ports enough for one residential subscriber? How about SMB where no global IP is required.
One would think a 1000 ports would be enough, but if you have a dozen devices at home all browsing and doing various things, and with IOT, etc, maybe not?
1000 ports doesn't mean you can have at max 1000 layer-4 sessions at once. It means you can have 1000 sessions to single destination IP+port. You can reuse same source port numbers for different destination IP or even destination port. We are seeing very good results with 256 ports per subscriber in the mobile scenario where consumer is mobile handset. So not directly translatable to broadband setup but still good datapoint. If you must go CGNAT today it's only reasonable to use PBA (so you log only block allocations) or pure deterministic where you have strict mapping between inside IP and outside IP+portrange so you don't need any logs at all. -- tarko
How big is your ip pool for CGNAT? On Wed, Apr 29, 2020 at 10:17 AM Robert Blayzor <rblayzor.bulk@inoc.net> wrote:
On 4/28/20 11:01 PM, Brandon Martin wrote:
Depending on how many IPs you need to reclaim and what your target IP:subscriber ratio is, you may be able to eliminate the need for a lot of logging by assigning a range of TCP/UDP ports to a single inside IP so that the TCP/UDP port number implies a specific subscriber.
You can't get rid of all the state tracking without also having the CPE know which ports to use (in which case you might as well use LW4o6 or MAP), but at least you can get it down to where you really only need to log (or block and dole out public IPs as needed) port-less protocols.
I'm wondering if there are any real world examples of this, namely in the realm of subscriber to IP and range of ports required, etc. ie: Is is a range of 1000 ports enough for one residential subscriber? How about SMB where no global IP is required.
One would think a 1000 ports would be enough, but if you have a dozen devices at home all browsing and doing various things, and with IOT, etc, maybe not?
-- inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP: https://pgp.inoc.net/rblayzor/
-- Sent from Gmail Mobile
On Wed, 29 Apr 2020, Robert Blayzor wrote:
One would think a 1000 ports would be enough, but if you have a dozen devices at home all browsing and doing various things, and with IOT, etc, maybe not?
https://www.juniper.net/documentation/en_US/junos/topics/concept/nat-best-pr... There are some numbers in there for instance talking about 1024 ports per subscriber as a good number. In presentations I have seen over time, people typically talk about 512-4096 as being a good number for the bulk port allocation size. -- Mikael Abrahamsson email: swmike@swm.pp.se
On 4/29/20 10:29 AM, Mikael Abrahamsson wrote:
There are some numbers in there for instance talking about 1024 ports per subscriber as a good number. In presentations I have seen over time, people typically talk about 512-4096 as being a good number for the bulk port allocation size.
So as a happy medium of about 2048 ports per subscriber, that's roughly a 32:1 NAT/IP over-subscription ? -- inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP: https://pgp.inoc.net/rblayzor/
In testing, I observed opening a website, for instance cnn.com can cause >200 ports/sessions to fire off. Although, many are short-lived sessions, but, ports requests nonetheless. Overall, I use about 1,500 public ip's for 50,000 private ip customers I allow 3,000 ports per customer ... 30 blocks of 100 each We started our port blocks at a nice round number, so that each pba dynamic assignment results in nice 100-199, next 200-299 .... good for parsing, grep'ing logs for doing subpoena info look-ups, etc. I see most customers hover well below 1,000 ports/sessions active, and what appear to be misbehaving hosts (malware, infected, bots, etc, unsure) hit up at the 3,000 max and trigger a ports exceeded error message. I see the 3k port limit as putting a cap on free-running suspicious hosts. We can then investigate and contact customer of the concern. -Aaron -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Robert Blayzor Sent: Wednesday, April 29, 2020 9:14 AM To: nanog@nanog.org Subject: Re: CGNAT Solutions On 4/28/20 11:01 PM, Brandon Martin wrote:
Depending on how many IPs you need to reclaim and what your target IP:subscriber ratio is, you may be able to eliminate the need for a lot of logging by assigning a range of TCP/UDP ports to a single inside IP so that the TCP/UDP port number implies a specific subscriber.
You can't get rid of all the state tracking without also having the CPE know which ports to use (in which case you might as well use LW4o6 or MAP), but at least you can get it down to where you really only need to log (or block and dole out public IPs as needed) port-less protocols.
I'm wondering if there are any real world examples of this, namely in the realm of subscriber to IP and range of ports required, etc. ie: Is is a range of 1000 ports enough for one residential subscriber? How about SMB where no global IP is required. One would think a 1000 ports would be enough, but if you have a dozen devices at home all browsing and doing various things, and with IOT, etc, maybe not? -- inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP: https://pgp.inoc.net/rblayzor/
I will say it is much better to consider 464XLAT with NAT64, if the CPEs allow it. https://datatracker.ietf.org/doc/rfc8683/ I’m right now doing a deployment for 25.000.000 customers of an ISP (GPON, DLS and cellular mix), all the testing has been done, and all doing fine. I’ve done it already for smaller ISPs, but the size of this project is more interesting to better demonstrate that it just works. I plan to do a presentation when the information can be made public … bit delay because the Covid-19 confinement. Regards, Jordi @jordipalet El 28/4/20 21:15, "NANOG en nombre de John Alcock" <nanog-bounces@nanog.org en nombre de john@alcock.org> escribió: Afternoon, I run a small ISP in Tennessee. COVID has forced a lot of people to work from home. I am starting to run low on IP's and need to consider CGNAT. I do have IPV6 space, but we all know that until we force everyone to move to IPV6, we need to keep IPV4 up and running. I could buy more space, but I am really wondering if that is the best option. It is expensive. I know CGNAT devices are expensive as well, but it looks like I could stretch it out a bit. My thinking is to convert about 50% of my subscribers to CGNAT. I am interested in vendors or devices you have used in the past. I already know about the pitfalls many of my subscribers will have with CGNAT such as VPN's, Gamers, etc. What are your thoughts on CGNAT vendors? A10Networks F5Networks Others? ********************************************** IPv4 is over Are you ready for the new Internet ? http://www.theipv6company.com The IPv6 Company This electronic message contains information which may be privileged or confidential. The information is intended to be for the exclusive use of the individual(s) named above and further non-explicilty authorized disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited and will be considered a criminal offense. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited, will be considered a criminal offense, so you must reply to the original sender to inform about this communication and delete it.
Take a look at DANOS for CG-NAT as a free solution or Netgate's TNSR has a CG-NAT feature https://www.tnsr.com/features On Tue, Apr 28, 2020 at 2:57 PM JORDI PALET MARTINEZ via NANOG < nanog@nanog.org> wrote:
I will say it is much better to consider 464XLAT with NAT64, if the CPEs allow it.
https://datatracker.ietf.org/doc/rfc8683/
I’m right now doing a deployment for 25.000.000 customers of an ISP (GPON, DLS and cellular mix), all the testing has been done, and all doing fine.
I’ve done it already for smaller ISPs, but the size of this project is more interesting to better demonstrate that it just works.
I plan to do a presentation when the information can be made public … bit delay because the Covid-19 confinement.
Regards,
Jordi
@jordipalet
El 28/4/20 21:15, "NANOG en nombre de John Alcock" < nanog-bounces@nanog.org en nombre de john@alcock.org> escribió:
Afternoon,
I run a small ISP in Tennessee. COVID has forced a lot of people to work from home. I am starting to run low on IP's and need to consider CGNAT.
I do have IPV6 space, but we all know that until we force everyone to move to IPV6, we need to keep IPV4 up and running.
I could buy more space, but I am really wondering if that is the best option. It is expensive. I know CGNAT devices are expensive as well, but it looks like I could stretch it out a bit.
My thinking is to convert about 50% of my subscribers to CGNAT.
I am interested in vendors or devices you have used in the past. I already know about the pitfalls many of my subscribers will have with CGNAT such as VPN's, Gamers, etc.
What are your thoughts on CGNAT vendors?
A10Networks
F5Networks
Others?
********************************************** IPv4 is over Are you ready for the new Internet ? http://www.theipv6company.com The IPv6 Company
This electronic message contains information which may be privileged or confidential. The information is intended to be for the exclusive use of the individual(s) named above and further non-explicilty authorized disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited and will be considered a criminal offense. If you are not the intended recipient be aware that any disclosure, copying, distribution or use of the contents of this information, even if partially, including attached files, is strictly prohibited, will be considered a criminal offense, so you must reply to the original sender to inform about this communication and delete it.
I haven't used them, but 6-WIND is pretty proud of their CGNAT performance. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "John Alcock" <john@alcock.org> To: nanog@nanog.org Sent: Tuesday, April 28, 2020 2:12:29 PM Subject: CGNAT Solutions Afternoon, I run a small ISP in Tennessee. COVID has forced a lot of people to work from home. I am starting to run low on IP's and need to consider CGNAT. I do have IPV6 space, but we all know that until we force everyone to move to IPV6, we need to keep IPV4 up and running. I could buy more space, but I am really wondering if that is the best option. It is expensive. I know CGNAT devices are expensive as well, but it looks like I could stretch it out a bit. My thinking is to convert about 50% of my subscribers to CGNAT. I am interested in vendors or devices you have used in the past. I already know about the pitfalls many of my subscribers will have with CGNAT such as VPN's, Gamers, etc. What are your thoughts on CGNAT vendors? A10Networks F5Networks Others?
Thank you everyone for the suggestions. To clarify small ISP. 12K subscribers 35 Gigs traffic at peak. Growing about 500 megs per month traffic. John On Tue, Apr 28, 2020 at 3:12 PM John Alcock <john@alcock.org> wrote:
Afternoon,
I run a small ISP in Tennessee. COVID has forced a lot of people to work from home. I am starting to run low on IP's and need to consider CGNAT.
I do have IPV6 space, but we all know that until we force everyone to move to IPV6, we need to keep IPV4 up and running.
I could buy more space, but I am really wondering if that is the best option. It is expensive. I know CGNAT devices are expensive as well, but it looks like I could stretch it out a bit.
My thinking is to convert about 50% of my subscribers to CGNAT.
I am interested in vendors or devices you have used in the past. I already know about the pitfalls many of my subscribers will have with CGNAT such as VPN's, Gamers, etc.
What are your thoughts on CGNAT vendors?
A10Networks F5Networks Others?
participants (14)
-
Aaron Gould -
Baldur Norddahl -
Brandon Martin -
Ca By -
james jones -
Jared Geiger -
John Alcock -
JORDI PALET MARTINEZ -
Masataka Ohta -
Mikael Abrahamsson -
Mike Hammett -
Robert Blayzor -
Tarko Tikan -
William Herrin