December 2024 - NANOG - lists.nanog.org

CGNAT growing pains
by Jon Lewis 17 Apr '25

17 Apr '25

We started rolling out CGNAT about 6 months ago. It was smooth sailing for the first few months, but we eventually did run into a number of issues. Our customer base is primarily FTTH with "dynamic" IP assignment via DHCP. Since connections are always-on, customer ONTs/routers get an IP assigned, and then when the lease is renewed, they request a new lease for the existing IP, and, in general, that request is granted. This gives customers the mistaken impression they have a static IP. So, my impression, from working with some customers who've needed to be moved from CGNAT back to public IP is that customers who are doing port-forwarding don't even bother with dynamic DNS. They just know they can connect to their IP as they've never seen it change. We do offer/sell static IP, but pre-CGNAT, it was strictly for business customers. i.e. A residential customer could only get static IP service by converting their account to a business account. That may change in the near future. One issue we didn't foresee has been IP Geo issues. i.e. We all knew that streaming services like Netflix use IP Geo to determine what content should be made available, but that's, AFAIK, limited by country or region. What we didn't anticipate is services like Hulu Live TV doing IP Geo down to the city level to determine which local channels are a subscriber's local channels. We're using Juniper MX gear and SPC3 cards for our CGNAT routers, each one having a single large external pool. Since we serve most of FL, one external pool can't IP Geo correctly for customers as far apart as Miami and Jacksonville hitting the same CGNAT router. We don't currently have an acceptable solution to this other than moving impacted customers off CGNAT. One of the great unknowns (at least for us) with CGNAT was what our PBA settings should be. i.e. How large each port-block should be, and how many port-blocks to allow per customer. We started with 256x4. It seemed to work. We eventually noticed that we were logging port-block exceeded errors. This is one aspect where Juniper's CGNAT support is lacking. There's a counter for these errors, and it's available via SNMP, but there's no way to attribute the errors to subscriber IPs. We're polling the mib and graphing it, so we know it's a continuing issue and can see when it's incrementing faster/slower, but Junos provides no means for determining if "PBEs" are all being caused by a single customer, a handful of customers, etc. We have a JTAC case open on this. As a quick & hopeful fix, we both increased the port-block size and block limit. That helped, but didn't stop the errors. It also cut our CGNAT ratio by more than half (64:1 -> 28:1), if we stay at this ratio, we'll need much larger external pools than originally anticipated. Tuning these settings is kind of painful as JTAC strongly recommends bouncing the CGNAT service anytime CGNAT related config changes are made. This means briefly breaking Internet access for all CGNAT'd customers. For the PBEs, JTAC's suggestions so far have been to shorten some of the timeouts in the config and to keep doing what we're doing, which is a cron job that essentially does a "show services nat source port-block", parses the output looking for subscriber IPs that have used up the ports in several of their port-blocks, then does a "show services sessions source-prefix ..." and logs all of this. This at least gives us snapshots of "who's a heavy user right now" and lets us look at how they were using all their ports. i.e. was it bittorent, are they compromised and scanning the internet for more systems to compromise, is it legit looking traffic - just lots of it, etc.? The latest CGNAT issue is a customer with a Palo Alto Networks firewall connected to our network and several of their employees are our FTTH customers. On their PANW firewall, they're doing IP Geo based filtering, limiting access to internal servers to "US IPs". Since we only CGNAT traffic to the external Internet, their on-net employees hit the firewall from their 100.64/10 IPs and get blocked. I suggested they whitelist 100.64/10, saying we block traffic from 100.64/10 from entering our network via peering and transit, so they can be assured anything from 100.64/10 came from inside our network / our customers. They say the firewall won't let them whitelist 100.64.0.0/10, giving an error that it's invalid IP space. I know we're not the first to implement CGNAT, so I'm curious if others have run into these sorts of issues, or others we haven't run into yet, and if so, how you solved them. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Blue Stream Fiber, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

10 16

Re: ECN
by Toke Høiland-Jørgensen 25 Mar '25

25 Mar '25

On 13 November 2019 17:20:18 CET, Matt Corallo <netadmin(a)as397444.net> wrote: >This sounds like a bug on Cloudflare’s end (cause trying to do anycast >TCP is... out of spec to say the least), not a bug in ECN/ECMP. Even without anycast, an ECMP shouldn't hash on the ECN bits. Doing so will split the flow over multiple paths; avoiding that is the whole point of doing the flow-based hashing in the first place. Anycast "only" turns a potential degradation of TCP performance into a hard failure... :) -Toke

6 8

DNSSEC & WIldcards
by Dennis Burgess 01 Feb '25

01 Feb '25

So have *.app.linktechs.net that I have been trying to get to work, we have DNSSEC on this, and its failing, but cannot for the life of me understand why. I think it may have something to do with proving it exists as a wildcard, but any DNSSEC experts want to take a stab at it ? Dennis Burgess

11 16

CAIDA AS Rank
by Mike Hammett 22 Jan '25

22 Jan '25

Is it broken for anyone else? https://asrank.caida.org/asns ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP

7 9

Implementing Decentralized RPKI with Blockchain Technology
by Brandon Z. 20 Jan '25

20 Jan '25

Hi there, Currently, due to political factors, some countries are not particularly proactive in deploying RPKI. Imagine if the RIR of a region were forced to revoke all IP resources of a particular country from RPKI, effectively isolating that country from the global internet. To address this, one approach is for autonomous networks within a region to establish two trusted RPKI CA servers: one from the major RIRs and another locally managed. The locally managed CA would take precedence, allowing autonomous networks to submit their IP resources to the RPKI server of their peers (and potentially backed by a national mandate to trust this CA). This setup could prevent a scenario where an entire country’s IP resources are revoked, leading to all IPs being marked as invalid. Another concept is to use blockchain technology. While cryptocurrencies use computational power to verify ownership, BGP could use peer count. If an IP resource is marked as valid by a majority of high-influence networks (with many peers), it could be trusted by the entire internet. Could this approach work? Perhaps there’s existing research on similar methods? *Brandon Z.* HUIZE LTD www.huize.asia <https://huize.asia/>| www.ixp.su | Twitter This e-mail and any attachments or any reproduction of this e-mail in whatever manner are confidential and for the use of the addressee(s) only. HUIZE LTD can’t take any liability and guarantee of the text of the email message and virus.

17 43

APRICOT 2025: CfP
by Mark Tinka 07 Jan '25

07 Jan '25

Hello all. The APRICOT 2025 Programme Committee is now seeking contributions for Presentations and Tutorials for the APRICOT 2025 Conference, which will be held in Petaling Jaya, Malaysia, 19th - 27th, February 2025. Details about the Call for Papers, and how you can submit a proposal for a conference, peering and tutorial talk, and the submission deadlines, can be found at the link below: https://2025.apricot.net/programme/callforpresentations We look forward to receiving your papers, and seeing you in Malaysia. Many thanks! Mark Tinka, Achie Atienza & Mark Duffell APRICOT 2025 Programme Committee Chairs

1 2

New home builders without wires
by Sean Donelan 06 Jan '25

06 Jan '25

As some may remember from earlier this year, my friend was buying a new "semi-custom" home. "Semi-custom" is a marketing term, meaning you get to choose (pay more) pre-determined builder options. It is not custom designed. The home builder was not installing any wired broadband utilities in the new neighborhood. No cable coax, no telephone DSL, no fiber optic. The only option was wireless, with a special deal with a specific 5G wireless cellular provider. Originally, the builder's sales agent (i.e. the people working in the model home selling houses) said new homes didn't need (and would not have) a wired "demarc" location and no ethernet or coax outlets. Not my house, but I was surprised when I heard that. I like wired connections when possible for any fixed devices, and WiFi only for mobile devices. I visited his new house over the Thanksgiving Holiday. The sales agent was partially wrong and partially correct. Never believe the sales agent spiel. The built house came with exactly FOUR wired ethernet outlets in the living room and each bedroom/office (x2 Cat6 jacks each outlet). But no wired DEMARC, no coax outlets, and no wired broadband utilities in the neighhood. The wired ethernet jacks were needed because the wireless 5G base station ended up in an upstairs bedroom window for signal strength reasons. The in-house wired ethernet was needed for a WiFi extender in the living room. I wouldn't be happy, but it seems to work for his family. The 5G deal was cheaper than what he was paying at his old house. According to the real estate realtor, not the builder's sales agent, broadband is now in the top three things home buyers want to know. Some states require the realtor MLS to disclose broadband access in the home listings. Broadband access disclosure not required in this state.

26 56

Best way to have redundancy announcing on separate routers
by Jean Franco 02 Jan '25

02 Jan '25

Hi Folks, I'm trying to achieve total redundancy on a multihomed environment: ISP 1 <=> Router 1 <= X => Router 2 <=> ISP 2 Where X is my Network. In the example below, he announces separate blocks to each ISP. https://www.networkstraining.com/cisco-bgp-configuration-tutorial/ I would like to do a failover model, where if one ISP goes down the other would take over. Please share your thoughts on this. Best regards,

16 29

Distributed Router Fabrics
by Mike Hammett 01 Jan '25

01 Jan '25

I've noticed that the whitebox hardware vendors are pushing distributed router fabrics, where you can keep buying pizza boxes and hooking them into a larger and larger fabric. Obviously, at some point, buying a big chassis makes more sense. Does it make sense building up to that point? What are your thoughts on that direction? ----- Mike Hammett [ http://www.ics-il.com/ | Intelligent Computing Solutions ] [ https://www.facebook.com/ICSIL ] [ https://plus.google.com/+IntelligentComputingSolutionsDeKalb ] [ https://www.linkedin.com/company/intelligent-computing-solutions ] [ https://twitter.com/ICSIL ] [ http://www.midwest-ix.com/ | Midwest Internet Exchange ] [ https://www.facebook.com/mdwestix ] [ https://www.linkedin.com/company/midwest-internet-exchange ] [ https://twitter.com/mdwestix ] [ http://www.thebrotherswisp.com/ | The Brothers WISP ] [ https://www.facebook.com/thebrotherswisp ] [ https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg ]

13 36

Spectrum Carrier Team
by Dennis Burgess - LTI Support 31 Dec '24

31 Dec '24

Looking for someone at Spectrum to help with a ticket. ECT-83933060 no one is picking up the ticket ☹ [LTI-Full_175px] Dennis Burgess MikroTik : Trainer, Network Associate, Routing Engineer, Wireless Engineer, Traffic Control Engineer, Inter-Networking Engineer, Security Engineer, Enterprise Wireless Engineer Hurricane Electric: IPv6 Sage Level Cambium: ePMP Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Create your own Tickets via https://hd.linktechs.net<https://hd.linktechs.net/> Create Wireless Coverage’s with www.towercoverage.com Need MikroTik Cloud Management: https://cloud.linktechs.net<https://cloud.linktechs.net/> Admiral MikroTik Services: http://admiral.linktechs.net

3 2