October 2024 - NANOG - lists.nanog.org

CGNAT growing pains
by Jon Lewis 17 Apr '25

17 Apr '25

We started rolling out CGNAT about 6 months ago. It was smooth sailing for the first few months, but we eventually did run into a number of issues. Our customer base is primarily FTTH with "dynamic" IP assignment via DHCP. Since connections are always-on, customer ONTs/routers get an IP assigned, and then when the lease is renewed, they request a new lease for the existing IP, and, in general, that request is granted. This gives customers the mistaken impression they have a static IP. So, my impression, from working with some customers who've needed to be moved from CGNAT back to public IP is that customers who are doing port-forwarding don't even bother with dynamic DNS. They just know they can connect to their IP as they've never seen it change. We do offer/sell static IP, but pre-CGNAT, it was strictly for business customers. i.e. A residential customer could only get static IP service by converting their account to a business account. That may change in the near future. One issue we didn't foresee has been IP Geo issues. i.e. We all knew that streaming services like Netflix use IP Geo to determine what content should be made available, but that's, AFAIK, limited by country or region. What we didn't anticipate is services like Hulu Live TV doing IP Geo down to the city level to determine which local channels are a subscriber's local channels. We're using Juniper MX gear and SPC3 cards for our CGNAT routers, each one having a single large external pool. Since we serve most of FL, one external pool can't IP Geo correctly for customers as far apart as Miami and Jacksonville hitting the same CGNAT router. We don't currently have an acceptable solution to this other than moving impacted customers off CGNAT. One of the great unknowns (at least for us) with CGNAT was what our PBA settings should be. i.e. How large each port-block should be, and how many port-blocks to allow per customer. We started with 256x4. It seemed to work. We eventually noticed that we were logging port-block exceeded errors. This is one aspect where Juniper's CGNAT support is lacking. There's a counter for these errors, and it's available via SNMP, but there's no way to attribute the errors to subscriber IPs. We're polling the mib and graphing it, so we know it's a continuing issue and can see when it's incrementing faster/slower, but Junos provides no means for determining if "PBEs" are all being caused by a single customer, a handful of customers, etc. We have a JTAC case open on this. As a quick & hopeful fix, we both increased the port-block size and block limit. That helped, but didn't stop the errors. It also cut our CGNAT ratio by more than half (64:1 -> 28:1), if we stay at this ratio, we'll need much larger external pools than originally anticipated. Tuning these settings is kind of painful as JTAC strongly recommends bouncing the CGNAT service anytime CGNAT related config changes are made. This means briefly breaking Internet access for all CGNAT'd customers. For the PBEs, JTAC's suggestions so far have been to shorten some of the timeouts in the config and to keep doing what we're doing, which is a cron job that essentially does a "show services nat source port-block", parses the output looking for subscriber IPs that have used up the ports in several of their port-blocks, then does a "show services sessions source-prefix ..." and logs all of this. This at least gives us snapshots of "who's a heavy user right now" and lets us look at how they were using all their ports. i.e. was it bittorent, are they compromised and scanning the internet for more systems to compromise, is it legit looking traffic - just lots of it, etc.? The latest CGNAT issue is a customer with a Palo Alto Networks firewall connected to our network and several of their employees are our FTTH customers. On their PANW firewall, they're doing IP Geo based filtering, limiting access to internal servers to "US IPs". Since we only CGNAT traffic to the external Internet, their on-net employees hit the firewall from their 100.64/10 IPs and get blocked. I suggested they whitelist 100.64/10, saying we block traffic from 100.64/10 from entering our network via peering and transit, so they can be assured anything from 100.64/10 came from inside our network / our customers. They say the firewall won't let them whitelist 100.64.0.0/10, giving an error that it's invalid IP space. I know we're not the first to implement CGNAT, so I'm curious if others have run into these sorts of issues, or others we haven't run into yet, and if so, how you solved them. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Blue Stream Fiber, Sr. Neteng | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

10 16

Re: ECN
by Toke Høiland-Jørgensen 25 Mar '25

25 Mar '25

On 13 November 2019 17:20:18 CET, Matt Corallo <netadmin(a)as397444.net> wrote: >This sounds like a bug on Cloudflare’s end (cause trying to do anycast >TCP is... out of spec to say the least), not a bug in ECN/ECMP. Even without anycast, an ECMP shouldn't hash on the ECN bits. Doing so will split the flow over multiple paths; avoiding that is the whole point of doing the flow-based hashing in the first place. Anycast "only" turns a potential degradation of TCP performance into a hard failure... :) -Toke

6 8

DNSSEC & WIldcards
by Dennis Burgess 01 Feb '25

01 Feb '25

So have *.app.linktechs.net that I have been trying to get to work, we have DNSSEC on this, and its failing, but cannot for the life of me understand why. I think it may have something to do with proving it exists as a wildcard, but any DNSSEC experts want to take a stab at it ? Dennis Burgess

11 16

APRICOT 2025: CfP
by Mark Tinka 07 Jan '25

07 Jan '25

Hello all. The APRICOT 2025 Programme Committee is now seeking contributions for Presentations and Tutorials for the APRICOT 2025 Conference, which will be held in Petaling Jaya, Malaysia, 19th - 27th, February 2025. Details about the Call for Papers, and how you can submit a proposal for a conference, peering and tutorial talk, and the submission deadlines, can be found at the link below: https://2025.apricot.net/programme/callforpresentations We look forward to receiving your papers, and seeing you in Malaysia. Many thanks! Mark Tinka, Achie Atienza & Mark Duffell APRICOT 2025 Programme Committee Chairs

1 2

Incorrect Reverse DNS in Verizon Fios NYC core router traceroute
by Neel Chauhan 18 Nov '24

18 Nov '24

Hi, I got Fios installed today in my NYC apartment, as I just moved back from Seattle after a 4-month stay in Connecticut. When doing a traceroute, I am noticing an incorrect Reverse DNS entry: [root@twin ~]# traceroute 1.1.1.1 traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets 1 _gateway (172.20.0.1) 0.145 ms 0.115 ms 0.106 ms 2 lo0-100.NYCMNY-VFTTP-372.verizon-gni.net (173.56.84.1) 2.079 ms 2.070 ms 2.061 ms 3 G0-3-4-4.PITBPA-LCR-21.verizon-gni.net (100.41.196.252) 3.751 ms T0-8-0-9.BSTNMA-LCR-21.verizon-gni.net (100.41.196.248) 4.454 ms 4.447 ms 4 * * * 5 * * * 6 162.158.61.113 (162.158.61.113) 7.246 ms 6.603 ms 162.158.61.117 (162.158.61.117) 4.944 ms 7 one.one.one.one (1.1.1.1) 6.077 ms 6.355 ms 6.346 ms [root@twin ~]# In short, hop 3 is "G0-3-4-4.PITBPA-LCR-21.verizon-gni.net (100.41.196.252)" which seems to be a Pittsburgh, PA reverse DNS entry even when I'm in NYC. While it doesn't affect the normal operation of the service, it might give the impression packets are being routed via Pittsburgh to someone less savvy. If someone works at the Verizon Fios/ILEC network engineering or sysadmin team could the 100.41.196.252 reverse DNS entry to say "G0-3-4-4.NYCMNY-LCR-21.verizon-gni.net" please get updated? Thanks! Neel Chauhan

5 4

SSH bruteforce attempts from Verizon to "AS17452 Bitstop Inc"
by Neel Chauhan 01 Nov '24

01 Nov '24

Hi, I am a customer of Verizon Fios in NYC and received a very interesting abuse complaint today from abuse(a)verizon.com. I got SSH bruteforce attempts between my IP address and this IPv4 prefix: 202.91.160.0/20 This is hosted on a network called "AS17452 Bitstop Inc". My connection runs Tor relays, but not exit relays. I doubt it's Tor since when I had Google Fiber or CenturyLink even when running Tor relays I never got those complaints. I use a MikroTik core router so maybe it got malware, although I recently updated it from 7.15.3 to 7.16.1. I decided to filter the prefix. Maybe a Windows PC on our network is infected. Maybe it's my Rocky Linux servers. Should I probably get a Supermicro/Deciso box and run an OPNsense firewall instead? But I never got complaints from Frontier or Optimum when I put MikroTik routers on both ISPs too, that for accounts not in my name. Maybe it's a false positive and just TCP forgery pretending to me be (I hope so). Just letting you know. -Neel === REDACTED COMPLAINT BELOW === Dear Verizon Online Customer, On 10-30-2024, your account was reported to have been used in an attempt to gain unauthorized access to another system, or to transmit malicious traffic to another Internet user. It is possible that a device connected to your network may have been infected by a virus or a botnet that is causing this action. Report and/or Logs: To assist you in understanding the situation, we have provided the relevant log data below, with timestamps adjusted to our GMT +8 timezone: DateTime Action AttackClass SourceIP Srcport Protocol DestinationIP DestPort 0 30-Oct-2024 13:37:21 BLOCKED attempted-recon 108.30.XXX.XXX 0 202.91.162.24 22 1 30-Oct-2024 13:49:38 BLOCKED attempted-recon 108.30.XXX.XXX 0 202.91.162.24 22 2 30-Oct-2024 14:00:01 BLOCKED attempted-recon 108.30.XXX.XXX 0 202.91.162.24 22 3 30-Oct-2024 14:10:12 BLOCKED attempted-recon 108.30.XXX.XXX 0 202.91.162.24 22 4 30-Oct-2024 15:17:15 DENIED 108.30.XXX.XXX 32769 TCP 202.91.162.17 22 5 30-Oct-2024 15:18:29 BLOCKED attempted-recon 108.30.XXX.XXX 0 202.91.162.39 22 6 30-Oct-2024 15:23:08 DENIED 108.30.XXX.XXX 54688 TCP 202.91.163.179 22 7 30-Oct-2024 15:30:22 BLOCKED attempted-recon 108.30.XXX.XXX 0 202.91.162.47 22 8 30-Oct-2024 15:47:32 BLOCKED attempted-recon 108.30.XXX.XXX 0 202.91.162.24 22 9 30-Oct-2024 15:58:03 DENIED 108.30.XXX.XXX 50405 TCP 202.91.163.143 22 Please immediately ensure your anti-virus and anti-malware software is properly updated. Please perform full system scans on your device(s). [Including - computers, tablets, cellular devices, network attached storage, security camera recorders (DVR or NVR), and IOT devices, where possible.] Additional information and removal guidance of detected malware may be found on the website of your scanner(s) manufacturer. It is difficult to verify the presence of an exact virus or malware infecting a device without a full system scan with up-to-date software. Installing the most recent firmware and software updates can also assist in securing your device(s). Please follow the device manufactures processes for any updates. If you are unable to take immediate action, it would be advisable to remove the device(s), which may be infected, from your network and the Internet connection until it has been properly cleaned. This may be easily done by unplugging the network cable that connects the device to the router. For wireless devices removing power from the device will keep it off it the network. Note: this information is being provided as a courtesy; you are solely responsible for any changes you make to your device(s) or network. Verizon Policy: If you do not take steps to resolve this issue, we may be forced to take further action. Actions could include the suspension or termination of your service until the issue is resolved, in order to ensure the safety of our network, and the safety of other Internet users. Please carefully review these agreements, which can be viewed at: http://www.verizon.com/about/terms/ Any future violation will result in further action being taken, up to, and including, the termination of your service. Sincerely, Verizon Global IP Abuse http://www.verizon.com/about/terms/ http://www.verizon.com/securityinfo Abuse(a)verizon.com

3 3

N93 Registration is OPEN! Election Results Are In...+ More!
by Nanog News 31 Oct '24

31 Oct '24

*** 🎃👻 Happy Halloween from NANOG!* ------------------------------------------------------------ Stay safe, secure your network, and enjoy the treats (online and off)! 🦇💻 *** NANOG Election Results are In* ------------------------------------------------------------ *Congratulations Board Members!* Steve Feldman and Elizabeth Culley were elected to the board. Their terms will run from 01 January 2025 to 31 December 2027. We would like to thank all six candidates. *** My Favorite NANOG 92 Moment* ------------------------------------------------------------ *Community Shares the Moments that Made Our Last Meeting* *NANOG 92 (21-23, Oct.) has come and gone, but the memories will last forever.* Whether it was a conversation that shifted perspective, a Social that deepened connections, or a talk that inspired future growth, it’s clear that N92 sparked unforgettable moments for attendees. *READ MORE (https://nanog.org/stories/articles/my-favorite-nanog-92-moment/ <https://nanog.org/stories/articles/my-favorite-nanog-92-moment/>)* *** N93 Registration is OPEN!* ------------------------------------------------------------ *Join Us in Atlanta 3 - 5, Feb. 2025!* Early birds get the worm. *Don't miss out on discounted early bird pricing until 10 November!* *REGISTER NOW (https://nanog.org/events/nanog-93/ <https://nanog.org/events/nanog-93/>)* *** N93 Call For Presentations!* ------------------------------------------------------------ *Program Committee is Seeking Talk Proposals for our Next Meeting* We welcome all talk topics but are aiming to highlight presentations throughout NANOG 93 focusing on Automation. *MORE INFO (https://nanog.org/program/call-presentations/ <https://nanog.org/program/call-presentations/>)*

1 0

Reminder to Submit Presentations for NANOG 93
by Stevan Plote 31 Oct '24

31 Oct '24

NANOG Community at large, The NANOG Program Committee (PC) would like to remind you that we are now accepting proposals for in-person or live remote presentations at all sessions of NANOG 93, taking place in Atlanta, GA on February 3-5, 2025. Below is a summary of key details and dates from the Call For Presentations on the NANOG website, which can be found at https://www.nanog.org/program/call-presentations/. *Requested Topics: * We are aiming to spotlight presentations throughout NANOG 93 focusing on Automation. Topics can range from best practices, to “what I learned setting up my first automated network”, and/or a panel discussion on how ISPs, Research + Education, Enterprise and other industry sectors have implemented automation. Based on feedback from our recent surveys, we have seen numerous requests for the following topics: ● Network Automation - practical implementations, how to get started ● Kubernetes - implementation and best practices ● Future of Networking - forecast for changes in technology, design, applications ● Research & Education - what research is happening now in network operations ● Security - various protocols of, developments in, problems/solutions ● Optical Networking - technology, practices, deployment options ● Tutorials - at all skill levels, IPv6, BGP, Segment Routing, DNS, MPLS, VxLAN We are looking to schedule* 1,600 minutes of content *between General Session and Breakout Rooms for NANOG 93, and have confirmed 255 minutes to date - so don’t wait! *Presentation abstracts are welcomed now and draft slides should be submitted no later than Monday, 06 January 2025 to be considered for NANOG 93.* Presentations may cover current technologies, soon-to-be deployed technologies, and industry innovation. Vendors are welcome to submit talks which cover relevant technologies and capabilities, but presentations should not be promotional or discuss proprietary solutions. The primary speaker, moderator, or author should submit a presentation proposal and abstract via the Program Committee Tool at: https://www.nanog.org/meetings/submit-presentation/ ● Sign in with your Profile Account ● Select the type of talk you propose to present, and complete the form Timeline for submission and proposal review: ● Submitter enters abstract (and draft slides if possible) in the Program Committee Tool prior to the deadline for slide submission. ● PC performs initial review and assigns a “shepherd” to help develop the submission — typically within 2 weeks. ● Submitter develops draft slides of talk, if not already submitted with the initial proposal. Please submit initial draft slides early — the PC does not evaluate submissions until draft slides are available for review. NANOG Staff is available to assist with slide templates upon request from the submitter. ● Panel and Track submissions should provide a topic list and intended/confirmed participants in the abstract. ● PC reviews the slides and continues to work with Submitter as needed to develop the topic and content. ● *FINAL SUBMISSION DEADLINE *Draft presentation slides should be submitted prior to the published deadline for slides (06 January 2025). ● PC evaluates submissions to determine presentations for the agenda (posted on 13 January 2025). ● Submitters notified. ● Agenda assembled and posted. ● *Final presentation slides* must be submitted prior to the published deadline for slides (27 January 2025 for in person + live remote presentations). If you think you have an interesting topic but want feedback or suggestions for developing an idea into a presentation, please email the PC ( nanogpc(a)nanog.org) and a representative will respond to you in a timely manner. Otherwise, submit your talk, tutorial, track, or panel proposal to the Program Committee Tool at your earliest convenience. We look forward to reviewing your submission! *NANOG 93 Calendar of Events* *Date* *Event/Deadline* Nov. 4, 2024 CFP Reminder Announcement Dec. 2, 2024 Call for Proposals Reminder #2 Jan. 6, 2025 DRAFT Presentation Slides Due Jan. 6, 2025 Topics List Published Jan. 13, 2025 Meeting Agenda Published Jan. 27, 2025 FINAL Slides Due Feb. 2, 2025 On-Site Lightning Talk Submissions Open *Final slides for accepted presentations must be submitted by Monday, January 27, 2025.* Materials received after that date, and prior to January 31 may be accepted based on Shepherd availability, but is not guaranteed. Materials received after January 31 will not be accepted. We look forward to seeing you in February! Sincerely, Stevan Plote Program Committee Chair Sent on behalf of the NANOG Program Committee

1 0

hbo max geolocation issue
by Mehmet 30 Oct '24

30 Oct '24

hello there i am dealing with hbo max geolocation issue, is there anyone on the list from HBO team who can assist (or point me where i can look for a solution) thank you

8 9

Re: october
by Mel Beckman 29 Oct '24

29 Oct '24

Randy, My apologies, I wasn’t aware. Do you have documentation of that? They appear to still be soliciting contributions, but I see the last award was issued it back in 2012. -mel > On Oct 29, 2024, at 1:34 PM, Randy Bush <randy(a)psg.com> wrote: > >> >> There is a memorial fund for his ISOC Service Award, to which >> interested people can contribute: > > don't. the award is formally declared done. > > randy

1 0