November 2021 - NANOG - lists.nanog.org

Validating multi-path in production?
by Adam Thompson 26 Nov '21

26 Nov '21

Hello all. Over time, we've run into occurrences of both bugs and human error, both in our own gear and in our partner networks' gear, specifically affecting multi-path forwarding, at pretty much all layers: Multi-chassis LAG, ECMP, and BGP MP. (Yes, I am a corner-case magnet. Lucky me.) Some of these issues were fairly obvious when they happened, but some were really hard to pin down. We've found that typical network monitoring tools (Observium & Smokeping, not to mention plain old ping and traceroute) can't really detect a hashing-related or multi-path-related problem: either the packets get through or they don't. Can anyone recommend either tools or techniques to validate that multi-path forwarding either is, or isn't, working correctly in a production network? I'm looking to add something to our test suite for when we make changes to critical network gear. Almost all the scenarios I want to test only involve two paths, if that helps. The best I've come up with so far is to have two test systems (typically VMs) that use adjacent IP addresses and adjacent MAC addresses, and test both inbound and outbound to/from those, blindly trusting/hoping that hashing algorithms will probably exercise both paths. Some of the problems we've seen show that merely looking at interface counters is insufficient, so I'm trying to find an explicit proof, not implicit. Any suggestions? Surely other vendors and/or admins have screwed this up in subtle ways enough times that this knowledge exists? (My Google-fu is usually pretty good, but I'm striking out - maybe I'm using the wrong terms.) -Adam Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson(a)merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/>

7 7

Looking for a contact at Citi
by Ed Ronayne 26 Nov '21

26 Nov '21

Hi, Apologies for the noise, shouting out to see if I can reach anyone in Citi Bank that could assist in resetting a VPN tunnel. Best Regards Ed

1 0

anyone use fbtracert successfully?
by Adam Thompson 25 Nov '21

25 Nov '21

The tool fbtracert (http://github.com/facebookarchive/fbtracert) was mentioned here recently as a way to get visibility into multi-pathing. Has anyone here ever used this tool successfully? Supposedly Facebook uses this tool internally, but… that doesn’t help much. I’ve tried it on 4 different platforms/OSes (WSL Ubuntu; RedHat; Debian; OpenBSD), and versions of Go (v1.10 through v1.16), in three very different environments (on-prem public IP; on-prem NAT’d; cloud public IP), and I’ve yet to see it produce any meaningful output – each run/iteration/thread only detects one, single, hop out of the entire chain of routers, making it less than useful. Granted, that’s not a full regression test by any means, but if anyone here has ever used it successfully, could you please let me know what sort of environment you ran it in/on? Thanks, -Adam Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athompson(a)merlin.mb.ca<mailto:athompson@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/>

5 6

Salesforce issues
by Erik Sundberg 25 Nov '21

25 Nov '21

We are receiving latency complaints to Salesforce, anyone else seeing this? Looks like Vocus Network might be possibly leaking some force.com (Salesforce) routes, causing 200ms of latency. Traffic is going Chicago -> Vocus Networks -> Australia -> Akamai Atlanta I have emailed both Vocus Networks and Akamai about this. dig force.com +short 104.109.10.129 <<< Possible Leaked via Vocus Networks 104.109.11.129 <<< Possible Leaked via Vocus Networks 184.25.179.132 <<< Possible Leaked via Vocus Networks 184.31.3.130 <<<<< Route ok (NTT, TATA, Akamai) 25ms 184.31.10.133 <<< Route ok (NTT, Akamai) 16ms 23.1.35.132 <<< Possible Leaked via Vocus Networks 23.1.99.130 <<< Route ok (Akamai) 15 ms 23.1.106.133 <<< Route ok (NTT, Akamai) 16ms 2021-11-24T13:47:59-0600 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. AS??? 172.19.1.2 (172.19.1.2) 0.0% 7 1.7 1.9 0.9 4.8 1.3 2. ??? 3. AS53828 207.200.193.193 (207.200.193.193) 0.0% 6 1.0 1.0 0.9 1.0 0.1 4. AS??? eqix-ch1.vocusconnect.com (208.115.136.142) 0.0% 6 41.0 40.8 40.8 41.0 0.1 5. AS4826 be101.bdr04.sjc01.ca.us.vocus.network (114.31.199.38) 40.0% 6 200.6 200.7 200.6 200.8 0.1 [MPLS: Lbl 24266 Exp 0 S 1 TTL 1] 6. AS4826 be100.bdr03.sjc01.ca.us.vocus.network (114.31.199.32) 0.0% 6 197.4 197.5 197.4 197.5 0.0 [MPLS: Lbl 24807 Exp 0 S 1 TTL 1] 7. AS4826 be202.cor02.syd04.nsw.vocus.network (114.31.199.42) 0.0% 6 197.4 197.4 197.3 197.5 0.1 [MPLS: Lbl 24932 Exp 0 S 1 TTL 1] 8. AS4826 be100.bdr04.syd03.nsw.vocus.network (114.31.192.45) 0.0% 6 197.5 197.5 197.4 197.5 0.0 9. AS4826 static-74.173.255.49.in-addr.VOCUS.net.au (49.255.173.74) 0.0% 6 200.2 200.7 197.4 205.2 3.0 10. AS20940 a23-1-240-228.deploy.static.akamaitechnologies.com (23.1.240.228) 0.0% 6 197.1 197.3 197.1 197.6 0.2 11. ??? 12. AS4826 static-73.173.255.49.in-addr.VOCUS.net.au (49.255.173.73) 0.0% 6 197.5 197.6 197.5 197.8 0.1 13. AS4826 be114.cor02.syd04.nsw.vocus.network (114.31.192.44) 0.0% 6 348.7 349.0 348.7 349.4 0.3 14. AS4826 be202.bdr03.sjc01.ca.us.vocus.network (114.31.199.43) 0.0% 6 349.0 349.1 348.9 349.3 0.1 15. AS3257 173.205.35.1 (173.205.35.1) 0.0% 6 348.8 351.0 348.6 362.1 5.4 16. AS3257 ae18-3416.cr11-fra2.ip4.gtt.net (89.149.180.134) 0.0% 6 356.0 350.5 349.1 356.0 2.7 17. AS3257 ip4.gtt.net (98.124.180.42) 0.0% 6 348.9 349.0 348.9 349.2 0.1 18. AS20940 ae12.r01.sjc01.icn.netarch.akamai.com (23.207.232.36) 0.0% 6 368.5 353.4 349.3 368.5 8.4 19. AS20940 ae11.r01.ord01.icn.netarch.akamai.com (23.32.63.46) 0.0% 6 393.7 393.8 393.7 394.1 0.2 20. AS20940 ae6.r02.iad01.icn.netarch.akamai.com (23.32.63.21) 0.0% 6 411.4 411.8 411.4 412.4 0.5 21. AS20940 ae2.r12.iad01.ien.netarch.akamai.com (23.203.152.41) 0.0% 6 411.9 415.1 411.9 417.4 2.9 AS20940 ae2.r11.iad01.ien.netarch.akamai.com (23.203.152.39) 22. ??? 23. AS33905 a184-25-179-132.deploy.static.akamaitechnologies.com (184.25.179.132) 0.0% 6 215.9 216.4 215.9 216.7 0.3 Erik Sundberg Sr. Network Engineer office 773.661.5532 mobile 708.710.7419 noc 866.892.0915 esundberg(a)nitelusa.com<mailto:esundberg@nitelusa.com> 888.450.2100 350 N Orleans St. #1300N Chicago, IL 60654 nitelusa.com<http://www.nitelusa.com/> [cid:image006.png@01D7E13A.E1A0E610]<http://www.nitelusa.com/> Smarter technology made simple ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

2 1

Google location question
by Chuck Church 24 Nov '21

24 Nov '21

NANOG, Sorry if this is the wrong forum, but I figured we could all use a distraction from IPV4 expansion for a bit. We're facing a problem with corporate use of browsers and google location services. Our users across North and South America seem to be getting location info for Rio de Janeiro, BR as of the last few days. All of our users when browsing hit general internet via a NAT pool for IPs out of Atlanta, GA. That hasn't changed in over a year. The IP space itself and the upstream routers PTR haven't changed for longer than that. For a normal laptop without a GPS chip and using default settings, it would seem that something else must be informing google of location. We've had a few theories. I discovered our Cisco wireless in RIO office is broadcasting SSID. The same SSID name we use in all offices. But I believe that google uses BSSID which is MAC based. And would be different between sites, as each have their own wireless controller, and certainly different APs. At least that's what I believe to be true, that google didn't associate our standard SSID name with a physical location. I'm curious if anyone has ever dealt with this before. I don't want to blame our workstation folks for something that might be a geolocation issue, either based on wi-fi being detected, or a messed up geolocation for our IP space. Is there any place in Google-land where you can see exactly why it thinks you're in a given location? I did see some articles on API usage to view values, but nothing that shows you the process when it takes all variables into effect. Thanks, Chuck

2 1

Quantifying the customer support and impact of cgnat for residential ipv4
by Eric Kuhnke 22 Nov '21

22 Nov '21

Looking for anecdotal examples of the following: If you put N number of individual DHCP client residential broadband customers behind cgnat for ipv4, what percent of customers contact support and become a support/troubleshooting case later. And what percent of customers have a significant problem with it, to the extent that they either need to be offered a $5-10/mo extra /32 dedicated real address, or possibly cancel? Hopefully on sample sizes of 5000 or more. All else assuming that the customers are also dual stack v4/v6 and can reach v6 things normally without any of that traffic going through the cgnat.

4 3

RPKI-Based Policy Without Route Refresh
by Mark Tinka 22 Nov '21

22 Nov '21

Randy will be presenting draft-ymbk-sidrops-rov-no-rr during RIPE-83, at around 1530hrs UTC: https://datatracker.ietf.org/doc/html/draft-ymbk-sidrops-rov-no-rr-02 Most grateful if you can join, and provide some initial feedback. Thanks. Mark.

1 0

Re: Redploying most of 127/8 as unicast public
by Mark Andrews 22 Nov '21

22 Nov '21

> On 18 Nov 2021, at 11:58, Joe Maimon <jmaimon(a)chl.com> wrote: > > > > Mark Andrews wrote: >> It’s a denial of service attack on the IETF process to keep bringing up drafts like this that are never going to be approved. 127/8 is in use. It isn’t free. > > There are so many things wrong with this statement that I am not even going to try to enumerate them. > > However suffice it to say that drafts like these are concrete documentation of non-groupthink and essentially you are advocating for self-censorship and loss of historical perspective. I’m advocating for not taking address away that have been allocated for a purpose. No one knows what the impact of doing that will be. Perhaps we should just take back 216.222.144.0/20? You obviously think taking back address that are in use to be a good thing. This is similar to taking back other address that are allocated but not advertised. > Which given the state of IPv6 transition, perhaps more of that in the past would have been nice. > > For example https://datatracker.ietf.org/doc/html/draft-fuller-240space-02 from 2008 which fell prey to the "by the time this is usable IPv6 will have taken over" groupthink. Again an oranges vs apples comparison. Class E is not 127/8. This is “Reserved" vs "Reserved for use on the host”. Totally different histories. Totally different expectations. > Objectively wrong. > > Predictive self-fulfilling circular arguments of this sort should no longer be given any weight, and along your lines, should never even be brought up. > >> >> Lots of bad attempts to justify a bad idea. >> >> "The IPv4 network 127/8 was first reserved by Jon Postel in 1981 [RFC0776]. Postel's policy was to reserve the first and last network of each class, and it does not appear that he had a specific plan for how to use 127/8.” >> >> Having a space for permission-less innovation and testing is a good thing. Jon understood that. > > Yes its a good idea to have space that is guaranteed to be available to every system regardless of its networking condition and that the host has deterministic control over the addressing used in that space. > > However, it turns out that /8 was much too large. The extreme few instances of its usefulness at that size pale in comparison with even the possibility of its usefulness to the public. > > So any attempt to adjust that should be given proper attention and serious thought. > >> >> "By contrast, IPv6, despite its vastly larger pool of available address space, allocates only a single local loopback address (::1) [RFC4291]. This appears to be an architectural vote of confidence in the idea that Internet protocols ultimately do not require millions of distinct loopback addresses.” >> >> This is an apples-to-oranges comparison. IPv6 has both link and site local addresses and an architecture to deliver packets to specific instances of each. This does not exist in the IPv4 world. > > SO an IPv6 only system without any network interfaces can run multiple discrete instances of the same daemon accepting connections on the same TCP port? Yes. I’ve been doing testing over the IPv6 loopback for 20+ years now. > Can I script that, can I template that with hardcoded addresses, same as I can now for 127/8? > > Good thing I can just use ::FFFF:127.0.0.1/104 You can script is to the same extent that you can hard code 127/8 addresses. I’ve used ULA addresses but conceptually they are the same. The lo0 interface also has more that 127.0.0.1 IPv4 addresses on it. % ifconfig lo0 inet6 lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet6 fd92:7065:b8e:ffff::1 prefixlen 64 inet6 fd92:7065:b8e:ffff::2 prefixlen 64 inet6 fd92:7065:b8e:ffff::3 prefixlen 64 inet6 fd92:7065:b8e:ffff::4 prefixlen 64 inet6 fd92:7065:b8e:ffff::5 prefixlen 64 inet6 fd92:7065:b8e:ffff::6 prefixlen 64 inet6 fd92:7065:b8e:ffff::7 prefixlen 64 inet6 fd92:7065:b8e:ffff::8 prefixlen 64 inet6 fd92:7065:b8e:ffff::9 prefixlen 64 inet6 fd92:7065:b8e:ffff::10 prefixlen 64 inet6 fd92:7065:b8e:ff::1 prefixlen 64 inet6 fd92:7065:b8e:ff::2 prefixlen 64 inet6 fd92:7065:b8e:99ff::1 prefixlen 64 inet6 fd92:7065:b8e:99ff::2 prefixlen 64 inet6 fd92:7065:b8e:fffe::a35:4 prefixlen 64 nd6 options=201<PERFORMNUD,DAD> % >> "In theory, having multiple local loopback addresses might be useful for increasing the number of distinct IPv4 sockets that can be used for inter-process communication within a host. The local loopback /16 network retained by this document will still permit billions of distinct concurrent loopback TCP connections within a single host, even if both the IP address and port number of one endpoint of each connection are fixed.” >> >> But it doesn’t deliver millions of end points. Sorry you simulation will not work because we don’t have more that 65000 end points anymore. Sorry RFC 1918 addresses are not always suitable. >> >> "Reserved for <use>" is not the same as “Reserved”. >> >> Mark > > Let them use IPv6 link local for their simulations. Which doesn’t work when you are simulating dual stack. >>> On 18 Nov 2021, at 10:45, scott <surfer(a)mauigateway.com> wrote: >>> >>> >>> >>> On 11/17/2021 1:29 PM, Jay R. Ashworth wrote: >>>> This seems like a really bad idea to me; am I really the only one who noticed? >>>> >>>> > > Its only a relevant idea if you still care about IPv4. In which case, it might be a good idea. > > Joe -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka(a)isc.org

6 17

Unicast etc meta-observation
by bzs＠theworld.com 21 Nov '21

21 Nov '21

Reading over many of these notes my observation is that many here are good at understanding the technical points of the proposals and throwing around 224/4 this and 127/8 that. Then the discussion mostly disintegrates into anecdotes with hands waving furiously, "my anecdote is VERY important!" What would be useful would be some attempt to quantify what those anecdotes are trying to express such as the costs and disruptions they imply. Something more like engineering or economics or engineering economics. Let's use a measure, "Cost And Disruption to the Internet's Infrastructure", call it the CDII. For example: 1. Proposing these address re-uses: CDII zero, other than the cost of endless discussion. 2. Writing them down into revised standards: CDII zero, other than the effort to agree and write them down. 3. Deploying an experiment: CDII about zero one hopes, again other than keeping "lab notebooks" and reporting back results. 4. Deploying wider, voluntary usage: CDII larger as this begins to require changes in, for example, actual routing, firewalls, etc in the wild and may begin to affect those not intentionally involved. 5. Full implementation: Still voluntary of course because everything is. But this might be the point where someone requesting IPv4 address space is assigned a block in these re-purposed spaces. CDII grows. There are sub-threads implied for the various repurposed blocks with different CDIIs. Many seem to feel, for example, that re-use of 224/4 would have relatively low CDII, 127/8 perhaps a somewhat higher CDII, etc. What's the detailed list? What are the relative CDIIs? And there are devils in the details of quantifying CDII per se. For example CDII for infrastructure operators vs end users for a given deployment. But it's not totally open-ended or unknowable or, perhaps put better, not impossible to enumerate and make intelligent estimates. It's not a very deep or bushy tree. Just a few choices at each decision point with CDIIs assigned. Plus some attempt to aggregate those CDIIs and decide what is acceptable and what is not, thresholds. Plus time horizons and quantifiable measures of success and failure along the way. Something like this could be more productive than describing one's connected lightbulb or networking in Armenia one more time even if such anecdotes would form the basis for that quantification. Since this and similar discussions have been going on literally for decades maybe the time is past due to try to quantify something like this proposed CDII model. -- -Barry Shein Software Tool & Die | bzs(a)TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD The World: Since 1989 | A Public Information Utility | *oo*

1 0

FERC releases final report on Texas power outages (2021)
by Sean Donelan 20 Nov '21

20 Nov '21

"Those Who Do Not Learn History Are Doomed To Repeat It." Federal Energy Regulatory Commission (FERC) has published its final report on the Texas 2021 power outages. According to FERC during the last 10 years, there have been four extreme cold weather events which have threatened the Texas power grid. That includes the 2011 Texas power outage during which hundreds of people died. https://www.ferc.gov/media/february-2021-cold-weather-outages-texas-and-sou… As found after past weather events, investigators determined that most of the power outages could have been avoided if power plants and wind turbines had been better protected against the cold and ice. Eighty-one percent of freeze-related generating unit outages occurred at temperatures above the units' stated ambient design temperature. Eighty-seven percent of unplanned generation outages were due to fuel issues related to natural gas. Natural gas fuel supply issues were caused by natural gas production declines. Some recommendations by FERC include: Generator owners should retrofit existing generating units, and when building new units account for weather events including extreme temperatures. Generator owners and operators should perform annual training on winterization plans. Develop corrective action plans should freeze-related outages occur. However, Texas maintains its electric grid as an isolated island, and hasn't followed past recommendations to avoid electric grid outages. https://www.mysanantonio.com/news/local/article/Federal-report-warns-Texas-…

5 5