AS7018 upstream loss via 32.130.88.22 (SF Bay Area) — per-flow, unidirectional
Seeing sustained upstream packet loss on AT&T Fiber (AS7018) in the SF Bay Area that looks path-dependent rather than congestion or CPE related. Posting in case anyone at AT&T can take a look, or has seen the same. Symptoms: Upstream collapses to ~2–5 Mbps on a 1 Gbps symmetric circuit Downstream unaffected — 219 measured transfers, 0 failures Host TCP counters show 5–14% retransmission on affected flows Per-flow: back-to-back transfers to the same destination, differing only in source port, alternate between full rate and collapsed. Once a flow establishes it stays in whichever state it started in. Affected destinations traverse 32.130.88.22 → 32.130.20.157. Destinations routing otherwise are unaffected (12 tested, 0 failures). Ruled out: CPE (3 gateways, 2 models), plan tier (5G → 1G), optics (no DDM alarms, PON O5), MTU (0% ICMP loss at all sizes), Wi-Fi (wired throughout). ~650 measurements over 7 days. Traceroutes and per-destination breakdown available on request. Anyone else seeing this on AS7018 in the Bay Area?
Hey Mike, I took a look and I did find a link that needed some attention. I have removed it from the path while the transport team works on it. Let me know if you are still seeing an issue. Josh From: mikecika--- via NANOG <nanog@lists.nanog.org> Sent: Monday, August 24, 2026 9:16 PM To: nanog@lists.nanog.org Cc: mikecika@gmail.com Subject: AS7018 upstream loss via 32.130.88.22 (SF Bay Area) — per-flow, unidirectional Seeing sustained upstream packet loss on AT&T Fiber (AS7018) in the SF Bay Area that looks path-dependent rather than congestion or CPE related. Posting in case anyone at AT&T can take a look, or has seen the same. Symptoms: Upstream Seeing sustained upstream packet loss on AT&T Fiber (AS7018) in the SF Bay Area that looks path-dependent rather than congestion or CPE related. Posting in case anyone at AT&T can take a look, or has seen the same. Symptoms: Upstream collapses to ~2–5 Mbps on a 1 Gbps symmetric circuit Downstream unaffected — 219 measured transfers, 0 failures Host TCP counters show 5–14% retransmission on affected flows Per-flow: back-to-back transfers to the same destination, differing only in source port, alternate between full rate and collapsed. Once a flow establishes it stays in whichever state it started in. Affected destinations traverse 32.130.88.22 → 32.130.20.157. Destinations routing otherwise are unaffected (12 tested, 0 failures). Ruled out: CPE (3 gateways, 2 models), plan tier (5G → 1G), optics (no DDM alarms, PON O5), MTU (0% ICMP loss at all sizes), Wi-Fi (wired throughout). ~650 measurements over 7 days. Traceroutes and per-destination breakdown available on request. Anyone else seeing this on AS7018 in the Bay Area? _______________________________________________ NANOG mailing list https://urldefense.com/v3/__https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/ESBPA7A4QTSLZZ6VQFYEPN25MQLKSV5L/__;!!BhdT!h0Luc1sfVVXxXajbYsfK7V_nfeJH3KWsbXxv9nQmp_A_zAxFalMdn2ukuMgDoHNw0sN6oYdpBuKFHjEp$<https://urldefense.com/v3/__https:/lists.nanog.org/archives/list/nanog@lists.nanog.org/message/ESBPA7A4QTSLZZ6VQFYEPN25MQLKSV5L/__;!!BhdT!h0Luc1sfVVXxXajbYsfK7V_nfeJH3KWsbXxv9nQmp_A_zAxFalMdn2ukuMgDoHNw0sN6oYdpBuKFHjEp$>
Maybe Josh fixed it, but I'm not seeing packet loss here. Odd that traceroutes stop at the 32.130... Because of that I cannot see if I am transiting 32.130.88.22 → 32.130.20.157 or not. al@post:/z/dns$ traceroute ibm.com traceroute to ibm.com (23.203.227.97), 30 hops max, 60 byte packets 1 routr0.sunnyside.com (192.147.248.2) 0.114 ms 0.139 ms * 2 12.250.77.157 (12.250.77.157) 3.149 ms 3.177 ms 3.208 ms 3 cr82.sj2ca.ip.att.net (12.122.128.202) 4.583 ms 4.602 ms 4.598 ms 4 ggr6.la2ca.ip.att.net (12.122.128.33) 2.938 ms 3.121 ms 3.207 ms 5 12.122.149.46 (12.122.149.46) 7.381 ms 7.404 ms 7.397 ms 6 12.122.149.225 (12.122.149.225) 6.951 ms 6.787 ms 6.781 ms 7 32.130.25.168 (32.130.25.168) 5.354 ms * * 8 * * 32.130.20.158 (32.130.20.158) 4.423 ms 9 32.130.104.99 (32.130.104.99) 4.816 ms 4.805 ms 4.792 ms 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * al@post:/z/dns$ ping -4 ibm.com PING ibm.com (23.203.227.97) 56(84) bytes of data. 64 bytes from a23-203-227-97.deploy.static.akamaitechnologies.com (23.203.227.97): icmp_seq=1 ttl=53 time=5.69 ms 64 bytes from a23-203-227-97.deploy.static.akamaitechnologies.com (23.203.227.97): icmp_seq=2 ttl=53 time=5.70 ms ^C --- ibm.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 5.690/5.696/5.703/0.075 ms On 8/24/2026 19:15, mikecika--- via NANOG wrote:
Seeing sustained upstream packet loss on AT&T Fiber (AS7018) in the SF Bay Area that looks path-dependent rather than congestion or CPE related. Posting in case anyone at AT&T can take a look, or has seen the same.
Symptoms:
Upstream collapses to ~2–5 Mbps on a 1 Gbps symmetric circuit Downstream unaffected — 219 measured transfers, 0 failures Host TCP counters show 5–14% retransmission on affected flows Per-flow: back-to-back transfers to the same destination, differing only in source port, alternate between full rate and collapsed. Once a flow establishes it stays in whichever state it started in.
Affected destinations traverse 32.130.88.22 → 32.130.20.157. Destinations routing otherwise are unaffected (12 tested, 0 failures).
Ruled out: CPE (3 gateways, 2 models), plan tier (5G → 1G), optics (no DDM alarms, PON O5), MTU (0% ICMP loss at all sizes), Wi-Fi (wired throughout).
~650 measurements over 7 days. Traceroutes and per-destination breakdown available on request.
Anyone else seeing this on AS7018 in the Bay Area? _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/ESBPA7A4...
Hop 9 is the last hop on AS7018. The hop you were concerned with is no longer in the path and it would have been one of the first few hops in your trace if it was there. Josh From: Al via NANOG <nanog@lists.nanog.org> Sent: Tuesday, August 25, 2026 11:00 AM To: mikecika--- via NANOG <nanog@lists.nanog.org> Cc: Al <awnanog@sunnyside.com> Subject: Re: AS7018 upstream loss via 32.130.88.22 (SF Bay Area) — per-flow, unidirectional Maybe Josh fixed it, but I'm not seeing packet loss here. Odd that traceroutes stop at the 32. 130. . . Because of that I cannot see if I am transiting 32. 130. 88. 22 → 32. 130. 20. 157 or not. al@ post: /z/dns$ traceroute ibm. com traceroute to ibm. com Maybe Josh fixed it, but I'm not seeing packet loss here. Odd that traceroutes stop at the 32.130... Because of that I cannot see if I am transiting 32.130.88.22 → 32.130.20.157 or not. al@post:/z/dns$ traceroute ibm.com traceroute to ibm.com (23.203.227.97), 30 hops max, 60 byte packets 1 routr0.sunnyside.com (192.147.248.2) 0.114 ms 0.139 ms * 2 12.250.77.157 (12.250.77.157) 3.149 ms 3.177 ms 3.208 ms 3 cr82.sj2ca.ip.att.net (12.122.128.202) 4.583 ms 4.602 ms 4.598 ms 4 ggr6.la2ca.ip.att.net (12.122.128.33) 2.938 ms 3.121 ms 3.207 ms 5 12.122.149.46 (12.122.149.46) 7.381 ms 7.404 ms 7.397 ms 6 12.122.149.225 (12.122.149.225) 6.951 ms 6.787 ms 6.781 ms 7 32.130.25.168 (32.130.25.168) 5.354 ms * * 8 * * 32.130.20.158 (32.130.20.158) 4.423 ms 9 32.130.104.99 (32.130.104.99) 4.816 ms 4.805 ms 4.792 ms 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * al@post:/z/dns$ ping -4 ibm.com PING ibm.com (23.203.227.97) 56(84) bytes of data. 64 bytes from a23-203-227-97.deploy.static.akamaitechnologies.com (23.203.227.97): icmp_seq=1 ttl=53 time=5.69 ms 64 bytes from a23-203-227-97.deploy.static.akamaitechnologies.com (23.203.227.97): icmp_seq=2 ttl=53 time=5.70 ms ^C --- ibm.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 5.690/5.696/5.703/0.075 ms On 8/24/2026 19:15, mikecika--- via NANOG wrote:
Seeing sustained upstream packet loss on AT&T Fiber (AS7018) in the SF Bay Area that looks path-dependent rather than congestion or CPE related. Posting in case anyone at AT&T can take a look, or has seen the same.
Symptoms:
Upstream collapses to ~2–5 Mbps on a 1 Gbps symmetric circuit
Downstream unaffected — 219 measured transfers, 0 failures
Host TCP counters show 5–14% retransmission on affected flows
Per-flow: back-to-back transfers to the same destination, differing only in source port, alternate between full rate and collapsed. Once a flow establishes it stays in whichever state it started in.
Affected destinations traverse 32.130.88.22 → 32.130.20.157. Destinations routing otherwise are unaffected (12 tested, 0 failures).
Ruled out: CPE (3 gateways, 2 models), plan tier (5G → 1G), optics (no DDM alarms, PON O5), MTU (0% ICMP loss at all sizes), Wi-Fi (wired throughout).
~650 measurements over 7 days. Traceroutes and per-destination breakdown available on request.
Anyone else seeing this on AS7018 in the Bay Area?
_______________________________________________
NANOG mailing list
_______________________________________________ NANOG mailing list https://urldefense.com/v3/__https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/2EEOI22MILJE3J3WIVKWXMKWTKWMPCXS/__;!!BhdT!mKTuNgiy4h822gcTRPjzmoo4wJG5rDiTBgSfHprGNhrh5KByYkKuD-Yw83n-nHAB86Q57YYKB-GUU4lX$<https://urldefense.com/v3/__https:/lists.nanog.org/archives/list/nanog@lists.nanog.org/message/2EEOI22MILJE3J3WIVKWXMKWTKWMPCXS/__;!!BhdT!mKTuNgiy4h822gcTRPjzmoo4wJG5rDiTBgSfHprGNhrh5KByYkKuD-Yw83n-nHAB86Q57YYKB-GUU4lX$>
On Tue, Aug 25, 2026 at 1:13 PM BEASLEY, JOSH via NANOG <nanog@lists.nanog.org> wrote:
Hop 9 is the last hop on AS7018. The hop you were concerned with is no longer in the path and it would have been one of the first few hops in your trace if it was there.
Josh
From: Al via NANOG <nanog@lists.nanog.org> Sent: Tuesday, August 25, 2026 11:00 AM To: mikecika--- via NANOG <nanog@lists.nanog.org> Cc: Al <awnanog@sunnyside.com> Subject: Re: AS7018 upstream loss via 32.130.88.22 (SF Bay Area) — per-flow, unidirectional
Maybe Josh fixed it, but I'm not seeing packet loss here. Odd that traceroutes stop at the 32. 130. . . Because of that I cannot see if I am transiting 32. 130. 88. 22 → 32. 130. 20. 157 or not. al@ post: /z/dns$ traceroute ibm. com traceroute to ibm. com
Maybe Josh fixed it, but I'm not seeing packet loss here.
Odd that traceroutes stop at the 32.130...
Because of that I cannot see if I am transiting 32.130.88.22 →
32.130.20.157 or not.
I'd guess that (since traceroutes to this space from outside 7018 seem to just die a the 7018 border) this is CGNat space for ATT Fiber customers... I suspect this all has to be fixed inside the 7018 cloud, and there's not much 'nanog' peoples can generally do for this problem, eh? :) (so calling the att support folks, booting your win98 computer and rebooting it with them on the phone - pstn - THEN getting them perf / traceroute data is required)
participants (4)
-
Al -
BEASLEY, JOSH -
Christopher Morrow -
mikecika＠gmail.com