Comcast doesn't appear to have any usable NOC contacts via whois, and this issue is apparently very widespread. Comcast obviously has multiple saturated paths out in this area, so if you're seeing issues getting to your customers on Comcast...well, it's probably Comcast. Sort of an ongoing/me too on last months thread about same... Outbound traffic to Comcast via basically any of our (AS13331) upstreams except Spectrum (AS11404) is experiencing very high packet loss once inside Comcast's network, we suspect a router or link very near to us in Seattle is failing. I've marked out the destination customer IPs but they're available to Comcast engineers if they get in touch with me directly or as noc@metapeer.com -- TIA -- and again, sorry to the list. Below are three failing/high loss traces, and then same three traces from our other DC which is routing out via spectrum. The loss is present on Cogent and L3 outgoing paths at the least, maybe others. As of now I've routed around the problem point for via the good path. fission:~# mtr -s 900 --report --report-cycles 30 1.2.3.4 fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev ve75-gw1-xmr.metapeer.com 0.0% 0.4 1.4 0.3 11.8 2.9 te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.9 0.6 0.5 0.9 0.1 te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 43.7 42.3 38.8 43.7 1.4 be-13-cr01.seattle.wa.ibone.comcast.net 3.3% 47.7 44.6 41.3 47.7 1.8 pos-0-7-0-0-cr01.denver.co.ibone.comcast.net 3.3% 69.4 68.7 65.0 71.3 1.5 he-5-15-0-0-cr01.350ecermak.il.ibone.comcast 6.7% 92.6 97.3 91.4 102.0 3.8 he-3-15-0-0-ar01.woburn.ma.boston.comcast.ne 10.0% 120.3 119.8 116.3 122.0 1.8 pos-0-1-0-0-ar01.needham.ma.boston.comcast.n 0.0% 122.0 120.5 117.1 123.9 1.8 po-80-ur01.deering.nh.boston.comcast.net 0.0% 121.4 121.3 118.3 133.8 2.8 po-21-ur01.concord.nh.boston.comcast.net 6.7% 119.6 130.1 119.3 272.8 31.8 te-1-0-0-ten01.concord.nh.boston.comcast.net 6.7% 118.7 120.0 117.6 121.4 1.3 ??? 100.0 0.0 0.0 0.0 0.0 0.0 You have new mail in /var/spool/mail/root fission:~# mtr -s 900 --report --report-cycles 30 5.6.7.8 fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev ve75-gw1-xmr.metapeer.com 0.0% 0.3 0.6 0.3 5.2 0.9 te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.8 0.6 0.5 0.8 0.1 te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 0.0% 41.4 42.6 38.5 43.8 1.3 be-13-cr01.seattle.wa.ibone.comcast.net 0.0% 45.6 44.1 40.4 46.7 1.8 pos-0-8-0-0-cr01.denver.co.ibone.comcast.net 6.7% 71.2 69.0 65.0 72.1 1.9 he-5-12-0-0-cr01.350ecermak.il.ibone.comcast 3.3% 98.1 96.3 86.4 101.2 3.9 he-0-15-0-0-ar01.pontiac.mi.michigan.comcast 10.0% 93.5 97.3 93.5 100.6 2.3 xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.c 0.0% 97.2 99.1 92.9 141.8 10.4 te-17-10-cdn05.rochestrhlls.mi.michigan.comc 3.3% 118.9 113.2 104.6 118.9 3.5 ??? 100.0 0.0 0.0 0.0 0.0 0.0 fission:~# mtr -s 900 --report --report-cycles 30 a.b.c.d fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev ve75-gw1-xmr.metapeer.com 0.0% 0.3 2.1 0.3 47.1 8.6 te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.7 0.6 0.5 0.8 0.1 te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 40.6 42.4 39.7 45.2 1.3 be-15-cr01.seattle.wa.ibone.comcast.net 0.0% 47.3 48.2 41.3 54.8 4.3 68.86.92.34 3.3% 43.7 45.5 39.5 101.7 11.6 be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 43.8 43.0 40.2 44.3 1.4 te-3-0-0-ten15.bellevue.wa.seattle.comcast.n 6.7% 42.7 44.1 42.1 45.7 1.1 ??? 100.0 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 1.2.3.4 HOST: phobos Loss% Snt Last Avg Best Wrst StDev 1. 207.229.74.1 0.0% 30 0.4 5.1 0.3 132.4 24.1 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 1.5 14.0 1.5 206.4 39.8 3. 23.30.206.33 0.0% 30 1.8 2.0 1.7 3.5 0.3 4. be-17-cr01.seattle.wa.ibone.comcast.net 0.0% 30 12.6 5.3 2.1 12.6 2.7 5. pos-0-4-0-0-cr01.denver.co.ibone.comcast.net 0.0% 30 28.5 29.2 27.2 31.4 1.3 6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 30 49.9 52.2 49.9 56.7 2.8 7. he-3-5-0-0-ar01.woburn.ma.boston.comcast.net 0.0% 30 78.5 79.5 78.3 81.0 1.0 8. pos-1-12-0-0-ar01.needham.ma.boston.comcast.net 0.0% 30 80.2 80.1 78.2 81.7 1.1 9. po-80-ur01.deering.nh.boston.comcast.net 0.0% 30 80.0 80.1 79.9 80.4 0.1 10. po-21-ur01.concord.nh.boston.comcast.net 0.0% 30 80.7 85.1 80.7 199.3 21.6 11. te-1-0-0-ten01.concord.nh.boston.comcast.net 0.0% 30 80.5 80.5 80.5 80.7 0.0 12. ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 5.6.7.8 HOST: phobos Loss% Snt Last Avg Best Wrst StDev 1. 207.229.74.1 0.0% 30 0.3 3.6 0.3 96.3 17.5 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 10.2 10.4 1.5 58.5 15.1 3. 23.30.206.33 0.0% 30 2.4 2.0 1.8 2.4 0.1 4. be-14-cr01.seattle.wa.ibone.comcast.net 0.0% 30 3.6 4.4 1.7 10.6 2.4 5. pos-0-3-0-0-cr01.denver.co.ibone.comcast.net 0.0% 30 31.1 29.5 27.2 31.6 1.3 6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 30 52.3 55.4 50.3 61.5 3.6 7. he-0-14-0-0-ar01.pontiac.mi.michigan.comcast.net 0.0% 30 58.9 58.0 56.1 60.1 1.2 8. xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.comcast.net 0.0% 30 71.2 58.9 56.6 83.6 6.4 9. te-17-10-cdn05.rochestrhlls.mi.michigan.comcast.net 0.0% 30 71.2 72.6 67.6 78.0 3.2 10. ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 a.b.c.d HOST: phobos Loss% Snt Last Avg Best Wrst StDev 1. 207.229.74.1 0.0% 30 0.7 5.1 0.3 143.4 26.1 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 1.5 3.1 1.5 30.1 5.6 3. 23.30.206.33 0.0% 30 1.9 2.0 1.9 2.2 0.1 4. be-12-cr01.seattle.wa.ibone.comcast.net 0.0% 30 6.9 5.1 1.7 12.3 3.1 5. 68.86.94.70 0.0% 30 1.8 2.4 1.8 17.3 2.8 6. be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 30 3.5 3.4 3.3 3.5 0.1 7. te-3-0-0-ten15.bellevue.wa.seattle.comcast.net 0.0% 30 3.3 3.2 3.2 3.4 0.0 8. ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler