Comcast NOC - issues to/from AS13331 (Seattle)
Comcast doesn't appear to have any usable NOC contacts via whois, and this issue is apparently very widespread. Comcast obviously has multiple saturated paths out in this area, so if you're seeing issues getting to your customers on Comcast...well, it's probably Comcast. Sort of an ongoing/me too on last months thread about same... Outbound traffic to Comcast via basically any of our (AS13331) upstreams except Spectrum (AS11404) is experiencing very high packet loss once inside Comcast's network, we suspect a router or link very near to us in Seattle is failing. I've marked out the destination customer IPs but they're available to Comcast engineers if they get in touch with me directly or as noc@metapeer.com -- TIA -- and again, sorry to the list. Below are three failing/high loss traces, and then same three traces from our other DC which is routing out via spectrum. The loss is present on Cogent and L3 outgoing paths at the least, maybe others. As of now I've routed around the problem point for via the good path. fission:~# mtr -s 900 --report --report-cycles 30 1.2.3.4 fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev ve75-gw1-xmr.metapeer.com 0.0% 0.4 1.4 0.3 11.8 2.9 te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.9 0.6 0.5 0.9 0.1 te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 43.7 42.3 38.8 43.7 1.4 be-13-cr01.seattle.wa.ibone.comcast.net 3.3% 47.7 44.6 41.3 47.7 1.8 pos-0-7-0-0-cr01.denver.co.ibone.comcast.net 3.3% 69.4 68.7 65.0 71.3 1.5 he-5-15-0-0-cr01.350ecermak.il.ibone.comcast 6.7% 92.6 97.3 91.4 102.0 3.8 he-3-15-0-0-ar01.woburn.ma.boston.comcast.ne 10.0% 120.3 119.8 116.3 122.0 1.8 pos-0-1-0-0-ar01.needham.ma.boston.comcast.n 0.0% 122.0 120.5 117.1 123.9 1.8 po-80-ur01.deering.nh.boston.comcast.net 0.0% 121.4 121.3 118.3 133.8 2.8 po-21-ur01.concord.nh.boston.comcast.net 6.7% 119.6 130.1 119.3 272.8 31.8 te-1-0-0-ten01.concord.nh.boston.comcast.net 6.7% 118.7 120.0 117.6 121.4 1.3 ??? 100.0 0.0 0.0 0.0 0.0 0.0 You have new mail in /var/spool/mail/root fission:~# mtr -s 900 --report --report-cycles 30 5.6.7.8 fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev ve75-gw1-xmr.metapeer.com 0.0% 0.3 0.6 0.3 5.2 0.9 te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.8 0.6 0.5 0.8 0.1 te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 0.0% 41.4 42.6 38.5 43.8 1.3 be-13-cr01.seattle.wa.ibone.comcast.net 0.0% 45.6 44.1 40.4 46.7 1.8 pos-0-8-0-0-cr01.denver.co.ibone.comcast.net 6.7% 71.2 69.0 65.0 72.1 1.9 he-5-12-0-0-cr01.350ecermak.il.ibone.comcast 3.3% 98.1 96.3 86.4 101.2 3.9 he-0-15-0-0-ar01.pontiac.mi.michigan.comcast 10.0% 93.5 97.3 93.5 100.6 2.3 xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.c 0.0% 97.2 99.1 92.9 141.8 10.4 te-17-10-cdn05.rochestrhlls.mi.michigan.comc 3.3% 118.9 113.2 104.6 118.9 3.5 ??? 100.0 0.0 0.0 0.0 0.0 0.0 fission:~# mtr -s 900 --report --report-cycles 30 a.b.c.d fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev ve75-gw1-xmr.metapeer.com 0.0% 0.3 2.1 0.3 47.1 8.6 te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.7 0.6 0.5 0.8 0.1 te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 40.6 42.4 39.7 45.2 1.3 be-15-cr01.seattle.wa.ibone.comcast.net 0.0% 47.3 48.2 41.3 54.8 4.3 68.86.92.34 3.3% 43.7 45.5 39.5 101.7 11.6 be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 43.8 43.0 40.2 44.3 1.4 te-3-0-0-ten15.bellevue.wa.seattle.comcast.n 6.7% 42.7 44.1 42.1 45.7 1.1 ??? 100.0 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 1.2.3.4 HOST: phobos Loss% Snt Last Avg Best Wrst StDev 1. 207.229.74.1 0.0% 30 0.4 5.1 0.3 132.4 24.1 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 1.5 14.0 1.5 206.4 39.8 3. 23.30.206.33 0.0% 30 1.8 2.0 1.7 3.5 0.3 4. be-17-cr01.seattle.wa.ibone.comcast.net 0.0% 30 12.6 5.3 2.1 12.6 2.7 5. pos-0-4-0-0-cr01.denver.co.ibone.comcast.net 0.0% 30 28.5 29.2 27.2 31.4 1.3 6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 30 49.9 52.2 49.9 56.7 2.8 7. he-3-5-0-0-ar01.woburn.ma.boston.comcast.net 0.0% 30 78.5 79.5 78.3 81.0 1.0 8. pos-1-12-0-0-ar01.needham.ma.boston.comcast.net 0.0% 30 80.2 80.1 78.2 81.7 1.1 9. po-80-ur01.deering.nh.boston.comcast.net 0.0% 30 80.0 80.1 79.9 80.4 0.1 10. po-21-ur01.concord.nh.boston.comcast.net 0.0% 30 80.7 85.1 80.7 199.3 21.6 11. te-1-0-0-ten01.concord.nh.boston.comcast.net 0.0% 30 80.5 80.5 80.5 80.7 0.0 12. ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 5.6.7.8 HOST: phobos Loss% Snt Last Avg Best Wrst StDev 1. 207.229.74.1 0.0% 30 0.3 3.6 0.3 96.3 17.5 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 10.2 10.4 1.5 58.5 15.1 3. 23.30.206.33 0.0% 30 2.4 2.0 1.8 2.4 0.1 4. be-14-cr01.seattle.wa.ibone.comcast.net 0.0% 30 3.6 4.4 1.7 10.6 2.4 5. pos-0-3-0-0-cr01.denver.co.ibone.comcast.net 0.0% 30 31.1 29.5 27.2 31.6 1.3 6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 30 52.3 55.4 50.3 61.5 3.6 7. he-0-14-0-0-ar01.pontiac.mi.michigan.comcast.net 0.0% 30 58.9 58.0 56.1 60.1 1.2 8. xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.comcast.net 0.0% 30 71.2 58.9 56.6 83.6 6.4 9. te-17-10-cdn05.rochestrhlls.mi.michigan.comcast.net 0.0% 30 71.2 72.6 67.6 78.0 3.2 10. ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 a.b.c.d HOST: phobos Loss% Snt Last Avg Best Wrst StDev 1. 207.229.74.1 0.0% 30 0.7 5.1 0.3 143.4 26.1 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 1.5 3.1 1.5 30.1 5.6 3. 23.30.206.33 0.0% 30 1.9 2.0 1.9 2.2 0.1 4. be-12-cr01.seattle.wa.ibone.comcast.net 0.0% 30 6.9 5.1 1.7 12.3 3.1 5. 68.86.94.70 0.0% 30 1.8 2.4 1.8 17.3 2.8 6. be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 30 3.5 3.4 3.3 3.5 0.1 7. te-3-0-0-ten15.bellevue.wa.seattle.comcast.net 0.0% 30 3.3 3.2 3.2 3.4 0.0 8. ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ -- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
We have same problem. Cogent noc says use another provider, Comcast will not upgrade unless they pay for the traffic b/c they send more then they take in. I attach Ren Provo for official statement. Vinod
Date: Mon, 22 Apr 2013 22:35:27 -0700 Subject: Comcast NOC - issues to/from AS13331 (Seattle) From: mloftis@wgops.com To: nanog@nanog.org
Comcast doesn't appear to have any usable NOC contacts via whois, and this issue is apparently very widespread. Comcast obviously has multiple saturated paths out in this area, so if you're seeing issues getting to your customers on Comcast...well, it's probably Comcast. Sort of an ongoing/me too on last months thread about same...
Outbound traffic to Comcast via basically any of our (AS13331) upstreams except Spectrum (AS11404) is experiencing very high packet loss once inside Comcast's network, we suspect a router or link very near to us in Seattle is failing. I've marked out the destination customer IPs but they're available to Comcast engineers if they get in touch with me directly or as noc@metapeer.com -- TIA -- and again, sorry to the list.
Below are three failing/high loss traces, and then same three traces from our other DC which is routing out via spectrum. The loss is present on Cogent and L3 outgoing paths at the least, maybe others. As of now I've routed around the problem point for via the good path.
fission:~# mtr -s 900 --report --report-cycles 30 1.2.3.4 fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev ve75-gw1-xmr.metapeer.com 0.0% 0.4 1.4 0.3 11.8 2.9 te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.9 0.6 0.5 0.9 0.1 te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 43.7 42.3 38.8 43.7 1.4 be-13-cr01.seattle.wa.ibone.comcast.net 3.3% 47.7 44.6 41.3 47.7 1.8 pos-0-7-0-0-cr01.denver.co.ibone.comcast.net 3.3% 69.4 68.7 65.0 71.3 1.5 he-5-15-0-0-cr01.350ecermak.il.ibone.comcast 6.7% 92.6 97.3 91.4 102.0 3.8 he-3-15-0-0-ar01.woburn.ma.boston.comcast.ne 10.0% 120.3 119.8 116.3 122.0 1.8 pos-0-1-0-0-ar01.needham.ma.boston.comcast.n 0.0% 122.0 120.5 117.1 123.9 1.8 po-80-ur01.deering.nh.boston.comcast.net 0.0% 121.4 121.3 118.3 133.8 2.8 po-21-ur01.concord.nh.boston.comcast.net 6.7% 119.6 130.1 119.3 272.8 31.8 te-1-0-0-ten01.concord.nh.boston.comcast.net 6.7% 118.7 120.0 117.6 121.4 1.3 ??? 100.0 0.0 0.0 0.0 0.0 0.0 You have new mail in /var/spool/mail/root fission:~# mtr -s 900 --report --report-cycles 30 5.6.7.8 fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev ve75-gw1-xmr.metapeer.com 0.0% 0.3 0.6 0.3 5.2 0.9 te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.8 0.6 0.5 0.8 0.1 te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 0.0% 41.4 42.6 38.5 43.8 1.3 be-13-cr01.seattle.wa.ibone.comcast.net 0.0% 45.6 44.1 40.4 46.7 1.8 pos-0-8-0-0-cr01.denver.co.ibone.comcast.net 6.7% 71.2 69.0 65.0 72.1 1.9 he-5-12-0-0-cr01.350ecermak.il.ibone.comcast 3.3% 98.1 96.3 86.4 101.2 3.9 he-0-15-0-0-ar01.pontiac.mi.michigan.comcast 10.0% 93.5 97.3 93.5 100.6 2.3 xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.c 0.0% 97.2 99.1 92.9 141.8 10.4 te-17-10-cdn05.rochestrhlls.mi.michigan.comc 3.3% 118.9 113.2 104.6 118.9 3.5 ??? 100.0 0.0 0.0 0.0 0.0 0.0 fission:~# mtr -s 900 --report --report-cycles 30 a.b.c.d fission.myfreecams.com Snt: 30 Loss% Last Avg Best Wrst StDev ve75-gw1-xmr.metapeer.com 0.0% 0.3 2.1 0.3 47.1 8.6 te0-7-0-15.ccr21.sea02.atlas.cogentco.com 0.0% 0.7 0.6 0.5 0.8 0.1 te-0-5-0-3-pe03.seattle.wa.ibone.comcast.net 6.7% 40.6 42.4 39.7 45.2 1.3 be-15-cr01.seattle.wa.ibone.comcast.net 0.0% 47.3 48.2 41.3 54.8 4.3 68.86.92.34 3.3% 43.7 45.5 39.5 101.7 11.6 be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 43.8 43.0 40.2 44.3 1.4 te-3-0-0-ten15.bellevue.wa.seattle.comcast.n 6.7% 42.7 44.1 42.1 45.7 1.1 ??? 100.0 0.0 0.0 0.0 0.0 0.0
mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 1.2.3.4 HOST: phobos Loss% Snt Last Avg Best Wrst StDev 1. 207.229.74.1 0.0% 30 0.4 5.1 0.3 132.4 24.1 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 1.5 14.0 1.5 206.4 39.8 3. 23.30.206.33 0.0% 30 1.8 2.0 1.7 3.5 0.3 4. be-17-cr01.seattle.wa.ibone.comcast.net 0.0% 30 12.6 5.3 2.1 12.6 2.7 5. pos-0-4-0-0-cr01.denver.co.ibone.comcast.net 0.0% 30 28.5 29.2 27.2 31.4 1.3 6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 30 49.9 52.2 49.9 56.7 2.8 7. he-3-5-0-0-ar01.woburn.ma.boston.comcast.net 0.0% 30 78.5 79.5 78.3 81.0 1.0 8. pos-1-12-0-0-ar01.needham.ma.boston.comcast.net 0.0% 30 80.2 80.1 78.2 81.7 1.1 9. po-80-ur01.deering.nh.boston.comcast.net 0.0% 30 80.0 80.1 79.9 80.4 0.1 10. po-21-ur01.concord.nh.boston.comcast.net 0.0% 30 80.7 85.1 80.7 199.3 21.6 11. te-1-0-0-ten01.concord.nh.boston.comcast.net 0.0% 30 80.5 80.5 80.5 80.7 0.0 12. ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 5.6.7.8 HOST: phobos Loss% Snt Last Avg Best Wrst StDev 1. 207.229.74.1 0.0% 30 0.3 3.6 0.3 96.3 17.5 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 10.2 10.4 1.5 58.5 15.1 3. 23.30.206.33 0.0% 30 2.4 2.0 1.8 2.4 0.1 4. be-14-cr01.seattle.wa.ibone.comcast.net 0.0% 30 3.6 4.4 1.7 10.6 2.4 5. pos-0-3-0-0-cr01.denver.co.ibone.comcast.net 0.0% 30 31.1 29.5 27.2 31.6 1.3 6. he-5-12-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 30 52.3 55.4 50.3 61.5 3.6 7. he-0-14-0-0-ar01.pontiac.mi.michigan.comcast.net 0.0% 30 58.9 58.0 56.1 60.1 1.2 8. xe-11-0-0-0-sur01.rochestrhlls.mi.michigan.comcast.net 0.0% 30 71.2 58.9 56.6 83.6 6.4 9. te-17-10-cdn05.rochestrhlls.mi.michigan.comcast.net 0.0% 30 71.2 72.6 67.6 78.0 3.2 10. ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$ mtr -s 900 --report --report-wide --report-cycles 30 a.b.c.d HOST: phobos Loss% Snt Last Avg Best Wrst StDev 1. 207.229.74.1 0.0% 30 0.7 5.1 0.3 143.4 26.1 2. agg1-sea-t7-8.bb.spectrumnet.us 0.0% 30 1.5 3.1 1.5 30.1 5.6 3. 23.30.206.33 0.0% 30 1.9 2.0 1.9 2.2 0.1 4. be-12-cr01.seattle.wa.ibone.comcast.net 0.0% 30 6.9 5.1 1.7 12.3 3.1 5. 68.86.94.70 0.0% 30 1.8 2.4 1.8 17.3 2.8 6. be-18-ur06.bellevue.wa.seattle.comcast.net 0.0% 30 3.5 3.4 3.3 3.5 0.1 7. te-3-0-0-ten15.bellevue.wa.seattle.comcast.net 0.0% 30 3.3 3.2 3.2 3.4 0.0 8. ??? 100.0 30 0.0 0.0 0.0 0.0 0.0 mloftis@phobos:~$
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
If you search the nanog archives for Comcast, you'll see a long history of discussions like this one, and the suspected/supposed reasons for their behavior in relation to the rest of the net. -Blake
participants (3)
-
Blake Dunlap -
Michael Loftis -
Vinod K