Unable to geto to www.cymru.com and 68.22.187.24 has been down for 5+ hours. Known issue?
On Monday, 31-October-2005 13:46, matthew zeier wrote:
Unable to geto to www.cymru.com and 68.22.187.24 has been down for 5+ hours. Known issue?
I think Ameritech/SBC/AT&T has problems here in Chicago. We have had problems getting to several sites here. Traces across Ameritech's network die in some spots. We are trying to get the issues resolved, but Ameritech's people are...less than clueful. Sargon
Getting hung up in savvis - can't ping through - 1. v140.core1.irv.intelenet.net 2. v8-ge4-1.border4.irv.intelenet.net 3. so-6-0-0.ar3.LAX1.gblx.net 4. so6-0-0-2488M.ar2.PAO2.gblx.net 5. bpr1-so-2-0-0.PaloAltoPaix.savvis.net 6. dcr2-so-3-3-0.SanFranciscosfo.savvis.net 7. dcr1-so-5-0-0.SanFranciscosfo.savvis.net 8. dcr2-so-0-0-0.Denver.savvis.net 9. dcr1-so-7-0-0.Chicago.savvis.net 10. acr2-so-1-0-0.Chicago.savvis.net 11. acr2-so-0-0-0.Chicago.savvis.net 12. s228110-1.savvis-internet.uschcg1-bsn.savvis.net 13. ??? Sargon wrote:
On Monday, 31-October-2005 13:46, matthew zeier wrote:
Unable to geto to www.cymru.com and 68.22.187.24 has been down for 5+ hours. Known issue?
I think Ameritech/SBC/AT&T has problems here in Chicago. We have had problems getting to several sites here. Traces across Ameritech's network die in some spots. We are trying to get the issues resolved, but Ameritech's people are...less than clueful.
Sargon
Odd - I am seeing my route die here (from AS7949): 1 router-sf-t3sf.wmis.net (69.39.94.5) 4 msec 8 msec 4 msec 2 69.39.68.169 [AS 12129] 4 msec 12 msec 4 msec 3 edge-1.sfld-mi.123.net (216.234.96.2) [AS 12129] 4 msec 4 msec 4 msec 4 POS2-1.GW4.DET5.ALTER.NET (65.206.123.245) [AS 701] 8 msec 8 msec 4 msec 5 500.at-6-3-0.CL2.DET5.ALTER.NET (152.63.27.106) [AS 701] 8 msec 8 msec 4 msec 6 0.so-5-0-0.XL2.BOS4.ALTER.NET (152.63.16.126) [AS 701] 40 msec 40 msec 36 msec 7 0.so-7-0-0.XR2.BOS4.ALTER.NET (152.63.16.134) [AS 701] 48 msec 36 msec 40 msec 8 178.at-3-0-0.HR1.BOS4.ALTER.NET (152.63.21.245) [AS 701] 36 msec 40 msec 36 msec 9 0.so-0-2-0.AR1.BOS24.ALTER.NET (152.63.3.14) [AS 701] 36 msec 40 msec 36 msec 10 * * * My BGP session to cymru has been down 5hs33min ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Matt Taber tabes@wmis.net WMIS Internet http://www.wmis.net "Accelerate ... It's a Speed Thing" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sargon wrote:
On Monday, 31-October-2005 13:46, matthew zeier wrote:
Unable to geto to www.cymru.com and 68.22.187.24 has been down for 5+ hours. Known issue?
I think Ameritech/SBC/AT&T has problems here in Chicago. We have had problems getting to several sites here. Traces across Ameritech's network die in some spots. We are trying to get the issues resolved, but Ameritech's people are...less than clueful.
Sargon
On Mon, 31 Oct 2005, Matt Taber wrote:
Odd - I am seeing my route die here (from AS7949):
1 router-sf-t3sf.wmis.net (69.39.94.5) 4 msec 8 msec 4 msec 2 69.39.68.169 [AS 12129] 4 msec 12 msec 4 msec 3 edge-1.sfld-mi.123.net (216.234.96.2) [AS 12129] 4 msec 4 msec 4 msec 4 POS2-1.GW4.DET5.ALTER.NET (65.206.123.245) [AS 701] 8 msec 8 msec 4 msec 5 500.at-6-3-0.CL2.DET5.ALTER.NET (152.63.27.106) [AS 701] 8 msec 8 msec 4 msec 6 0.so-5-0-0.XL2.BOS4.ALTER.NET (152.63.16.126) [AS 701] 40 msec 40 msec 36 msec 7 0.so-7-0-0.XR2.BOS4.ALTER.NET (152.63.16.134) [AS 701] 48 msec 36 msec 40 msec 8 178.at-3-0-0.HR1.BOS4.ALTER.NET (152.63.21.245) [AS 701] 36 msec 40 msec 36 msec 9 0.so-0-2-0.AR1.BOS24.ALTER.NET (152.63.3.14) [AS 701] 36 msec 40 msec 36 msec 10 * * *
My BGP session to cymru has been down 5hs33min
ah-ha! me thinks someone is in the midst of a new customer turnup, though it's back to replying to tcp/80 requests via savvis now.
Hi, NANOGers. Just a quick interruption in your day to update you on the routing fun we had. :) The problem was two-fold: 1. An errant provider router that randomly selected which prefixes to propagate. Mix that with uRPF, and you have FUN FUN FUN! :) 2. An interesting use of LOCAL_PREF by the provider, now mitigated. Our tests from multiple external pods and route servers show that we're globally visible again. If you're having any difficulty reaching us, please let us know at <team-cymru@cymru.com>. Again we apologize for the inconvenience! Thanks, Rob, for Team Cymru. -- Rob Thomas Team Cymru http://www.cymru.com/ ASSERT(coffee != empty);
On Mon, 31 Oct 2005, Roy Arends wrote:
On Mon, 31 Oct 2005, matthew zeier wrote:
Unable to geto to www.cymru.com and 68.22.187.24 has been down for 5+ hours. Known issue?
www.cymru.com resolves to 68.22.187.27 which is reachable from AS1103.
just to through more fuel on the fire, from a 701 perspective it hops though savvis and dies here: 11 acr2-so-1-0-0.Chicago.savvis.net (208.172.3.86) 25.014 ms 24.956 ms 25.833 ms 12 s228110-1.savvis-internet.uschcg1-bsn.savvis.net (64.240.84.82) 30.395 ms 29.377 ms 29.371 ms 13 *^C I think this is the upstream device to the 'customer' network... perhaps firewall problems? or a local host problem. Generally they are pretty quick to fix these sorts of things.
Maybe its just _scary_ Halloween routing? It works from my view of SBC/Ameritech... even though it looks like its going through a Cisco Systems customer connection. 6 ex1-g9-0-0.pxatga.sbcglobal.net (151.164.249.9) 17.078 ms 16.964 ms 16.89 1 ms 7 bb2-p3-0.atlnga.sbcglobal.net (151.164.42.14) 17.586 ms 75.782 ms 17.365 ms 8 bb1-p14-0.atlnga.sbcglobal.net (151.164.189.21) 17.599 ms 17.469 ms 17.37 0 ms 9 core1-p5-0.cratga.sbcglobal.net (151.164.40.230) 17.232 ms 17.243 ms 17.2 13 ms 10 core3-p3-0.crchil.sbcglobal.net (151.164.241.122) 31.975 ms 31.739 ms 31. 704 ms 11 core1-p9-0.crchil.sbcglobal.net (151.164.243.18) 32.315 ms 32.086 ms 31.9 77 ms 12 bb1-p6-0.chcgil.ameritech.net (151.164.42.184) 66.854 ms 33.085 ms 110.77 5 ms 13 ded2-g1-3-0.chcgil.ameritech.net (151.164.241.113) 33.132 ms 64.020 ms 33 .133 ms 14 Cisco-Systems-1054141.cust-rtr.ameritech.net (65.42.139.42) 41.344 ms 37.3 96 ms 37.158 ms Happy Halloween, Deepak Christopher L. Morrow wrote:
On Mon, 31 Oct 2005, Roy Arends wrote:
On Mon, 31 Oct 2005, matthew zeier wrote:
Unable to geto to www.cymru.com and 68.22.187.24 has been down for 5+ hours. Known issue?
www.cymru.com resolves to 68.22.187.27 which is reachable from AS1103.
just to through more fuel on the fire, from a 701 perspective it hops though savvis and dies here:
11 acr2-so-1-0-0.Chicago.savvis.net (208.172.3.86) 25.014 ms 24.956 ms 25.833 ms 12 s228110-1.savvis-internet.uschcg1-bsn.savvis.net (64.240.84.82) 30.395 ms 29.377 ms 29.371 ms 13 *^C
I think this is the upstream device to the 'customer' network... perhaps firewall problems? or a local host problem. Generally they are pretty quick to fix these sorts of things.
Hi, NANOGers. ] Maybe its just _scary_ Halloween routing? It's definitely Halloween, but this isn't as scary as it looks. :) ] 14 Cisco-Systems-1054141.cust-rtr.ameritech.net... Back when we were part of Cisco, SBC named our link to them accordingly. Now that we're on our own, I should probably ask them to change that. Thanks, Rob. -- Rob Thomas Team Cymru http://www.cymru.com/ ASSERT(coffee != empty);
On Mon, Oct 31, 2005 at 08:10:37PM +0000, Christopher L. Morrow wrote:
On Mon, 31 Oct 2005, Roy Arends wrote:
On Mon, 31 Oct 2005, matthew zeier wrote:
Unable to geto to www.cymru.com and 68.22.187.24 has been down for 5+ hours. Known issue?
www.cymru.com resolves to 68.22.187.27 which is reachable from AS1103.
just to through more fuel on the fire, from a 701 perspective it hops though savvis and dies here:
11 acr2-so-1-0-0.Chicago.savvis.net (208.172.3.86) 25.014 ms 24.956 ms 25.833 ms 12 s228110-1.savvis-internet.uschcg1-bsn.savvis.net (64.240.84.82) 30.395 ms 29.377 ms 29.371 ms 13 *^C
I think this is the upstream device to the 'customer' network... perhaps firewall problems? or a local host problem. Generally they are pretty quick to fix these sorts of things.
So, how many people from nanog DOES it take to troubleshoot a simple network outage anyways? :) With each passing day I find myself wishing more and more that there was a Cisco Certified Traceroute Engineer program, and a convenient system to ignore emails with traceroutes from anyone else. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Hi, NANOGers. ] Unable to geto to www.cymru.com and 68.22.187.24 has been down ] for 5+ hours. Known issue? We brought up a new BGP peer today, oddly enough to add additional redundancy and improved performance. :) The turn-up appears to have gone awry somewhere in the provider's network, and we're debugging that now. The primary site has been online the entire time, but the routing oddness has caused everyone a bit of pain. We are also seeing some Chicago-based oddness that others are enduring. We're pretty sure it has nothing to do with one of our members, an ardent Cubs fan, cursing the Sox. ;) We continue to debug it with our peers. Stay tuned! Apologies for the inconvenience. Thanks, Rob. -- Rob Thomas Team Cymru http://www.cymru.com/ ASSERT(coffee != empty);
participants (8)
-
Christopher L. Morrow -
Deepak Jain -
Matt Taber -
matthew zeier -
Richard A Steenbergen -
Rob Thomas -
Roy Arends -
Sargon