Also peering problems in Chicago, had to drop BGP to them until they

Appears to have been resolved after seven hours. My ticket just says: "We isolated the routing issue and resolved it. The issue was due to a misconfiguration on one our core routers." Now that the issue is corrected, interestingly enough, my trace goes from Tampa to Chicago instead of Dallas. David -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Cool Hand Luke Sent: Wednesday, November 19, 2014 11:23 AM To: nanog@nanog.org Subject: Re: level3 issue in chicago (was: FYI, Level 3 issues in Dallas) like david below, we've been getting reports this morning from customers unable to reach various web sites. investigating a number of these reports, the one commonality is level3 in chicago. sites are reachable from level3/cincinnati but not level3/chicago. traceroutes make it one hop past our bgp peer and seem to die there. for one particular remote web server (node1104.follett.com, and possibly others) traceroutes and pings from level3's looking glass (using chicago as the site) complete successfully, while neither does when ran from my gear through the same site. On 11/19/2014 10:46 AM, garya@completeweb.net wrote: figure it out.

Reported it 4 hours ago, no good response yet. Gary

...

We have some customers unable to access their websites, seeing this on the way to them:

4 ae-0-11.bar2.Tampa1.Level3.net (4.69.137.110) 0.343 ms 0.423 ms 0.406 ms 5 ae-12-12.ebr1.Dallas1.Level3.net (4.69.137.118) 23.940 ms 23.470 ms 23.426 ms 6 ae-71-71.csw2.Dallas1.Level3.net (4.69.151.137) 24.026 ms ae-81-81.csw3.Dallas1.Level3.net (4.69.151.149) 24.007 ms ae-61-61.csw1.Dallas1.Level3.net (4.69.151.125) 23.855 ms 7 ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 23.361 ms ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 23.329 ms ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 23.936 ms 8 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.378 ms vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.387 ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.963 ms 9 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 23.357 ms ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 22.779 ms ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 62.640 ms 10 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 24.179 ms 23.297 ms 23.403 ms 11 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 65.617 ms ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 23.286 ms ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 23.553 ms 12 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.303 ms vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.027 ms 24.028 ms 13 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 24.018 ms ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 23.374 ms ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203) 23.342 ms 14 vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 24.068 ms vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.361 ms 23.348 ms 15 ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 24.023 ms ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 22.740 ms ae-4-90.edge10.Dallas1.Level3.net (4.69.145.210) 23.963 ms 16 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.420 ms vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.400 ms vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 25.317 ms 17 ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82) 23.985 ms ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 23.476 ms ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 51.762 ms 18 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 23.416 ms vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 24.062 ms vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.051 ms 19 ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 44.798 ms ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 22.856 ms ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 22.891 ms 20 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.433 ms vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 25.333 ms vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.367 ms 21 ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 23.475 ms ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 23.476 ms ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 64.753 ms 22 vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 24.008 ms 24.043 ms vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.508 ms 23 ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139) 23.505 ms ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 23.958 ms ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.058 ms 24 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.429 ms vlan90.csw4.Dallas1.Level3.net (4.69.145.254) 23.623 ms 25.547 ms 25 ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82) 23.491 ms ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144) 23.551 ms ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11) 24.078 ms 26 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 24.472 ms 24.144 ms 24.080 ms 27 ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18) 28.121 ms ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146) 22.848 ms ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.090 ms 28 vlan60.csw1.Dallas1.Level3.net (4.69.145.62) 23.509 ms 24.056 ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.522 ms 29 ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75) 24.161 ms 24.153

...

ms ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 24.130 ms 30 vlan70.csw2.Dallas1.Level3.net (4.69.145.126) 23.606 ms 23.482 ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190) 23.580 ms

fyi: On 11/20/2014 02:42 AM, cool hand luke wrote:

On 11/19/2014 07:29 PM, David Hubbard wrote:

...

Appears to have been resolved after seven hours. My ticket just says:

"We isolated the routing issue and resolved it. The issue was due to a misconfiguration on one our core routers."

Now that the issue is corrected, interestingly enough, my trace goes from Tampa to Chicago instead of Dallas.

our issue was resolved around the same time but is apparently now occurring again, since 0638 utc -- routing issue affecting (for us) traffic on level3 between chicago, illinois, and cincinnati, ohio. i imagine it's affecting other locations as well.

routing issue experienced from 06:37 - 11:02 utc. notes from level 3: "11/20/2014 11:30:19 GMT - After investigating the issue I found that this circuit was affected by an outage on our network. A non-service affecting maintenance had unintended results causing routing problems in the Chicago area. All traffic through the Chicago area was affected by this issue. If you have any additional destinations that are still seeing issues please contact us as soon as possible as the issue may not be completely resolved yet." /chl