RE: Level 3 TPA routing today?
From: Jon Lewis [mailto:jlewis@lewis.org]
We have gigE to Level3 in Orlando, and saw something happen around 1pm today. Customers were complaining of latency and packet loss, and our traffic to/from L3 dropped noticably if only for a few minutes.
It sounded like based on Craig's post yesterday they did have some known issue and were working to resolve it around that time, ~8 pm EST. I turned our gig link to them back on around 4 AM EST and have not had any complaints so far today. Traffic is up compared to yesterday afternoon too, back where it would normally be. The tech I spoke to this morning said he had no knowledge of any issues yesterday, of course my ticket also had none of the information I sent in to them yesterday or even a clear description of what the problem was.... Dave
On Wed, Aug 27, 2008 at 4:40 PM, David Hubbard < dhubbard@dino.hostasaurus.com> wrote:
From: Jon Lewis [mailto:jlewis@lewis.org]
We have gigE to Level3 in Orlando, and saw something happen around 1pm today. Customers were complaining of latency and packet loss, and our traffic to/from L3 dropped noticably if only for a few minutes.
It sounded like based on Craig's post yesterday they did have some known issue and were working to resolve it around that time, ~8 pm EST. I turned our gig link to them back on around 4 AM EST and have not had any complaints so far today. Traffic is up compared to yesterday afternoon too, back where it would normally be. The tech I spoke to this morning said he had no knowledge of any issues yesterday, of course my ticket also had none of the information I sent in to them yesterday or even a clear description of what the problem was....
Dave
The speicific issue we were seeing yesterday seems to have been resolved last night some time and we aren't seeing any instances of it any longer. --chip -- Just my $.02, your mileage may vary, batteries not included, etc....
On Wed, 27 Aug 2008, David Hubbard wrote:
be. The tech I spoke to this morning said he had no knowledge of any issues yesterday, of course my ticket also had none of the information I sent in to them yesterday or even a clear description of what the problem was....
We opened a ticket for today's event and got the same response. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Most likely the issue was communication between the NOC and the service management center. The NOC deals with the core facing events versus the SMC which takes the incoming calls from the customers. In this case the issue was identified and resolved in the NOC. Perhaps the RFO was not posted internally or whomever you talked with didn't check the status updates or something. Lot's of things could have resulted in a tech not knowing about this type of issue. Anyway, to tie up loose ends, there was a problem on a core router that was isolated and then repaired in Atlanta. regards -Craig On Aug 27, 2008, at 5:02 PM, Jon Lewis wrote:
On Wed, 27 Aug 2008, David Hubbard wrote:
be. The tech I spoke to this morning said he had no knowledge of any issues yesterday, of course my ticket also had none of the information I sent in to them yesterday or even a clear description of what the problem was....
We opened a ticket for today's event and got the same response.
---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Has anyone noticed significant Level3 transit issues this evening? [wrl@<REDACTED> ~]$ traceroute ae-23-52.car3.Chicago1.Level3.net traceroute to ae-23-52.car3.Chicago1.Level3.net (4.68.101.39), 30 hops max, 40 byte packets [...] 4 ge-6-1-101.hsa1.Cleveland1.Level3.net (64.156.66.29) 2.627 ms !H [wrl@<REDACTED> ~]$ [wrl@<REDACTED> ~]$ traceroute vlan79.csw2.Dallas1.Level3.net traceroute to vlan79.csw2.Dallas1.Level3.net (4.68.19.126), 30 hops max, 40 byte packets [...] 4 ge-6-1-101.hsa1.Cleveland1.Level3.net (64.156.66.29) 3.166 ms !H * * [wrl@<REDACTED> ~]$ [root@<REDACTED> ~]# traceroute www.he.net traceroute to www.he.net (216.218.186.2), 30 hops max, 40 byte packets [...] 3 te-3-1.car1.Cincinnati1.Level3.net (4.78.216.9) 29.279 ms 18.127 ms te-3-2.car1.Cincinnati1.Level3.net (4.78.216.13) 20.737 ms 4 ae-2-5.bar1.Cincinnati1.Level3.net (4.69.132.206) 20.193 ms 24.019 ms 24.360 ms 5 ae-10-10.ebr2.Chicago1.Level3.net (4.69.136.214) 35.255 ms 35.884 ms 35.595 ms 6 ae-23-56.car3.Chicago1.Level3.net (4.68.101.167) 28.672 ms ae-23-54.car3.Chicago1.Level3.net (4.68.101.103) 30.715 ms ae-23-56.car3.Chicago1.Level3.net (4.68.101.167) 32.160 ms 7 glbx-level3-te.Chicago1.level3.net (4.68.110.194) 97.377 ms 97.080 ms 96.625 ms 8 * * * 9 * * * [root@<REDACTED> ~]# On Wed, 27 Aug 2008, Craig Pierantozzi wrote:
Most likely the issue was communication between the NOC and the service management center. The NOC deals with the core facing events versus the SMC which takes the incoming calls from the customers. In this case the issue was identified and resolved in the NOC.
Perhaps the RFO was not posted internally or whomever you talked with didn't check the status updates or something. Lot's of things could have resulted in a tech not knowing about this type of issue.
Anyway, to tie up loose ends, there was a problem on a core router that was isolated and then repaired in Atlanta.
regards -Craig
On Aug 27, 2008, at 5:02 PM, Jon Lewis wrote:
On Wed, 27 Aug 2008, David Hubbard wrote:
be. The tech I spoke to this morning said he had no knowledge of any issues yesterday, of course my ticket also had none of the information I sent in to them yesterday or even a clear description of what the problem was....
We opened a ticket for today's event and got the same response.
---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
-- William R. Lorenz "Whether you think you can or think you can't, your right." -- Henry Ford
Some infrastructure blocks are not routed to portions of the network but should not affect ultimate reachability as long as the correct loopbacks and directly connected networks are advertised properly. regards On Aug 27, 2008, at 6:42 PM, William R. Lorenz wrote:
Has anyone noticed significant Level3 transit issues this evening?
[wrl@<REDACTED> ~]$ traceroute ae-23-52.car3.Chicago1.Level3.net traceroute to ae-23-52.car3.Chicago1.Level3.net (4.68.101.39), 30 hops max, 40 byte packets [...] 4 ge-6-1-101.hsa1.Cleveland1.Level3.net (64.156.66.29) 2.627 ms !H [wrl@<REDACTED> ~]$
[wrl@<REDACTED> ~]$ traceroute vlan79.csw2.Dallas1.Level3.net traceroute to vlan79.csw2.Dallas1.Level3.net (4.68.19.126), 30 hops max, 40 byte packets [...] 4 ge-6-1-101.hsa1.Cleveland1.Level3.net (64.156.66.29) 3.166 ms !H * * [wrl@<REDACTED> ~]$
participants (5)
-
chip -
Craig Pierantozzi -
David Hubbard -
Jon Lewis -
William R. Lorenz