What country, location, where you fed from?? -----Original Message----- From: marco [mailto:marco@zero11.com] Sent: December 28, 2008 12:59 PM To: nanog@nanog.org Subject: Level 3 issues is anyone having issues with Level3? ---------------------------------------------------------------------------- "The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you."
http://www.internetpulse.net/ (if you can get to it). Does not look pretty for L3. I can't get to most web sites if I go via Level3 (Cleveland, OH). Ping/traceroute look good though. marco wrote:
is anyone having issues with Level3?
Ive got connection issues from Colorado to new York on level3 that have been restored, but still nothing from Chicago to Colorado, and way too many other places to list. Anyone have a ticket number with level3? -----Original Message----- From: Pierre-Henri [mailto:phach34@gmail.com] Sent: Sunday, December 28, 2008 11:06 AM To: marco Cc: nanog@nanog.org Subject: Re: Level 3 issues marco a écrit :
is anyone having issues with Level3?
hi, theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes. Don't know if there is a link with your question, but . Pierre-Henri
Blake Pfankuch wrote:
Ive got connection issues from Colorado to new York on level3 that have been restored, but still nothing from Chicago to Colorado, and way too many other places to list. Anyone have a ticket number with level3?
-----Original Message----- From: Pierre-Henri [mailto:phach34@gmail.com] Sent: Sunday, December 28, 2008 11:06 AM To: marco Cc: nanog@nanog.org Subject: Re: Level 3 issues
marco a écrit :
is anyone having issues with Level3?
hi, theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes. Don't know if there is a link with your question, but .
Pierre-Henri
I have a ticket in with L3, their NOC is pretty overwhelmed with phone calls.
marco a écrit :
is anyone having issues with Level3?
hi, theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes. Don't know if there is a link with your question, but it's strange... Pierre-Henri
Ahh.. yes seeing that now here from Toronto ON - didn't see this issue when the original poster sent the first message... it's now happening here too... Shutting down their session until something looks "better" -----Original Message----- From: Pierre-Henri [mailto:phach34@gmail.com] Sent: December 28, 2008 1:06 PM To: marco Cc: nanog@nanog.org Subject: Re: Level 3 issues marco a écrit :
is anyone having issues with Level3?
hi, theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes. Don't know if there is a link with your question, but it's strange... Pierre-Henri ---------------------------------------------------------------------------- "The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you."
Same issue here from Chicago and Montreal. Seems anything routing through Washington.Level3 is going to null. The rest of the level3 network seems to be ok. 6 ae-32-52.ebr2.Chicago1.Level3.net (4.68.101.62) 0.976 ms 10.344 ms 0.866 ms 7 ae-5.ebr2.Chicago2.Level3.net (4.69.140.194) 1.245 ms 0.991 ms 0.978 ms 8 ae-2-2.ebr2.Washington1.Level3.net (4.69.132.70) 18.608 ms 18.961 ms 18.583 ms 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * ... 4 car1.Montreal2.Level3.net (67.215.0.146) 0.657 ms 0.791 ms 0.699 ms 5 ae-5-5.ebr4.NewYork1.Level3.net (4.69.141.6) 17.764 ms 8.490 ms 18.197 ms 6 ae-94-94.csw4.NewYork1.Level3.net (4.69.134.126) 15.541 ms 8.286 ms 17.098 ms 7 ae-93-93.ebr3.NewYork1.Level3.net (4.69.134.109) 11.384 ms ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65) 9.100 ms 8.614 ms 8 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 13.840 ms 15.584 ms 17.443 ms 9 ae-94-94.csw4.Washington1.Level3.net (4.69.134.190) 23.420 ms 25.569 ms 18.042 ms 10 ae-4-99.edge2.Washington4.Level3.net (4.68.17.211) 14.052 ms 14.028 ms 13.610 ms 11 * * * 12 * * * 13 * * * 14 * * * 15 * Paul Stewart wrote:
Ahh.. yes seeing that now here from Toronto ON - didn't see this issue when the original poster sent the first message... it's now happening here too...
Shutting down their session until something looks "better"
-----Original Message----- From: Pierre-Henri [mailto:phach34@gmail.com] Sent: December 28, 2008 1:06 PM To: marco Cc: nanog@nanog.org Subject: Re: Level 3 issues
marco a écrit :
is anyone having issues with Level3?
hi, theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes. Don't know if there is a link with your question, but it's strange...
Pierre-Henri
----------------------------------------------------------------------------
"The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you."
-- GloboTech Communications Phone: 1-514-907-0050 x 215 Toll Free: 1-(888)-GTCOMM1 Fax: 1-(514)-907-0750 paul@gtcomm.net http://www.gtcomm.net
Paul wrote:
Same issue here from Chicago and Montreal. Seems anything routing through Washington.Level3 is going to null. The rest of the level3 network seems to be ok. 6 ae-32-52.ebr2.Chicago1.Level3.net (4.68.101.62) 0.976 ms 10.344 ms 0.866 ms 7 ae-5.ebr2.Chicago2.Level3.net (4.69.140.194) 1.245 ms 0.991 ms 0.978 ms 8 ae-2-2.ebr2.Washington1.Level3.net (4.69.132.70) 18.608 ms 18.961 ms 18.583 ms 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * *
... 4 car1.Montreal2.Level3.net (67.215.0.146) 0.657 ms 0.791 ms 0.699 ms 5 ae-5-5.ebr4.NewYork1.Level3.net (4.69.141.6) 17.764 ms 8.490 ms 18.197 ms 6 ae-94-94.csw4.NewYork1.Level3.net (4.69.134.126) 15.541 ms 8.286 ms 17.098 ms 7 ae-93-93.ebr3.NewYork1.Level3.net (4.69.134.109) 11.384 ms ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65) 9.100 ms 8.614 ms 8 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 13.840 ms 15.584 ms 17.443 ms 9 ae-94-94.csw4.Washington1.Level3.net (4.69.134.190) 23.420 ms 25.569 ms 18.042 ms 10 ae-4-99.edge2.Washington4.Level3.net (4.68.17.211) 14.052 ms 14.028 ms 13.610 ms 11 * * * 12 * * * 13 * * * 14 * * * 15 *
Paul Stewart wrote:
Ahh.. yes seeing that now here from Toronto ON - didn't see this issue when the original poster sent the first message... it's now happening here too...
Shutting down their session until something looks "better"
-----Original Message----- From: Pierre-Henri [mailto:phach34@gmail.com] Sent: December 28, 2008 1:06 PM To: marco Cc: nanog@nanog.org Subject: Re: Level 3 issues
marco a écrit :
is anyone having issues with Level3?
hi, theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes. Don't know if there is a link with your question, but it's strange...
Pierre-Henri
----------------------------------------------------------------------------
"The information transmitsted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you."
According to L3, this issue should be fixed and we should start seeing the traffic normalizing. Can anyone confirm?
According to L3, this issue should be fixed and we should start seeing
the traffic normalizing. Can anyone confirm?
Here in Richmond Virginia, everything seems to be back to normal now. Traffic coming from my Comcast connection can get through L3 now. 7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net [68. 86.91.121] 8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net [4.79.231 .9] 9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net [4.68.17.190] 10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net [4.69.134.1 85] 11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net [4.69.132.94] 12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net [4.69.134.74] 13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net [4.68.16.142]
Confirmed here as well. Jon ----- Original Message ----- From: "Jason Cheslock" <sangreviento@gmail.com> To: "marco" <marco@zero11.com> Cc: nanog@nanog.org Sent: Sunday, December 28, 2008 1:35:45 PM GMT -05:00 US/Canada Eastern Subject: Re: Level 3 issues According to L3, this issue should be fixed and we should start seeing
the traffic normalizing. Can anyone confirm?
Here in Richmond Virginia, everything seems to be back to normal now. Traffic coming from my Comcast connection can get through L3 now. 7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net [68. 86.91.121] 8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net [4.79.231 .9] 9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net [4.68.17.190] 10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net [4.69.134.1 85] 11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net [4.69.132.94] 12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net [4.69.134.74] 13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net [4.68.16.142]
Seems to be normalizing here in Colorado as well, however still having occasional packet loss to NY. -----Original Message----- From: Jon Wolberg [mailto:jon@defenderhosting.com] Sent: Sunday, December 28, 2008 11:40 AM To: Jason Cheslock Cc: nanog@nanog.org Subject: Re: Level 3 issues Confirmed here as well. Jon ----- Original Message ----- From: "Jason Cheslock" <sangreviento@gmail.com> To: "marco" <marco@zero11.com> Cc: nanog@nanog.org Sent: Sunday, December 28, 2008 1:35:45 PM GMT -05:00 US/Canada Eastern Subject: Re: Level 3 issues According to L3, this issue should be fixed and we should start seeing
the traffic normalizing. Can anyone confirm?
Here in Richmond Virginia, everything seems to be back to normal now. Traffic coming from my Comcast connection can get through L3 now. 7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net [68. 86.91.121] 8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net [4.79.231 .9] 9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net [4.68.17.190] 10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net [4.69.134.1 85] 11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net [4.69.132.94] 12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net [4.69.134.74] 13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net [4.68.16.142]
Looks like most providers here in the east coast are routing through level3 again, and I'm not seeing any packet loss or latency anymore. On Sun, Dec 28, 2008 at 1:47 PM, Blake Pfankuch <bpfankuch@cpgreeley.com>wrote:
Seems to be normalizing here in Colorado as well, however still having occasional packet loss to NY.
-----Original Message----- From: Jon Wolberg [mailto:jon@defenderhosting.com] Sent: Sunday, December 28, 2008 11:40 AM To: Jason Cheslock Cc: nanog@nanog.org Subject: Re: Level 3 issues
Confirmed here as well.
Jon
----- Original Message ----- From: "Jason Cheslock" <sangreviento@gmail.com> To: "marco" <marco@zero11.com> Cc: nanog@nanog.org Sent: Sunday, December 28, 2008 1:35:45 PM GMT -05:00 US/Canada Eastern Subject: Re: Level 3 issues
According to L3, this issue should be fixed and we should start seeing
the traffic normalizing. Can anyone confirm?
Here in Richmond Virginia, everything seems to be back to normal now. Traffic coming from my Comcast connection can get through L3 now.
7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net [68. 86.91.121] 8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net [4.79.231 .9] 9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net [4.68.17.190]
10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net [4.69.134.1 85] 11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net [4.69.132.94] 12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net [4.69.134.74]
13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net [4.68.16.142]
-- Derek Bodner subscribedlists@derekbodner.com
Any word on the actual cause of the issue? From: Derek Bodner [mailto:subscribedlists@derekbodner.com] Sent: Sunday, December 28, 2008 11:53 AM To: Blake Pfankuch Cc: Jon Wolberg; Jason Cheslock; nanog@nanog.org Subject: Re: Level 3 issues Looks like most providers here in the east coast are routing through level3 again, and I'm not seeing any packet loss or latency anymore. On Sun, Dec 28, 2008 at 1:47 PM, Blake Pfankuch <bpfankuch@cpgreeley.com<mailto:bpfankuch@cpgreeley.com>> wrote: Seems to be normalizing here in Colorado as well, however still having occasional packet loss to NY. -----Original Message----- From: Jon Wolberg [mailto:jon@defenderhosting.com<mailto:jon@defenderhosting.com>] Sent: Sunday, December 28, 2008 11:40 AM To: Jason Cheslock Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Level 3 issues Confirmed here as well. Jon ----- Original Message ----- From: "Jason Cheslock" <sangreviento@gmail.com<mailto:sangreviento@gmail.com>> To: "marco" <marco@zero11.com<mailto:marco@zero11.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Sunday, December 28, 2008 1:35:45 PM GMT -05:00 US/Canada Eastern Subject: Re: Level 3 issues According to L3, this issue should be fixed and we should start seeing
the traffic normalizing. Can anyone confirm?
Here in Richmond Virginia, everything seems to be back to normal now. Traffic coming from my Comcast connection can get through L3 now. 7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net<http://te-0-3-0-0-cr01.mclean.va.ibone.comcast.net> [68. 86.91.121] 8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net<http://xe-11-1-0.edge1.Washington1.Level3.net> [4.79.231 .9] 9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net<http://vlan89.csw3.Washington1.Level3.net> [4.68.17.190] 10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net<http://ae-84-84.ebr4.Washington1.Level3.net> [4.69.134.1 85] 11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net<http://ae-3-3.ebr1.NewYork1.Level3.net> [4.69.132.94] 12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net<http://ae-81-81.csw3.NewYork1.Level3.net> [4.69.134.74] 13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net<http://ae-3-89.edge1.NewYork1.Level3.net> [4.68.16.142] -- Derek Bodner subscribedlists@derekbodner.com<mailto:subscribedlists@derekbodner.com>
Blake Pfankuch wrote:
Any word on the actual cause of the issue?
From: Derek Bodner [mailto:subscribedlists@derekbodner.com] Sent: Sunday, December 28, 2008 11:53 AM To: Blake Pfankuch Cc: Jon Wolberg; Jason Cheslock; nanog@nanog.org Subject: Re: Level 3 issues
Looks like most providers here in the east coast are routing through level3 again, and I'm not seeing any packet loss or latency anymore. On Sun, Dec 28, 2008 at 1:47 PM, Blake Pfankuch <bpfankuch@cpgreeley.com<mailto:bpfankuch@cpgreeley.com>> wrote: Seems to be normalizing here in Colorado as well, however still having occasional packet loss to NY.
-----Original Message----- From: Jon Wolberg [mailto:jon@defenderhosting.com<mailto:jon@defenderhosting.com>] Sent: Sunday, December 28, 2008 11:40 AM To: Jason Cheslock Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Level 3 issues
Confirmed here as well.
Jon
----- Original Message ----- From: "Jason Cheslock" <sangreviento@gmail.com<mailto:sangreviento@gmail.com>> To: "marco" <marco@zero11.com<mailto:marco@zero11.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Sunday, December 28, 2008 1:35:45 PM GMT -05:00 US/Canada Eastern Subject: Re: Level 3 issues
According to L3, this issue should be fixed and we should start seeing
the traffic normalizing. Can anyone confirm?
Here in Richmond Virginia, everything seems to be back to normal now. Traffic coming from my Comcast connection can get through L3 now.
7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net<http://te-0-3-0-0-cr01.mclean.va.ibone.comcast.net> [68. 86.91.121] 8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net<http://xe-11-1-0.edge1.Washington1.Level3.net> [4.79.231 .9] 9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net<http://vlan89.csw3.Washington1.Level3.net> [4.68.17.190]
10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net<http://ae-84-84.ebr4.Washington1.Level3.net> [4.69.134.1 85] 11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net<http://ae-3-3.ebr1.NewYork1.Level3.net> [4.69.132.94] 12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net<http://ae-81-81.csw3.NewYork1.Level3.net> [4.69.134.74]
13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net<http://ae-3-89.edge1.NewYork1.Level3.net> [4.68.16.142]
-- Derek Bodner subscribedlists@derekbodner.com<mailto:subscribedlists@derekbodner.com>
From what I heard, it was some some malfunction with a router in Washington D.C. which terminated a 100GB bundle from Paris. It was carring about 50GB at the time of the failure.
Not sure why routes within the US would be effected.
marco wrote:
From what I heard, it was some some malfunction with a router in Washington D.C. which terminated a 100GB bundle from Paris. It was carring about 50GB at the time of the failure.
Not sure why routes within the US would be effected.
We connect to level3 in Ashburn/DC and saw traffic drop 50% in both directions on that port. Testing showed 100% loss on 50% of the flows. We shut that port down and now it won't come back up. I have link but no arp for their IP. This is a new link that was turned up in the past few weeks. - Kevin
We saw our bandwidth drop on our Level3 OC-48 to about half of what we were doing. We had to stop announcing our subnets to Level3 to get traffic to fail over properly throughout the world. We have a ticket open with Level3's NOC but have not received word on what happened or when to expect a resolution. Kevin Loch wrote:
marco wrote:
From what I heard, it was some some malfunction with a router in Washington D.C. which terminated a 100GB bundle from Paris. It was carring about 50GB at the time of the failure.
Not sure why routes within the US would be effected.
We connect to level3 in Ashburn/DC and saw traffic drop 50% in both directions on that port. Testing showed 100% loss on 50% of the flows. We shut that port down and now it won't come back up. I have link but no arp for their IP. This is a new link that was turned up in the past few weeks.
- Kevin
-- Steve King Network Engineer - Liquid Web, Inc. Cisco Certified Network Associate CompTIA Linux+ Certified Professional CompTIA A+ Certified Professional
Steven King wrote:
We saw our bandwidth drop on our Level3 OC-48 to about half of what we were doing. We had to stop announcing our subnets to Level3 to get traffic to fail over properly throughout the world. We have a ticket open with Level3's NOC but have not received word on what happened or when to expect a resolution.
Kevin Loch wrote:
marco wrote:
From what I heard, it was some some malfunction with a router in
Washington D.C. which terminated a 100GB bundle from Paris. It was carring about 50GB at the time of the failure.
Not sure why routes within the US would be effected.
We connect to level3 in Ashburn/DC and saw traffic drop 50% in both directions on that port. Testing showed 100% loss on 50% of the flows. We shut that port down and now it won't come back up. I have link but no arp for their IP. This is a new link that was turned up in the past few weeks.
- Kevin
http://www.internetpulse.net/Main.aspx shows that everything is back to normal.
Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Kevin Loch [mailto:kloch@kl.net] Sent: Sunday, December 28, 2008 12:22 PM Cc: nanog@nanog.org Subject: Re: Level 3 issues marco wrote:
From what I heard, it was some some malfunction with a router in Washington D.C. which terminated a 100GB bundle from Paris. It was carring about 50GB at the time of the failure.
Not sure why routes within the US would be effected. <<<<<<<<<<<<<<<<<<<< did ya think about BGP extensions... It could be a host of things, like communitites, MEDs, you name it.
We connect to level3 in Ashburn/DC and saw traffic drop 50% in both directions on that port. Testing showed 100% loss on 50% of the flows. We shut that port down and now it won't come back up. I have link but no arp for their IP. This is a new link that was turned up in the past few weeks. - Kevin ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
On 12/28/08, Blake Pfankuch <bpfankuch@cpgreeley.com> wrote:
Any word on the actual cause of the issue?
Given the lurking presence of wannabe press vultures here, I doubt you'll see anything forthcoming from the technical folks about what actually happened. This is not to say that people haven't been informed of the issue, it's simply that NANOG is no longer a friendly hapy techie-only place where such information can be shared without it being seized on and quoted without permission by the press. Bitter? Just a bit, yes. After having what I mentioned here get quoted by the press without permission, it's very clear that there will be no more technical commentary/feedback/information flow through this channel from me or any of the other people at the company for which I work, more's the pity. If you can find a forum where engineers can share information freely without having to worry about being quoted by the press, you might try enquiring there. ^_^; Matt
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 IMHO, this is exactly what service providers love to hear in order for them not to be forth coming. regards, /virendra Matthew Petach wrote:
On 12/28/08, Blake Pfankuch <bpfankuch@cpgreeley.com> wrote:
Any word on the actual cause of the issue?
Given the lurking presence of wannabe press vultures here, I doubt you'll see anything forthcoming from the technical folks about what actually happened. This is not to say that people haven't been informed of the issue, it's simply that NANOG is no longer a friendly hapy techie-only place where such information can be shared without it being seized on and quoted without permission by the press.
Bitter? Just a bit, yes. After having what I mentioned here get quoted by the press without permission, it's very clear that there will be no more technical commentary/feedback/information flow through this channel from me or any of the other people at the company for which I work, more's the pity. If you can find a forum where engineers can share information freely without having to worry about being quoted by the press, you might try enquiring there. ^_^;
Matt
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJV+UUpbZvCIJx1bcRAoxfAKCFn9inJ1Pq2Z0W9uuu1YeMGjf0LACeJVca GkejywYfBp2yUjTFJwrOLFU= =d1BQ -----END PGP SIGNATURE-----
On Sun, Dec 28, 2008 at 11:21:50AM -0800, Matthew Petach wrote:
Given the lurking presence of wannabe press vultures here, I doubt you'll see anything forthcoming from the technical folks about what actually happened. This is not to say that people haven't been informed of the issue, it's simply that NANOG is no longer a friendly hapy techie-only place where such information can be shared without it being seized on and quoted without permission by the press.
It was good advice then; it's even better advice now: Never say anything in an electronic message that you wouldn't want appearing, and attributed to you, in tomorrow morning's front-page headline in the New York Times. --- Colonel David Russell, former head of DARPA's Information Processing Techniques Office ---Rsk
It seems that there was fiber cut because of train derailment around NY area. Alex Blake Pfankuch wrote:
Any word on the actual cause of the issue?
From: Derek Bodner [mailto:subscribedlists@derekbodner.com] Sent: Sunday, December 28, 2008 11:53 AM To: Blake Pfankuch Cc: Jon Wolberg; Jason Cheslock; nanog@nanog.org Subject: Re: Level 3 issues
Looks like most providers here in the east coast are routing through level3 again, and I'm not seeing any packet loss or latency anymore. On Sun, Dec 28, 2008 at 1:47 PM, Blake Pfankuch <bpfankuch@cpgreeley.com<mailto:bpfankuch@cpgreeley.com>> wrote: Seems to be normalizing here in Colorado as well, however still having occasional packet loss to NY.
-----Original Message----- From: Jon Wolberg [mailto:jon@defenderhosting.com<mailto:jon@defenderhosting.com>] Sent: Sunday, December 28, 2008 11:40 AM To: Jason Cheslock Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Level 3 issues
Confirmed here as well.
Jon
----- Original Message ----- From: "Jason Cheslock" <sangreviento@gmail.com<mailto:sangreviento@gmail.com>> To: "marco" <marco@zero11.com<mailto:marco@zero11.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Sunday, December 28, 2008 1:35:45 PM GMT -05:00 US/Canada Eastern Subject: Re: Level 3 issues
According to L3, this issue should be fixed and we should start seeing
the traffic normalizing. Can anyone confirm?
Here in Richmond Virginia, everything seems to be back to normal now. Traffic coming from my Comcast connection can get through L3 now.
7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net<http://te-0-3-0-0-cr01.mclean.va.ibone.comcast.net> [68. 86.91.121] 8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net<http://xe-11-1-0.edge1.Washington1.Level3.net> [4.79.231 .9] 9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net<http://vlan89.csw3.Washington1.Level3.net> [4.68.17.190]
10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net<http://ae-84-84.ebr4.Washington1.Level3.net> [4.69.134.1 85] 11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net<http://ae-3-3.ebr1.NewYork1.Level3.net> [4.69.132.94] 12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net<http://ae-81-81.csw3.NewYork1.Level3.net> [4.69.134.74]
13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net<http://ae-3-89.edge1.NewYork1.Level3.net> [4.68.16.142]
-- Derek Bodner subscribedlists@derekbodner.com<mailto:subscribedlists@derekbodner.com>
I have heard this story several times. The train derailment was yesterday in New York unless it has not made it to news.google.com on a search for train derail. Issues did not start until 1030 MST. It seems highly unlikely that a train derailment yesterday caused major network issues today. -----Original Message----- From: Alex H. Ryu [mailto:r.hyunseog@ieee.org] Sent: Sunday, December 28, 2008 2:44 PM To: Blake Pfankuch Cc: Derek Bodner; nanog@nanog.org Subject: Re: Level 3 issues It seems that there was fiber cut because of train derailment around NY area. Alex Blake Pfankuch wrote:
Any word on the actual cause of the issue?
From: Derek Bodner [mailto:subscribedlists@derekbodner.com] Sent: Sunday, December 28, 2008 11:53 AM To: Blake Pfankuch Cc: Jon Wolberg; Jason Cheslock; nanog@nanog.org Subject: Re: Level 3 issues
Looks like most providers here in the east coast are routing through level3 again, and I'm not seeing any packet loss or latency anymore. On Sun, Dec 28, 2008 at 1:47 PM, Blake Pfankuch <bpfankuch@cpgreeley.com<mailto:bpfankuch@cpgreeley.com>> wrote: Seems to be normalizing here in Colorado as well, however still having occasional packet loss to NY.
-----Original Message----- From: Jon Wolberg [mailto:jon@defenderhosting.com<mailto:jon@defenderhosting.com>] Sent: Sunday, December 28, 2008 11:40 AM To: Jason Cheslock Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Level 3 issues
Confirmed here as well.
Jon
----- Original Message ----- From: "Jason Cheslock" <sangreviento@gmail.com<mailto:sangreviento@gmail.com>> To: "marco" <marco@zero11.com<mailto:marco@zero11.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Sunday, December 28, 2008 1:35:45 PM GMT -05:00 US/Canada Eastern Subject: Re: Level 3 issues
According to L3, this issue should be fixed and we should start seeing
the traffic normalizing. Can anyone confirm?
Here in Richmond Virginia, everything seems to be back to normal now. Traffic coming from my Comcast connection can get through L3 now.
7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net<http://te-0-3-0-0-cr01.mclean.va.ibone.comcast.net> [68. 86.91.121] 8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net<http://xe-11-1-0.edge1.Washington1.Level3.net> [4.79.231 .9] 9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net<http://vlan89.csw3.Washington1.Level3.net> [4.68.17.190]
10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net<http://ae-84-84.ebr4.Washington1.Level3.net> [4.69.134.1 85] 11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net<http://ae-3-3.ebr1.NewYork1.Level3.net> [4.69.132.94] 12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net<http://ae-81-81.csw3.NewYork1.Level3.net> [4.69.134.74]
13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net<http://ae-3-89.edge1.NewYork1.Level3.net> [4.68.16.142]
-- Derek Bodner subscribedlists@derekbodner.com<mailto:subscribedlists@derekbodner.com>
On Sun, Dec 28, 2008 at 17:17, Blake Pfankuch <bpfankuch@cpgreeley.com> wrote:
It seems highly unlikely that a train derailment yesterday caused major network issues today.
Have you ever seen cleanup efforts after a major accident. Cleanup usually involves more backhoes, and other major equipment, than a normal well planned construction site. -Jim P.
On Sun, 28 Dec 2008, Jim Popovitch wrote:
On Sun, Dec 28, 2008 at 17:17, Blake Pfankuch <bpfankuch@cpgreeley.com> wrote:
It seems highly unlikely that a train derailment yesterday caused major network issues today.
Have you ever seen cleanup efforts after a major accident. Cleanup usually involves more backhoes, and other major equipment, than a normal well planned construction site.
True enough. In their haste to clean up from that derailment, it's possible that the backhoes severed the other side of that diverse (cough) optical ring, which was probably on the other side of the same set of railroad tracks, if not in the same conduit/bank as the cable that got hit :) jms
You know, it gets pretty thick through here, when all you people slam on someone, to justify pent up angst or whatever the cause may be. I worked for Level 3 as a NOC engr, and they follow standards as other companies do, and for that matter a standard that I AM SURE all of you follow in some form, degree, or another within the employ you are with, so shut up already won't you. Give me a 10-minute break already. Half of the crap that you guys serve up is crap, just that, CRAP! Get to talking real 'net stuff, not filler, fodder, just facts man. Oh yeah the other thing, quit your whining. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Blake Pfankuch [mailto:bpfankuch@cpgreeley.com] Sent: Sunday, December 28, 2008 3:18 PM To: Alex H. Ryu Cc: nanog@nanog.org Subject: RE: Level 3 issues I have heard this story several times. The train derailment was yesterday in New York unless it has not made it to news.google.com on a search for train derail. Issues did not start until 1030 MST. It seems highly unlikely that a train derailment yesterday caused major network issues today. -----Original Message----- From: Alex H. Ryu [mailto:r.hyunseog@ieee.org] Sent: Sunday, December 28, 2008 2:44 PM To: Blake Pfankuch Cc: Derek Bodner; nanog@nanog.org Subject: Re: Level 3 issues It seems that there was fiber cut because of train derailment around NY area. Alex Blake Pfankuch wrote:
Any word on the actual cause of the issue?
From: Derek Bodner [mailto:subscribedlists@derekbodner.com] Sent: Sunday, December 28, 2008 11:53 AM To: Blake Pfankuch Cc: Jon Wolberg; Jason Cheslock; nanog@nanog.org Subject: Re: Level 3 issues
Looks like most providers here in the east coast are routing through level3 again, and I'm not seeing any packet loss or latency anymore. On Sun, Dec 28, 2008 at 1:47 PM, Blake Pfankuch <bpfankuch@cpgreeley.com<mailto:bpfankuch@cpgreeley.com>> wrote: Seems to be normalizing here in Colorado as well, however still having occasional packet loss to NY.
-----Original Message----- From: Jon Wolberg [mailto:jon@defenderhosting.com<mailto:jon@defenderhosting.com>] Sent: Sunday, December 28, 2008 11:40 AM To: Jason Cheslock Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Level 3 issues
Confirmed here as well.
Jon
----- Original Message ----- From: "Jason Cheslock" <sangreviento@gmail.com<mailto:sangreviento@gmail.com>> To: "marco" <marco@zero11.com<mailto:marco@zero11.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Sent: Sunday, December 28, 2008 1:35:45 PM GMT -05:00 US/Canada Eastern Subject: Re: Level 3 issues
According to L3, this issue should be fixed and we should start seeing
the traffic normalizing. Can anyone confirm?
Here in Richmond Virginia, everything seems to be back to normal now. Traffic coming from my Comcast connection can get through L3 now.
7 11 ms 13 ms 11 ms te-0-3-0-0-cr01.mclean.va.ibone.comcast.net<http://te-0-3-0-0-cr01.mclea n.va.ibone.comcast.net> [68. 86.91.121] 8 10 ms 11 ms 12 ms xe-11-1-0.edge1.Washington1.Level3.net<http://xe-11-1-0.edge1.Washingt on1.Level3.net> [4.79.231 .9] 9 12 ms 17 ms 18 ms vlan89.csw3.Washington1.Level3.net<http://vlan89.csw3.Washington1.Leve l3.net> [4.68.17.190]
10 12 ms 17 ms 17 ms ae-84-84.ebr4.Washington1.Level3.net<http://ae-84-84.ebr4.Washington1. Level3.net> [4.69.134.1 85] 11 16 ms 26 ms 16 ms ae-3-3.ebr1.NewYork1.Level3.net<http://ae-3-3.ebr1.NewYork1.Level3.net
[4.69.132.94] 12 32 ms 30 ms 17 ms ae-81-81.csw3.NewYork1.Level3.net<http://ae-81-81.csw3.NewYork1.Level3 .net> [4.69.134.74]
13 15 ms 19 ms 16 ms ae-3-89.edge1.NewYork1.Level3.net<http://ae-3-89.edge1.NewYork1.Level3 .net> [4.68.16.142]
-- Derek Bodner subscribedlists@derekbodner.com<mailto:subscribedlists@derekbodner.com
______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
On Mon, Dec 29, 2008 at 11:33 AM, Murphy, Jay, DOH <Jay.Murphy@state.nm.us> wrote:
You know, it gets pretty thick through here, when all you people slam on someone, to justify pent up angst or whatever the cause may be. I worked for Level 3 as a NOC engr, and they follow standards as other companies do, and for that matter a standard that I AM SURE all of you follow in some form, degree, or another within the employ you are with, so shut up already won't you. Give me a 10-minute break already. Half of the crap that you guys serve up is crap, just that, CRAP! Get to talking real 'net stuff, not filler, fodder, just facts man. Oh yeah the other thing, quit your whining.
<ironylol/> -- -- Todd Vierling <tv@duh.org> <tv@pobox.com> <todd@vierling.name>
I think that most of the anger was directed at the wanna-be reporters/journalists that visit this list. Todd Vierling wrote:
On Mon, Dec 29, 2008 at 11:33 AM, Murphy, Jay, DOH <Jay.Murphy@state.nm.us> wrote:
You know, it gets pretty thick through here, when all you people slam on someone, to justify pent up angst or whatever the cause may be. I worked for Level 3 as a NOC engr, and they follow standards as other companies do, and for that matter a standard that I AM SURE all of you follow in some form, degree, or another within the employ you are with, so shut up already won't you. Give me a 10-minute break already. Half of the crap that you guys serve up is crap, just that, CRAP! Get to talking real 'net stuff, not filler, fodder, just facts man. Oh yeah the other thing, quit your whining.
<ironylol/>
Duly meditated upon. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 "We move the information that moves your world." -----Original Message----- From: marco [mailto:marco@zero11.com] Sent: Monday, December 29, 2008 11:20 AM To: nanog@nanog.org Subject: Re: Level 3 issues I think that most of the anger was directed at the wanna-be reporters/journalists that visit this list. Todd Vierling wrote:
On Mon, Dec 29, 2008 at 11:33 AM, Murphy, Jay, DOH <Jay.Murphy@state.nm.us> wrote:
You know, it gets pretty thick through here, when all you people slam
on someone, to justify pent up angst or whatever the cause may be. I
worked for Level 3 as a NOC engr, and they follow standards as other companies do, and for that matter a standard that I AM SURE all of you follow in some form, degree, or another within the employ you are
with, so shut up already won't you. Give me a 10-minute break already. Half of the crap that you guys serve up is crap, just that, CRAP! Get to talking real 'net stuff, not filler, fodder, just facts man. Oh yeah the other thing, quit your whining.
<ironylol/>
______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
Dennis Springer Network Engineer III Charter Communications 12405 Powerscourt Dr. 2nd floor St. Louis, MO 63131 Phone: (314) 288-3425 Cell: (314) 607-9824 -----Original Message----- From: Murphy, Jay, DOH [mailto:Jay.Murphy@state.nm.us] Sent: Monday, December 29, 2008 12:26 PM To: marco; nanog@nanog.org Subject: RE: Level 3 issues Duly meditated upon. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 "We move the information that moves your world." -----Original Message----- From: marco [mailto:marco@zero11.com] Sent: Monday, December 29, 2008 11:20 AM To: nanog@nanog.org Subject: Re: Level 3 issues I think that most of the anger was directed at the wanna-be reporters/journalists that visit this list. Todd Vierling wrote:
On Mon, Dec 29, 2008 at 11:33 AM, Murphy, Jay, DOH <Jay.Murphy@state.nm.us> wrote:
You know, it gets pretty thick through here, when all you people slam
on someone, to justify pent up angst or whatever the cause may be. I
worked for Level 3 as a NOC engr, and they follow standards as other companies do, and for that matter a standard that I AM SURE all of you follow in some form, degree, or another within the employ you are
with, so shut up already won't you. Give me a 10-minute break already. Half of the crap that you guys serve up is crap, just that, CRAP! Get to talking real 'net stuff, not filler, fodder, just facts man. Oh yeah the other thing, quit your whining.
<ironylol/>
______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Springer, Dennis D [mailto:Dennis.Springer@chartercom.com] Sent: Monday, December 29, 2008 3:07 PM To: nanog@nanog.org Subject: unsubscribe Dennis Springer Network Engineer III Charter Communications 12405 Powerscourt Dr. 2nd floor St. Louis, MO 63131 Phone: (314) 288-3425 Cell: (314) 607-9824 -----Original Message----- From: Murphy, Jay, DOH [mailto:Jay.Murphy@state.nm.us] Sent: Monday, December 29, 2008 12:26 PM To: marco; nanog@nanog.org Subject: RE: Level 3 issues Duly meditated upon. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 "We move the information that moves your world." -----Original Message----- From: marco [mailto:marco@zero11.com] Sent: Monday, December 29, 2008 11:20 AM To: nanog@nanog.org Subject: Re: Level 3 issues I think that most of the anger was directed at the wanna-be reporters/journalists that visit this list. Todd Vierling wrote:
On Mon, Dec 29, 2008 at 11:33 AM, Murphy, Jay, DOH <Jay.Murphy@state.nm.us> wrote:
You know, it gets pretty thick through here, when all you people slam
on someone, to justify pent up angst or whatever the cause may be. I
worked for Level 3 as a NOC engr, and they follow standards as other companies do, and for that matter a standard that I AM SURE all of you follow in some form, degree, or another within the employ you are
with, so shut up already won't you. Give me a 10-minute break already. Half of the crap that you guys serve up is crap, just that, CRAP! Get to talking real 'net stuff, not filler, fodder, just facts man. Oh yeah the other thing, quit your whining.
<ironylol/>
______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________
nanog-request@merit.edu?body=unsubscribe To Unsubscribe. James Thomas -----Original Message----- From: Murphy, Jay, DOH [mailto:Jay.Murphy@state.nm.us] Sent: Monday, December 29, 2008 5:31 PM To: Springer, Dennis D; nanog@nanog.org Subject: RE: unsubscribe Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Springer, Dennis D [mailto:Dennis.Springer@chartercom.com] Sent: Monday, December 29, 2008 3:07 PM To: nanog@nanog.org Subject: unsubscribe Dennis Springer Network Engineer III Charter Communications 12405 Powerscourt Dr. 2nd floor St. Louis, MO 63131 Phone: (314) 288-3425 Cell: (314) 607-9824 -----Original Message----- From: Murphy, Jay, DOH [mailto:Jay.Murphy@state.nm.us] Sent: Monday, December 29, 2008 12:26 PM To: marco; nanog@nanog.org Subject: RE: Level 3 issues Duly meditated upon. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 "We move the information that moves your world." -----Original Message----- From: marco [mailto:marco@zero11.com] Sent: Monday, December 29, 2008 11:20 AM To: nanog@nanog.org Subject: Re: Level 3 issues I think that most of the anger was directed at the wanna-be reporters/journalists that visit this list. Todd Vierling wrote:
On Mon, Dec 29, 2008 at 11:33 AM, Murphy, Jay, DOH <Jay.Murphy@state.nm.us> wrote:
You know, it gets pretty thick through here, when all you people slam
on someone, to justify pent up angst or whatever the cause may be. I
worked for Level 3 as a NOC engr, and they follow standards as other companies do, and for that matter a standard that I AM SURE all of you follow in some form, degree, or another within the employ you are
with, so shut up already won't you. Give me a 10-minute break already. Half of the crap that you guys serve up is crap, just that, CRAP! Get to talking real 'net stuff, not filler, fodder, just facts man. Oh yeah the other thing, quit your whining.
<ironylol/>
______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________
Pardons, several keys were depressed by mechanical miscalculation. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: James Thomas [mailto:jthomas@crucialservers.net] Sent: Monday, December 29, 2008 3:41 PM To: Murphy, Jay, DOH; 'Springer, Dennis D'; nanog@nanog.org Subject: RE: unsubscribe nanog-request@merit.edu?body=unsubscribe To Unsubscribe. James Thomas -----Original Message----- From: Murphy, Jay, DOH [mailto:Jay.Murphy@state.nm.us] Sent: Monday, December 29, 2008 5:31 PM To: Springer, Dennis D; nanog@nanog.org Subject: RE: unsubscribe Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Springer, Dennis D [mailto:Dennis.Springer@chartercom.com] Sent: Monday, December 29, 2008 3:07 PM To: nanog@nanog.org Subject: unsubscribe Dennis Springer Network Engineer III Charter Communications 12405 Powerscourt Dr. 2nd floor St. Louis, MO 63131 Phone: (314) 288-3425 Cell: (314) 607-9824 -----Original Message----- From: Murphy, Jay, DOH [mailto:Jay.Murphy@state.nm.us] Sent: Monday, December 29, 2008 12:26 PM To: marco; nanog@nanog.org Subject: RE: Level 3 issues Duly meditated upon. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 "We move the information that moves your world." -----Original Message----- From: marco [mailto:marco@zero11.com] Sent: Monday, December 29, 2008 11:20 AM To: nanog@nanog.org Subject: Re: Level 3 issues I think that most of the anger was directed at the wanna-be reporters/journalists that visit this list. Todd Vierling wrote:
On Mon, Dec 29, 2008 at 11:33 AM, Murphy, Jay, DOH <Jay.Murphy@state.nm.us> wrote:
You know, it gets pretty thick through here, when all you people slam
on someone, to justify pent up angst or whatever the cause may be. I
worked for Level 3 as a NOC engr, and they follow standards as other companies do, and for that matter a standard that I AM SURE all of you follow in some form, degree, or another within the employ you are
with, so shut up already won't you. Give me a 10-minute break already. Half of the crap that you guys serve up is crap, just that, CRAP! Get to talking real 'net stuff, not filler, fodder, just facts man. Oh yeah the other thing, quit your whining.
<ironylol/>
______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
On 12/29/08, marco <marco@zero11.com> wrote:
I think that most of the anger was directed at the wanna-be reporters/journalists that visit this list.
Yes. A nice set of examples of why that anger exists can be found here: http://news.search.yahoo.com/news/search?p=petach+kablooie :( Matt
Todd Vierling wrote:
On Mon, Dec 29, 2008 at 11:33 AM, Murphy, Jay, DOH <Jay.Murphy@state.nm.us> wrote:
You know, it gets pretty thick through here, when all you people slam on someone, to justify pent up angst or whatever the cause may be. I worked for Level 3 as a NOC engr, and they follow standards as other companies do, and for that matter a standard that I AM SURE all of you follow in some form, degree, or another within the employ you are with, so shut up already won't you. Give me a 10-minute break already. Half of the crap that you guys serve up is crap, just that, CRAP! Get to talking real 'net stuff, not filler, fodder, just facts man. Oh yeah the other thing, quit your whining.
<ironylol/>
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Yo Matt! On Mon, 29 Dec 2008, Matthew Petach wrote:
Yes. A nice set of examples of why that anger exists can be found here: http://news.search.yahoo.com/news/search?p=petach+kablooie
Well... I sorta liked that story. I did not think you came off badly in it either. RGDS GARY - --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97701 gem@rellim.com Tel:+1(541)382-8588 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) iD8DBQFJWViQBmnRqz71OvMRAr0uAJ9fwOgXpktikAnkzZfdYLpJqsGmCgCfX/H7 MBBrbspBEOWMmTcgDX7LK9M= =7bbY -----END PGP SIGNATURE-----
2008/12/28 marco <marco@zero11.com>
Paul wrote:
Same issue here from Chicago and Montreal. Seems anything routing through Washington.Level3 is going to null. The rest of the level3 network seems to be ok. 6 ae-32-52.ebr2.Chicago1.Level3.net (4.68.101.62) 0.976 ms 10.344 ms 0.866 ms 7 ae-5.ebr2.Chicago2.Level3.net (4.69.140.194) 1.245 ms 0.991 ms 0.978 ms 8 ae-2-2.ebr2.Washington1.Level3.net (4.69.132.70) 18.608 ms 18.961 ms 18.583 ms 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * *
... 4 car1.Montreal2.Level3.net (67.215.0.146) 0.657 ms 0.791 ms 0.699 ms 5 ae-5-5.ebr4.NewYork1.Level3.net (4.69.141.6) 17.764 ms 8.490 ms 18.197 ms 6 ae-94-94.csw4.NewYork1.Level3.net (4.69.134.126) 15.541 ms 8.286 ms 17.098 ms 7 ae-93-93.ebr3.NewYork1.Level3.net (4.69.134.109) 11.384 ms ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65) 9.100 ms 8.614 ms 8 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 13.840 ms 15.584 ms 17.443 ms 9 ae-94-94.csw4.Washington1.Level3.net (4.69.134.190) 23.420 ms 25.569 ms 18.042 ms 10 ae-4-99.edge2.Washington4.Level3.net (4.68.17.211) 14.052 ms 14.028 ms 13.610 ms 11 * * * 12 * * * 13 * * * 14 * * * 15 *
Paul Stewart wrote:
Ahh.. yes seeing that now here from Toronto ON - didn't see this issue when the original poster sent the first message... it's now happening here too...
Shutting down their session until something looks "better"
-----Original Message----- From: Pierre-Henri [mailto:phach34@gmail.com] Sent: December 28, 2008 1:06 PM To: marco Cc: nanog@nanog.org Subject: Re: Level 3 issues
marco a écrit :
is anyone having issues with Level3?
hi, theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes. Don't know if there is a link with your question, but it's strange...
Pierre-Henri
"The information transmitsted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you."
According to L3, this issue should be fixed and we should start seeing the traffic normalizing. Can anyone confirm?
Everything seems to be back to normal in France -- Johan Denoyer jdenoy@jdlabs.fr JD Labs Linkedin: www.linkedin.com/in/jdenoy
I'm showing significant latency and loss over my L3 stuff. interetpulse.net showing the same thing too, seems to be a substantial problem. Pierre-Henri wrote:
marco a écrit :
is anyone having issues with Level3?
hi, theplanet.com and many websites (cnn.com ; amazon.com ; ... ) have not been accessible from France (Orange, home connection) for about 30 minutes. Don't know if there is a link with your question, but it's strange...
Pierre-Henri
participants (25)
-
Alex H. Ryu -
Blake Pfankuch -
David Coulson -
Derek Bodner -
Gary E. Miller -
James Thomas -
Jason Cheslock -
Jim Popovitch -
Johan Denoyer -
Jon Wolberg -
Justin M. Streiner -
Kevin Loch -
marco -
Matt Kelly -
Matthew Petach -
Murphy, Jay, DOH -
Paul -
Paul Stewart -
Pierre-Henri -
Rich Kulawiec -
Springer, Dennis D -
Steven King -
Thomas Beecher -
Todd Vierling -
virendra rode