Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail. One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets 4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477 ms 5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H (From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by" I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
Can't get to level3.net 63.211.236.36 or www.level3.net 4.68.95.28 from Pittsburgh either and I peer directly with level3 with a full BGP feed. On Wed, Apr 15, 2009 at 3:35 PM, J. Oquendo <sil@infiltrated.net> wrote:
Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail.
One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets
4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477 ms 5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H
(From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H
Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by"
I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary.
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP
"It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett
227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
--
Have you been able to in the past?? The site is used for other purposes, and the front end site that you will see is www.level3.com, not net. So which one? Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Charles Mills [mailto:w3yni1@gmail.com] Sent: Wednesday, April 15, 2009 1:46 PM To: J. Oquendo Cc: nanog@nanog.org Subject: Re: Level3 funkiness Can't get to level3.net 63.211.236.36 or www.level3.net 4.68.95.28 from Pittsburgh either and I peer directly with level3 with a full BGP feed. On Wed, Apr 15, 2009 at 3:35 PM, J. Oquendo <sil@infiltrated.net> wrote:
Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail.
One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets
4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477
ms
5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H
(From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H
Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by"
I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary.
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP
"It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett
227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
-- ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
On Wed, 15 Apr 2009, Murphy, Jay, DOH wrote:
Have you been able to in the past?? The site is used for other purposes, and the front end site that you will see is www.level3.com, not net. So which one?
Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851
"We move the information that moves your world."
Yes discovered that then thought about reposting full traceroute feeds. It was the *.com I can get through now from 4 out of like 8 addresses. Actually on the phone with Level3 right now =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
Have you been able to in the past?? The site is used for other
Yea the .com addr is on the same subnet, unless it has been carved into a /30. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: J. Oquendo [mailto:sil@infiltrated.net] Sent: Wednesday, April 15, 2009 1:56 PM To: Murphy, Jay, DOH Cc: nanog@nanog.org Subject: Re: Level3 funkiness On Wed, 15 Apr 2009, Murphy, Jay, DOH wrote: purposes,
and the front end site that you will see is www.level3.com, not net. So which one?
Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851
"We move the information that moves your world."
Yes discovered that then thought about reposting full traceroute feeds. It was the *.com I can get through now from 4 out of like 8 addresses. Actually on the phone with Level3 right now =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
* sil@infiltrated.net (J. Oquendo) [Wed 15 Apr 2009, 22:31 CEST]:
Yes discovered that then thought about reposting full traceroute feeds. It was the *.com I can get through now from 4 out of like 8 addresses. Actually on the phone with Level3 right now
Wait, what? Are you seriously calling Level 3, a global operator of IP and transport networks, to ask them about why their *web site* is down? Furrfu. -- Niels.
I don't think you will ever get a "true" answer, maybe someone just forgot to re-reg the domain ;) -----Original Message----- From: Niels Bakker [mailto:niels=nanog@bakker.net] Sent: Wednesday, April 15, 2009 5:13 PM To: nanog@nanog.org Subject: Re: Level3 funkiness * sil@infiltrated.net (J. Oquendo) [Wed 15 Apr 2009, 22:31 CEST]:
Yes discovered that then thought about reposting full traceroute feeds. It was the *.com I can get through now from 4 out of like 8 addresses. Actually on the phone with Level3 right now
Wait, what? Are you seriously calling Level 3, a global operator of IP and transport networks, to ask them about why their *web site* is down? Furrfu. -- Niels.
In Chicago, traceroutes are dying in the same place (Denver). Peered out of 350 Cermak. -brandon On Wed, Apr 15, 2009 at 2:45 PM, Charles Mills <w3yni1@gmail.com> wrote:
Can't get to level3.net 63.211.236.36 or www.level3.net 4.68.95.28 from Pittsburgh either and I peer directly with level3 with a full BGP feed.
On Wed, Apr 15, 2009 at 3:35 PM, J. Oquendo <sil@infiltrated.net> wrote:
Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail.
One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets
4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477
ms
5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H
(From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H
Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by"
I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary.
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP
"It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett
227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
--
-- Brandon Galbraith Voice: 630.400.6992
maybe host problem? I can reach to www.level3.com, but not www.level3.net. It seems both are belonging to same subnet. Brandon Galbraith wrote:
In Chicago, traceroutes are dying in the same place (Denver). Peered out of 350 Cermak.
-brandon
On Wed, Apr 15, 2009 at 2:45 PM, Charles Mills <w3yni1@gmail.com> wrote:
Can't get to level3.net 63.211.236.36 or www.level3.net 4.68.95.28 from Pittsburgh either and I peer directly with level3 with a full BGP feed.
On Wed, Apr 15, 2009 at 3:35 PM, J. Oquendo <sil@infiltrated.net> wrote:
Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail.
One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets
4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477
ms
5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019
ms
10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166
ms
9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616
ms
16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115
ms
38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H
(From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H
Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by"
I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary.
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP
"It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett
227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
--
Can't get to level3.net 63.211.236.36 or www.level3.net 4.68.95.28 from Pittsburgh either and I peer directly with level3 with a full BGP feed.
On Wed, Apr 15, 2009 at 3:35 PM, J. Oquendo <sil@infiltrated.net> wrote:
Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail.
One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte
4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms
21.477 ms
5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H
(From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte
Listen the two are different, level3.com, and level3.net, the two are colo'd at the same place, thus the reason for the Denver "dying" end point. It's .net as you can see; try surfing to 4.6 8.95.11 yes, 4.68.95.28, no...It's just how the DNS PTR for the box is set. It has nothing to do with the naming convention of the routers or other network elements. Are the two confused? Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Brandon Galbraith [mailto:brandon.galbraith@gmail.com] Sent: Wednesday, April 15, 2009 1:54 PM To: Charles Mills Cc: nanog@nanog.org Subject: Re: Level3 funkiness In Chicago, traceroutes are dying in the same place (Denver). Peered out of 350 Cermak. -brandon On Wed, Apr 15, 2009 at 2:45 PM, Charles Mills <w3yni1@gmail.com> wrote: packets packets
11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H
Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by"
I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary.
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP
"It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett
227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
--
-- Brandon Galbraith Voice: 630.400.6992 ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
Yes, I die on your hop14 with TWTelecom -----Original Message----- From: J. Oquendo [mailto:sil@infiltrated.net] Sent: Wednesday, April 15, 2009 3:36 PM To: nanog@nanog.org Subject: Level3 funkiness Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail. One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets 4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477 ms 5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H (From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by" I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
-----Original Message----- From: J. Oquendo [mailto:sil@infiltrated.net] Sent: Wednesday, April 15, 2009 15:36 To: nanog@nanog.org Subject: Level3 funkiness
Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail.
One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets
4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477 ms 5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74->74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84->84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62->62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72->72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9->1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net >(4.68.107.163) 75.797 ms !H
(From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6->0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H
Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by"
I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary.
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+>=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP
"It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett
227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
Looks similar from Sprints perspective via https://www.sprint.net/lg/lg_start.php as well... Sprint Source Region: Anaheim, CA (sl-bb20-ana) IP Destination: 63.211.236.36 Performing: ICMP Traceroute Tracing the route to 63.211.236.36 1 sl-crs2-ana-0-13-5-0.sprintlink.net (144.232.1.177) 4 msec 0 msec 0 msec 2 144.232.19.227 4 msec 24 msec sl-st30-la-0-4-5-0.sprintlink.net (144.232.24.41) 0 msec 3 xe-7-3-0.edge1.losangeles9.Level3.net (4.68.111.169) [AS 3356] 4 msec xe-7-2-0.edge1.losangeles9.Level3.net (4.68.111.89) [AS 3356] 0 msec 0 msec 4 vlan69.csw1.LosAngeles1.Level3.net (4.68.20.62) [AS 3356] 16 msec vlan99.csw4.LosAngeles1.Level3.net (4.68.20.254) [AS 3356] 12 msec vlan89.csw3.LosAngeles1.Level3.net (4.68.20.190) [AS 3356] 24 msec 5 ae-63-63.ebr3.LosAngeles1.Level3.net (4.69.137.33) [AS 3356] 4 msec 12 msec ae-93-93.ebr3.LosAngeles1.Level3.net (4.69.137.45) [AS 3356] 4 msec 6 ae-2.ebr3.SanJose1.Level3.net (4.69.132.9) [AS 3356] 40 msec 16 msec 16 msec 7 ae-73-73.csw2.SanJose1.Level3.net (4.69.134.230) [AS 3356] 16 msec ae-63-63.csw1.SanJose1.Level3.net (4.69.134.226) [AS 3356] 12 msec 12 msec 8 ae-82-82.ebr2.SanJose1.Level3.net (4.69.134.217) [AS 3356] 12 msec ae-92-92.ebr2.SanJose1.Level3.net (4.69.134.221) [AS 3356] 32 msec 12 msec 9 ae-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] 60 msec 48 msec 52 msec 10 ge-6-1.hsa1.Denver1.Level3.net (4.68.107.67) [AS 3356] !H ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) [AS 3356] !H * Completed - Wed Apr 15 15:43:21 EDT 2009 Sprint Source Region: Atlanta, GA (sl-bb20-atl) IP Destination: 63.211.236.36 Performing: ICMP Traceroute Tracing the route to 63.211.236.36 1 sl-bb21-atl-14-0.sprintlink.net (144.232.12.142) 0 msec 0 msec 0 msec 2 sl-crs1-atl-0-4-0-3.sprintlink.net (144.232.12.148) 4 msec 0 msec 0 msec 3 sl-crs1-rly-0-0-5-0.sprintlink.net (144.232.20.177) 20 msec 16 msec 16 msec 4 sl-crs1-dc-0-8-0-0.sprintlink.net (144.232.19.213) 20 msec 20 msec 24 msec 5 sl-st31-ash-0-4-0-1.sprintlink.net (144.232.19.4) 20 msec 20 msec 24 msec 6 4.68.63.205 [AS 3356] 20 msec te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 20 msec 24 msec 7 vlan79.csw2.Washington1.Level3.net (4.68.17.126) [AS 3356] 24 msec 28 msec vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 28 msec 8 ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) [AS 3356] 24 msec ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 24 msec 24 msec 9 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 28 msec 28 msec 28 msec 10 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 28 msec 28 msec 28 msec 11 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 48 msec 48 msec 52 msec 12 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H * Completed - Wed Apr 15 15:44:17 EDT 2009 Sprint Source Region: Chicago, IL (sl-bb20-chi) IP Destination: 63.211.236.36 Performing: ICMP Traceroute Tracing the route to 63.211.236.36 1 sl-crs1-chi-0-0-0-0.sprintlink.net (144.232.26.115) 0 msec 4 msec 0 msec 2 sl-st20-chi-4-0-0.sprintlink.net (144.232.18.153) 0 msec sl-st20-chi-13-0-0.sprintlink.net (144.232.20.3) 0 msec 0 msec 3 144.232.19.174 0 msec 0 msec 4 msec 4 ae-31-53.ebr1.Chicago1.Level3.net (4.68.101.94) [AS 3356] 8 msec ae-31-51.ebr1.Chicago1.Level3.net (4.68.101.30) [AS 3356] 8 msec ae-31-55.ebr1.Chicago1.Level3.net (4.68.101.158) [AS 3356] 4 msec 5 ae-6.ebr1.Chicago2.Level3.net (4.69.140.190) [AS 3356] 0 msec 4 msec 0 msec 6 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 44 msec 44 msec 36 msec 7 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) [AS 3356] !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H * Completed - Wed Apr 15 15:44:49 EDT 2009 Sprint Source Region: Relay, MD (sl-bb20-rly) IP Destination: 63.211.236.36 Performing: ICMP Traceroute Tracing the route to 63.211.236.36 1 sl-crs2-rly-0-1-0-0.sprintlink.net (144.232.3.96) 0 msec 0 msec 4 msec 2 sl-crs2-dc-0-12-2-0.sprintlink.net (144.232.19.221) 0 msec 0 msec 0 msec 3 sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 0 msec sl-st31-ash-0-1-0-0.sprintlink.net (144.232.19.230) 0 msec sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 4 msec 4 4.68.63.205 [AS 3356] 4 msec te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 0 msec 4.68.63.205 [AS 3356] 0 msec 5 vlan69.csw1.Washington1.Level3.net (4.68.17.62) [AS 3356] 0 msec vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 4 msec 12 msec 6 ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 4 msec ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) [AS 3356] 0 msec ae-82-82.ebr2.Washington1.Level3.net (4.69.134.153) [AS 3356] 4 msec 7 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 20 msec 20 msec 20 msec 8 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 20 msec 24 msec 20 msec 9 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 52 msec 64 msec 52 msec 10 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H * !H Completed - Wed Apr 15 15:45:37 EDT 2009
Same result from Cogent in Texas. Dying at ge-6-2.hsa1.Denver1.Level3.net # traceroute level3.net traceroute to 63.211.236.36 (63.211.236.36), 30 hops max, 46 byte packets 1 gi1-1.ccr01.aus02.atlas.cogentco.com (38.104.4.37) 0.493 ms 0.393 ms 0.496 ms 2 te4-4.ccr01.aus01.atlas.cogentco.com (154.54.25.157) 1.040 ms te4-3.ccr01.aus01.atlas.cogentco.com (154.54.25.153) 0.645 ms te4-4.ccr01.aus01.atlas.cogentco.com (154.54.25.157) 0.662 ms 3 te8-2.mpd01.iah01.atlas.cogentco.com (154.54.1.66) 3.972 ms 3.999 ms 3.926 ms 4 te4-1.mpd01.dfw01.atlas.cogentco.com (154.54.2.14) 9.289 ms * 9.249 ms 5 te3-3.mpd01.dfw03.atlas.cogentco.com (154.54.6.94) 9.227 ms te7-3.mpd01.dfw03.atlas.cogentco.com (154.54.6.66) 9.670 ms te3-3.mpd01.dfw03.atlas.cogentco.com (154.54.6.94) 9.806 ms 6 te-3-2.car3.Dallas1.Level3.net (4.68.110.109) 9.670 ms 9.813 ms 9.431 ms 7 * vlan79.csw2.Dallas1.Level3.net (4.68.19.126) 19.579 ms 20.438 ms 8 ae-72-72.ebr2.Dallas1.Level3.net (4.69.136.141) 15.812 ms 15.417 ms 18.290 ms 9 ae-2.ebr1.Denver1.Level3.net (4.69.132.105) 26.905 ms * 24.321 ms 10 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 24.518 ms !H * 24.644 ms !H Alex Thurlow Blastro Networks http://www.blastro.com http://www.roxwel.com http://www.yallwire.com On 4/15/2009 2:49 PM, Dixon, Justin wrote:
-----Original Message----- From: J. Oquendo [mailto:sil@infiltrated.net] Sent: Wednesday, April 15, 2009 15:36 To: nanog@nanog.org Subject: Level3 funkiness
Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail.
One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets
4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477
ms
5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms
16.393 ms
6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019
ms 10.833 ms
7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms
14.474 ms
8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166
ms 9.725 ms
9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616
ms 16.275 ms
10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms
ae-74->74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84->84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms
11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms
ae-62->62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72->72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms
12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms
45.774 ms
13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115
ms 38.135 ms
14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms
67.937 ms
15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H
ge-9->1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net>(4.68.107.163) 75.797 ms !H
(From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H *
ge-6->0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H
Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by"
I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary.
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+>=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP
"It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett
227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
Looks similar from Sprints perspective via https://www.sprint.net/lg/lg_start.php as well...
Sprint Source Region: Anaheim, CA (sl-bb20-ana) IP Destination: 63.211.236.36 Performing: ICMP Traceroute Tracing the route to 63.211.236.36 1 sl-crs2-ana-0-13-5-0.sprintlink.net (144.232.1.177) 4 msec 0 msec 0 msec 2 144.232.19.227 4 msec 24 msec sl-st30-la-0-4-5-0.sprintlink.net (144.232.24.41) 0 msec 3 xe-7-3-0.edge1.losangeles9.Level3.net (4.68.111.169) [AS 3356] 4 msec xe-7-2-0.edge1.losangeles9.Level3.net (4.68.111.89) [AS 3356] 0 msec 0 msec 4 vlan69.csw1.LosAngeles1.Level3.net (4.68.20.62) [AS 3356] 16 msec vlan99.csw4.LosAngeles1.Level3.net (4.68.20.254) [AS 3356] 12 msec vlan89.csw3.LosAngeles1.Level3.net (4.68.20.190) [AS 3356] 24 msec 5 ae-63-63.ebr3.LosAngeles1.Level3.net (4.69.137.33) [AS 3356] 4 msec 12 msec ae-93-93.ebr3.LosAngeles1.Level3.net (4.69.137.45) [AS 3356] 4 msec 6 ae-2.ebr3.SanJose1.Level3.net (4.69.132.9) [AS 3356] 40 msec 16 msec 16 msec 7 ae-73-73.csw2.SanJose1.Level3.net (4.69.134.230) [AS 3356] 16 msec ae-63-63.csw1.SanJose1.Level3.net (4.69.134.226) [AS 3356] 12 msec 12 msec 8 ae-82-82.ebr2.SanJose1.Level3.net (4.69.134.217) [AS 3356] 12 msec ae-92-92.ebr2.SanJose1.Level3.net (4.69.134.221) [AS 3356] 32 msec 12 msec 9 ae-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] 60 msec 48 msec 52 msec 10 ge-6-1.hsa1.Denver1.Level3.net (4.68.107.67) [AS 3356] !H ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) [AS 3356] !H *
Completed - Wed Apr 15 15:43:21 EDT 2009
Sprint Source Region: Atlanta, GA (sl-bb20-atl) IP Destination: 63.211.236.36 Performing: ICMP Traceroute Tracing the route to 63.211.236.36 1 sl-bb21-atl-14-0.sprintlink.net (144.232.12.142) 0 msec 0 msec 0 msec 2 sl-crs1-atl-0-4-0-3.sprintlink.net (144.232.12.148) 4 msec 0 msec 0 msec 3 sl-crs1-rly-0-0-5-0.sprintlink.net (144.232.20.177) 20 msec 16 msec 16 msec 4 sl-crs1-dc-0-8-0-0.sprintlink.net (144.232.19.213) 20 msec 20 msec 24 msec 5 sl-st31-ash-0-4-0-1.sprintlink.net (144.232.19.4) 20 msec 20 msec 24 msec 6 4.68.63.205 [AS 3356] 20 msec te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 20 msec 24 msec 7 vlan79.csw2.Washington1.Level3.net (4.68.17.126) [AS 3356] 24 msec 28 msec vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 28 msec 8 ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) [AS 3356] 24 msec ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 24 msec 24 msec 9 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 28 msec 28 msec 28 msec 10 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 28 msec 28 msec 28 msec 11 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 48 msec 48 msec 52 msec 12 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H * Completed - Wed Apr 15 15:44:17 EDT 2009
Sprint Source Region: Chicago, IL (sl-bb20-chi) IP Destination: 63.211.236.36 Performing: ICMP Traceroute Tracing the route to 63.211.236.36 1 sl-crs1-chi-0-0-0-0.sprintlink.net (144.232.26.115) 0 msec 4 msec 0 msec 2 sl-st20-chi-4-0-0.sprintlink.net (144.232.18.153) 0 msec sl-st20-chi-13-0-0.sprintlink.net (144.232.20.3) 0 msec 0 msec 3 144.232.19.174 0 msec 0 msec 4 msec 4 ae-31-53.ebr1.Chicago1.Level3.net (4.68.101.94) [AS 3356] 8 msec ae-31-51.ebr1.Chicago1.Level3.net (4.68.101.30) [AS 3356] 8 msec ae-31-55.ebr1.Chicago1.Level3.net (4.68.101.158) [AS 3356] 4 msec 5 ae-6.ebr1.Chicago2.Level3.net (4.69.140.190) [AS 3356] 0 msec 4 msec 0 msec 6 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 44 msec 44 msec 36 msec 7 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) [AS 3356] !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H *
Completed - Wed Apr 15 15:44:49 EDT 2009
Sprint Source Region: Relay, MD (sl-bb20-rly) IP Destination: 63.211.236.36 Performing: ICMP Traceroute Tracing the route to 63.211.236.36 1 sl-crs2-rly-0-1-0-0.sprintlink.net (144.232.3.96) 0 msec 0 msec 4 msec 2 sl-crs2-dc-0-12-2-0.sprintlink.net (144.232.19.221) 0 msec 0 msec 0 msec 3 sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 0 msec sl-st31-ash-0-1-0-0.sprintlink.net (144.232.19.230) 0 msec sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 4 msec 4 4.68.63.205 [AS 3356] 4 msec te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 0 msec 4.68.63.205 [AS 3356] 0 msec 5 vlan69.csw1.Washington1.Level3.net (4.68.17.62) [AS 3356] 0 msec vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 4 msec 12 msec 6 ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 4 msec ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) [AS 3356] 0 msec ae-82-82.ebr2.Washington1.Level3.net (4.69.134.153) [AS 3356] 4 msec 7 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 20 msec 20 msec 20 msec 8 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 20 msec 24 msec 20 msec 9 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 52 msec 64 msec 52 msec 10 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H * !H
Completed - Wed Apr 15 15:45:37 EDT 2009
-- Alex Thurlow Blastro Networks http://www.blastro.com http://www.roxwel.com http://www.yallwire.com
On Wed, Apr 15, 2009 at 4:07 PM, Alex Thurlow <alex@blastro.com> wrote:
Same result from Cogent in Texas. Dying at ge-6-2.hsa1.Denver1.Level3.net
# traceroute level3.net
I didn't know that an unreachable A record indicated that (3) was down :-) http://www.level3.com/lookingglass/ I can reach Denver and www.level3.com, et. Al. I seem to be able to reach everything. Is the NOC number still 1-877-2LEVEL3? Martin -- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
-----Original Message----- From: J. Oquendo [mailto:sil@infiltrated.net] Sent: Wednesday, April 15, 2009 3:36 PM To: nanog@nanog.org Subject: Level3 funkiness
Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail.
My Level3 connection is working normally, I can reach their site, and I'm peered in Atlanta. Jason A. Bertoch Network Administrator jason@electronet.net Electronet Broadband Communications 3411 Capital Medical Blvd. Tallahassee, FL 32308 (V) 850.222.0229 (F) 850.222.8771
Yea Jason, level3.net is unreachable....I am sure it is filtering ICMP, or blocking certain ports, sessions, or services, dude. The level3.net server is used for other purposes as stated in the previous thread, so necessarily it is not a question of destination unknown, DNS, or access for that matter....just you no touchy my server, that's it. I know. Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Jason Bertoch [mailto:jason@electronet.net] Sent: Wednesday, April 15, 2009 1:52 PM To: nanog@nanog.org Subject: RE: Level3 funkiness
-----Original Message----- From: J. Oquendo [mailto:sil@infiltrated.net] Sent: Wednesday, April 15, 2009 3:36 PM To: nanog@nanog.org Subject: Level3 funkiness
Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail.
My Level3 connection is working normally, I can reach their site, and I'm peered in Atlanta. Jason A. Bertoch Network Administrator jason@electronet.net Electronet Broadband Communications 3411 Capital Medical Blvd. Tallahassee, FL 32308 (V) 850.222.0229 (F) 850.222.8771 ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
destination unreachable on qwest out of St.Paul/Minneapolis. Level3.com does work. Dylan Ebner, Network Engineer -----Original Message----- From: J. Oquendo [mailto:sil@infiltrated.net] Sent: Wednesday, April 15, 2009 2:36 PM To: nanog@nanog.org Subject: Level3 funkiness Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail. One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets 4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477 ms 5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H (From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by" I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
2 dvr-edge-05.inet.qwest.net (72.165.27.181) 27.696 ms 27.688 ms 28.022 ms 3 dvr-core-01.inet.qwest.net (205.171.10.89) 28.010 ms 28.001 ms 27.990 ms 4 * * 67.14.2.89 (67.14.2.89) 50.773 ms 5 xe-8-2-0.edge2.dallas3.level3.net (4.68.63.53) 51.120 ms xe-8-1-0.edge2.dallas3.level3.net (4.68.63.49) 51.107 ms 51.099 ms 6 vlan79.csw2.Dallas1.Level3.net (4.68.19.126) 56.763 ms 37.806 ms vlan89.csw3.Dallas1.Level3.net (4.68.19.190) 33.368 ms 7 ae-82-82.ebr2.Dallas1.Level3.net (4.69.136.145) 35.514 ms ae-72-72.ebr2.Dallas1.Level3.net (4.69.136.141) 44.125 ms ae-62-62.ebr2.Dallas1.Level3.net (4.69.136.137) 44.120 ms 8 ae-2.ebr1.Denver1.Level3.net (4.69.132.105) 50.913 ms 50.895 ms 50.522 ms 9 ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 45.675 ms !H ge-6-1.hsa1.Denver1.Level3.net (4.68.107.67) 46.875 ms !H * -----Original Message----- From: J. Oquendo [mailto:sil@infiltrated.net] Sent: Wednesday, April 15, 2009 1:36 PM To: nanog@nanog.org Subject: Level3 funkiness Anyone else experience sporadic funkiness via Level3? I can't even reach the main website from who knows how many networks I've tried. Also friends and former colleagues have tried to reach the site to no avail. One of my machines on AT&T: # traceroute level3.net traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets 4 cr1.n54ny.ip.att.net (12.122.105.58) 11.285 ms 21.702 ms 21.477 ms 5 ggr2.n54ny.ip.att.net (12.122.131.141) 12.712 ms 10.194 ms 16.393 ms 6 so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149) 9.975 ms 10.019 ms 10.833 ms 7 vlan79.csw2.NewYork1.Level3.net (4.68.16.126) 10.162 ms 10.189 ms 14.474 ms 8 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) 15.763 ms 11.166 ms 9.725 ms 9 ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93) 16.139 ms 30.616 ms 16.275 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 15.684 ms ae-74-74.csw2.Washington1.Level3.net (4.69.134.182) 21.870 ms ae-84-84.csw3.Washington1.Level3.net (4.69.134.186) 28.729 ms 11 ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) 17.035 ms ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 17.041 ms ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) 21.940 ms 12 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) 31.671 ms 42.407 ms 45.774 ms 13 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) 31.922 ms 32.115 ms 38.135 ms 14 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) 75.265 ms 67.528 ms 67.937 ms 15 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) 62.587 ms !H ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) 62.543 ms !H ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) 75.797 ms !H (From Texas through Above.net) $ traceroute level3.net|tail -n 1 traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets 11 ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) 21.473 ms !H * ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 21.547 ms !H Confirmed it can't be reached from Travelers Ins, The Hartford, none of my connections. Anyone else seeing issues? I'm seeing drop off from clients going through their Atlanta interconnects with Charter and two other providers, which I can't make sense of. I DO KNOW they experienced some sort of issue with a TDM switch or so they said... Very broad statements: "We know teh interwebs are down please stand by" I know websites are one thing, but the chances of the website going down, a TDM switch being wacky and now clients traversing their networks complaining all at once seems a little out of the ordinary. =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
On Wed, 15 Apr 2009, Blake Pfankuch wrote:
2 dvr-edge-05.inet.qwest.net (72.165.27.181) 27.696 ms 27.688 ms 28.022 ms 3 dvr-core-01.inet.qwest.net (205.171.10.89) 28.010 ms 28.001 ms 27.990 ms 4 * * 67.14.2.89 (67.14.2.89) 50.773 ms 5 xe-8-2-0.edge2.dallas3.level3.net (4.68.63.53) 51.120 ms xe-8-1-0.edge2.dallas3.level3.net (4.68.63.49) 51.107 ms 51.099 ms 6 vlan79.csw2.Dallas1.Level3.net (4.68.19.126) 56.763 ms 37.806 ms vlan89.csw3.Dallas1.Level3.net (4.68.19.190) 33.368 ms 7 ae-82-82.ebr2.Dallas1.Level3.net (4.69.136.145) 35.514 ms ae-72-72.ebr2.Dallas1.Level3.net (4.69.136.141) 44.125 ms ae-62-62.ebr2.Dallas1.Level3.net (4.69.136.137) 44.120 ms 8 ae-2.ebr1.Denver1.Level3.net (4.69.132.105) 50.913 ms 50.895 ms 50.522 ms 9 ge-6-0.hsa1.Denver1.Level3.net (4.68.107.3) 45.675 ms !H ge-6-1.hsa1.Denver1.Level3.net (4.68.107.67) 46.875 ms !H *
Thanks to all who replied. I should have actually reposted traces but it would have annoyed. I had experienced issues connecting to Level3.com, couldn't reach lg.level3 from a wide range of sites. I have a ticket on "watch" mode of sorts concerning a TDM switch having gone down in Connecticut. Just seems a little strange yet another client passing through an interchange with Level3 is now affected as well. I'll let them deal with their own provider. As far as I can tell from the ITSP side of things, I'm alright now. Anyone doing VoIP would have seen the Level3 hiccup up here (Stamford area'ish). =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 227C 5D35 7DCB 0893 95AA 4771 1DCE 1FD1 5CCD 6B5E http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
participants (13)
-
Alex H. Ryu -
Alex Thurlow -
Blake Pfankuch -
Brandon Galbraith -
Charles Mills -
Dave Larter -
Dixon, Justin -
Dylan Ebner -
J. Oquendo -
Jason Bertoch -
Martin Hannigan -
Murphy, Jay, DOH -
Niels Bakker