Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers. Mark
We just got a notification from one of our upstream Voice providers that they are seeing problems both inbound and outbound that seems to be Level 3 related *Ben Hatton* Network Engineer Haefele TV Inc. d:(607)589-8000 bhatton@htva.net www.htva.net On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
We can't make any LD calls through Level3 either. Thank you, Shane Heupel
On Oct 4, 2016, at 10:06 AM, Benjamin Hatton <bhatton@htva.net> wrote:
We just got a notification from one of our upstream Voice providers that they are seeing problems both inbound and outbound that seems to be Level 3 related
*Ben Hatton*
Network Engineer
Haefele TV Inc.
d:(607)589-8000
bhatton@htva.net
www.htva.net
On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
Boise Idaho is affected....... -----Original Message----- From: NANOG [mailto:nanog-bounces+ascher=slhs.org@nanog.org] On Behalf Of Benjamin Hatton Sent: Tuesday, October 04, 2016 9:04 AM To: nanog@nanog.org Subject: [BULK] Re: Level 3 voice outage Importance: High WARNING: This email originated outside of St. Luke’s email system. DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe. We just got a notification from one of our upstream Voice providers that they are seeing problems both inbound and outbound that seems to be Level 3 related *Ben Hatton* Network Engineer Haefele TV Inc. d:(607)589-8000 bhatton@htva.net www.htva.net On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
"This message is intended for the use of the person or entity to which it is addressed and may contain information that is confidential or privileged, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this information is strictly prohibited. If you have received this message by error, please notify us immediately and destroy the related message."
Our upstream Voice provider is also having inbound issues with numbers that were ported into their system, but the main trunk number is working. -Tim -----Original Message----- From: NANOG [mailto:nanog-bounces+timrutherford=c4.net@nanog.org] On Behalf Of Benjamin Hatton Sent: Tuesday, October 4, 2016 11:04 AM To: nanog@nanog.org Subject: Re: Level 3 voice outage We just got a notification from one of our upstream Voice providers that they are seeing problems both inbound and outbound that seems to be Level 3 related *Ben Hatton* Network Engineer Haefele TV Inc. d:(607)589-8000 bhatton@htva.net www.htva.net On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
Twitter is blowing up ( https://twitter.com/hashtag/outage ) and their website (www.level3.com) is down for me so yeah... I'd say they're having a frowny face day. On Tue, Oct 4, 2016 at 10:20 AM, <timrutherford@c4.net> wrote:
Our upstream Voice provider is also having inbound issues with numbers that were ported into their system, but the main trunk number is working.
-Tim
-----Original Message----- From: NANOG [mailto:nanog-bounces+timrutherford=c4.net@nanog.org] On Behalf Of Benjamin Hatton Sent: Tuesday, October 4, 2016 11:04 AM To: nanog@nanog.org Subject: Re: Level 3 voice outage
We just got a notification from one of our upstream Voice providers that they are seeing problems both inbound and outbound that seems to be Level 3 related
*Ben Hatton*
Network Engineer
Haefele TV Inc.
d:(607)589-8000
bhatton@htva.net
www.htva.net
On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
We are back in action here in MA. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of timrutherford@c4.net Sent: Tuesday, October 4, 2016 11:20 AM To: nanog@nanog.org Subject: RE: Level 3 voice outage Our upstream Voice provider is also having inbound issues with numbers that were ported into their system, but the main trunk number is working. -Tim -----Original Message----- From: NANOG [mailto:nanog-bounces+timrutherford=c4.net@nanog.org] On Behalf Of Benjamin Hatton Sent: Tuesday, October 4, 2016 11:04 AM To: nanog@nanog.org Subject: Re: Level 3 voice outage We just got a notification from one of our upstream Voice providers that they are seeing problems both inbound and outbound that seems to be Level 3 related *Ben Hatton* Network Engineer Haefele TV Inc. d:(607)589-8000 bhatton@htva.net www.htva.net On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
On Tue, 4 Oct 2016, Mark Stevens wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Yes, major outage: http://downdetector.com/status/level3 Customers all over the mid-atlantic region are down.
Level 3 has issues nationwide today. We have voice outages at our offices in multiple cities, Chicago and out west. On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
There's a long thread over at the outages list. Sign-up for the outages list is at https://puck.nether.net/mailman/listinfo/outages and the archive for the whole thread starts at https://puck.nether.net/pipermail/outages/2016-October/009609.html. Matt Freitag Network Engineer I Information Technology Michigan Technological University (906) 487-3696 <%28906%29%20487-3696> https://www.mtu.edu/ https://www.it.mtu.edu/ On Tue, Oct 4, 2016 at 11:06 AM, Kevin Kadow <kkadow@gmail.com> wrote:
Level 3 has issues nationwide today. We have voice outages at our offices in multiple cities, Chicago and out west.
On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
Our provider is down as well. Saying Level 3 issues. Anyone know what's going on or and ETR? On Tue, Oct 4, 2016 at 10:06 AM, Kevin Kadow <kkadow@gmail.com> wrote:
Level 3 has issues nationwide today. We have voice outages at our offices in multiple cities, Chicago and out west.
On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
-- Adair Winter VP, Network Operations / Co-Owner Amarillo Wireless | 806.316.5071 C: 806.231.7180 http://www.amarillowireless.net <http://www.amarillowireless.net>
Major Level 3 outage; they are currently not taking support calls. On 10/4/16, 8:01 AM, "NANOG on behalf of Mark Stevens" <nanog-bounces@nanog.org on behalf of manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
There's all sorts of people online reporting problems with Level3 voice services - we're down for outbound calling, inbound is spotty. Their portal is unreachable, can't get to any of their #'s...seems like the world is on fire over there. We were a previous TW Telecom customer before Level3 bought them out, the service quality has definitely degraded since then. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mark Stevens Sent: Tuesday, October 04, 2016 10:02 AM To: nanog@nanog.org Subject: Level 3 voice outage Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers. Mark *********************************************************************************** This email message, including any attachments, is intended solely for the designated person or entity to which the message is addressed, and may contain confidential information. It is strictly prohibited to review, distribute, copy, or print this email, including attachments, by any person or entity other than the designated addressee. If you have erroneously received this message, please contact the sender immediately and remove the data from any computer on which the message resides.
When will L3 notify their customers for the outage?!? According to l3 twitter account their are aware of the voice impact and working on it On Oct 4, 2016 11:03 AM, "Mark Stevens" <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
Patience Obi Wan ! They are investigating the root cause and like most root causes they don’t just hold out a flag and say here I am !!!
On Oct 4, 2016, at 10:32, Ivaylo Katovski <ivokatovski@gmail.com> wrote:
When will L3 notify their customers for the outage?!? According to l3 twitter account their are aware of the voice impact and working on it
On Oct 4, 2016 11:03 AM, "Mark Stevens" <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
-- Jason Hellenthal JJH48-ARIN
We are seeing high latencies and loss of transit through Level3 in Southern California. Level 3 has opened Global Ticket 115724: Network devices in multiple markets in North America are rebooting, impacting IP services.” This could be DoS attack.
On Oct 4, 2016, at 11:06 AM, Jason Hellenthal <jhellenthal@dataix.net> wrote:
Patience Obi Wan ! They are investigating the root cause and like most root causes they don’t just hold out a flag and say here I am !!!
On Oct 4, 2016, at 10:32, Ivaylo Katovski <ivokatovski@gmail.com> wrote:
When will L3 notify their customers for the outage?!? According to l3 twitter account their are aware of the voice impact and working on it
On Oct 4, 2016 11:03 AM, "Mark Stevens" <manager@monmouth.com> wrote:
Is anyone noticing issue with Level 3 voice? I can't even call their 800 number using one of my other carriers.
Mark
-- Jason Hellenthal JJH48-ARIN
Sure. The recent release of the IoT DDoS attack code in the wild. -mel
On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu wrote:
On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said:
This could be DoS attack.
Or a missing comma in a code update.
Or a fumble-fingered NOC monkey.
Or....
You have any reason to suspect a DoS attack rather than all the other possibilities?
Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had it been an approved chance it would have been rolled back i guess... On the other hand, a zero day could apply... Em 04/10/2016 19:54, "Mel Beckman" <mel@beckman.org> escreveu:
Sure. The recent release of the IoT DDoS attack code in the wild.
-mel
On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu wrote:
On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said:
This could be DoS attack.
Or a missing comma in a code update.
Or a fumble-fingered NOC monkey.
Or....
You have any reason to suspect a DoS attack rather than all the other possibilities?
765 Gbps per second directed at a router’s interface IP might give the router pause, so to speak :) -mel On Oct 4, 2016, at 12:10 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>> wrote: Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had it been an approved chance it would have been rolled back i guess... On the other hand, a zero day could apply... Em 04/10/2016 19:54, "Mel Beckman" <mel@beckman.org<mailto:mel@beckman.org>> escreveu: Sure. The recent release of the IoT DDoS attack code in the wild. -mel
On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu<mailto:Valdis.Kletnieks@vt.edu> wrote:
On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said:
This could be DoS attack.
Or a missing comma in a code update.
Or a fumble-fingered NOC monkey.
Or....
You have any reason to suspect a DoS attack rather than all the other possibilities?
I won't believe a company like Level3 would not deploy backplane protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network didn't pause or rebooted routers. And i guess both companies have had their share of (D)DoS in the past, so they had the time to get up to the challenge. Now... there where times where one malformed IP packet would cause a memory leak leading to a router reboot... :) On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman <mel@beckman.org> wrote:
765 Gbps per second directed at a router’s interface IP might give the router pause, so to speak :)
-mel
On Oct 4, 2016, at 12:10 PM, Marco Teixeira <admin@marcoteixeira.com> wrote:
Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had it been an approved chance it would have been rolled back i guess... On the other hand, a zero day could apply...
Em 04/10/2016 19:54, "Mel Beckman" <mel@beckman.org> escreveu:
Sure. The recent release of the IoT DDoS attack code in the wild.
-mel
On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu wrote:
On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said:
This could be DoS attack.
Or a missing comma in a code update.
Or a fumble-fingered NOC monkey.
Or....
You have any reason to suspect a DoS attack rather than all the other possibilities?
... but back then they didn't have ITIL and the likes (at least not that i knew... but then again, i was just starting with networking back then), where a ChM process must produce risk, impact, etc, reports, and at least one peer review meeting for approval :) = Marco On Tue, Oct 4, 2016 at 8:33 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Tue, 04 Oct 2016 20:10:44 +0100, Marco Teixeira said:
Had it been an approved chance it would have been rolled back i guess...
See the 1990 ATT long-distance collapse for a worked example.
participants (17)
-
Adair Winter -
Asche, Ron -
Benjamin Hatton -
C. Jon Larsen -
Ivaylo Katovski -
Jason Hellenthal -
Ken Warnimont -
Kevin Kadow -
Marco Teixeira -
Mark Horton -
Mark Stevens -
Matt Freitag -
Mel Beckman -
Sean -
Shane Heupel -
timrutherford＠c4.net -
Valdis.Kletnieks＠vt.edu