Yeap, i know, it was what i understood, as it is my opinion that a zero day would fit better... in the pure speculation world :) At the end of the day... maybe some undocumented fault int some obscure functionality that was activated/deployed a long time ago, and just revealed it self now... There are so many things that can go wrong on complex networks even with all the controls imposed on changes... On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie <shawnritchie@gmail.com> wrote:
Well, Level3 has by no means said that this was the result of a DDoS, that's just speculation on behalf of folks who do not work at Level3 so far.
On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira <admin@marcoteixeira.com> wrote:
I won't believe a company like Level3 would not deploy backplane protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network didn't pause or rebooted routers. And i guess both companies have had their share of (D)DoS in the past, so they had the time to get up to the challenge. Now... there where times where one malformed IP packet would cause a memory leak leading to a router reboot... :)
On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman <mel@beckman.org> wrote:
765 Gbps per second directed at a router’s interface IP might give the router pause, so to speak :)
-mel
On Oct 4, 2016, at 12:10 PM, Marco Teixeira <admin@marcoteixeira.com> wrote:
Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had it been an approved chance it would have been rolled back i guess... On the other hand, a zero day could apply...
Em 04/10/2016 19:54, "Mel Beckman" <mel@beckman.org> escreveu:
Sure. The recent release of the IoT DDoS attack code in the wild.
-mel
On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu wrote:
On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said:
This could be DoS attack.
Or a missing comma in a code update.
Or a fumble-fingered NOC monkey.
Or....
You have any reason to suspect a DoS attack rather than all the other possibilities?
--
-- Shawn
Possibly somebody YANGed when they should have yinged :) -mel beckman On Oct 4, 2016, at 1:06 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>> wrote: Yeap, i know, it was what i understood, as it is my opinion that a zero day would fit better... in the pure speculation world :) At the end of the day... maybe some undocumented fault int some obscure functionality that was activated/deployed a long time ago, and just revealed it self now... There are so many things that can go wrong on complex networks even with all the controls imposed on changes... On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie <shawnritchie@gmail.com<mailto:shawnritchie@gmail.com>> wrote: Well, Level3 has by no means said that this was the result of a DDoS, that's just speculation on behalf of folks who do not work at Level3 so far. On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>> wrote: I won't believe a company like Level3 would not deploy backplane protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network didn't pause or rebooted routers. And i guess both companies have had their share of (D)DoS in the past, so they had the time to get up to the challenge. Now... there where times where one malformed IP packet would cause a memory leak leading to a router reboot... :)? On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
765 Gbps per second directed at a router's interface IP might give the router pause, so to speak :)
-mel
On Oct 4, 2016, at 12:10 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>> wrote:
Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had it been an approved chance it would have been rolled back i guess... On the other hand, a zero day could apply...
Em 04/10/2016 19:54, "Mel Beckman" <mel@beckman.org<mailto:mel@beckman.org>> escreveu:
Sure. The recent release of the IoT DDoS attack code in the wild.
-mel
On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu<mailto:Valdis.Kletnieks@vt.edu> wrote:
On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said:
This could be DoS attack.
Or a missing comma in a code update.
Or a fumble-fingered NOC monkey.
Or....
You have any reason to suspect a DoS attack rather than all the other possibilities?
-- -- Shawn
Looks like a fat finger event...
From Level 3: "On October 4, our voice network experienced a service disruption affecting some of our customers in North America due to a configuration error. We know how important these services are to our customers. As an organization, we're putting processes in place to prevent issues like this from recurring in the future. We were able to restore all services by 9:31am Mountain time."
http://www.theregister.co.uk/2016/10/05/level3_voip_blackout_cause/ -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mel Beckman Sent: Tuesday, October 4, 2016 1:09 PM To: Marco Teixeira <admin@marcoteixeira.com> Cc: Shawn Ritchie <shawnritchie@gmail.com>; NANOG list <nanog@nanog.org> Subject: Re: Level 3 voice outage Possibly somebody YANGed when they should have yinged :) -mel beckman On Oct 4, 2016, at 1:06 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>> wrote: Yeap, i know, it was what i understood, as it is my opinion that a zero day would fit better... in the pure speculation world :) At the end of the day... maybe some undocumented fault int some obscure functionality that was activated/deployed a long time ago, and just revealed it self now... There are so many things that can go wrong on complex networks even with all the controls imposed on changes... On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie <shawnritchie@gmail.com<mailto:shawnritchie@gmail.com>> wrote: Well, Level3 has by no means said that this was the result of a DDoS, that's just speculation on behalf of folks who do not work at Level3 so far. On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>> wrote: I won't believe a company like Level3 would not deploy backplane protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network didn't pause or rebooted routers. And i guess both companies have had their share of (D)DoS in the past, so they had the time to get up to the challenge. Now... there where times where one malformed IP packet would cause a memory leak leading to a router reboot... :)? On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
765 Gbps per second directed at a router's interface IP might give the router pause, so to speak :)
-mel
On Oct 4, 2016, at 12:10 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>> wrote:
Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had it been an approved chance it would have been rolled back i guess... On the other hand, a zero day could apply...
Em 04/10/2016 19:54, "Mel Beckman" <mel@beckman.org<mailto:mel@beckman.org>> escreveu:
Sure. The recent release of the IoT DDoS attack code in the wild.
-mel
On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu<mailto:Valdis.Kletnieks@vt.edu> wrote:
On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said:
This could be DoS attack.
Or a missing comma in a code update.
Or a fumble-fingered NOC monkey.
Or....
You have any reason to suspect a DoS attack rather than all the other possibilities?
-- -- Shawn This electronic mail transmission contains information from Warner Pacific Insurance Services that may be confidential or privileged. Such information is solely for the intended recipient, and use by any other party is not authorized. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this message, its contents or any attachments is prohibited. Any wrongful interception of this message is punishable as a Federal Crime. If you have received this message in error, please notify the sender immediately by telephone (800) 801-2300 or by electronic mail at postmaster@warnerpacific.com.
It’s good to see them acknowledging this. -mel On Oct 5, 2016, at 10:10 AM, Gareth Tupper <Gareth.Tupper@warnerpacific.com<mailto:Gareth.Tupper@warnerpacific.com>> wrote: Looks like a fat finger event... From Level 3: "On October 4, our voice network experienced a service disruption affecting some of our customers in North America due to a configuration error. We know how important these services are to our customers. As an organization, we're putting processes in place to prevent issues like this from recurring in the future. We were able to restore all services by 9:31am Mountain time." http://www.theregister.co.uk/2016/10/05/level3_voip_blackout_cause/ -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mel Beckman Sent: Tuesday, October 4, 2016 1:09 PM To: Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>> Cc: Shawn Ritchie <shawnritchie@gmail.com<mailto:shawnritchie@gmail.com>>; NANOG list <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Level 3 voice outage Possibly somebody YANGed when they should have yinged :) -mel beckman On Oct 4, 2016, at 1:06 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com><mailto:admin@marcoteixeira.com>> wrote: Yeap, i know, it was what i understood, as it is my opinion that a zero day would fit better... in the pure speculation world :) At the end of the day... maybe some undocumented fault int some obscure functionality that was activated/deployed a long time ago, and just revealed it self now... There are so many things that can go wrong on complex networks even with all the controls imposed on changes... On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie <shawnritchie@gmail.com<mailto:shawnritchie@gmail.com><mailto:shawnritchie@gmail.com>> wrote: Well, Level3 has by no means said that this was the result of a DDoS, that's just speculation on behalf of folks who do not work at Level3 so far. On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com><mailto:admin@marcoteixeira.com>> wrote: I won't believe a company like Level3 would not deploy backplane protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network didn't pause or rebooted routers. And i guess both companies have had their share of (D)DoS in the past, so they had the time to get up to the challenge. Now... there where times where one malformed IP packet would cause a memory leak leading to a router reboot... :)? On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org><mailto:mel@beckman.org>> wrote: 765 Gbps per second directed at a router's interface IP might give the router pause, so to speak :) -mel On Oct 4, 2016, at 12:10 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com><mailto:admin@marcoteixeira.com>> wrote: Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had it been an approved chance it would have been rolled back i guess... On the other hand, a zero day could apply... Em 04/10/2016 19:54, "Mel Beckman" <mel@beckman.org<mailto:mel@beckman.org><mailto:mel@beckman.org>> escreveu: Sure. The recent release of the IoT DDoS attack code in the wild. -mel On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu<mailto:Valdis.Kletnieks@vt.edu><mailto:Valdis.Kletnieks@vt.edu> wrote: On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said: This could be DoS attack. Or a missing comma in a code update. Or a fumble-fingered NOC monkey. Or.... You have any reason to suspect a DoS attack rather than all the other possibilities? -- -- Shawn This electronic mail transmission contains information from Warner Pacific Insurance Services that may be confidential or privileged. Such information is solely for the intended recipient, and use by any other party is not authorized. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this message, its contents or any attachments is prohibited. Any wrongful interception of this message is punishable as a Federal Crime. If you have received this message in error, please notify the sender immediately by telephone (800) 801-2300 or by electronic mail at postmaster@warnerpacific.com<mailto:postmaster@warnerpacific.com>.
Can anyone who was affected by last week's outage confirm that 911 services were impacted (I assume they were)? Anyone know if the current outage is in any way related to this one from last week? http://downdetector.com/status/level3/map/ On 10/05/2016 01:24 PM, Mel Beckman wrote:
It’s good to see them acknowledging this.
-mel
On Oct 5, 2016, at 10:10 AM, Gareth Tupper <Gareth.Tupper@warnerpacific.com<mailto:Gareth.Tupper@warnerpacific.com>> wrote:
Looks like a fat finger event...
From Level 3: "On October 4, our voice network experienced a service disruption affecting some of our customers in North America due to a configuration error. We know how important these services are to our customers. As an organization, we're putting processes in place to prevent issues like this from recurring in the future. We were able to restore all services by 9:31am Mountain time."
http://www.theregister.co.uk/2016/10/05/level3_voip_blackout_cause/
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mel Beckman Sent: Tuesday, October 4, 2016 1:09 PM To: Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com>> Cc: Shawn Ritchie <shawnritchie@gmail.com<mailto:shawnritchie@gmail.com>>; NANOG list <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Level 3 voice outage
Possibly somebody YANGed when they should have yinged :)
-mel beckman
On Oct 4, 2016, at 1:06 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com><mailto:admin@marcoteixeira.com>> wrote:
Yeap, i know, it was what i understood, as it is my opinion that a zero day would fit better... in the pure speculation world :) At the end of the day... maybe some undocumented fault int some obscure functionality that was activated/deployed a long time ago, and just revealed it self now... There are so many things that can go wrong on complex networks even with all the controls imposed on changes...
On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie <shawnritchie@gmail.com<mailto:shawnritchie@gmail.com><mailto:shawnritchie@gmail.com>> wrote: Well, Level3 has by no means said that this was the result of a DDoS, that's just speculation on behalf of folks who do not work at Level3 so far.
On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com><mailto:admin@marcoteixeira.com>> wrote: I won't believe a company like Level3 would not deploy backplane protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network didn't pause or rebooted routers. And i guess both companies have had their share of (D)DoS in the past, so they had the time to get up to the challenge. Now... there where times where one malformed IP packet would cause a memory leak leading to a router reboot... :)?
On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org><mailto:mel@beckman.org>> wrote:
765 Gbps per second directed at a router's interface IP might give the router pause, so to speak :)
-mel
On Oct 4, 2016, at 12:10 PM, Marco Teixeira <admin@marcoteixeira.com<mailto:admin@marcoteixeira.com><mailto:admin@marcoteixeira.com>> wrote:
Multiple reboots across several markets... Does not seem something that full pipes would trigger. Had it been an approved chance it would have been rolled back i guess... On the other hand, a zero day could apply...
Em 04/10/2016 19:54, "Mel Beckman" <mel@beckman.org<mailto:mel@beckman.org><mailto:mel@beckman.org>> escreveu:
Sure. The recent release of the IoT DDoS attack code in the wild.
-mel
On Oct 4, 2016, at 11:42 AM, Valdis.Kletnieks@vt.edu<mailto:Valdis.Kletnieks@vt.edu><mailto:Valdis.Kletnieks@vt.edu> wrote:
On Tue, 04 Oct 2016 18:14:54 -0000, Mel Beckman said:
This could be DoS attack.
Or a missing comma in a code update.
Or a fumble-fingered NOC monkey.
Or....
You have any reason to suspect a DoS attack rather than all the other possibilities?
--
-- Shawn
This electronic mail transmission contains information from Warner Pacific Insurance Services that may be confidential or privileged. Such information is solely for the intended recipient, and use by any other party is not authorized. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of this message, its contents or any attachments is prohibited. Any wrongful interception of this message is punishable as a Federal Crime. If you have received this message in error, please notify the sender immediately by telephone (800) 801-2300 or by electronic mail at postmaster@warnerpacific.com<mailto:postmaster@warnerpacific.com>.
participants (4)
-
Gareth Tupper -
Marco Teixeira -
Mel Beckman -
voytek