600,000 routers bricked
https://www.linkedin.com/pulse/600000-families-using-one-internet-provider-h... -- https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast Dave Täht CSO, LibreQos
And then when it became clear that the issue wasn't being addressed, they forcibly turned off those 600,000 routers. I am finding it difficult not to applaud that action.
The concern is that someone would shut off the routers or compromise them, so they compromised and shut them off? On Sun, Jun 2, 2024 at 4:03 PM Dave Taht <dave.taht@gmail.com> wrote:
https://www.linkedin.com/pulse/600000-families-using-one-internet-provider-h...
-- https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast Dave Täht CSO, LibreQos
After reading the actual report, I think bruce is making assumptions about the attackers' motivations that may or may not be the case. https://blog.lumen.com/the-pumpkin-eclipse/ Still, 600k routers gone in 72 hours is quite a lot. If they were also being actively used in a botnet, good riddance. On Sun, Jun 2, 2024 at 1:05 PM Josh Luthman <josh@imaginenetworksllc.com> wrote:
And then when it became clear that the issue wasn't being addressed, they forcibly turned off those 600,000 routers. I am finding it difficult not to applaud that action.
The concern is that someone would shut off the routers or compromise them, so they compromised and shut them off?
On Sun, Jun 2, 2024 at 4:03 PM Dave Taht <dave.taht@gmail.com> wrote:
https://www.linkedin.com/pulse/600000-families-using-one-internet-provider-h...
-- https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast Dave Täht CSO, LibreQos
-- https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast Dave Täht CSO, LibreQos
let's hope that this action didn't harm anyone - particularly a vulnerable person who might have an emergency system using IP to send alerts On Mon, 3 Jun 2024 at 01:22, Josh Luthman <josh@imaginenetworksllc.com> wrote:
And then when it became clear that the issue wasn't being addressed, they forcibly turned off those 600,000 routers. I am finding it difficult not to applaud that action.
The concern is that someone would shut off the routers or compromise them, so they compromised and shut them off?
On Sun, Jun 2, 2024 at 4:03 PM Dave Taht <dave.taht@gmail.com> wrote:
https://www.linkedin.com/pulse/600000-families-using-one-internet-provider-h...
-- https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast Dave Täht CSO, LibreQos
That post from Mr. Perens about this is honestly really shitty. 1. Is he right that Lumen has to shoulder blame for not keeping CPE updated with exploit free software? Certainly. 2. Making a claim that all 600k of these routers were being used as botnet zombies without any supporting evidence is really poor form. 3. Even if we assert that 50% of these devices were exploited for botnet activity, that means 50% WEREN'T. We shouldn't be applauding 300k people/businesses that just had their internet connectivity yeeted away from them through zero fault or their own. 4. "I've never heard of these router manufactures" is exceptionally ignorant. ActionTec has been around since the early 90s. Sagemcom wasn't someone I've heard of before , but so what. Yes, CPE provided by ISPs can be a problem. But applauding asshats who bricked all this stuff as some noble event that should be "applauded" as he says is really, really stupid. It's not going to meaningfully move the needle with how ISPs handle this stuff, and all it did was inconvenience a LOT of end users. On Sun, Jun 2, 2024 at 4:04 PM Dave Taht <dave.taht@gmail.com> wrote:
https://www.linkedin.com/pulse/600000-families-using-one-internet-provider-h...
-- https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast Dave Täht CSO, LibreQos
In the second paragraph, he cites his source: https://blog.lumen.com/the-pumpkin-eclipse/ Lumen’s Black Lotus Labs detected the event; the post answers all of your concerns. Further, they remark that this was an especially sophisticated infection, that hid its tracks well. Lee From: NANOG <nanog-bounces+leehoward=hilcostreambank.com@nanog.org> On Behalf Of Tom Beecher Sent: Sunday, June 2, 2024 4:23 PM To: Dave Taht <dave.taht@gmail.com> Cc: NANOG <nanog@nanog.org> Subject: Re: 600,000 routers bricked This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments. That post from Mr. Perens about this is honestly really shitty. 1. Is he right that Lumen has to shoulder blame for not keeping CPE updated with exploit free software? Certainly. 2. Making a claim that all 600k of these routers were being used as botnet zombies without any supporting evidence is really poor form. 3. Even if we assert that 50% of these devices were exploited for botnet activity, that means 50% WEREN'T. We shouldn't be applauding 300k people/businesses that just had their internet connectivity yeeted away from them through zero fault or their own. 4. "I've never heard of these router manufactures" is exceptionally ignorant. ActionTec has been around since the early 90s. Sagemcom wasn't someone I've heard of before , but so what. Yes, CPE provided by ISPs can be a problem. But applauding asshats who bricked all this stuff as some noble event that should be "applauded" as he says is really, really stupid. It's not going to meaningfully move the needle with how ISPs handle this stuff, and all it did was inconvenience a LOT of end users. On Sun, Jun 2, 2024 at 4:04 PM Dave Taht <dave.taht@gmail.com<mailto:dave.taht@gmail.com>> wrote: https://www.linkedin.com/pulse/600000-families-using-one-internet-provider-h... -- https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast Dave Täht CSO, LibreQos
Lumen’s Black Lotus Labs detected the event; the post answers all of your concerns.
The source document from Black Lotus details the behavior of the malware used to brick the equipment. It does NOT make any statements or claims that the targeted devices were being used in botnet activity, which is the accusation made by Mr. Perens in his post. On Mon, Jun 3, 2024 at 9:27 AM Howard, Lee <LeeHoward@hilcostreambank.com> wrote:
In the second paragraph, he cites his source: https://blog.lumen.com/the-pumpkin-eclipse/
Lumen’s Black Lotus Labs detected the event; the post answers all of your concerns. Further, they remark that this was an especially sophisticated infection, that hid its tracks well.
Lee
*From:* NANOG <nanog-bounces+leehoward=hilcostreambank.com@nanog.org> *On Behalf Of *Tom Beecher *Sent:* Sunday, June 2, 2024 4:23 PM *To:* Dave Taht <dave.taht@gmail.com> *Cc:* NANOG <nanog@nanog.org> *Subject:* Re: 600,000 routers bricked
*This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments.*
That post from Mr. Perens about this is honestly really shitty.
1. Is he right that Lumen has to shoulder blame for not keeping CPE updated with exploit free software? Certainly.
2. Making a claim that all 600k of these routers were being used as botnet zombies without any supporting evidence is really poor form.
3. Even if we assert that 50% of these devices were exploited for botnet activity, that means 50% WEREN'T. We shouldn't be applauding 300k people/businesses that just had their internet connectivity yeeted away from them through zero fault or their own.
4. "I've never heard of these router manufactures" is exceptionally ignorant. ActionTec has been around since the early 90s. Sagemcom wasn't someone I've heard of before , but so what.
Yes, CPE provided by ISPs can be a problem. But applauding asshats who bricked all this stuff as some noble event that should be "applauded" as he says is really, really stupid. It's not going to meaningfully move the needle with how ISPs handle this stuff, and all it did was inconvenience a LOT of end users.
On Sun, Jun 2, 2024 at 4:04 PM Dave Taht <dave.taht@gmail.com> wrote:
https://www.linkedin.com/pulse/600000-families-using-one-internet-provider-h...
--
https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast
Dave Täht CSO, LibreQos
I'm sorry, but if you have the wherewithal to commandeer 600,000 devices well enough to permanantly brick them, you have the wherewithal to commandeer them and load a patched version of software on them closing up the vulnerability. If there's no fixed version of software available for the platform, then you cannot fault the ISP for not patching the devices. If there IS a fixed version of the software available, this person should have used the botnet c2 to distribute and apply the fixed firmware, thus solving the problem while not killing connectivity for innocent end users. The decision to take destructive action is indefensible. The right choice should been to update the devices with patched software if it was available, and if it wasn't, to leave them alone and instead focus on trying to develop a fixed version of software. Now, if they were simply inept, and were trying to load fixed software onto the devices but failed to test their process adequately first, then at least their heart was in the right place, even if their understanding of how to do large-scale firmware upgrades safely wasn't. But that's certainly not what that article would lead us to suspect was the intended outcome. Matt On Sun, Jun 2, 2024, 16:47 Dave Taht <dave.taht@gmail.com> wrote:
https://www.linkedin.com/pulse/600000-families-using-one-internet-provider-h...
-- https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast Dave Täht CSO, LibreQos
It's important to note though that if you quietly (or even publicly) patch 600k devices to fix a bug, nobody cares. Plus, doing so is still a crime: it's 600k instances of accessing a computer system without permission. It's also far, FAR easier to write a stream of 0s to the bootloader than it is to decompile and debug bad firmware. Now if you brick the 600k devices, it gets attention. I'm NOT saying this is the appropriate or morally righteous thing to do, but like any other form of protest, the point is not to solve a single instance of a problem, it's to draw attention to the wider systemic issue: some ISPs not patching or life-cycling their CPEs. Depriving access to the Internet (and potentially 911) to 600k households is still wrong, no matter the intent. -Matt On Mon, Jun 3, 2024 at 11:10 AM Matthew Petach <mpetach@netflight.com> wrote:
I'm sorry, but if you have the wherewithal to commandeer 600,000 devices well enough to permanantly brick them, you have the wherewithal to commandeer them and load a patched version of software on them closing up the vulnerability.
If there's no fixed version of software available for the platform, then you cannot fault the ISP for not patching the devices.
If there IS a fixed version of the software available, this person should have used the botnet c2 to distribute and apply the fixed firmware, thus solving the problem while not killing connectivity for innocent end users.
The decision to take destructive action is indefensible. The right choice should been to update the devices with patched software if it was available, and if it wasn't, to leave them alone and instead focus on trying to develop a fixed version of software.
Now, if they were simply inept, and were trying to load fixed software onto the devices but failed to test their process adequately first, then at least their heart was in the right place, even if their understanding of how to do large-scale firmware upgrades safely wasn't.
But that's certainly not what that article would lead us to suspect was the intended outcome.
Matt
On Sun, Jun 2, 2024, 16:47 Dave Taht <dave.taht@gmail.com> wrote:
https://www.linkedin.com/pulse/600000-families-using-one-internet-provider-h...
-- https://www.youtube.com/watch?v=BVFWSyMp3xg&t=1098s Waves Podcast Dave Täht CSO, LibreQos
-- Matt Erculiani
On Mon, Jun 3, 2024 at 1:40 PM Matt Erculiani <merculiani@gmail.com> wrote:
It's important to note though that if you quietly (or even publicly) patch 600k devices to fix a bug, nobody cares. Plus, doing so is still a crime: it's 600k instances of accessing a computer system without permission. It's also far, FAR easier to write a stream of 0s to the bootloader than it is to decompile and debug bad firmware.
Lumen USED TO HAVE a walled-garden they dropped people into when their links/network ran amok.. at least in legacy-qwest/century-link consumer connectivity situations. maybe that's gone now? maybe the part of the affected network for this incident didn't have that capability?
If you do a bit more digging the ISP is not Lumen ... It is a well known ISP and I recall reading about this outage when it happened. I don’t know if indeed this was a botched attempt to gather a bot network or like some said an intentional act to get attention. Robert Jacobs | Data Center Manager Direct: 832-615-7742 Main: 832-615-8000 Fax: 713-510-1650 5959 Corporate Dr. Suite 3300; Houston, TX 77036 A Certified Woman-Owned Business 24x7x365 Customer Support: 832-615-8000 | support@pslightwave.com This electronic message contains information from PS Lightwave which may be privileged and confidential. The information is intended to be for the use of individual(s) or entity named above. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic message in error, please notify me by telephone or e-mail immediately. -----Original Message----- From: NANOG <nanog-bounces+rjacobs=pslightwave.com@nanog.org> On Behalf Of Christopher Morrow Sent: Monday, June 3, 2024 1:04 PM To: Matt Erculiani <merculiani@gmail.com> Cc: NANOG <nanog@nanog.org> Subject: Re: 600,000 routers bricked CAUTION: External Email. Do not click links or open attachments unless you recognize the sender and know the content is safe. On Mon, Jun 3, 2024 at 1:40 PM Matt Erculiani <merculiani@gmail.com> wrote:
It's important to note though that if you quietly (or even publicly) patch 600k devices to fix a bug, nobody cares. Plus, doing so is still a crime: it's 600k instances of accessing a computer system without permission. It's also far, FAR easier to write a stream of 0s to the bootloader than it is to decompile and debug bad firmware.
Lumen USED TO HAVE a walled-garden they dropped people into when their links/network ran amok.. at least in legacy-qwest/century-link consumer connectivity situations. maybe that's gone now? maybe the part of the affected network for this incident didn't have that capability?
It appears that Robert Jacobs <rjacobs@pslightwave.com> said:
-=-=-=-=-=-
If you do a bit more digging the ISP is not Lumen ... It is a well known ISP
It's Windstream. and I recall reading about this
outage when it happened. I don’t know if indeed this was a botched attempt to gather a bot network or like some said an intentional act to get attention.
Nobody else knows either. For me the most interesting question is where did Windstream get 600,000 replacement routers and how long did it take. The original attack was three days.
participants (10)
-
Alan Buxey -
Christopher Morrow -
Dave Taht -
Howard, Lee -
John Levine -
Josh Luthman -
Matt Erculiani -
Matthew Petach -
Robert Jacobs -
Tom Beecher