Does anyone have information on a widespread Rogers outage in Canada. I have customers with multiple sites down. Thanks Rob [cid:image001.jpg@01D892AF.488E0A30] Robert DeVita CEO and Founder t: (469) 581-2160<tel:(469)%20581-2160> | m: (469) 441-8864<tel:(469)%20441-8864> e: radevita@mejeticks.com<mailto:radevita@mejeticks.com> | w: mejeticks.com<http://www.example.com/> a: 3100 Carlisle St , Dallas , 75204 [cid:image002.png@01D892AF.488E0A30]<https://www.linkedin.com/company/mejeticks/> [cid:image003.png@01D892AF.488E0A30]<https://twitter.com/mejeticks> [cid:image004.png@01D892AF.488E0A30]<https://www.facebook.com/mejeticks>
On 7/8/22 07:44, Robert DeVita wrote:
Does anyone have information on a widespread Rogers outage in Canada. I have customers with multiple sites down.
There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage. Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building. I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information. Sent from my inoperable Rogers Mobile via emergency eSIM. Regards, Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours>, Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit. Regards, Josh Green On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org> wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net> wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
-- *Josh Green.*
Whatever they did, it has also taken out SS7/PSTN 911 services for many millions of people. https://www.cbc.ca/news/business/rogers-outage-cell-mobile-wifi-1.6514373 On Fri, 8 Jul 2022 at 11:44, Snowmobile2004 <greenjosh6499@gmail.com> wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours>, Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org> wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net> wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
-- *Josh Green.*
i cant see BGP taking out SS7. -jim On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com> wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours>, Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org> wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net> wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
-- *Josh Green.*
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone. I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail. This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed. On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com> wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com> wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours>, Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org> wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net> wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
-- *Josh Green.*
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house. Mutatis mutandis when Rogers is down things that are Rogers dependent will not work. Some people are so retarded it is astonishing! -- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
-----Original Message----- From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of Eric Kuhnke Sent: Friday, 8 July, 2022 13:34 To: jim deleskie <deleskie@gmail.com> Cc: NANOG list <nanog@nanog.org> Subject: Re: Rogers Outage Canada
I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail.
This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed.
On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com <mailto:deleskie@gmail.com> > wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net <mailto:jay@west.net> > wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net <mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
--
Josh Green.
I see the point you’re trying to make but using the word retarded in this context is not only dumb in itself but offense. Please be more respectful on this list. Thanks
On Jul 9, 2022, at 10:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
-----Original Message----- From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of Eric Kuhnke Sent: Friday, 8 July, 2022 13:34 To: jim deleskie <deleskie@gmail.com> Cc: NANOG list <nanog@nanog.org> Subject: Re: Rogers Outage Canada
I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail.
This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed.
On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com <mailto:deleskie@gmail.com> > wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net <mailto:jay@west.net> > wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net <mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
--
Josh Green.
On 7/9/22 09:54, JASON BOTHE via NANOG wrote:
I see the point you’re trying to make but using the word retarded in this context is not only dumb in itself but offense. Please be more respectful on this list.
Shall we take another spin on the euphemism treadmill? -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
Normally I'd take offense, but in my experience of endless technician calls, hundreds of hours on the phone with support. I can assure Rogers staff are the utmost incompetent out of any I've seen. Only talked to one support tech with them that even knew what DOCSIS was. How can teaching employees about the technology they sell not be part of training? Only reason they are even able to operate right now is reselling Xfinity products from Comcast after wasting billions trying to roll their own solutions. Regards, Andrew Paolucci Sent from Proton Mail mobile -------- Original Message -------- On Jul. 9, 2022, 3:18 p.m., Jay Hennigan wrote:
On 7/9/22 09:54, JASON BOTHE via NANOG wrote: > I see the point you’re trying to make but using the word retarded in this context is not only dumb in itself but offense. Please be more respectful on this list. Shall we take another spin on the euphemism treadmill? -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
Just leaving this (yes, satire site) link here https://www.thebeaverton.com/2022/07/rogers-proudly-announces-10-service-charge-for-unplug-and-relax-nationwide-outage/<https://www.thebeaverton.com/2022/07/rogers-proudly-announces-10-service-charge-for-unplug-and-relax-nationwide-outage/?fbclid=IwAR2ylrdNYhKLddbQNQ23WdAZT5V8wEPrBSi-JKp32bdaiXVjEHZdfxwfV6A&fs=e&s=cl#l5eav3aeuldc0yfzwjk> --srs ________________________________ From: NANOG <nanog-bounces+ops.lists=gmail.com@nanog.org> on behalf of Andrew Paolucci via NANOG <nanog@nanog.org> Sent: Saturday, July 9, 2022 12:49:09 PM To: jay@west.net <jay@west.net>; nanog@nanog.org <nanog@nanog.org> Subject: Re: Rogers Outage Canada Normally I'd take offense, but in my experience of endless technician calls, hundreds of hours on the phone with support. I can assure Rogers staff are the utmost incompetent out of any I've seen. Only talked to one support tech with them that even knew what DOCSIS was. How can teaching employees about the technology they sell not be part of training? Only reason they are even able to operate right now is reselling Xfinity products from Comcast after wasting billions trying to roll their own solutions. Regards, Andrew Paolucci Sent from Proton Mail mobile -------- Original Message -------- On Jul. 9, 2022, 3:18 p.m., Jay Hennigan < jay@west.net> wrote: On 7/9/22 09:54, JASON BOTHE via NANOG wrote: > I see the point you’re trying to make but using the word retarded in this context is not only dumb in itself but offense. Please be more respectful on this list. Shall we take another spin on the euphemism treadmill? -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not. Does this person need directions to some local mental healthcare resources? On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
-----Original Message----- From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of Eric Kuhnke Sent: Friday, 8 July, 2022 13:34 To: jim deleskie <deleskie@gmail.com> Cc: NANOG list <nanog@nanog.org> Subject: Re: Rogers Outage Canada
I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail.
This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed.
On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com <mailto:deleskie@gmail.com> > wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net <mailto:jay@west.net> > wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net <mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
--
Josh Green.
Lest we ALL MOVE ON…. Yes he said “RETARD” = The delay in processing the current status of a Situation… Lets move on to bigger n scarier Post mortems: Was This or Was this NOT A CYBERATTACK ??? Done. Lets deal with REAL threats not PC references. Time to evaluate n re eval Our NTS!! Veni vidi vici 2022 On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not.
Does this person need directions to some local mental healthcare resources?
On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
-----Original Message----- From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of Eric Kuhnke Sent: Friday, 8 July, 2022 13:34 To: jim deleskie <deleskie@gmail.com> Cc: NANOG list <nanog@nanog.org> Subject: Re: Rogers Outage Canada
I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail.
This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed.
On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com <mailto:deleskie@gmail.com> > wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net <mailto:jay@west.net> > wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net <mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
--
Josh Green.
--
Liz ******* 416.660.5456
We should move on and focus on constructive comments vs. kicking an operator when they are down. That said, demeaning comments, no matter what they are or whom they are targeted at, is not healthy and is not helpful. regards, Victor K On Sat, Jul 9, 2022 at 9:11 PM L F <liz.fazekas@gmail.com> wrote:
Lest we ALL MOVE ON….
Yes he said
“RETARD” =
The delay in processing the current status of a Situation…
Lets move on to bigger n scarier Post mortems:
Was This or Was this NOT
A CYBERATTACK ???
Done.
Lets deal with REAL threats not PC references.
Time to evaluate n re eval Our NTS!!
Veni vidi vici 2022
On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not.
Does this person need directions to some local mental healthcare resources?
On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
-----Original Message----- From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of Eric Kuhnke Sent: Friday, 8 July, 2022 13:34 To: jim deleskie <deleskie@gmail.com> Cc: NANOG list <nanog@nanog.org> Subject: Re: Rogers Outage Canada
I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail.
This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed.
On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com <mailto:deleskie@gmail.com> > wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net <mailto:jay@west.net> > wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net <mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
--
Josh Green.
--
Liz ******* 416.660.5456
Seeing the little of that thread. I never understood why someone thinks their experience with Level 1 thru 3 staff reflect the actual situation. That was clearly an issue with network engineering and their processes. Because a 14h downtime for, as Rogers says, "a specific equipment" is too big of a spin. ----- Alain Hebertahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911http://www.pubnix.net Fax: 514-990-9443 On 7/9/22 21:18, Victor Kuarsingh wrote:
We should move on and focus on constructive comments vs. kicking an operator when they are down.
That said, demeaning comments, no matter what they are or whom they are targeted at, is not healthy and is not helpful. regards,
Victor K
On Sat, Jul 9, 2022 at 9:11 PM L F <liz.fazekas@gmail.com> wrote:
Lest we ALL MOVE ON….
Yes he said
“RETARD” =
The delay in processing the current status of a Situation…
Lets move on to bigger n scarier Post mortems:
Was This or Was this NOT
A CYBERATTACK ???
Done.
Lets deal with REAL threats not PC references.
Time to evaluate n re eval Our NTS!!
Veni vidi vici 2022
On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not.
Does this person need directions to some local mental healthcare resources?
On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
>I can't either, but the reality right now seems to be that 911 calls are >failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
>-----Original Message----- >From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of >Eric Kuhnke >Sent: Friday, 8 July, 2022 13:34 >To: jim deleskie <deleskie@gmail.com> >Cc: NANOG list <nanog@nanog.org> >Subject: Re: Rogers Outage Canada > > >I have seen anecdotal reports that the mobile network is in a half broken >state that phones remain registered to, so a 911 call will attempt and >then fail. > > >This is unlike what would happen if you had a US/Canada cellphone with >battery power but no SIM card in it that would search for any available >network in RF range for a 911 call if needed. > > >On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com ><mailto:deleskie@gmail.com> > wrote: > > > i cant see BGP taking out SS7. > > -jim > > On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 ><greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote: > > > According to Cloudflare Radar ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers >BGP announcements spiked massively to levels 536,777% higher than normal >(343,601 vs 64 normally) just minutes before the outage. I would not be >surprised if this happened to be the culprit. > > Regards, > Josh Green > > On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG ><nanog@nanog.org <mailto:nanog@nanog.org> > wrote: > > > In the early hours of the morning around 2-3am my modem >got hit with a configuration update that caused a DHCP release that >wasn't renewed for about two hours, after rollback the connection was >fine for 3 hours before this network wide outage. > > > Maybe a failed night time update was attempted again >during office hours, I've heard daytime guys are still WFH and night >shift is in building. > > > I expect we'll never get a real explanation. Rogers is >notorious for withholding any type of helpful or technical information. > > > Sent from my inoperable Rogers Mobile via emergency eSIM. > > > Regards, > > Andrew Paolucci > -------- Original Message -------- > On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net ><mailto:jay@west.net> > wrote: > > > On 7/8/22 07:44, Robert DeVita wrote: > Does anyone >have information on a widespread Rogers outage in Canada. I > have >customers with multiple sites down. There's discussion on the Outages >mailing list. Seems widespread, affecting all services, mobile, voice, >Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net ><mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - >WB6RDV > > > > -- > > Josh Green.
-- Liz ******* 416.660.5456
fyi - see BOLD. A Message from Rogers President and CEO Dear Valued Customer, As you know, we experienced a service outage across the Rogers, Fido, Chatr and Cityfone wireless networks on Friday. I am reaching out to share that our services have been restored, and our networks and systems are close to fully operational. Our technical teams are continuing to monitor for any remaining intermittent issues. I also want to outline an action plan we are putting in place to address what happened. I want to share what we know about what happened on Friday. We now believe *we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction. We disconnected the specific equipment and redirected traffic,* which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels. On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas@gmail.com> wrote:
Lest we ALL MOVE ON….
Yes he said
“RETARD” =
The delay in processing the current status of a Situation…
Lets move on to bigger n scarier Post mortems:
Was This or Was this NOT
A CYBERATTACK ???
Done.
Lets deal with REAL threats not PC references.
Time to evaluate n re eval Our NTS!!
Veni vidi vici 2022
On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not.
Does this person need directions to some local mental healthcare resources?
On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
-----Original Message----- From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of Eric Kuhnke Sent: Friday, 8 July, 2022 13:34 To: jim deleskie <deleskie@gmail.com> Cc: NANOG list <nanog@nanog.org> Subject: Re: Rogers Outage Canada
I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail.
This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed.
On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com <mailto:deleskie@gmail.com> > wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net <mailto:jay@west.net> > wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net <mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
--
Josh Green.
Given the outage was so bad it was disrupting select E911 services nationwide for something like 24+ hours, it's great to see such an in depth analysis and plan of action to prevent such things in the future. bravo rogers On 7/10/2022 2:55 PM, L F wrote:
fyi - see BOLD.
A Message from Rogers President and CEO
Dear Valued Customer,
As you know, we experienced a service outage across the Rogers, Fido, Chatr and Cityfone wireless networks on Friday.
I am reaching out to share that our services have been restored, and our networks and systems are close to fully operational. Our technical teams are continuing to monitor for any remaining intermittent issues. I also want to outline an action plan we are putting in place to address what happened.
I want to share what we know about what happened on Friday. We now believe *we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction. We disconnected the specific equipment and redirected traffic,*which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.
On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas@gmail.com> wrote:
Lest we ALL MOVE ON….
Yes he said
“RETARD” =
The delay in processing the current status of a Situation…
Lets move on to bigger n scarier Post mortems:
Was This or Was this NOT
A CYBERATTACK ???
Done.
Lets deal with REAL threats not PC references.
Time to evaluate n re eval Our NTS!!
Veni vidi vici 2022
On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not.
Does this person need directions to some local mental healthcare resources?
On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
>I can't either, but the reality right now seems to be that 911 calls are >failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
>-----Original Message----- >From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of >Eric Kuhnke >Sent: Friday, 8 July, 2022 13:34 >To: jim deleskie <deleskie@gmail.com> >Cc: NANOG list <nanog@nanog.org> >Subject: Re: Rogers Outage Canada > > >I have seen anecdotal reports that the mobile network is in a half broken >state that phones remain registered to, so a 911 call will attempt and >then fail. > > >This is unlike what would happen if you had a US/Canada cellphone with >battery power but no SIM card in it that would search for any available >network in RF range for a 911 call if needed. > > >On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com ><mailto:deleskie@gmail.com> > wrote: > > > i cant see BGP taking out SS7. > > -jim > > On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 ><greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote: > > > According to Cloudflare Radar ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers >BGP announcements spiked massively to levels 536,777% higher than normal >(343,601 vs 64 normally) just minutes before the outage. I would not be >surprised if this happened to be the culprit. > > Regards, > Josh Green > > On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG ><nanog@nanog.org <mailto:nanog@nanog.org> > wrote: > > > In the early hours of the morning around 2-3am my modem >got hit with a configuration update that caused a DHCP release that >wasn't renewed for about two hours, after rollback the connection was >fine for 3 hours before this network wide outage. > > > Maybe a failed night time update was attempted again >during office hours, I've heard daytime guys are still WFH and night >shift is in building. > > > I expect we'll never get a real explanation. Rogers is >notorious for withholding any type of helpful or technical information. > > > Sent from my inoperable Rogers Mobile via emergency eSIM. > > > Regards, > > Andrew Paolucci > -------- Original Message -------- > On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net ><mailto:jay@west.net> > wrote: > > > On 7/8/22 07:44, Robert DeVita wrote: > Does anyone >have information on a widespread Rogers outage in Canada. I > have >customers with multiple sites down. There's discussion on the Outages >mailing list. Seems widespread, affecting all services, mobile, voice, >Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net ><mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - >WB6RDV > > > > -- > > Josh Green.
-- Jon Sands MFI Labs https://fohdeesha.com/
What in depth analysis have you seen? Seems to me, this was a failure in a known maintenance activity, and they simply disconnected the devices under maintenance from the network. Shane On Mon, Jul 11, 2022 at 5:41 AM Jon Sands <fohdeesha@gmail.com> wrote:
Given the outage was so bad it was disrupting select E911 services nationwide for something like 24+ hours, it's great to see such an in depth analysis and plan of action to prevent such things in the future. bravo rogers
On 7/10/2022 2:55 PM, L F wrote:
fyi - see BOLD.
A Message from Rogers President and CEO
Dear Valued Customer,
As you know, we experienced a service outage across the Rogers, Fido, Chatr and Cityfone wireless networks on Friday.
I am reaching out to share that our services have been restored, and our networks and systems are close to fully operational. Our technical teams are continuing to monitor for any remaining intermittent issues. I also want to outline an action plan we are putting in place to address what happened.
I want to share what we know about what happened on Friday. We now believe *we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction. We disconnected the specific equipment and redirected traffic,* which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.
On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas@gmail.com> wrote:
Lest we ALL MOVE ON….
Yes he said
“RETARD” =
The delay in processing the current status of a Situation…
Lets move on to bigger n scarier Post mortems:
Was This or Was this NOT
A CYBERATTACK ???
Done.
Lets deal with REAL threats not PC references.
Time to evaluate n re eval Our NTS!!
Veni vidi vici 2022
On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not.
Does this person need directions to some local mental healthcare resources?
On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
-----Original Message----- From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of Eric Kuhnke Sent: Friday, 8 July, 2022 13:34 To: jim deleskie <deleskie@gmail.com> Cc: NANOG list <nanog@nanog.org> Subject: Re: Rogers Outage Canada
I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail.
This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed.
On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com <mailto:deleskie@gmail.com> > wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net <mailto:jay@west.net> > wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net <mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
--
Josh Green.
-- Jon Sands MFI Labshttps://fohdeesha.com/
This is the most they can and will say. For liabilities reasons, specifics are likely not in the cards. As most services ride over common service networks, its quite possible that a network substrate failure can have a number of upstream service impacts. The point here is that the CEO is directly addressing the customer base, which is needed here. regards, Victor K On Mon, Jul 11, 2022 at 10:11 AM Shane Ronan <shane@ronan-online.com> wrote: > What in depth analysis have you seen? Seems to me, this was a failure in a > known maintenance activity, and they simply disconnected the devices under > maintenance from the network. > > Shane > > On Mon, Jul 11, 2022 at 5:41 AM Jon Sands <fohdeesha@gmail.com> wrote: > >> Given the outage was so bad it was disrupting select E911 services >> nationwide for something like 24+ hours, it's great to see such an in depth >> analysis and plan of action to prevent such things in the future. bravo >> rogers >> >> On 7/10/2022 2:55 PM, L F wrote: >> >> fyi - see BOLD. >> >> A Message from Rogers President and CEO >> >> >> Dear Valued Customer, >> >> As you know, we experienced a service outage across the Rogers, Fido, >> Chatr and Cityfone wireless networks on Friday. >> >> I am reaching out to share that our services have been restored, and our >> networks and systems are close to fully operational. Our technical teams >> are continuing to monitor for any remaining intermittent issues. I also >> want to outline an action plan we are putting in place to address what >> happened. >> >> I want to share what we know about what happened on Friday. We now >> believe *we’ve narrowed the cause to a network system failure following >> a maintenance update in our core network, which caused some of our routers >> to malfunction. We disconnected the specific equipment and redirected >> traffic,* which allowed our network and services to come back online >> over time as we managed traffic volumes returning to normal levels. >> >> >> >> On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas@gmail.com> wrote: >> >>> >>> Lest we ALL MOVE ON…. >>> >>> >>> Yes he said >>> >>> “RETARD” = >>> >>> The delay in processing the current status of a Situation… >>> >>> >>> Lets move on to bigger n scarier Post mortems: >>> >>> >>> Was This or Was this NOT >>> >>> A CYBERATTACK ??? >>> >>> Done. >>> >>> Lets deal with REAL threats not PC references. >>> >>> Time to evaluate n re eval Our NTS!! >>> >>> Veni vidi vici 2022 >>> >>> >>> >>> >>> On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> >>> wrote: >>> >>>> Can we have a discussion with the list admins about a list member >>>> appending a threat of violence to their outbound emails? Whether serious >>>> or not. >>>> >>>> Does this person need directions to some local mental healthcare >>>> resources? >>>> >>>> >>>> On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote: >>>> >>>>> >>>>> >I can't either, but the reality right now seems to be that 911 calls >>>>> are >>>>> >failing for anyone on a Rogers cellphone. >>>>> >>>>> This is par for the course. These people chose to deal with Rogers >>>>> despite knowing the consequences. It is like if you bought a Rogers >>>>> Snowblower and it did not work. That would mean that people who bought the >>>>> Rogers Snowblower will not be using it to get rid of the snow that is >>>>> preventing them from leaving their house. >>>>> >>>>> Mutatis mutandis when Rogers is down things that are Rogers dependent >>>>> will not work. >>>>> >>>>> Some people are so retarded it is astonishing! >>>>> >>>>> -- >>>>> (CAUTION) You are advised that if you attack my person or property, >>>>> you will be put down in accordance with the provisions of section 34 & 35 >>>>> of the Criminal Code respectively. If you are brandishing (or in >>>>> possession) of a weapon then lethal force will be applied to your person in >>>>> accordance with the law. This means that your misadventures may end in >>>>> your death. Consider yourself cautioned and govern your actions >>>>> appropriately. >>>>> >>>>> >-----Original Message----- >>>>> >From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf >>>>> Of >>>>> >Eric Kuhnke >>>>> >Sent: Friday, 8 July, 2022 13:34 >>>>> >To: jim deleskie <deleskie@gmail.com> >>>>> >Cc: NANOG list <nanog@nanog.org> >>>>> >Subject: Re: Rogers Outage Canada >>>>> > >>>>> > >>>>> >I have seen anecdotal reports that the mobile network is in a half >>>>> broken >>>>> >state that phones remain registered to, so a 911 call will attempt and >>>>> >then fail. >>>>> > >>>>> > >>>>> >This is unlike what would happen if you had a US/Canada cellphone with >>>>> >battery power but no SIM card in it that would search for any >>>>> available >>>>> >network in RF range for a 911 call if needed. >>>>> > >>>>> > >>>>> >On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com >>>>> ><mailto:deleskie@gmail.com> > wrote: >>>>> > >>>>> > >>>>> > i cant see BGP taking out SS7. >>>>> > >>>>> > -jim >>>>> > >>>>> > On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 >>>>> ><greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote: >>>>> > >>>>> > >>>>> > According to Cloudflare Radar >>>>> ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , >>>>> Rogers >>>>> >BGP announcements spiked massively to levels 536,777% higher than >>>>> normal >>>>> >(343,601 vs 64 normally) just minutes before the outage. I would not >>>>> be >>>>> >surprised if this happened to be the culprit. >>>>> > >>>>> > Regards, >>>>> > Josh Green >>>>> > >>>>> > On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via >>>>> NANOG >>>>> ><nanog@nanog.org <mailto:nanog@nanog.org> > wrote: >>>>> > >>>>> > >>>>> > In the early hours of the morning around 2-3am >>>>> my modem >>>>> >got hit with a configuration update that caused a DHCP release that >>>>> >wasn't renewed for about two hours, after rollback the connection was >>>>> >fine for 3 hours before this network wide outage. >>>>> > >>>>> > >>>>> > Maybe a failed night time update was attempted >>>>> again >>>>> >during office hours, I've heard daytime guys are still WFH and night >>>>> >shift is in building. >>>>> > >>>>> > >>>>> > I expect we'll never get a real explanation. >>>>> Rogers is >>>>> >notorious for withholding any type of helpful or technical >>>>> information. >>>>> > >>>>> > >>>>> > Sent from my inoperable Rogers Mobile via >>>>> emergency eSIM. >>>>> > >>>>> > >>>>> > Regards, >>>>> > >>>>> > Andrew Paolucci >>>>> > -------- Original Message -------- >>>>> > On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < >>>>> jay@west.net >>>>> ><mailto:jay@west.net> > wrote: >>>>> > >>>>> > >>>>> > On 7/8/22 07:44, Robert DeVita wrote: >>>>> > Does anyone >>>>> >have information on a widespread Rogers outage in Canada. I > have >>>>> >customers with multiple sites down. There's discussion on the Outages >>>>> >mailing list. Seems widespread, affecting all services, mobile, voice, >>>>> >Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net >>>>> ><mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 >>>>> - >>>>> >WB6RDV >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > >>>>> > Josh Green. >>>>> >>>>> >>>>> >>>>> >> >> >> -- >> Jon Sands >> MFI Labshttps://fohdeesha.com/ >> >>
As a Rogers data center customer I received this on Saturday: You have received this bulletin because you are an official contact for your Rogers Data Centres Server Colocation service The nationwide network outage has now been resolved. Rogers Data Centres is reporting all network and dependent services as operating normally. If you are experiencing any ongoing issues please reach out to the Data Centres help desk An official Incident Report will be shared in 5 to 7 Business days. We thank you for your patience during the restoration. -- Keith Stokes SalonBiz, Inc On Jul 11, 2022, at 10:16 AM, Victor Kuarsingh <victor@jvknet.com<mailto:victor@jvknet.com>> wrote: This is the most they can and will say. For liabilities reasons, specifics are likely not in the cards. As most services ride over common service networks, its quite possible that a network substrate failure can have a number of upstream service impacts. The point here is that the CEO is directly addressing the customer base, which is needed here. regards, Victor K On Mon, Jul 11, 2022 at 10:11 AM Shane Ronan <shane@ronan-online.com<mailto:shane@ronan-online.com>> wrote: What in depth analysis have you seen? Seems to me, this was a failure in a known maintenance activity, and they simply disconnected the devices under maintenance from the network. Shane On Mon, Jul 11, 2022 at 5:41 AM Jon Sands <fohdeesha@gmail.com<mailto:fohdeesha@gmail.com>> wrote: Given the outage was so bad it was disrupting select E911 services nationwide for something like 24+ hours, it's great to see such an in depth analysis and plan of action to prevent such things in the future. bravo rogers On 7/10/2022 2:55 PM, L F wrote: fyi - see BOLD. A Message from Rogers President and CEO Dear Valued Customer, As you know, we experienced a service outage across the Rogers, Fido, Chatr and Cityfone wireless networks on Friday. I am reaching out to share that our services have been restored, and our networks and systems are close to fully operational. Our technical teams are continuing to monitor for any remaining intermittent issues. I also want to outline an action plan we are putting in place to address what happened. I want to share what we know about what happened on Friday. We now believe we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction. We disconnected the specific equipment and redirected traffic, which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels. On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas@gmail.com<mailto:liz.fazekas@gmail.com>> wrote: Lest we ALL MOVE ON…. Yes he said “RETARD” = The delay in processing the current status of a Situation… Lets move on to bigger n scarier Post mortems: Was This or Was this NOT A CYBERATTACK ??? Done. Lets deal with REAL threats not PC references. Time to evaluate n re eval Our NTS!! Veni vidi vici 2022 On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com<mailto:eric.kuhnke@gmail.com>> wrote: Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not. Does this person need directions to some local mental healthcare resources? On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com<mailto:kmedcalf@dessus.com>> wrote:
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house. Mutatis mutandis when Rogers is down things that are Rogers dependent will not work. Some people are so retarded it is astonishing! -- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
-----Original Message----- From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org<mailto:dessus.com@nanog.org>> On Behalf Of Eric Kuhnke Sent: Friday, 8 July, 2022 13:34 To: jim deleskie <deleskie@gmail.com<mailto:deleskie@gmail.com>> Cc: NANOG list <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: Rogers Outage Canada
I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail.
This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed.
On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com<mailto:deleskie@gmail.com> <mailto:deleskie@gmail.com<mailto:deleskie@gmail.com>> > wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com<mailto:greenjosh6499@gmail.com> <mailto:greenjosh6499@gmail.com<mailto:greenjosh6499@gmail.com>> > wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org<mailto:nanog@nanog.org> <mailto:nanog@nanog.org<mailto:nanog@nanog.org>> > wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net<mailto:jay@west.net> <mailto:jay@west.net<mailto:jay@west.net>> > wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net<mailto:jay@west.net> <mailto:jay@west.net<mailto:jay@west.net>> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
--
Josh Green.
-- Jon Sands MFI Labs https://fohdeesha.com/
On Jul 11, 2022, at 11:15 AM, Victor Kuarsingh <victor@jvknet.com> wrote:
This is the most they can and will say. For liabilities reasons, specifics are likely not in the cards. As most services ride over common service networks, its quite possible that a network substrate failure can have a number of upstream service impacts. The point here is that the CEO is directly addressing the customer base, which is needed here.
Yes, unless you get a side-leak from someone in-the-know, I expect either direct or CRTC will be the best source. I know a few of us are concerned about if we have any similar risk of impact to our environments, so we will be watching close for any hints. - Jared
* victor@jvknet.com (Victor Kuarsingh) [Mon 11 Jul 2022, 17:17 CEST]:
This is the most they can and will say. For liabilities reasons, specifics are likely not in the cards.
I doubt it. Given that emergency services were impacted you can count on a proper assessment being made available to a parliamentary inquiry. Too much broke to cover this up any further. -- Niels.
It's hard to believe that a same time maintenance affecting so many devices in the core network could be approved. Core networks are build with redundancy, so that failures can't completely destroy the whole network. If it would be just "particular devices" they could be isolated much faster. In this case the outage was so huge that it affected mostly all network gear, maybe all devices from one vendor. And even after most of the services restored we still observed partial connectivity in some areas that means the impact was mush deeper, in some places down to access layer. Kind regards, Andrey Shane Ronan писал(а) 2022-07-11 10:09:
What in depth analysis have you seen? Seems to me, this was a failure in a known maintenance activity, and they simply disconnected the devices under maintenance from the network.
Shane
On Mon, Jul 11, 2022 at 9:01 AM Andrey Kostin <ankost@podolsk.ru> wrote:
It's hard to believe that a same time maintenance affecting so many devices in the core network could be approved. Core networks are build with redundancy, so that failures can't completely destroy the whole network.
I think you might need to re-evaluate your assumption about how core networks are built. A well-designed core network will have layers of redundancy built in, with easy isolation of fault layers, yes. I've seen (and sometimes worked on) too many networks that didn't have enough budget for redundancy, and were built as a string of pearls, one router to the next; if any router in the string of pearls broke, the entire string of pearls would come crashing down, to abuse a metaphor just a bit too much. Really well-thought out redundancy takes a design team that has enough experience and enough focused hours in the day to think through different failure modes and lay out the design ahead of time, before purchases get made. Many real-world networks share the same engineers between design, deployment, and operation of the network--and in that model, operation and deployment always win over design when it comes time to allocate engineering hours. Likeise, if you didn't have the luxury of being able to lay out the design ahead of time, before purchasing hardware and leasing facilities, you're likely doing the best you can with locations that were contracted before you came into the picture, using hardware that was decided on before you had an opportunity to suggest better alternatives. Taking it a step further, and thinking about the large Facebook outage, even if you did well in the design phase, and chose two different vendors, with hardware redundancy and site redundancy in your entire core network, did you also think about redundancy and diversity for the O&M side of the house? Does each redundant data plane have a diverse control plane and management plane, or would an errant redistribution of BGP into IGP wipe out both data planes, and both hardware vendors at the same time? Likewise, if a bad configuration push isolates your core network nodes from the "God box" that controls the device configurations, do you have redundancy in connectivity to that "God box" so that you can restore known-good configurations to your core network sites, or are you stuck dispatching engineers with laptops and USB sticks with configs on them to get back to a working condition again? As you follow the control of core networks back up the chain, you ultimately realize that no network is truly redundant and diverse. Every network ultimately comes back to a single point of failure, and the only distinction you can make is how far up the ladder you climb before you discover that single point of failure. Thanks! Matt
I’m “guessing” based on all the services that were impacted the outage was likely cause by a change that caused a routing change in their multi-service network which overloaded many network devices, and by isolating the source the routes or traffic the rest of the network was able to recover. But just a guess. Shane
On Jul 11, 2022, at 4:22 PM, Matthew Petach <mpetach@netflight.com> wrote:
On Mon, Jul 11, 2022 at 9:01 AM Andrey Kostin <ankost@podolsk.ru> wrote: It's hard to believe that a same time maintenance affecting so many devices in the core network could be approved. Core networks are build with redundancy, so that failures can't completely destroy the whole network.
I think you might need to re-evaluate your assumption about how core networks are built.
A well-designed core network will have layers of redundancy built in, with easy isolation of fault layers, yes.
I've seen (and sometimes worked on) too many networks that didn't have enough budget for redundancy, and were built as a string of pearls, one router to the next; if any router in the string of pearls broke, the entire string of pearls would come crashing down, to abuse a metaphor just a bit too much.
Really well-thought out redundancy takes a design team that has enough experience and enough focused hours in the day to think through different failure modes and lay out the design ahead of time, before purchases get made. Many real-world networks share the same engineers between design, deployment, and operation of the network--and in that model, operation and deployment always win over design when it comes time to allocate engineering hours. Likeise, if you didn't have the luxury of being able to lay out the design ahead of time, before purchasing hardware and leasing facilities, you're likely doing the best you can with locations that were contracted before you came into the picture, using hardware that was decided on before you had an opportunity to suggest better alternatives.
Taking it a step further, and thinking about the large Facebook outage, even if you did well in the design phase, and chose two different vendors, with hardware redundancy and site redundancy in your entire core network, did you also think about redundancy and diversity for the O&M side of the house? Does each redundant data plane have a diverse control plane and management plane, or would an errant redistribution of BGP into IGP wipe out both data planes, and both hardware vendors at the same time? Likewise, if a bad configuration push isolates your core network nodes from the "God box" that controls the device configurations, do you have redundancy in connectivity to that "God box" so that you can restore known-good configurations to your core network sites, or are you stuck dispatching engineers with laptops and USB sticks with configs on them to get back to a working condition again?
As you follow the control of core networks back up the chain, you ultimately realize that no network is truly redundant and diverse. Every network ultimately comes back to a single point of failure, and the only distinction you can make is how far up the ladder you climb before you discover that single point of failure.
Thanks!
Matt
This is actually not the case. Cable service in some regional areas was restored as late as on Monday and even in the same geographical area there was a partial connectivity, so it looks less probable that network could be overloaded only in some isolated segments longer than the rest of the network and require manual intervention to fix it. OTOH, overloading network core of such big network should be hardly possible. Possible route "poisoning" could be isolated, so that it doesn't affect the whole network cost to cost. It must be very unlucky coincidence of such event to happen on such big scale. Maybe caused by HW or SW bug. I'm talking just about probabilities of different scenarios and trying to compare them. So far, IMO provided version doesn't look very convincing. Kind regards, Andrey Kostin sronan@ronan-online.com писал(а) 2022-07-11 20:18:
I’m “guessing” based on all the services that were impacted the outage was likely cause by a change that caused a routing change in their multi-service network which overloaded many network devices, and by isolating the source the routes or traffic the rest of the network was able to recover.
But just a guess.
Shane
FYI --- this is now making email rounds and is available to view and sign on CHANGE.ORG On Friday, Rogers experienced its second nationwide outage in as many years. Hundreds of thousands of Canadians were unable to access internet and cell services for hours, despite paying some of the highest fees in the world. Petition starter Celina is calling on the CRTC to protect Canadians from exorbitant cell and internet fees and more than 275,000 frustrated Canadians have joined her. You can show your support by signing the petition now. <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQAAAAC> *Canadians Are Being SCAMMED - Tell the CRTC We Demand Better Cell Phone Service! <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQAAAAC>* *276,541 have signed* Celina London’s petition. Let’s get to 300,000! Sign now with a click <https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAAAAAg~~> CANADIANS ARE BEING SCAMMED! Here in Canada we pay extremely high prices for our cell phones and cable in comparison to countries like Australia, United Kingdom and United States of America. As of August 2019, a plan with 30GB data from Australia’s Vodafone network is $45. In Canada, major providers like Rogers and Telus charge $95 for 20GB at max speed. This is unethical and not okay. It’s time we stand up for our hard-earned dollars and demand limitations be put in place to stop the wallet gouging of hard-working Canadians. We must force our telecommunications companies to provide Canadians better service for the astronomically high price they’re forced to pay for what is a necessity to function in our high-tech society in 2019. Please act now and sign your name below to show the CRTC (Canadian Radio-Television and Telecommunications Commission) that we’ve had ENOUGH and demand a change to our current telecommunications laws and regulations to implement a limit on price and an immediate increase in service provided for both existing and new customers. Once you’ve signed please take the time to share with your friends and family as this affects us all and together we can make a big difference and be heard! Thank you! Celina London - concerned and annoyed Canadian. Sign now with a click <https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAAAAAg~~> Visit petition page <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQAAAAC> *At Change.org, we believe in the voice of everyday people.* Is there something that you want to change? *Start a petition today <https://click.e.change.org/f/a/pbTbHw7ifps5zEHVN4Z5sQ~~/AANj1QA~/RgRkruHwP0TzaHR0cHM6Ly93d3cuY2hhbmdlLm9yZy9zdGFydC1hLXBldGl0aW9uP2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnV0bV9jYW1wYWlnbj02MmI5ZjNkMzYzMDQ0ZWExYmFhY2ZiYWRkMzYyMWI5OCZ1dG1fY29udGVudD1pbml0aWFsX3YwXzBfMiZ1dG1fbWVkaXVtPWVtYWlsJnV0bV9zb3VyY2U9YWFfYXVnbWVudGVkX3NpZ25fYXNrJnV0bV90ZXJtPWNzVwNzcGNCCmK79VzMYpjFNHRSEGRucGNvbkBnbWFpbC5jb21YBAAAAAI~>* The person (or organization) who started this petition is not affiliated with Change.org. Change.org did not create this petition and is not responsible for the petition content. On Mon, Jul 11, 2022 at 5:40 AM Jon Sands <fohdeesha@gmail.com> wrote:
Given the outage was so bad it was disrupting select E911 services nationwide for something like 24+ hours, it's great to see such an in depth analysis and plan of action to prevent such things in the future. bravo rogers
On 7/10/2022 2:55 PM, L F wrote:
fyi - see BOLD.
A Message from Rogers President and CEO
Dear Valued Customer,
As you know, we experienced a service outage across the Rogers, Fido, Chatr and Cityfone wireless networks on Friday.
I am reaching out to share that our services have been restored, and our networks and systems are close to fully operational. Our technical teams are continuing to monitor for any remaining intermittent issues. I also want to outline an action plan we are putting in place to address what happened.
I want to share what we know about what happened on Friday. We now believe *we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction. We disconnected the specific equipment and redirected traffic,* which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.
On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas@gmail.com> wrote:
Lest we ALL MOVE ON….
Yes he said
“RETARD” =
The delay in processing the current status of a Situation…
Lets move on to bigger n scarier Post mortems:
Was This or Was this NOT
A CYBERATTACK ???
Done.
Lets deal with REAL threats not PC references.
Time to evaluate n re eval Our NTS!!
Veni vidi vici 2022
On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not.
Does this person need directions to some local mental healthcare resources?
On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
I can't either, but the reality right now seems to be that 911 calls are failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
-----Original Message----- From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of Eric Kuhnke Sent: Friday, 8 July, 2022 13:34 To: jim deleskie <deleskie@gmail.com> Cc: NANOG list <nanog@nanog.org> Subject: Re: Rogers Outage Canada
I have seen anecdotal reports that the mobile network is in a half broken state that phones remain registered to, so a 911 call will attempt and then fail.
This is unlike what would happen if you had a US/Canada cellphone with battery power but no SIM card in it that would search for any available network in RF range for a 911 call if needed.
On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com <mailto:deleskie@gmail.com> > wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org <mailto:nanog@nanog.org> > wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net <mailto:jay@west.net> > wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net <mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
--
Josh Green.
-- Jon Sands MFI Labshttps://fohdeesha.com/
341,296 have signed. Let’s get to 500,000! At 500,000 signatures, this petition becomes one of the top signed on Change.org! On Mon, Jul 11, 2022 at 3:44 PM L F <liz.fazekas@gmail.com> wrote: > FYI --- this is now making email rounds and is available to view and sign > on CHANGE.ORG > > > > On Friday, Rogers experienced its second nationwide outage in as many > years. Hundreds of thousands of Canadians were unable to access internet > and cell services for hours, despite paying some of the highest fees in the > world. Petition starter Celina is calling on the CRTC to protect Canadians > from exorbitant cell and internet fees and more than 275,000 frustrated > Canadians have joined her. You can show your support by signing the > petition now. > > <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQAAAAC> > *Canadians Are Being SCAMMED - Tell the CRTC We Demand Better Cell Phone > Service! > <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQAAAAC>* > > *276,541 have signed* Celina London’s petition. Let’s get to 300,000! > Sign now with a click > <https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAAAAAg~~> > > CANADIANS ARE BEING SCAMMED! > > Here in Canada we pay extremely high prices for our cell phones and cable > in comparison to countries like Australia, United Kingdom and United States > of America. > > As of August 2019, a plan with 30GB data from Australia’s Vodafone network > is $45. In Canada, major providers like Rogers and Telus charge $95 for > 20GB at max speed. > > This is unethical and not okay. It’s time we stand up for our hard-earned > dollars and demand limitations be put in place to stop the wallet gouging > of hard-working Canadians. We must force our telecommunications companies > to provide Canadians better service for the astronomically high price > they’re forced to pay for what is a necessity to function in our high-tech > society in 2019. > > Please act now and sign your name below to show the CRTC (Canadian > Radio-Television and Telecommunications Commission) that we’ve had ENOUGH > and demand a change to our current telecommunications laws and regulations > to implement a limit on price and an immediate increase in service provided > for both existing and new customers. > > Once you’ve signed please take the time to share with your friends and > family as this affects us all and together we can make a big difference and > be heard! > > Thank you! > > Celina London - concerned and annoyed Canadian. > > > Sign now with a click > <https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAAAAAg~~> > > Visit petition page > <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQAAAAC> > > *At Change.org, we believe in the voice of everyday people.* Is there > something that you want to change? > *Start a petition today > <https://click.e.change.org/f/a/pbTbHw7ifps5zEHVN4Z5sQ~~/AANj1QA~/RgRkruHwP0TzaHR0cHM6Ly93d3cuY2hhbmdlLm9yZy9zdGFydC1hLXBldGl0aW9uP2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnV0bV9jYW1wYWlnbj02MmI5ZjNkMzYzMDQ0ZWExYmFhY2ZiYWRkMzYyMWI5OCZ1dG1fY29udGVudD1pbml0aWFsX3YwXzBfMiZ1dG1fbWVkaXVtPWVtYWlsJnV0bV9zb3VyY2U9YWFfYXVnbWVudGVkX3NpZ25fYXNrJnV0bV90ZXJtPWNzVwNzcGNCCmK79VzMYpjFNHRSEGRucGNvbkBnbWFpbC5jb21YBAAAAAI~>* > > The person (or organization) who started this petition is not affiliated > with Change.org. Change.org did not create this petition and is not > responsible for the petition content. > > On Mon, Jul 11, 2022 at 5:40 AM Jon Sands <fohdeesha@gmail.com> wrote: > >> Given the outage was so bad it was disrupting select E911 services >> nationwide for something like 24+ hours, it's great to see such an in depth >> analysis and plan of action to prevent such things in the future. bravo >> rogers >> >> On 7/10/2022 2:55 PM, L F wrote: >> >> fyi - see BOLD. >> >> A Message from Rogers President and CEO >> >> >> Dear Valued Customer, >> >> As you know, we experienced a service outage across the Rogers, Fido, >> Chatr and Cityfone wireless networks on Friday. >> >> I am reaching out to share that our services have been restored, and our >> networks and systems are close to fully operational. Our technical teams >> are continuing to monitor for any remaining intermittent issues. I also >> want to outline an action plan we are putting in place to address what >> happened. >> >> I want to share what we know about what happened on Friday. We now >> believe *we’ve narrowed the cause to a network system failure following >> a maintenance update in our core network, which caused some of our routers >> to malfunction. We disconnected the specific equipment and redirected >> traffic,* which allowed our network and services to come back online >> over time as we managed traffic volumes returning to normal levels. >> >> >> >> On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas@gmail.com> wrote: >> >>> >>> Lest we ALL MOVE ON…. >>> >>> >>> Yes he said >>> >>> “RETARD” = >>> >>> The delay in processing the current status of a Situation… >>> >>> >>> Lets move on to bigger n scarier Post mortems: >>> >>> >>> Was This or Was this NOT >>> >>> A CYBERATTACK ??? >>> >>> Done. >>> >>> Lets deal with REAL threats not PC references. >>> >>> Time to evaluate n re eval Our NTS!! >>> >>> Veni vidi vici 2022 >>> >>> >>> >>> >>> On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> >>> wrote: >>> >>>> Can we have a discussion with the list admins about a list member >>>> appending a threat of violence to their outbound emails? Whether serious >>>> or not. >>>> >>>> Does this person need directions to some local mental healthcare >>>> resources? >>>> >>>> >>>> On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote: >>>> >>>>> >>>>> >I can't either, but the reality right now seems to be that 911 calls >>>>> are >>>>> >failing for anyone on a Rogers cellphone. >>>>> >>>>> This is par for the course. These people chose to deal with Rogers >>>>> despite knowing the consequences. It is like if you bought a Rogers >>>>> Snowblower and it did not work. That would mean that people who bought the >>>>> Rogers Snowblower will not be using it to get rid of the snow that is >>>>> preventing them from leaving their house. >>>>> >>>>> Mutatis mutandis when Rogers is down things that are Rogers dependent >>>>> will not work. >>>>> >>>>> Some people are so retarded it is astonishing! >>>>> >>>>> -- >>>>> (CAUTION) You are advised that if you attack my person or property, >>>>> you will be put down in accordance with the provisions of section 34 & 35 >>>>> of the Criminal Code respectively. If you are brandishing (or in >>>>> possession) of a weapon then lethal force will be applied to your person in >>>>> accordance with the law. This means that your misadventures may end in >>>>> your death. Consider yourself cautioned and govern your actions >>>>> appropriately. >>>>> >>>>> >-----Original Message----- >>>>> >From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf >>>>> Of >>>>> >Eric Kuhnke >>>>> >Sent: Friday, 8 July, 2022 13:34 >>>>> >To: jim deleskie <deleskie@gmail.com> >>>>> >Cc: NANOG list <nanog@nanog.org> >>>>> >Subject: Re: Rogers Outage Canada >>>>> > >>>>> > >>>>> >I have seen anecdotal reports that the mobile network is in a half >>>>> broken >>>>> >state that phones remain registered to, so a 911 call will attempt and >>>>> >then fail. >>>>> > >>>>> > >>>>> >This is unlike what would happen if you had a US/Canada cellphone with >>>>> >battery power but no SIM card in it that would search for any >>>>> available >>>>> >network in RF range for a 911 call if needed. >>>>> > >>>>> > >>>>> >On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com >>>>> ><mailto:deleskie@gmail.com> > wrote: >>>>> > >>>>> > >>>>> > i cant see BGP taking out SS7. >>>>> > >>>>> > -jim >>>>> > >>>>> > On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 >>>>> ><greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote: >>>>> > >>>>> > >>>>> > According to Cloudflare Radar >>>>> ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , >>>>> Rogers >>>>> >BGP announcements spiked massively to levels 536,777% higher than >>>>> normal >>>>> >(343,601 vs 64 normally) just minutes before the outage. I would not >>>>> be >>>>> >surprised if this happened to be the culprit. >>>>> > >>>>> > Regards, >>>>> > Josh Green >>>>> > >>>>> > On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via >>>>> NANOG >>>>> ><nanog@nanog.org <mailto:nanog@nanog.org> > wrote: >>>>> > >>>>> > >>>>> > In the early hours of the morning around 2-3am >>>>> my modem >>>>> >got hit with a configuration update that caused a DHCP release that >>>>> >wasn't renewed for about two hours, after rollback the connection was >>>>> >fine for 3 hours before this network wide outage. >>>>> > >>>>> > >>>>> > Maybe a failed night time update was attempted >>>>> again >>>>> >during office hours, I've heard daytime guys are still WFH and night >>>>> >shift is in building. >>>>> > >>>>> > >>>>> > I expect we'll never get a real explanation. >>>>> Rogers is >>>>> >notorious for withholding any type of helpful or technical >>>>> information. >>>>> > >>>>> > >>>>> > Sent from my inoperable Rogers Mobile via >>>>> emergency eSIM. >>>>> > >>>>> > >>>>> > Regards, >>>>> > >>>>> > Andrew Paolucci >>>>> > -------- Original Message -------- >>>>> > On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < >>>>> jay@west.net >>>>> ><mailto:jay@west.net> > wrote: >>>>> > >>>>> > >>>>> > On 7/8/22 07:44, Robert DeVita wrote: >>>>> > Does anyone >>>>> >have information on a widespread Rogers outage in Canada. I > have >>>>> >customers with multiple sites down. There's discussion on the Outages >>>>> >mailing list. Seems widespread, affecting all services, mobile, voice, >>>>> >Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net >>>>> ><mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 >>>>> - >>>>> >WB6RDV >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > >>>>> > Josh Green. >>>>> >>>>> >>>>> >>>>> >> >> >> -- >> Jon Sands >> MFI Labshttps://fohdeesha.com/ >> >> -- Liz ******* 416.660.5456
Class action lawsuit has been filled in Quebec as well. https://lpclex.com/rogers/ ------- Original Message ------- On Monday, July 11th, 2022 at 3:45 PM, L F <liz.fazekas@gmail.com> wrote:
341,296 have signed. Let’s get to 500,000!
At 500,000 signatures, this petition becomes one of the top signed on Change.org!
On Mon, Jul 11, 2022 at 3:44 PM L F <liz.fazekas@gmail.com> wrote:
FYI --- this is now making email rounds and is available to view and sign on CHANGE.ORG
On Friday, Rogers experienced its second nationwide outage in as many years. Hundreds of thousands of Canadians were unable to access internet and cell services for hours, despite paying some of the highest fees in the world. Petition starter Celina is calling on the CRTC to protect Canadians from exorbitant cell and internet fees and more than 275,000 frustrated Canadians have joined her. You can show your support by signing the petition now.
https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4S...
[Canadians Are Being SCAMMED - Tell the CRTC We Demand Better Cell Phone Service!](https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4S...)
276,541 have signed Celina London’s petition. Let’s get to 300,000!
[Sign now with a click](https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4S...)
CANADIANS ARE BEING SCAMMED!
Here in Canada we pay extremely high prices for our cell phones and cable in comparison to countries like Australia, United Kingdom and United States of America.
As of August 2019, a plan with 30GB data from Australia’s Vodafone network is $45. In Canada, major providers like Rogers and Telus charge $95 for 20GB at max speed.
This is unethical and not okay. It’s time we stand up for our hard-earned dollars and demand limitations be put in place to stop the wallet gouging of hard-working Canadians. We must force our telecommunications companies to provide Canadians better service for the astronomically high price they’re forced to pay for what is a necessity to function in our high-tech society in 2019.
Please act now and sign your name below to show the CRTC (Canadian Radio-Television and Telecommunications Commission) that we’ve had ENOUGH and demand a change to our current telecommunications laws and regulations to implement a limit on price and an immediate increase in service provided for both existing and new customers.
Once you’ve signed please take the time to share with your friends and family as this affects us all and together we can make a big difference and be heard!
Thank you!
Celina London - concerned and annoyed Canadian.
[Sign now with a click](https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4S...)
[Visit petition page](https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4S...)
At Change.org, we believe in the voice of everyday people. Is there something that you want to change?
[Start a petition today](https://click.e.change.org/f/a/pbTbHw7ifps5zEHVN4Z5sQ~~/AANj1QA~/RgRkruHwP0T...)
The person (or organization) who started this petition is not affiliated with Change.org. Change.org did not create this petition and is not responsible for the petition content.
On Mon, Jul 11, 2022 at 5:40 AM Jon Sands <fohdeesha@gmail.com> wrote:
Given the outage was so bad it was disrupting select E911 services nationwide for something like 24+ hours, it's great to see such an in depth analysis and plan of action to prevent such things in the future. bravo rogers
On 7/10/2022 2:55 PM, L F wrote:
fyi - see BOLD.
A Message from Rogers President and CEO
Dear Valued Customer,
As you know, we experienced a service outage across the Rogers, Fido, Chatr and Cityfone wireless networks on Friday.
I am reaching out to share that our services have been restored, and our networks and systems are close to fully operational. Our technical teams are continuing to monitor for any remaining intermittent issues. I also want to outline an action plan we are putting in place to address what happened.
I want to share what we know about what happened on Friday. We now believe we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction. We disconnected the specific equipment and redirected traffic, which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.
On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas@gmail.com> wrote:
Lest we ALL MOVE ON….
Yes he said
“RETARD” =
The delay in processing the current status of a Situation…
Lets move on to bigger n scarier Post mortems:
Was This or Was this NOT
A CYBERATTACK ???
Done.
Lets deal with REAL threats not PC references.
Time to evaluate n re eval Our NTS!!
Veni vidi vici 2022
On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not.
Does this person need directions to some local mental healthcare resources?
On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
>>I can't either, but the reality right now seems to be that 911 calls are >>failing for anyone on a Rogers cellphone. > > This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house. > > Mutatis mutandis when Rogers is down things that are Rogers dependent will not work. > > Some people are so retarded it is astonishing! > > -- > (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately. > >>-----Original Message----- >>From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of >>Eric Kuhnke >>Sent: Friday, 8 July, 2022 13:34 >>To: jim deleskie <deleskie@gmail.com> >>Cc: NANOG list <nanog@nanog.org> >>Subject: Re: Rogers Outage Canada >> >> >>I have seen anecdotal reports that the mobile network is in a half broken >>state that phones remain registered to, so a 911 call will attempt and >>then fail. >> >> >>This is unlike what would happen if you had a US/Canada cellphone with >>battery power but no SIM card in it that would search for any available >>network in RF range for a 911 call if needed. >> >> >>On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com >><mailto:deleskie@gmail.com> > wrote: >> >> >> i cant see BGP taking out SS7. >> >> -jim >> >> On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 >><greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote: >> >> >> According to Cloudflare Radar >><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers >>BGP announcements spiked massively to levels 536,777% higher than normal >>(343,601 vs 64 normally) just minutes before the outage. I would not be >>surprised if this happened to be the culprit. >> >> Regards, >> Josh Green >> >> On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG >><nanog@nanog.org <mailto:nanog@nanog.org> > wrote: >> >> >> In the early hours of the morning around 2-3am my modem >>got hit with a configuration update that caused a DHCP release that >>wasn't renewed for about two hours, after rollback the connection was >>fine for 3 hours before this network wide outage. >> >> >> Maybe a failed night time update was attempted again >>during office hours, I've heard daytime guys are still WFH and night >>shift is in building. >> >> >> I expect we'll never get a real explanation. Rogers is >>notorious for withholding any type of helpful or technical information. >> >> >> Sent from my inoperable Rogers Mobile via emergency eSIM. >> >> >> Regards, >> >> Andrew Paolucci >> -------- Original Message -------- >> On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net >><mailto:jay@west.net> > wrote: >> >> >> On 7/8/22 07:44, Robert DeVita wrote: > Does anyone >>have information on a widespread Rogers outage in Canada. I > have >>customers with multiple sites down. There's discussion on the Outages >>mailing list. Seems widespread, affecting all services, mobile, voice, >>Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net >><mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - >>WB6RDV >> >> >> >> -- >> >> Josh Green.
-- Jon Sands MFI Labs https://fohdeesha.com/
--
Liz ******* 416.660.5456
Looks like new agreements might be put in place to prevent this from happening again in the future. https://www.ctvnews.ca/business/champagne-directs-major-telecoms-to-come-up-... -Josh On Mon, Jul 11, 2022 at 7:35 PM Andrew Paolucci via NANOG <nanog@nanog.org> wrote:
Class action lawsuit has been filled in Quebec as well.
------- Original Message -------
On Monday, July 11th, 2022 at 3:45 PM, L F <liz.fazekas@gmail.com> wrote:
341,296 have signed. Let’s get to 500,000! At 500,000 signatures, this petition becomes one of the top signed on Change.org!
On Mon, Jul 11, 2022 at 3:44 PM L F <liz.fazekas@gmail.com> wrote:
FYI --- this is now making email rounds and is available to view and sign on CHANGE.ORG
On Friday, Rogers experienced its second nationwide outage in as many years. Hundreds of thousands of Canadians were unable to access internet and cell services for hours, despite paying some of the highest fees in the world. Petition starter Celina is calling on the CRTC to protect Canadians from exorbitant cell and internet fees and more than 275,000 frustrated Canadians have joined her. You can show your support by signing the petition now.
<https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQAAAAC> *Canadians Are Being SCAMMED - Tell the CRTC We Demand Better Cell Phone Service! <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQAAAAC>*
*276,541 have signed* Celina London’s petition. Let’s get to 300,000! Sign now with a click <https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAAAAAg~~>
CANADIANS ARE BEING SCAMMED!
Here in Canada we pay extremely high prices for our cell phones and cable in comparison to countries like Australia, United Kingdom and United States of America.
As of August 2019, a plan with 30GB data from Australia’s Vodafone network is $45. In Canada, major providers like Rogers and Telus charge $95 for 20GB at max speed.
This is unethical and not okay. It’s time we stand up for our hard-earned dollars and demand limitations be put in place to stop the wallet gouging of hard-working Canadians. We must force our telecommunications companies to provide Canadians better service for the astronomically high price they’re forced to pay for what is a necessity to function in our high-tech society in 2019.
Please act now and sign your name below to show the CRTC (Canadian Radio-Television and Telecommunications Commission) that we’ve had ENOUGH and demand a change to our current telecommunications laws and regulations to implement a limit on price and an immediate increase in service provided for both existing and new customers.
Once you’ve signed please take the time to share with your friends and family as this affects us all and together we can make a big difference and be heard!
Thank you!
Celina London - concerned and annoyed Canadian.
*At Change.org, we believe in the voice of everyday people.* Is there something that you want to change? *Start a petition today <https://click.e.change.org/f/a/pbTbHw7ifps5zEHVN4Z5sQ~~/AANj1QA~/RgRkruHwP0TzaHR0cHM6Ly93d3cuY2hhbmdlLm9yZy9zdGFydC1hLXBldGl0aW9uP2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnV0bV9jYW1wYWlnbj02MmI5ZjNkMzYzMDQ0ZWExYmFhY2ZiYWRkMzYyMWI5OCZ1dG1fY29udGVudD1pbml0aWFsX3YwXzBfMiZ1dG1fbWVkaXVtPWVtYWlsJnV0bV9zb3VyY2U9YWFfYXVnbWVudGVkX3NpZ25fYXNrJnV0bV90ZXJtPWNzVwNzcGNCCmK79VzMYpjFNHRSEGRucGNvbkBnbWFpbC5jb21YBAAAAAI~>*
The person (or organization) who started this petition is not affiliated with Change.org. Change.org did not create this petition and is not responsible for the petition content.
On Mon, Jul 11, 2022 at 5:40 AM Jon Sands <fohdeesha@gmail.com> wrote:
Given the outage was so bad it was disrupting select E911 services nationwide for something like 24+ hours, it's great to see such an in depth analysis and plan of action to prevent such things in the future. bravo rogers
On 7/10/2022 2:55 PM, L F wrote:
fyi - see BOLD.
A Message from Rogers President and CEO
Dear Valued Customer,
As you know, we experienced a service outage across the Rogers, Fido, Chatr and Cityfone wireless networks on Friday.
I am reaching out to share that our services have been restored, and our networks and systems are close to fully operational. Our technical teams are continuing to monitor for any remaining intermittent issues. I also want to outline an action plan we are putting in place to address what happened.
I want to share what we know about what happened on Friday. We now believe *we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction. We disconnected the specific equipment and redirected traffic,* which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.
On Sat, Jul 9, 2022 at 9:09 PM L F <liz.fazekas@gmail.com> wrote:
Lest we ALL MOVE ON….
Yes he said
“RETARD” =
The delay in processing the current status of a Situation…
Lets move on to bigger n scarier Post mortems:
Was This or Was this NOT
A CYBERATTACK ???
Done.
Lets deal with REAL threats not PC references.
Time to evaluate n re eval Our NTS!!
Veni vidi vici 2022
On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke <eric.kuhnke@gmail.com> wrote:
Can we have a discussion with the list admins about a list member appending a threat of violence to their outbound emails? Whether serious or not.
Does this person need directions to some local mental healthcare resources?
On Sat, 9 Jul 2022 at 08:48, Keith Medcalf <kmedcalf@dessus.com> wrote:
>I can't either, but the reality right now seems to be that 911 calls are >failing for anyone on a Rogers cellphone.
This is par for the course. These people chose to deal with Rogers despite knowing the consequences. It is like if you bought a Rogers Snowblower and it did not work. That would mean that people who bought the Rogers Snowblower will not be using it to get rid of the snow that is preventing them from leaving their house.
Mutatis mutandis when Rogers is down things that are Rogers dependent will not work.
Some people are so retarded it is astonishing!
-- (CAUTION) You are advised that if you attack my person or property, you will be put down in accordance with the provisions of section 34 & 35 of the Criminal Code respectively. If you are brandishing (or in possession) of a weapon then lethal force will be applied to your person in accordance with the law. This means that your misadventures may end in your death. Consider yourself cautioned and govern your actions appropriately.
>-----Original Message----- >From: NANOG <nanog-bounces+kmedcalf=dessus.com@nanog.org> On Behalf Of >Eric Kuhnke >Sent: Friday, 8 July, 2022 13:34 >To: jim deleskie <deleskie@gmail.com> >Cc: NANOG list <nanog@nanog.org> >Subject: Re: Rogers Outage Canada > > >I have seen anecdotal reports that the mobile network is in a half broken >state that phones remain registered to, so a 911 call will attempt and >then fail. > > >This is unlike what would happen if you had a US/Canada cellphone with >battery power but no SIM card in it that would search for any available >network in RF range for a 911 call if needed. > > >On Fri, 8 Jul 2022 at 12:31, jim deleskie <deleskie@gmail.com ><mailto:deleskie@gmail.com> > wrote: > > > i cant see BGP taking out SS7. > > -jim > > On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 ><greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com> > wrote: > > > According to Cloudflare Radar ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers >BGP announcements spiked massively to levels 536,777% higher than normal >(343,601 vs 64 normally) just minutes before the outage. I would not be >surprised if this happened to be the culprit. > > Regards, > Josh Green > > On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG ><nanog@nanog.org <mailto:nanog@nanog.org> > wrote: > > > In the early hours of the morning around 2-3am my modem >got hit with a configuration update that caused a DHCP release that >wasn't renewed for about two hours, after rollback the connection was >fine for 3 hours before this network wide outage. > > > Maybe a failed night time update was attempted again >during office hours, I've heard daytime guys are still WFH and night >shift is in building. > > > I expect we'll never get a real explanation. Rogers is >notorious for withholding any type of helpful or technical information. > > > Sent from my inoperable Rogers Mobile via emergency eSIM. > > > Regards, > > Andrew Paolucci > -------- Original Message -------- > On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net ><mailto:jay@west.net> > wrote: > > > On 7/8/22 07:44, Robert DeVita wrote: > Does anyone >have information on a widespread Rogers outage in Canada. I > have >customers with multiple sites down. There's discussion on the Outages >mailing list. Seems widespread, affecting all services, mobile, voice, >Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net ><mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - >WB6RDV > > > > -- > > Josh Green.
-- Jon Sands MFI Labshttps://fohdeesha.com/
-- Liz ******* 416.660.5456
--
*Josh Green.*
Seems like their external BGP peers are dead. If their internal network is the same, and they carry M2PA etc, or use BGP/MPLS to carry SS7 traffic, or their softswitches/STPs rely on BGP/MPLS in some way then SS7 might exist, but not be functional. -- Pete Baldwin On 2022-07-08 15:30, jim deleskie wrote:
i cant see BGP taking out SS7.
-jim
On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004 <greenjosh6499@gmail.com <mailto:greenjosh6499@gmail.com>> wrote:
According to Cloudflare Radar <https://radar.cloudflare.com/asn/812?date_filter=last_24_hours>, Rogers BGP announcements spiked massively to levels 536,777% higher than normal (343,601 vs 64 normally) just minutes before the outage. I would not be surprised if this happened to be the culprit.
Regards, Josh Green
On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG <nanog@nanog.org <mailto:nanog@nanog.org>> wrote:
In the early hours of the morning around 2-3am my modem got hit with a configuration update that caused a DHCP release that wasn't renewed for about two hours, after rollback the connection was fine for 3 hours before this network wide outage.
Maybe a failed night time update was attempted again during office hours, I've heard daytime guys are still WFH and night shift is in building.
I expect we'll never get a real explanation. Rogers is notorious for withholding any type of helpful or technical information.
Sent from my inoperable Rogers Mobile via emergency eSIM.
Regards,
Andrew Paolucci -------- Original Message -------- On Jul. 8, 2022, 1:48 p.m., Jay Hennigan < jay@west.net <mailto:jay@west.net>> wrote:
On 7/8/22 07:44, Robert DeVita wrote: > Does anyone have information on a widespread Rogers outage in Canada. I > have customers with multiple sites down. There's discussion on the Outages mailing list. Seems widespread, affecting all services, mobile, voice, Internet. No cause or ETR posted yet. -- Jay Hennigan - jay@west.net <mailto:jay@west.net> Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
-- *Josh Green.*
participants (21)
-
Alain Hebert -
Andrew Paolucci -
Andrey Kostin -
Eric Kuhnke -
Jared Mauch -
JASON BOTHE -
Jay Hennigan -
jim deleskie -
Jon Sands -
Keith Medcalf -
Keith Stokes -
L F -
Matthew Petach -
Niels Bakker -
Pete Baldwin - TCC -
Robert DeVita -
Shane Ronan -
Snowmobile2004 -
sronan＠ronan-online.com -
Suresh Ramasubramanian -
Victor Kuarsingh