Appears to have affected at least some FirstNet SIMS as well. On Thu, Feb 22, 2024, 08:10 Ray Orsini via NANOG <nanog@nanog.org> wrote:

We're affected as well. Unable to dial out. I haven't found any official statement though.

[image: OIT Website] <https://www.oit.co/> Ray Orsini​​​​ Chief Executive Officer OIT, LLC 305.967.6756 x1009 <305.967.6756%20x1009> | 305.571.6272 ray@oit.co | [image: https://www.oit.co] <https://www.oit.co/> www.oit.co <https://www.oit.co/> oit.co/ray [image: Facebook] <https://go.oit.co/facebook> [image: LinkedIn] <https://go.oit.co/linkedin> [image: Twitter] <https://go.oit.co/twitter> [image: YouTube] <https://go.oit.co/youtube> How are we doing? We'd love to hear your feedback. https://go.oit.co/review <https://go.oit.co/review> ------------------------------ *From:* NANOG <nanog-bounces+ray=oit.co@nanog.org> on behalf of Robert DeVita <radevita@mejeticks.com> *Sent:* Thursday, February 22, 2024 8:03 AM *To:* nanog@nanog.org <nanog@nanog.org> *Subject:* Any info on AT&T Wireless Outage?

Reports have it starting at 4:30 a.m.. SOS on all phones..

*Robert DeVita**​**​**​**​*

*CEO and Founder*

t: (469) 581-2160 <(469)%20581-2160>

|

m: (469) 441-8864 <(469)%20441-8864>

e: radevita@mejeticks.com

|

w: mejeticks.com <https://www.mejeticks.com/>

a:

2323 N Akard Street

,

Dallas

,

75201

<https://www.linkedin.com/company/mejeticks/>

<https://twitter.com/mejeticks>

<https://www.facebook.com/mejeticks>

<https://linktr.ee/mejeticks>

No issues on FirstNet here in Houston metro, but AT&T consumer core still appears to be non-functional. ________________________________ From: NANOG on behalf of Ryan A. Krenzischek via NANOG Sent: Thursday, February 22, 2024 8:03 AM To: Ray Orsini Cc: nanog@nanog.org Subject: Re: Any info on AT&T Wireless Outage? The same as well for FirstNet but am now able to make calls. Others who are on AT&T are unable to receive or send calls. Enabling wifi calling on a regular AT&T phone (android) results in a 502 bad gateway error message. Ryan On Feb 22, 2024, at 08:11, Ray Orsini via NANOG <nanog@nanog.org> wrote:  We're affected as well. Unable to dial out. I haven't found any official statement though.

I’ve got about 500 Cradlepoints with AT&T throughout the US. I saw about 20 of them go down, all different locations, all around the same time (around 5am CST). FWIW these are across 3 different AT&T OU’s… AT&T IoT (with a FirstNet APN), AT&T 4G Business Broadband, and straight AT&T Business/Retail 4G. The response from my account manager is slightly comical: "Some of our customers are experiencing wireless service interruptions this morning. We are working urgently to restore service to them. We encourage the use of Wi-Fi calling until service is restored." Let me get right on activating Wifi Calling on my Cradlepoints… Chris Trudeau Network Engineer, Nitel From: NANOG <nanog-bounces+ctrudeau=nitelusa.com@nanog.org> On Behalf Of R. Leigh Hennig Sent: Thursday, February 22, 2024 8:17 AM To: Robert DeVita <radevita@mejeticks.com> Cc: nanog@nanog.org Subject: Re: Any info on AT&T Wireless Outage? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Word around the campfire is that it’s a Cisco issue. On Feb 22, 2024, at 8:03 AM, Robert DeVita <radevita@mejeticks.com<mailto:radevita@mejeticks.com>> wrote: Reports have it starting at 4:30 a.m.. SOS on all phones.. [cid:image001.jpg@01DA655D.39E82510] Robert DeVita​​​​ CEO and Founder t: (469) 581-2160<tel:(469)%20581-2160> | m: (469) 441-8864<tel:(469)%20441-8864> e: radevita@mejeticks.com<mailto:radevita@mejeticks.com> | w: mejeticks.com<https://www.mejeticks.com/> a: 2323 N Akard Street , Dallas , 75201 [cid:image002.png@01DA655D.39E82510]<https://www.linkedin.com/company/mejeticks/> [cid:image003.png@01DA655D.39E82510]<https://twitter.com/mejeticks> [cid:image004.png@01DA655D.39E82510]<https://www.facebook.com/mejeticks> [cid:image005.png@01DA655D.39E82510]<https://linktr.ee/mejeticks> ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

As widespread as it seemed to be, it feels like it would be quite a trick if it were a single piece of hardware. Firmware load that ended badly, I wonder? On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG <nanog@nanog.org> wrote:

Do you have the ability to expand on this at all? Do you mean a hardware failure of some kind IE router, optitcs, etc?

*From:* NANOG <nanog-bounces+gleato=advance-trading.com@nanog.org> *On Behalf Of *R. Leigh Hennig *Sent:* Thursday, February 22, 2024 8:17 AM *To:* Robert DeVita <radevita@mejeticks.com> *Cc:* nanog@nanog.org *Subject:* Re: Any info on AT&T Wireless Outage?

Word around the campfire is that it’s a Cisco issue.

On Feb 22, 2024, at 8:03 AM, Robert DeVita <radevita@mejeticks.com> wrote:

Reports have it starting at 4:30 a.m.. SOS on all phones..

*Robert DeVita**​**​**​**​*

*CEO and Founder*

t: (469) 581-2160 <(469)%20581-2160>

|

m: (469) 441-8864 <(469)%20441-8864>

e: radevita@mejeticks.com

|

w: mejeticks.com <https://www.mejeticks.com/>

a:

2323 N Akard Street

,

Dallas

,

75201

<https://www.linkedin.com/company/mejeticks/>

<https://twitter.com/mejeticks>

<https://www.facebook.com/mejeticks>

<https://linktr.ee/mejeticks>

The risk of trading futures and options can be substantial. All information, publications, and material used and distributed by Advance Trading Inc. shall be construed as a solicitation. ATI does not maintain an independent research department as defined in CFTC Regulation 1.71. Information obtained from third-party sources is believed to be reliable, but its accuracy is not guaranteed by Advance Trading Inc. Past performance is not necessarily indicative of future results.

I have several friends in the Nashville area who did not have AT&T Home broadband service this morning, could be unrelated but would point to a different issue than the SIMs. On Thu, Feb 22, 2024 at 1:16 PM John Councilman <jcouncilman@gmail.com> wrote:

From what I've read, they lost their database of SIM cards. I could be wrong of course.

On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel <dorn@hetzel.org> wrote:

...

As widespread as it seemed to be, it feels like it would be quite a trick if it were a single piece of hardware. Firmware load that ended badly, I wonder?

On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG <nanog@nanog.org> wrote:

...

Do you have the ability to expand on this at all? Do you mean a hardware failure of some kind IE router, optitcs, etc?

*From:* NANOG <nanog-bounces+gleato=advance-trading.com@nanog.org> *On Behalf Of *R. Leigh Hennig *Sent:* Thursday, February 22, 2024 8:17 AM *To:* Robert DeVita <radevita@mejeticks.com> *Cc:* nanog@nanog.org *Subject:* Re: Any info on AT&T Wireless Outage?

Word around the campfire is that it’s a Cisco issue.

On Feb 22, 2024, at 8:03 AM, Robert DeVita <radevita@mejeticks.com> wrote:

Reports have it starting at 4:30 a.m.. SOS on all phones..

*Robert DeVita**​**​**​**​*

*CEO and Founder*

t: (469) 581-2160 <(469)%20581-2160>

|

m: (469) 441-8864 <(469)%20441-8864>

e: radevita@mejeticks.com

|

w: mejeticks.com <https://www.mejeticks.com/>

a:

2323 N Akard Street

,

Dallas

,

75201

<https://www.linkedin.com/company/mejeticks/>

<https://twitter.com/mejeticks>

<https://www.facebook.com/mejeticks>

<https://linktr.ee/mejeticks>

The risk of trading futures and options can be substantial. All information, publications, and material used and distributed by Advance Trading Inc. shall be construed as a solicitation. ATI does not maintain an independent research department as defined in CFTC Regulation 1.71. Information obtained from third-party sources is believed to be reliable, but its accuracy is not guaranteed by Advance Trading Inc. Past performance is not necessarily indicative of future results.

Could it be someone fat fingered in preparation of this? "Starting in March, FirstNet will be the first and only wireless network to provide America’s first responders with always-on priority and preemption across 5G, expanding to include all AT&T 5G commercial spectrum. " https://www.firstnet.gov/newsroom/press-releases/firstnet-authority-att-anno... FirstNet Authority, AT&T Announce 10-Year Investment to Transform America’s Public Safety Broadband Network | First Responder Network Authority<https://www.firstnet.gov/newsroom/press-releases/firstnet-authority-att-announce-10-year-investment-transform-americas> The First Responder Network Authority is launching the next phase of FirstNet through a series of strategic investments totaling more than $8 billion over 10 years. The FirstNet Authority and its network partner, AT&T, join to unveil the latest network investment of $6.3 billion, delivering full 5G capabilities on FirstNet, expanded mission-critical services, and enhanced coverage. www.firstnet.gov ________________________________ From: NANOG <nanog-bounces+chris=kimball.cc@nanog.org> on behalf of Brandon Jackson <bjackson@napshome.net> Sent: Thursday, February 22, 2024 8:06 PM To: Patric Morgan <patric.morgan1@gmail.com> Cc: NANOG <nanog@nanog.org> Subject: Re: Any info on AT&T Wireless Outage? If it's AT&T wireless Home broadband Service then it would be affected just the same. It's still a cellular modem. It still has a SIM card. It's no different than a hotspot really, or just about any other cellular mobile device on the network. If it's wireline service though, that had nothing to do with this and would have been some separate issue. On Thu, Feb 22, 2024, 14:42 Patric Morgan <patric.morgan1@gmail.com<mailto:patric.morgan1@gmail.com>> wrote: I have several friends in the Nashville area who did not have AT&T Home broadband service this morning, could be unrelated but would point to a different issue than the SIMs. On Thu, Feb 22, 2024 at 1:16 PM John Councilman <jcouncilman@gmail.com<mailto:jcouncilman@gmail.com>> wrote: From what I've read, they lost their database of SIM cards. I could be wrong of course. On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel <dorn@hetzel.org<mailto:dorn@hetzel.org>> wrote: As widespread as it seemed to be, it feels like it would be quite a trick if it were a single piece of hardware. Firmware load that ended badly, I wonder? On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> wrote: Do you have the ability to expand on this at all? Do you mean a hardware failure of some kind IE router, optitcs, etc? From: NANOG <nanog-bounces+gleato=advance-trading.com@nanog.org<mailto:advance-trading.com@nanog.org>> On Behalf Of R. Leigh Hennig Sent: Thursday, February 22, 2024 8:17 AM To: Robert DeVita <radevita@mejeticks.com<mailto:radevita@mejeticks.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Any info on AT&T Wireless Outage? Word around the campfire is that it’s a Cisco issue. On Feb 22, 2024, at 8:03 AM, Robert DeVita <radevita@mejeticks.com<mailto:radevita@mejeticks.com>> wrote: Reports have it starting at 4:30 a.m.. SOS on all phones.. [X] Robert DeVita​​​​ CEO and Founder t: (469) 581-2160<tel:(469)%20581-2160> | m: (469) 441-8864<tel:(469)%20441-8864> e: radevita@mejeticks.com<mailto:radevita@mejeticks.com> | w: mejeticks.com<https://www.mejeticks.com/> a: 2323 N Akard Street , Dallas , 75201 [X]<https://www.linkedin.com/company/mejeticks/> [X]<https://twitter.com/mejeticks> [X]<https://www.facebook.com/mejeticks> [X]<https://linktr.ee/mejeticks> The risk of trading futures and options can be substantial. All information, publications, and material used and distributed by Advance Trading Inc. shall be construed as a solicitation. ATI does not maintain an independent research department as defined in CFTC Regulation 1.71. Information obtained from third-party sources is believed to be reliable, but its accuracy is not guaranteed by Advance Trading Inc. Past performance is not necessarily indicative of future results.

I read it as “someone pushed an ACL that wasn’t properly reviewed and it really screwed things up."

On Feb 27, 2024, at 21:41, Mark Seiden <mis@seiden.com> wrote:

aside from the official pablum that was released about an “incorrect process used” (which says exactly nothing) does anyone actually know anything accurate and more specific about the root cause?

(and why it took 11 hours to recover?)

...

On Feb 22, 2024, at 11:15 AM, John Councilman <jcouncilman@gmail.com> wrote:

From what I've read, they lost their database of SIM cards. I could be wrong of course.

On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel <dorn@hetzel.org <mailto:dorn@hetzel.org>> wrote:

...

As widespread as it seemed to be, it feels like it would be quite a trick if it were a single piece of hardware. Firmware load that ended badly, I wonder?

On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG <nanog@nanog.org <mailto:nanog@nanog.org>> wrote:

...

Do you have the ability to expand on this at all? Do you mean a hardware failure of some kind IE router, optitcs, etc?

From: NANOG <nanog-bounces+gleato=advance-trading.com@nanog.org <mailto:advance-trading.com@nanog.org>> On Behalf Of R. Leigh Hennig Sent: Thursday, February 22, 2024 8:17 AM To: Robert DeVita <radevita@mejeticks.com <mailto:radevita@mejeticks.com>> Cc: nanog@nanog.org <mailto:nanog@nanog.org> Subject: Re: Any info on AT&T Wireless Outage?

Word around the campfire is that it’s a Cisco issue.

On Feb 22, 2024, at 8:03 AM, Robert DeVita <radevita@mejeticks.com <mailto:radevita@mejeticks.com>> wrote:

Reports have it starting at 4:30 a.m.. SOS on all phones..

Robert DeVita​​​​

CEO and Founder

t: (469) 581-2160 <tel:(469)%20581-2160> |

m: (469) 441-8864 <tel:(469)%20441-8864> e: radevita@mejeticks.com <mailto:radevita@mejeticks.com> |

w: mejeticks.com <https://www.mejeticks.com/> a:

2323 N Akard Street

,

Dallas

,

75201

<https://www.linkedin.com/company/mejeticks/> <https://twitter.com/mejeticks> <https://www.facebook.com/mejeticks> <https://linktr.ee/mejeticks>

The risk of trading futures and options can be substantial. All information, publications, and material used and distributed by Advance Trading Inc. shall be construed as a solicitation. ATI does not maintain an independent research department as defined in CFTC Regulation 1.71. Information obtained from third-party sources is believed to be reliable, but its accuracy is not guaranteed by Advance Trading Inc. Past performance is not necessarily indicative of future results.

Where did you see this? Erik Prince was on the PBD podcast saying he has a 70% chance in his head it was China. I tend to learn towards human error from my experience in the IT biz. - J On Wed, Feb 28, 2024 at 10:58 AM <joel@joelesler.net> wrote:

I read it as “someone pushed an ACL that wasn’t properly reviewed and it really screwed things up."

On Feb 27, 2024, at 21:41, Mark Seiden <mis@seiden.com> wrote:

aside from the official pablum that was released about an “incorrect process used” (which says exactly nothing) does anyone actually know anything accurate and more specific about the root cause?

(and why it took 11 hours to recover?)

On Feb 22, 2024, at 11:15 AM, John Councilman <jcouncilman@gmail.com> wrote:

From what I've read, they lost their database of SIM cards. I could be wrong of course.

On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel <dorn@hetzel.org> wrote:

...

As widespread as it seemed to be, it feels like it would be quite a trick if it were a single piece of hardware. Firmware load that ended badly, I wonder?

On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG <nanog@nanog.org> wrote:

...

Do you have the ability to expand on this at all? Do you mean a hardware failure of some kind IE router, optitcs, etc?

*From:* NANOG <nanog-bounces+gleato=advance-trading.com@nanog.org> *On Behalf Of *R. Leigh Hennig *Sent:* Thursday, February 22, 2024 8:17 AM *To:* Robert DeVita <radevita@mejeticks.com> *Cc:* nanog@nanog.org *Subject:* Re: Any info on AT&T Wireless Outage?

Word around the campfire is that it’s a Cisco issue.

On Feb 22, 2024, at 8:03 AM, Robert DeVita <radevita@mejeticks.com> wrote:

Reports have it starting at 4:30 a.m.. SOS on all phones..

*Robert DeVita**​**​**​**​*

*CEO and Founder*

t: (469) 581-2160 <(469)%20581-2160>

|

m: (469) 441-8864 <(469)%20441-8864>

e: radevita@mejeticks.com

|

w: mejeticks.com <https://www.mejeticks.com/>

a:

2323 N Akard Street

,

Dallas

,

75201

<https://www.linkedin.com/company/mejeticks/>

<https://twitter.com/mejeticks>

<https://www.facebook.com/mejeticks>

<https://linktr.ee/mejeticks>

The risk of trading futures and options can be substantial. All information, publications, and material used and distributed by Advance Trading Inc. shall be construed as a solicitation. ATI does not maintain an independent research department as defined in CFTC Regulation 1.71. Information obtained from third-party sources is believed to be reliable, but its accuracy is not guaranteed by Advance Trading Inc. Past performance is not necessarily indicative of future results.

Erik Prince was on the PBD podcast saying he has a 70% chance in his head it was China.

Why do we care what his (almost certainly uninformed) opinion is? On Thu, Feb 29, 2024 at 2:56 PM Javier J <javier@advancedmachines.us> wrote:

Where did you see this? Erik Prince was on the PBD podcast saying he has a 70% chance in his head it was China. I tend to learn towards human error from my experience in the IT biz.

- J

On Wed, Feb 28, 2024 at 10:58 AM <joel@joelesler.net> wrote:

...

I read it as “someone pushed an ACL that wasn’t properly reviewed and it really screwed things up."

On Feb 27, 2024, at 21:41, Mark Seiden <mis@seiden.com> wrote:

aside from the official pablum that was released about an “incorrect process used” (which says exactly nothing) does anyone actually know anything accurate and more specific about the root cause?

(and why it took 11 hours to recover?)

On Feb 22, 2024, at 11:15 AM, John Councilman <jcouncilman@gmail.com> wrote:

From what I've read, they lost their database of SIM cards. I could be wrong of course.

On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel <dorn@hetzel.org> wrote:

...

As widespread as it seemed to be, it feels like it would be quite a trick if it were a single piece of hardware. Firmware load that ended badly, I wonder?

On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG <nanog@nanog.org> wrote:

...

Do you have the ability to expand on this at all? Do you mean a hardware failure of some kind IE router, optitcs, etc?

*From:* NANOG <nanog-bounces+gleato=advance-trading.com@nanog.org> *On Behalf Of *R. Leigh Hennig *Sent:* Thursday, February 22, 2024 8:17 AM *To:* Robert DeVita <radevita@mejeticks.com> *Cc:* nanog@nanog.org *Subject:* Re: Any info on AT&T Wireless Outage?

Word around the campfire is that it’s a Cisco issue.

On Feb 22, 2024, at 8:03 AM, Robert DeVita <radevita@mejeticks.com> wrote:

Reports have it starting at 4:30 a.m.. SOS on all phones..

*Robert DeVita**​**​**​**​*

*CEO and Founder*

t: (469) 581-2160 <(469)%20581-2160>

|

m: (469) 441-8864 <(469)%20441-8864>

e: radevita@mejeticks.com

|

w: mejeticks.com <https://www.mejeticks.com/>

a:

2323 N Akard Street

,

Dallas

,

75201

<https://www.linkedin.com/company/mejeticks/>

<https://twitter.com/mejeticks>

<https://www.facebook.com/mejeticks>

<https://linktr.ee/mejeticks>

The risk of trading futures and options can be substantial. All information, publications, and material used and distributed by Advance Trading Inc. shall be construed as a solicitation. ATI does not maintain an independent research department as defined in CFTC Regulation 1.71. Information obtained from third-party sources is believed to be reliable, but its accuracy is not guaranteed by Advance Trading Inc. Past performance is not necessarily indicative of future results.

The $5 you got back should help alleviate any concerns going forward Robert DeVita Founder & CEO Mejeticks c. 469-441-8864 e. radevita@mejeticks.com ________________________________ From: NANOG <nanog-bounces+radevita=mejeticks.com@nanog.org> on behalf of Mark Seiden <mis@seiden.com> Sent: Tuesday, February 27, 2024 8:41:49 PM To: John Councilman <jcouncilman@gmail.com> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Any info on AT&T Wireless Outage? aside from the official pablum that was released about an “incorrect process used” (which says exactly nothing) does anyone actually know anything accurate and more specific about the root cause? (and why it took 11 hours to recover?) On Feb 22, 2024, at 11:15 AM, John Councilman <jcouncilman@gmail.com> wrote: From what I've read, they lost their database of SIM cards. I could be wrong of course. On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel <dorn@hetzel.org<mailto:dorn@hetzel.org>> wrote: As widespread as it seemed to be, it feels like it would be quite a trick if it were a single piece of hardware. Firmware load that ended badly, I wonder? On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG <nanog@nanog.org<mailto:nanog@nanog.org>> wrote: Do you have the ability to expand on this at all? Do you mean a hardware failure of some kind IE router, optitcs, etc? From: NANOG <nanog-bounces+gleato=advance-trading.com@nanog.org<mailto:advance-trading.com@nanog.org>> On Behalf Of R. Leigh Hennig Sent: Thursday, February 22, 2024 8:17 AM To: Robert DeVita <radevita@mejeticks.com<mailto:radevita@mejeticks.com>> Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Any info on AT&T Wireless Outage? Word around the campfire is that it’s a Cisco issue. On Feb 22, 2024, at 8:03 AM, Robert DeVita <radevita@mejeticks.com<mailto:radevita@mejeticks.com>> wrote: Reports have it starting at 4:30 a.m.. SOS on all phones.. [X] Robert DeVita​​​​ CEO and Founder t: (469) 581-2160<tel:(469)%20581-2160> | m: (469) 441-8864<tel:(469)%20441-8864> e: radevita@mejeticks.com<mailto:radevita@mejeticks.com> | w: mejeticks.com<https://www.mejeticks.com/> a: 2323 N Akard Street , Dallas , 75201 [X]<https://www.linkedin.com/company/mejeticks/> [X]<https://twitter.com/mejeticks> [X]<https://www.facebook.com/mejeticks> [X]<https://linktr.ee/mejeticks> The risk of trading futures and options can be substantial. All information, publications, and material used and distributed by Advance Trading Inc. shall be construed as a solicitation. ATI does not maintain an independent research department as defined in CFTC Regulation 1.71. Information obtained from third-party sources is believed to be reliable, but its accuracy is not guaranteed by Advance Trading Inc. Past performance is not necessarily indicative of future results.