all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019
Does anyone have any more information on this? Users on Twitter report that T-Mobile said "that there is a known issue of texts being resent/spoofed and said not to worry about it." https://twitter.com/ThelocalfilmMN/status/1192434609197375488 https://investors.sprint.com/news-and-events/press-releases/press-release-de... thoughts? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The information contained in this electronic message may be confidential, and the message is for the use of intended recipients only. If you are not the intended recipient, do not disseminate, copy, or disclose this communication or its contents. If you have received this communication in error, please immediately notify me by replying to the email or call MIS Alliance at 617-500-1700 and permanently delete this communication.
On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why? Cheers, b.
From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex... It seems there is a company that has everyone's text messages.. "Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week." -Brandon On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca> wrote:
On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why?
Cheers, b.
I believe Syniverse only comes into play when you text someone on a different carrier than your own. Syniverse is basically the middle-man for that message delivery, and a server of theirs just spooled ~150k messages until someone rebooted/fixed that server. It sounds like these messages were never originally delivered to begin with, so "re-sent" is not exactly accurate. -- Trevor Manternach On Fri, Nov 8, 2019 at 10:56 AM Brandon Svec <bsvec@teamonesolutions.com> wrote:
From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex...
It seems there is a company that has everyone's text messages..
"Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week." -Brandon
On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca> wrote:
On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why?
Cheers, b.
“During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement. how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome?
On Nov 8, 2019, at 12:54 PM, Brandon Svec <bsvec@teamonesolutions.com> wrote:
From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex...
It seems there is a company that has everyone's text messages..
"Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week." -Brandon
On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca> wrote: On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why?
Cheers, b.
We apologize for finally getting around to our job and doing what we were paid to do... On Fri, Nov 8, 2019 at 1:27 PM Matt Hoppes < mattlists@rivervalleyinternet.net> wrote:
“During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement.
how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome?
On Nov 8, 2019, at 12:54 PM, Brandon Svec <bsvec@teamonesolutions.com> wrote:
From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex...
It seems there is a company that has everyone's text messages..
"Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week." -Brandon
On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca> wrote:
On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why?
Cheers, b.
-- :o@>
Esp on Valentine’s day. Of all the days that clear communication is important. I’d be very interested in their reasoning for why these messages were not sent and held. From: NANOG <nanog-bounces@nanog.org> On Behalf Of Oliver O'Boyle Sent: Friday, November 08, 2019 1:31 PM To: Matt Hoppes <mattlists@rivervalleyinternet.net> Cc: North American Network Operators' Group <nanog@nanog.org> Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019 We apologize for finally getting around to our job and doing what we were paid to do... On Fri, Nov 8, 2019 at 1:27 PM Matt Hoppes <mattlists@rivervalleyinternet.net<mailto:mattlists@rivervalleyinternet.net>> wrote: “During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement. how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome? On Nov 8, 2019, at 12:54 PM, Brandon Svec <bsvec@teamonesolutions.com<mailto:bsvec@teamonesolutions.com>> wrote: From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex... It seems there is a company that has everyone's text messages.. "Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week." -Brandon On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca<mailto:brian@interlinx.bc.ca>> wrote: On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why? Cheers, b. -- :o@>
Reading Syniverse's cause of trouble (lame excuse) tells me their data handling processes are poor and seemingly shady since I do not buy reason for the trouble. On 11/8/2019 1:34 PM, Kain, Becki (.) wrote:
Esp on Valentine’s day. Of all the days that clear communication is important. I’d be very interested in their reasoning for why these messages were not sent and held.
*From:* NANOG <nanog-bounces@nanog.org> *On Behalf Of *Oliver O'Boyle *Sent:* Friday, November 08, 2019 1:31 PM *To:* Matt Hoppes <mattlists@rivervalleyinternet.net> *Cc:* North American Network Operators' Group <nanog@nanog.org> *Subject:* Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019
We apologize for finally getting around to our job and doing what we were paid to do...
On Fri, Nov 8, 2019 at 1:27 PM Matt Hoppes <mattlists@rivervalleyinternet.net <mailto:mattlists@rivervalleyinternet.net>> wrote:
“During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement.
how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome?
On Nov 8, 2019, at 12:54 PM, Brandon Svec <bsvec@teamonesolutions.com <mailto:bsvec@teamonesolutions.com>> wrote:
From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex...
It seems there is a company that has everyone's text messages..
"Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week."
-Brandon
On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca <mailto:brian@interlinx.bc.ca>> wrote:
On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote: > Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why?
Cheers, b.
--
:o@>
Playing devil’s advocate, perhaps they were under emergency court order to not deliver texts for a certain duration, market, who knows what, and that order just ended, but some type of non-disclosure / secrecy directive continues to exist… may have just had to come up with something to say because their other agreements would not have permitted discarding the texts… 😊 David From: NANOG <nanog-bounces+dhubbard=dino.hostasaurus.com@nanog.org> on behalf of Mark Stevens <manager@monmouth.com> Date: Friday, November 8, 2019 at 1:45 PM To: "nanog@nanog.org" <nanog@nanog.org> Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019 Reading Syniverse's cause of trouble (lame excuse) tells me their data handling processes are poor and seemingly shady since I do not buy reason for the trouble. On 11/8/2019 1:34 PM, Kain, Becki (.) wrote: Esp on Valentine’s day. Of all the days that clear communication is important. I’d be very interested in their reasoning for why these messages were not sent and held. From: NANOG <nanog-bounces@nanog.org><mailto:nanog-bounces@nanog.org> On Behalf Of Oliver O'Boyle Sent: Friday, November 08, 2019 1:31 PM To: Matt Hoppes <mattlists@rivervalleyinternet.net><mailto:mattlists@rivervalleyinternet.net> Cc: North American Network Operators' Group <nanog@nanog.org><mailto:nanog@nanog.org> Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019 We apologize for finally getting around to our job and doing what we were paid to do... On Fri, Nov 8, 2019 at 1:27 PM Matt Hoppes <mattlists@rivervalleyinternet.net<mailto:mattlists@rivervalleyinternet.net>> wrote: “During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement. how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome? On Nov 8, 2019, at 12:54 PM, Brandon Svec <bsvec@teamonesolutions.com<mailto:bsvec@teamonesolutions.com>> wrote: From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex... It seems there is a company that has everyone's text messages.. "Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week." -Brandon On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca<mailto:brian@interlinx.bc.ca>> wrote: On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why? Cheers, b. -- :o@>
That’d be an incredibly obtuse, excessive, and horrible order. And it’d be the very first time that’s ever happened... -Ben Cannon CEO 6x7 Networks & 6x7 Telecom, LLC ben@6by7.net <mailto:ben@6by7.net>
On Nov 8, 2019, at 10:50 AM, David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Playing devil’s advocate, perhaps they were under emergency court order to not deliver texts for a certain duration, market, who knows what, and that order just ended, but some type of non-disclosure / secrecy directive continues to exist… may have just had to come up with something to say because their other agreements would not have permitted discarding the texts… 😊
David
From: NANOG <nanog-bounces+dhubbard=dino.hostasaurus.com@nanog.org> on behalf of Mark Stevens <manager@monmouth.com> Date: Friday, November 8, 2019 at 1:45 PM To: "nanog@nanog.org" <nanog@nanog.org> Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019
Reading Syniverse's cause of trouble (lame excuse) tells me their data handling processes are poor and seemingly shady since I do not buy reason for the trouble.
On 11/8/2019 1:34 PM, Kain, Becki (.) wrote:
Esp on Valentine’s day. Of all the days that clear communication is important. I’d be very interested in their reasoning for why these messages were not sent and held.
From: NANOG <nanog-bounces@nanog.org> <mailto:nanog-bounces@nanog.org> On Behalf Of Oliver O'Boyle Sent: Friday, November 08, 2019 1:31 PM To: Matt Hoppes <mattlists@rivervalleyinternet.net> <mailto:mattlists@rivervalleyinternet.net> Cc: North American Network Operators' Group <nanog@nanog.org> <mailto:nanog@nanog.org> Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019
We apologize for finally getting around to our job and doing what we were paid to do...
On Fri, Nov 8, 2019 at 1:27 PM Matt Hoppes <mattlists@rivervalleyinternet.net <mailto:mattlists@rivervalleyinternet.net>> wrote:
“During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement.
how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome?
On Nov 8, 2019, at 12:54 PM, Brandon Svec <bsvec@teamonesolutions.com <mailto:bsvec@teamonesolutions.com>> wrote:
From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex... <https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-text-messages-sent-valentines-day/2527660001/>
It seems there is a company that has everyone's text messages..
"Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week." -Brandon
On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca <mailto:brian@interlinx.bc.ca>> wrote:
On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why?
Cheers, b.
-- :o@>
What likely happened is that messages were queued on host to go out, SMPP binds go down, queue fills up, host crashes. Then someone realizes the host is down and brings it back up and the queue empties when the load is low. Since it included many carriers, might have been a message routing server in the middle of their platform. On Fri, Nov 8, 2019 at 10:52 AM David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Playing devil’s advocate, perhaps they were under emergency court order to not deliver texts for a certain duration, market, who knows what, and that order just ended, but some type of non-disclosure / secrecy directive continues to exist… may have just had to come up with something to say because their other agreements would not have permitted discarding the texts… 😊
David
*From: *NANOG <nanog-bounces+dhubbard=dino.hostasaurus.com@nanog.org> on behalf of Mark Stevens <manager@monmouth.com> *Date: *Friday, November 8, 2019 at 1:45 PM *To: *"nanog@nanog.org" <nanog@nanog.org> *Subject: *Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019
Reading Syniverse's cause of trouble (lame excuse) tells me their data handling processes are poor and seemingly shady since I do not buy reason for the trouble.
On 11/8/2019 1:34 PM, Kain, Becki (.) wrote:
Esp on Valentine’s day. Of all the days that clear communication is important. I’d be very interested in their reasoning for why these messages were not sent and held.
*From:* NANOG <nanog-bounces@nanog.org> <nanog-bounces@nanog.org> *On Behalf Of *Oliver O'Boyle *Sent:* Friday, November 08, 2019 1:31 PM *To:* Matt Hoppes <mattlists@rivervalleyinternet.net> <mattlists@rivervalleyinternet.net> *Cc:* North American Network Operators' Group <nanog@nanog.org> <nanog@nanog.org> *Subject:* Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019
We apologize for finally getting around to our job and doing what we were paid to do...
On Fri, Nov 8, 2019 at 1:27 PM Matt Hoppes < mattlists@rivervalleyinternet.net> wrote:
“During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement.
how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome?
On Nov 8, 2019, at 12:54 PM, Brandon Svec <bsvec@teamonesolutions.com> wrote:
From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex...
It seems there is a company that has everyone's text messages..
"Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week."
-Brandon
On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca> wrote:
On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why?
Cheers, b.
--
:o@>
If they just realized a server was down, from 2/14, just now, I’d say they have bigger support issues From: NANOG <nanog-bounces@nanog.org> On Behalf Of Jared Geiger Sent: Friday, November 08, 2019 2:23 PM To: nanog@nanog.org Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019 What likely happened is that messages were queued on host to go out, SMPP binds go down, queue fills up, host crashes. Then someone realizes the host is down and brings it back up and the queue empties when the load is low. Since it included many carriers, might have been a message routing server in the middle of their platform. On Fri, Nov 8, 2019 at 10:52 AM David Hubbard <dhubbard@dino.hostasaurus.com<mailto:dhubbard@dino.hostasaurus.com>> wrote: Playing devil’s advocate, perhaps they were under emergency court order to not deliver texts for a certain duration, market, who knows what, and that order just ended, but some type of non-disclosure / secrecy directive continues to exist… may have just had to come up with something to say because their other agreements would not have permitted discarding the texts… 😊 David From: NANOG <nanog-bounces+dhubbard=dino.hostasaurus.com@nanog.org<mailto:dino.hostasaurus.com@nanog.org>> on behalf of Mark Stevens <manager@monmouth.com<mailto:manager@monmouth.com>> Date: Friday, November 8, 2019 at 1:45 PM To: "nanog@nanog.org<mailto:nanog@nanog.org>" <nanog@nanog.org<mailto:nanog@nanog.org>> Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019 Reading Syniverse's cause of trouble (lame excuse) tells me their data handling processes are poor and seemingly shady since I do not buy reason for the trouble. On 11/8/2019 1:34 PM, Kain, Becki (.) wrote: Esp on Valentine’s day. Of all the days that clear communication is important. I’d be very interested in their reasoning for why these messages were not sent and held. From: NANOG <nanog-bounces@nanog.org><mailto:nanog-bounces@nanog.org> On Behalf Of Oliver O'Boyle Sent: Friday, November 08, 2019 1:31 PM To: Matt Hoppes <mattlists@rivervalleyinternet.net><mailto:mattlists@rivervalleyinternet.net> Cc: North American Network Operators' Group <nanog@nanog.org><mailto:nanog@nanog.org> Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019 We apologize for finally getting around to our job and doing what we were paid to do... On Fri, Nov 8, 2019 at 1:27 PM Matt Hoppes <mattlists@rivervalleyinternet.net<mailto:mattlists@rivervalleyinternet.net>> wrote: “During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement. how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome? On Nov 8, 2019, at 12:54 PM, Brandon Svec <bsvec@teamonesolutions.com<mailto:bsvec@teamonesolutions.com>> wrote: From: https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-tex... It seems there is a company that has everyone's text messages.. "Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week." -Brandon On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <brian@interlinx.bc.ca<mailto:brian@interlinx.bc.ca>> wrote: On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why? Cheers, b. -- :o@>
On Fri, 08 Nov 2019 11:23:17 -0800, Jared Geiger said:
What likely happened is that messages were queued on host to go out, SMPP binds go down, queue fills up, host crashes. Then someone realizes the host is down and brings it back up and the queue empties when the load is low.
What I've seen happen more often than that: Server goes partly belly-up, queue fills up. Backup process runs, backing up the queue. (Optionally here: Reboot the server and lose the queue). Much later, the server hits another issue that requires recovering from backups - and they restore a truly ancient copy. I recently got a replay of a bunch of email messages from 2002. I admit not at all understanding what procedure failures (multiple) resulted in reloading a mail spool from 2002.
What I've seen happen more often than that:
Server goes partly belly-up, queue fills up. Backup process runs, backing up the queue. (Optionally here: Reboot the server and lose the queue). Much later, the server hits another issue that requires recovering from backups - and they restore a truly ancient copy.
Particularly as mail servers tend to check for expired messages *after* a delivery attempt. This means a restored queue containing ancient messages will most likely be given one last delivery attempt prior to bouncing. One real example comes from the qmail-send man page: queuelifetime Number of seconds a message can stay in the queue. Default: 604800 (one week). After this time expires, qmail-send will try the message once more, but it will treat any temporary delivery failures as permanent failures. Combine that with the fact that it's not unheard of for SMS servers to be derived from mail servers (since they do virtually the same thing) and an accidental queue restore or server revivication seems the most plausible. Mark.
Under emergency court order not to deliver texts? Not delivering tens of thousands of messages would appear to be abuse of the legal process if it were true. Scary.... At 01:50 PM 08/11/2019, David Hubbard wrote:
Playing devilâs advocate, perhaps they were under emergency court order to not deliver texts for a certain duration, market, who knows what, and that order just ended, but some type of non-disclosure / secrecy directive continues to exist may have just had to come up with something to say because their other agreements would not have permitted discarding the texts
David
From: NANOG <nanog-bounces+dhubbard=dino.hostasaurus.com@nanog.org> on behalf of Mark Stevens <manager@monmouth.com> Date: Friday, November 8, 2019 at 1:45 PM To: "nanog@nanog.org" <nanog@nanog.org> Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019
Reading Syniverse's cause of trouble (lame excuse) tells me their data handling processes are poor and seemingly shady since I do not buy reason for the trouble.
On 11/8/2019 1:34 PM, Kain, Becki (.) wrote: Esp on Valentineâs day. Of all the days that clear communication is important. Iâd be very interested in their reasoning for why these messages were not sent and held.
From: NANOG <mailto:nanog-bounces@nanog.org><nanog-bounces@nanog.org> On Behalf Of Oliver O'Boyle Sent: Friday, November 08, 2019 1:31 PM To: Matt Hoppes <mailto:mattlists@rivervalleyinternet.net><mattlists@rivervalleyinternet.net> Cc: North American Network Operators' Group <mailto:nanog@nanog.org><nanog@nanog.org> Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019
We apologize for finally getting around to our job and doing what we were paid to do...
On Fri, Nov 8, 2019 at 1:27 PM Matt Hoppes <<mailto:mattlists@rivervalleyinternet.net>mattlists@rivervalleyinternet.net> wrote: âDuring an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operatorsâ subscribers," Syniverse said in a statement.
how do you inadvertently send messages that were supposed to be sent but worked and sent? Isnât that the desired outcome?
On Nov 8, 2019, at 12:54 PM, Brandon Svec <<mailto:bsvec@teamonesolutions.com>bsvec@teamonesolutions.com> wrote: From: <https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-text-messages-sent-valentines-day/2527660001/>https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-text-messages-sent-valentines-day/2527660001/
It seems there is a company that has everyone's text messages..
"Some mobile carriers rely on a third-party text platform called Syniverse to relay messages. The vendor said in a statement that its IT staff unknowingly caused the texts to be delivered this week." -Brandon
On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell <<mailto:brian@interlinx.bc.ca>brian@interlinx.bc.ca> wrote: On Thu, 2019-11-07 at 22:42 +0000, Chris Kimball via NANOG wrote:
Does anyone have any more information on this?
Yeah, like who (in the private sector -- we all knew the NSA already are doing this) has access to and is archiving *everyone*s text messages? And why?
Cheers, b.
-- :o@>
-- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
On Fri, Nov 08, 2019 at 01:43:41PM -0500, Mark Stevens wrote:
Reading Syniverse's cause of trouble (lame excuse) tells me their data handling processes are poor and seemingly shady since I do not buy reason for the trouble.
Agreed. So how many other messages have been delayed, lost, forwarded incorrectly, or...sold to third parties? (Note that I'm not saying Syniverse did that last one. What I'm saying is that an operation like this inevitably affords plenty of opportunities for employees to engage in a little freelance capitalism of their own or for third parties to simply help themselves.) ---rsk
On 11/8/19 10:34 AM, Kain, Becki (.) wrote:
Esp on Valentine’s day. Of all the days that clear communication is important. I’d be very interested in their reasoning for why these messages were not sent and held.
Roses are red, Violets are blue, Hope we're still together When this reaches you. (Sorry, it's Friday afternoon. I'll show myself out.) Jim **
On Nov 8, 2019, at 1:26 PM, Matt Hoppes <mattlists@rivervalleyinternet.net> wrote:
“During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement.
how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome?
I run mailing lists. I’ve had times where I find something stuck in the system and instead of just deleting it, I actually try to make sure it goes out based on the original intent. This has resulted in me sending out e-mails a year or two later at times. - Jared
Jared Mauch wrote on 11/8/2019 12:33 PM:
On Nov 8, 2019, at 1:26 PM, Matt Hoppes <mattlists@rivervalleyinternet.net> wrote:
“During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement.
how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome? I run mailing lists. I’ve had times where I find something stuck in the system and instead of just deleting it, I actually try to make sure it goes out based on the original intent. This has resulted in me sending out e-mails a year or two later at times.
- Jared
Timing can be critical, which is why SMTP servers often expire and return queued messages after 12-72hrs (maybe a week at most). Any messages that can't be returned are eventually discarded and a message is sent to the mail server's administrator. Sounds like none of that actually happens within Syniverse's TXT/SMS delivery system. Someone is asleep at the wheel if 100-200k messages are stuck in queue for months.
This can be a "curse" of highly available servers which stay up for a year or more, some of mine will. A mail delivery process locks messages in the queue for delivery and then the process hangs. Subsequent delivery attempts will honor the lock so they never go out, nor are they even timed out. It's not a terrible idea to have a scheduled process, like once a day, which kills all delivery processes just for this reason, or any which are more than, say, an hour or two old. It's an easy script to write and mail delivery programs are or should be resiliant to receiving a kill signal. There are other scenarios possible but one would have to know their entire software and network architecture to speculate. -- -Barry Shein Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD The World: Since 1989 | A Public Information Utility | *oo*
On Fri, 8 Nov 2019, Jared Mauch wrote:
I run mailing lists. I’ve had times where I find something stuck in the system and instead of just deleting it, I actually try to make sure it goes out based on the original intent. This has resulted in me sending out e-mails a year or two later at times.
Ok, you run mailing lists mostly on an amatuer (personal, unpaid) basis. Every commercial organization delivering customer records should have a record retention/archive schedule. Holding on to customer data longer than necessary for business purposes is just increasing your liability when something goes wrong. And it always goes wrong. Many tech startup companies never think about record retention schedules, or their privacy policy says 'indefinitely', which means the lawyer wrote something down in the policy but no one really thought about it. Western Union learned that lesson with telegrams a hundred years ago. Tech firms keep re-learning old lessons, the hard way.
It doesn't seem to be simply a matter of backlogged messages finally going out. My friend replied to the mystery messages received from me and I thought she was accidentally responding on the wrong thread. Her texts seemed spontaneous and disjointed which is why I assumed she was on the wrong thread. When we talked about it, it became clear she thought she was responding to me and sent me a screenshot of the messages she was replying to. I keep a copy of every message so I was able to locate the point in time in the past where this dialog happened and found the 2/14 timestamps. But here's the thing. She had interacted with me correctly at the time back on 2/14. The message did not get stuck and undelivered. This was a resend of a set of completed messages. On Fri, Nov 8, 2019 at 12:07 PM Sean Donelan <sean@donelan.com> wrote:
On Fri, 8 Nov 2019, Jared Mauch wrote:
I run mailing lists. I’ve had times where I find something stuck in the system and instead of just deleting it, I actually try to make sure it goes out based on the original intent. This has resulted in me sending out e-mails a year or two later at times.
Ok, you run mailing lists mostly on an amatuer (personal, unpaid) basis.
Every commercial organization delivering customer records should have a record retention/archive schedule. Holding on to customer data longer than necessary for business purposes is just increasing your liability when something goes wrong. And it always goes wrong.
Many tech startup companies never think about record retention schedules, or their privacy policy says 'indefinitely', which means the lawyer wrote something down in the policy but no one really thought about it.
Western Union learned that lesson with telegrams a hundred years ago. Tech firms keep re-learning old lessons, the hard way.
On Fri, 8 Nov 2019, Matt Hoppes wrote:
“During an internal maintenance cycle last night, 168,149 previously undelivered text messages were inadvertently sent to multiple mobile operators’ subscribers," Syniverse said in a statement.
how do you inadvertently send messages that were supposed to be sent but worked and sent? Isn’t that the desired outcome?
Monitoring and audits usually come after a failure of some sort. Nobody thought they needed to make sure all servers are checked for queued unsent messages, because the software will *always* do the "right thing." I'm sure email didn't have the 5 day deletion after non-delivery when it first started out either. Someone got an email a few months late and decided some cleanup needed to happen. Now you've got custom software running everywhere and similar alerting and purging requirements were not made explicitly on how long to hold onto the messages. I run a phone company and we do hold messages that cannot be delivered for a period of time less than a week, but I get paged when that queue holds more than X messages or any one message exceeds Y time since attempted send. It's not hard, but I've seen lots of pretty obvious issues like this overlooked and virtually every company regardless of size, even Amazon. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
OK, I understand the part about text messages from February 2019 being sent on now, but…
On Nov 7, 2019, at 23:42, Chris Kimball via NANOG <nanog@nanog.org> wrote:
https://investors.sprint.com/news-and-events/press-releases/press-release-de...
Are we sure this isn’t a press release from 2008? Grüße, Carsten
AT&T, Sprint, T-Mobile and Verizon Form Joint Venture to Transform Messaging Experience Oct 24, 2019 The Cross-Carrier Messaging Initiative will help drive the next generation of messaging for consumers and businesses. Looks to be within the last month! -----Original Message----- From: Carsten Bormann <cabo@tzi.org> Sent: Friday, November 8, 2019 2:29 PM To: Chris Kimball <CKimball@misalliance.com> Cc: nanog@nanog.org Subject: Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019 [EXTERNAL] OK, I understand the part about text messages from February 2019 being sent on now, but…
On Nov 7, 2019, at 23:42, Chris Kimball via NANOG <nanog@nanog.org> wrote:
https://investors.sprint.com/news-and-events/press-releases/press-release-de...
Are we sure this isn’t a press release from 2008? Grüße, Carsten - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The information contained in this electronic message may be confidential, and the message is for the use of intended recipients only. If you are not the intended recipient, do not disseminate, copy, or disclose this communication or its contents. If you have received this communication in error, please immediately notify me by replying to the email or call MIS Alliance at 617-500-1700 and permanently delete this communication.
On Nov 8, 2019, at 20:38, Chris Kimball <CKimball@misalliance.com> wrote:
Oct 24, 2019
I’ve seen the date. But have you seen the content?
The Cross-Carrier Messaging Initiative will help drive the next generation of messaging for consumers and businesses.
Hello?
Looks to be within the last month!
Of 2006? Grüße, Carsten
participants (23)
-
Ben Cannon -
Blake Hudson -
Brandon Svec -
Brian J. Murrell -
bzs＠theworld.com -
Carsten Bormann -
Chris Kimball -
Clayton Zekelman -
David Hubbard -
Jared Geiger -
Jared Mauch -
Jim Shankland -
Jim Stapleton -
Kain, Becki (.) -
Mark Delany -
Mark Stevens -
Matt Hoppes -
Oliver O'Boyle -
Peter Beckman -
Rich Kulawiec -
Sean Donelan -
Trevor Manternach -
Valdis Klētnieks