Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
I've heard from people having the issue in South Africa as well. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Matt Hoppes" <mattlists@rivervalleyinternet.net> To: nanog@nanog.org Sent: Tuesday, December 31, 2019 4:08:17 PM Subject: GPS Sync Outage Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
Yes, lots of our key wireless network timing elements reported GPS timing sync failure during the past two hours. Appears to be resolving now, within the past few minutes. On 12/31/2019 02:08 PM, Matt Hoppes wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
“It’s a glitch in the Matrix.” Momentarily donning my tinfoil hat here, as I certainly don’t claim to know the nitty gritty of how GPS timing works, but… What if, since the GPS system is operated by the US Air Force, there was something slipped into the GPS system relating to timing or accuracy or similar due to the events in the tinderbox of the Middle East? </tinfoil_hat> ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
On Dec 31, 2019, at 4:20 PM, Alex Bradley <alex@alexbradley.net> wrote:
Yes, lots of our key wireless network timing elements reported GPS timing sync failure during the past two hours.
Appears to be resolving now, within the past few minutes.
On 12/31/2019 02:08 PM, Matt Hoppes wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
GPS != GLONASS On Mon, Jan 6, 2020 at 1:32 PM Andy Ringsmuth <andy@andyring.com> wrote:
“It’s a glitch in the Matrix.”
Momentarily donning my tinfoil hat here, as I certainly don’t claim to know the nitty gritty of how GPS timing works, but…
What if, since the GPS system is operated by the US Air Force, there was something slipped into the GPS system relating to timing or accuracy or similar due to the events in the tinderbox of the Middle East?
</tinfoil_hat>
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
On Dec 31, 2019, at 4:20 PM, Alex Bradley <alex@alexbradley.net> wrote:
Yes, lots of our key wireless network timing elements reported GPS timing sync failure during the past two hours.
Appears to be resolving now, within the past few minutes.
On 12/31/2019 02:08 PM, Matt Hoppes wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
Only if it was the Russian Air Force... On Mon, Jan 6, 2020 at 10:31 AM Andy Ringsmuth <andy@andyring.com> wrote:
“It’s a glitch in the Matrix.”
Momentarily donning my tinfoil hat here, as I certainly don’t claim to know the nitty gritty of how GPS timing works, but…
What if, since the GPS system is operated by the US Air Force, there was something slipped into the GPS system relating to timing or accuracy or similar due to the events in the tinderbox of the Middle East?
</tinfoil_hat>
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
-- Jeff Shultz -- Like us on Social Media for News, Promotions, and other information!! <https://www.facebook.com/SCTCWEB/> <https://www.instagram.com/sctc_503/> <https://www.yelp.com/biz/sctc-stayton-3> <https://www.youtube.com/c/sctcvideos> _**** This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. ****_
On Tue, Dec 31, 2019 at 05:08:17PM -0500, Matt Hoppes wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
On what hardware/firmware are you seeing this? Nothing unusual in the Bay Area so far, in other words "All Quiet on the Western Front". I am seeing receivers synced to at least 6 or 7 sats. Two different locations; one Garmin 18LVC, one uBlox 6M hardware. There is also no chatter about it on the time-nuts list where I would suspect to see it first. -andreas
On Dec 31, 2019, at 5:32 PM, Andreas Ott <andreas@naund.org> wrote:
On Tue, Dec 31, 2019 at 05:08:17PM -0500, Matt Hoppes wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
On what hardware/firmware are you seeing this?
Nothing unusual in the Bay Area so far, in other words "All Quiet on the Western Front". I am seeing receivers synced to at least 6 or 7 sats. Two different locations; one Garmin 18LVC, one uBlox 6M hardware. There is also no chatter about it on the time-nuts list where I would suspect to see it first.
There’s a fair amount of hardware that uses GPS sync for wireless and I’m guessing this is likely a vendor bug w/ UBNT hardware, but it’s also good to know if it’s broader than that. I know there’s some interesting java bugs related to the switchover for 2020 as well, eg: https://twitter.com/NmVAson/status/1207820284268597249 I know of at least once place that was hit by this. - Jared
One of my hats is to design/manufacture/sell GPS third party timing receivers for Cambium Radios. It seems like something happened around 2PM MST today which caused (at a minimum) certain Globaltop/Sierra Wireless GPS modules to quit receiving signals from the GPS constellations. Because these modules are pretty well respected as far as the quality of the timing signal (normally) and relative low cost, they tend to be integrated in a lot of devices, including my products, the official Cambium Products, and I understand the gear from other manufacturers. So far, a power cycle/reset of the GPS module seems to solve the problem. I haven't had enough time to process all of the data I have to know more details yet, but I can confirm the widespread nature of this event. On Tue, Dec 31, 2019 at 5:39 PM Jared Mauch <jared@puck.nether.net> wrote:
On Dec 31, 2019, at 5:32 PM, Andreas Ott <andreas@naund.org> wrote:
On Tue, Dec 31, 2019 at 05:08:17PM -0500, Matt Hoppes wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
On what hardware/firmware are you seeing this?
Nothing unusual in the Bay Area so far, in other words "All Quiet on the Western Front". I am seeing receivers synced to at least 6 or 7 sats. Two different locations; one Garmin 18LVC, one uBlox 6M hardware. There is also no chatter about it on the time-nuts list where I would suspect to see it first.
There’s a fair amount of hardware that uses GPS sync for wireless and I’m guessing this is likely a vendor bug w/ UBNT hardware, but it’s also good to know if it’s broader than that.
I know there’s some interesting java bugs related to the switchover for 2020 as well, eg:
https://twitter.com/NmVAson/status/1207820284268597249
I know of at least once place that was hit by this.
- Jared
-- - Forrest
Was there ever any additional information to come out about this? My Cambuim lost sync starting at 21:00:23 UTC 12/31/2019 and didn’t come back until I rebooted the unit… Brandon Price Senior Network Engineer City of Sherwood, Sherwood Broadband Desk: 503.625.4258 Cell: 971.979.2182 From: NANOG <nanog-bounces@nanog.org> On Behalf Of Forrest Christian (List Account) Sent: Tuesday, December 31, 2019 5:12 PM To: Jared Mauch <jared@puck.nether.net> Cc: nanog list <nanog@nanog.org> Subject: Re: GPS Sync Outage CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you are expecting this email and/or know the content is safe. One of my hats is to design/manufacture/sell GPS third party timing receivers for Cambium Radios. It seems like something happened around 2PM MST today which caused (at a minimum) certain Globaltop/Sierra Wireless GPS modules to quit receiving signals from the GPS constellations. Because these modules are pretty well respected as far as the quality of the timing signal (normally) and relative low cost, they tend to be integrated in a lot of devices, including my products, the official Cambium Products, and I understand the gear from other manufacturers. So far, a power cycle/reset of the GPS module seems to solve the problem. I haven't had enough time to process all of the data I have to know more details yet, but I can confirm the widespread nature of this event. On Tue, Dec 31, 2019 at 5:39 PM Jared Mauch <jared@puck.nether.net<mailto:jared@puck.nether.net>> wrote:
On Dec 31, 2019, at 5:32 PM, Andreas Ott <andreas@naund.org<mailto:andreas@naund.org>> wrote:
On Tue, Dec 31, 2019 at 05:08:17PM -0500, Matt Hoppes wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
On what hardware/firmware are you seeing this?
Nothing unusual in the Bay Area so far, in other words "All Quiet on the Western Front". I am seeing receivers synced to at least 6 or 7 sats. Two different locations; one Garmin 18LVC, one uBlox 6M hardware. There is also no chatter about it on the time-nuts list where I would suspect to see it first.
There’s a fair amount of hardware that uses GPS sync for wireless and I’m guessing this is likely a vendor bug w/ UBNT hardware, but it’s also good to know if it’s broader than that. I know there’s some interesting java bugs related to the switchover for 2020 as well, eg: https://twitter.com/NmVAson/status/1207820284268597249<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2FNmVAson%2Fstatus%2F1207820284268597249&data=02%7C01%7Cpriceb%40sherwoodoregon.gov%7C2646aa8cafe04edeea0208d78e580ad4%7C3e3836d4f74c41a59e565410b65447f9%7C0%7C0%7C637134381103987580&sdata=lF4bclIIso7IUwoWnZGgKlBBMYwUHuuF%2FqDot0kYdrE%3D&reserved=0> I know of at least once place that was hit by this. - Jared -- - Forrest
GLONASS rollover. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Brandon Price" <PriceB@SherwoodOregon.gov> To: "Forrest Christian (List Account)" <lists@packetflux.com>, "Jared Mauch" <jared@puck.nether.net> Cc: "nanog list" <nanog@nanog.org> Sent: Thursday, January 2, 2020 2:40:33 PM Subject: RE: GPS Sync Outage Was there ever any additional information to come out about this? My Cambuim lost sync starting at 21:00:23 UTC 12/31/2019 and didn’t come back until I rebooted the unit… Brandon Price Senior Network Engineer City of Sherwood, Sherwood Broadband Desk: 503.625.4258 Cell: 971.979.2182 From: NANOG <nanog-bounces@nanog.org> On Behalf Of Forrest Christian (List Account) Sent: Tuesday, December 31, 2019 5:12 PM To: Jared Mauch <jared@puck.nether.net> Cc: nanog list <nanog@nanog.org> Subject: Re: GPS Sync Outage CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you are expecting this email and/or know the content is safe. One of my hats is to design/manufacture/sell GPS third party timing receivers for Cambium Radios. It seems like something happened around 2PM MST today which caused (at a minimum) certain Globaltop/Sierra Wireless GPS modules to quit receiving signals from the GPS constellations. Because these modules are pretty well respected as far as the quality of the timing signal (normally) and relative low cost, they tend to be integrated in a lot of devices, including my products, the official Cambium Products, and I understand the gear from other manufacturers. So far, a power cycle/reset of the GPS module seems to solve the problem. I haven't had enough time to process all of the data I have to know more details yet, but I can confirm the widespread nature of this event. On Tue, Dec 31, 2019 at 5:39 PM Jared Mauch < jared@puck.nether.net > wrote:
On Dec 31, 2019, at 5:32 PM, Andreas Ott < andreas@naund.org > wrote:
On Tue, Dec 31, 2019 at 05:08:17PM -0500, Matt Hoppes wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
On what hardware/firmware are you seeing this?
Nothing unusual in the Bay Area so far, in other words "All Quiet on the Western Front". I am seeing receivers synced to at least 6 or 7 sats. Two different locations; one Garmin 18LVC, one uBlox 6M hardware. There is also no chatter about it on the time-nuts list where I would suspect to see it first.
There’s a fair amount of hardware that uses GPS sync for wireless and I’m guessing this is likely a vendor bug w/ UBNT hardware, but it’s also good to know if it’s broader than that. I know there’s some interesting java bugs related to the switchover for 2020 as well, eg: https://twitter.com/NmVAson/status/1207820284268597249 I know of at least once place that was hit by this. - Jared -- - Forrest
The following is from one ntpd instance in Chennai, India. We use Garmin GPS 18x LVC receivers with our ntpd instances. ntpd interfaces directly to the receiver using its NMEA driver and also receives a PPS signal from the GPS receiver. During last night, the ntpd process suddenly dropped its GPS reference clock. This is the first time we've observed this happening. ntpq -np showed an 'x' in the remote field where an 'o' should be. 'x' means "Out of tolerance, do not use (discarded by intersection algorithm);" (https://nlug.ml1.co.uk/2012/01/ntpq-p-output/831) It was noticed in the morning. A drift is observed in /var/log/peerstats.20191231. Corresponding to the log times of where the drift starts, the $GPRMC sentences from /var/log/clockstats.20191231 show the following: 58848 73710.974 127.127.20.0 $GPRMC,202830,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73726.974 127.127.20.0 $GPRMC,202846,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73742.974 127.127.20.0 $GPRMC,202902,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73758.974 127.127.20.0 $GPRMC,202918,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73774.974 127.127.20.0 $GPRMC,202934,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73790.974 127.127.20.0 $GPRMC,202950,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73806.974 127.127.20.0 $GPRMC,203006,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73822.974 127.127.20.0 $GPRMC,203025,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73838.973 127.127.20.0 $GPRMC,203041,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73854.973 127.127.20.0 $GPRMC,203057,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73870.973 127.127.20.0 $GPRMC,203113,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73886.974 127.127.20.0 $GPRMC,203129,A,____.____,_,_____.____,_,000.0,029.5,311219,001.5,W,D*__ 58848 73902.973 127.127.20.0 $GPRMC,203145,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__ 58848 73918.973 127.127.20.0 $GPRMC,203201,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__ 58848 73934.973 127.127.20.0 $GPRMC,203217,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__ 58848 73950.973 127.127.20.0 $GPRMC,203233,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__ 58848 73966.973 127.127.20.0 $GPRMC,203249,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__ 58848 73982.973 127.127.20.0 $GPRMC,203305,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__ 58848 73998.973 127.127.20.0 $GPRMC,203321,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__ 58848 74014.973 127.127.20.0 $GPRMC,203337,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__ 58848 74030.973 127.127.20.0 $GPRMC,203350,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__ 58848 74046.973 127.127.20.0 $GPRMC,203406,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__ 58848 74062.973 127.127.20.0 $GPRMC,203422,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__ 58848 74078.973 127.127.20.0 $GPRMC,203438,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__ 58848 74094.973 127.127.20.0 $GPRMC,203454,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__ 58848 74110.973 127.127.20.0 $GPRMC,203510,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,A*__ 58848 74126.973 127.127.20.0 $GPRMC,203526,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__ 58848 74142.974 127.127.20.0 $GPRMC,203542,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__ 58848 74158.973 127.127.20.0 $GPRMC,203558,A,____.____,_,_____.____,_,000.0,004.7,311219,001.5,W,D*__ These messages are logged once every 16 seconds. The $GPRMC time (UTC) jumped from 203006 to 203025 (delta 19 seconds), and then corrected back from 203337 to 203350 (delta 13 seconds). Note that these $GPRMC lines are the (derived) output of the GPS receiver. ntpd was restarted this morning, and appears to use the GPS timesource properly.
Matt, I saw a similar issue today with Cambium PMP450i’s losing sync from our PacketFlux rack injectors. May not be useful information to you know considering there’s chatter about this already on social media. Thanks, Bryton J. Herdes Network Engineer Jackson Electric Cooperative 100 Cooperative Way Bay City, TX 77414 Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, copy, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. On Dec 31, 2019, at 4:08 PM, Matt Hoppes <mattlists@rivervalleyinternet.net<mailto:mattlists@rivervalleyinternet.net>> wrote: Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
Over the weekend, tests confirmed that all of these issues are related to a end of the year GLONASS day rollover bug in the GNSS receiver. A power cycle seems to correct it until the next rollover in 2024. If you're still seeing these issues then power cycle the device using a method appropriate to the device. On the RackInjector in particular there is a power control button on the GNSS statusimulation screen. On Mon, Jan 6, 2020, 8:50 AM Bryton Herdes <bherdes@myjec.coop> wrote:
Matt,
I saw a similar issue today with Cambium PMP450i’s losing sync from our PacketFlux rack injectors. May not be useful information to you know considering there’s chatter about this already on social media.
Thanks,
Bryton J. Herdes Network Engineer Jackson Electric Cooperative 100 Cooperative Way Bay City, TX 77414
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, copy, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
On Dec 31, 2019, at 4:08 PM, Matt Hoppes < mattlists@rivervalleyinternet.net> wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
I hate composing email on my phone. I always seems to mangle something. Clarification: On the RackInjector, you can power cycle the GNSS receiver using a button on the GNSS status page. On Mon, Jan 6, 2020, 9:00 AM Forrest Christian (List Account) < lists@packetflux.com> wrote:
Over the weekend, tests confirmed that all of these issues are related to a end of the year GLONASS day rollover bug in the GNSS receiver. A power cycle seems to correct it until the next rollover in 2024.
If you're still seeing these issues then power cycle the device using a method appropriate to the device. On the RackInjector in particular there is a power control button on the GNSS statusimulation screen.
On Mon, Jan 6, 2020, 8:50 AM Bryton Herdes <bherdes@myjec.coop> wrote:
Matt,
I saw a similar issue today with Cambium PMP450i’s losing sync from our PacketFlux rack injectors. May not be useful information to you know considering there’s chatter about this already on social media.
Thanks,
Bryton J. Herdes Network Engineer Jackson Electric Cooperative 100 Cooperative Way Bay City, TX 77414
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, copy, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
On Dec 31, 2019, at 4:08 PM, Matt Hoppes < mattlists@rivervalleyinternet.net> wrote:
Is anyone else seeing GPS timing source outages across the U. S. In the last two hours?
participants (13)
-
Alex Bradley -
Andreas Ott -
Andy Ringsmuth -
Brandon Price -
Bryton Herdes -
Forrest Christian (List Account) -
Jared Mauch -
Jeff Shultz -
Matt Hoppes -
Mike Hammett -
Mukund Sivaraman -
Ryan A. Krenzischek -
Tom Beecher