NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic Storm Watch
<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> SWPC Issues Its First G4 Watch Since 2005 | NOAA / NWS Space Weather Prediction Center<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> swpc.noaa.gov<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> [favicon.ico]<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> "Multiple CMEs erupted associated with flare activity from Region 3664 on 07-09 May. These CMEs are expected to merge with potential arrival expected by early May 11 on the UTC day.” (Low but distinct possibility of effects to radio and transmission systems) FYI, /John John Curran President and CEO American Registry for Internet Numbers
We just had two TM1000 TimeMachine brand GPS NTP servers lose clock sync at the same time, in two different cities (LA and Santa Barbara). The outage lasted about five minutes, during which the NTP servers were responding, but with time that was 1900 seconds out of sync. The devices showed satellite lock on 8 birds (not all the same ones). I've never seen this behavior before with years of NTP clock experience. It could be that these inexpensive NTP servers aren't very selective about bogus inputs, as I would have expected them to lose synch in the event of a GPS signal failure. Instead they produced garbage. Our PRTG NTP monitor logged the problem this way: Sensor SNTP (SNTP) *** Device 10.2.10.90-TimeMachine NTP server (10.2.10.90) New Status at 5/10/2024 12:49:52 PM (Pacific Standard Time): Down Last Message: The target server did not return a valid time. To resolve this issue, use a packet analyzing tool and do a trace of the NTP packets to check if all fields are correctly populated. (code: PE085) ________________________________ From: NANOG <nanog-bounces+mel=beckman.org@nanog.org> on behalf of John Curran <jcurran@arin.net> Sent: Friday, May 10, 2024 10:54 AM To: NANOG <nanog@nanog.org> Subject: NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic Storm Watch <https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> SWPC Issues Its First G4 Watch Since 2005 | NOAA / NWS Space Weather Prediction Center<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> swpc.noaa.gov<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> [favicon.ico]<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> "Multiple CMEs erupted associated with flare activity from Region 3664 on 07-09 May. These CMEs are expected to merge with potential arrival expected by early May 11 on the UTC day.” (Low but distinct possibility of effects to radio and transmission systems) FYI, /John John Curran President and CEO American Registry for Internet Numbers
How odd. Both clocks are stratum 1? Were the associated servers chiming off other servers as well? Cheers, -- jra ----- Original Message -----
From: "Mel Beckman" <mel@beckman.org> To: "John Curran" <jcurran@arin.net>, "NANOG" <nanog@nanog.org> Sent: Friday, May 10, 2024 4:29:13 PM Subject: Re: NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic Storm Watch
We just had two TM1000 TimeMachine brand GPS NTP servers lose clock sync at the same time, in two different cities (LA and Santa Barbara). The outage lasted about five minutes, during which the NTP servers were responding, but with time that was 1900 seconds out of sync. The devices showed satellite lock on 8 birds (not all the same ones). I've never seen this behavior before with years of NTP clock experience.
It could be that these inexpensive NTP servers aren't very selective about bogus inputs, as I would have expected them to lose synch in the event of a GPS signal failure. Instead they produced garbage. Our PRTG NTP monitor logged the problem this way:
Sensor SNTP (SNTP) *** Device 10.2.10.90-TimeMachine NTP server (10.2.10.90) New Status at 5/10/2024 12:49:52 PM (Pacific Standard Time): Down Last Message: The target server did not return a valid time. To resolve this issue, use a packet analyzing tool and do a trace of the NTP packets to check if all fields are correctly populated. (code: PE085)
________________________________ From: NANOG <nanog-bounces+mel=beckman.org@nanog.org> on behalf of John Curran <jcurran@arin.net> Sent: Friday, May 10, 2024 10:54 AM To: NANOG <nanog@nanog.org> Subject: NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic Storm Watch
<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> SWPC Issues Its First G4 Watch Since 2005 | NOAA / NWS Space Weather Prediction Center<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> swpc.noaa.gov<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> [favicon.ico]<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005>
"Multiple CMEs erupted associated with flare activity from Region 3664 on 07-09 May. These CMEs are expected to merge with potential arrival expected by early May 11 on the UTC day.”
(Low but distinct possibility of effects to radio and transmission systems)
FYI, /John
John Curran President and CEO American Registry for Internet Numbers
-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
We had some Meinberg's which did something similar but different some time ago. NTP was out of sync with GPS. We had a CheckMk instance which detected drift between sources in our network. Turns out there was one or more configs in the Meinberg that enable failover from one source to another, and to ensure GPS and NTP are working together rather than independently. Maybe your TimeMachines have similar config variabilities. On 2024-05-10 14:29, Mel Beckman wrote:
We just had two TM1000 TimeMachine brand GPS NTP servers lose clock sync at the same time, in two different cities (LA and Santa Barbara). The outage lasted about five minutes, during which the NTP servers were responding, but with time that was 1900 seconds out of sync. The devices showed satellite lock on 8 birds (not all the same ones). I've never seen this behavior before with years of NTP clock experience.
It could be that these inexpensive NTP servers aren't very selective about bogus inputs, as I would have expected them to lose synch in the event of a GPS signal failure. Instead they produced garbage. Our PRTG NTP monitor logged the problem this way:
Sensor SNTP (SNTP) *** Device 10.2.10.90-TimeMachine NTP server (10.2.10.90) New Status at 5/10/2024 12:49:52 PM (Pacific Standard Time): Down Last Message: The target server did not return a valid time. To resolve this issue, use a packet analyzing tool and do a trace of the NTP packets to check if all fields are correctly populated. (code: PE085)
------------------------------------------------------------------------ *From:* NANOG <nanog-bounces+mel=beckman.org@nanog.org> on behalf of John Curran <jcurran@arin.net> *Sent:* Friday, May 10, 2024 10:54 AM *To:* NANOG <nanog@nanog.org> *Subject:* NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic Storm Watch
SWPC Issues Its First G4 Watch Since 2005 | NOAA / NWS Space Weather Prediction Center <https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> swpc.noaa.gov <https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> favicon.ico <https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005>
"Multiple CMEs erupted associated with flare activity from Region 3664 on 07-09 May. These CMEs are expected to merge with potential arrival expected by early May 11 on the UTC day.”
(Low but distinct possibility of effects to radio and transmission systems)
FYI, /John
John Curran President and CEO American Registry for Internet Numbers
They are identical units running the same firmware versions. The TM 1000A doesn’t give you an option for any kind of backup syncing to NTP on the network. They are pure GPS servers; theoretically they either serve the correct time or no time at all. So technically to give such as our time deviation would be a bug of some kind. Unfortunately, I don’t have the actual packets received, so I can’t decode them to see if this was just bad interpretation on the part of PRTG. Needless to say we’re watching these closely, plus all of our other GPS NTP servers. I am dumping PCAP from the corresponding firewalls to a database, since the volume of NTP traffic is low. -mel On May 10, 2024, at 3:09 PM, Raymond Burkholder <ray@oneunified.net> wrote: We had some Meinberg's which did something similar but different some time ago. NTP was out of sync with GPS. We had a CheckMk instance which detected drift between sources in our network. Turns out there was one or more configs in the Meinberg that enable failover from one source to another, and to ensure GPS and NTP are working together rather than independently. Maybe your TimeMachines have similar config variabilities. On 2024-05-10 14:29, Mel Beckman wrote: We just had two TM1000 TimeMachine brand GPS NTP servers lose clock sync at the same time, in two different cities (LA and Santa Barbara). The outage lasted about five minutes, during which the NTP servers were responding, but with time that was 1900 seconds out of sync. The devices showed satellite lock on 8 birds (not all the same ones). I've never seen this behavior before with years of NTP clock experience. It could be that these inexpensive NTP servers aren't very selective about bogus inputs, as I would have expected them to lose synch in the event of a GPS signal failure. Instead they produced garbage. Our PRTG NTP monitor logged the problem this way: Sensor SNTP (SNTP) *** Device 10.2.10.90-TimeMachine NTP server (10.2.10.90) New Status at 5/10/2024 12:49:52 PM (Pacific Standard Time): Down Last Message: The target server did not return a valid time. To resolve this issue, use a packet analyzing tool and do a trace of the NTP packets to check if all fields are correctly populated. (code: PE085) ________________________________ From: NANOG <nanog-bounces+mel=beckman.org@nanog.org><mailto:nanog-bounces+mel=beckman.org@nanog.org> on behalf of John Curran <jcurran@arin.net><mailto:jcurran@arin.net> Sent: Friday, May 10, 2024 10:54 AM To: NANOG <nanog@nanog.org><mailto:nanog@nanog.org> Subject: NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic Storm Watch SWPC Issues Its First G4 Watch Since 2005 | NOAA / NWS Space Weather Prediction Center<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> swpc.noaa.gov<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> [favicon.ico]<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005> "Multiple CMEs erupted associated with flare activity from Region 3664 on 07-09 May. These CMEs are expected to merge with potential arrival expected by early May 11 on the UTC day.” (Low but distinct possibility of effects to radio and transmission systems) FYI, /John John Curran President and CEO American Registry for Internet Numbers
participants (5)
-
Jay R. Ashworth -
John Curran -
Mel Beckman -
Randy Bush -
Raymond Burkholder