We had some Meinberg's which did something similar but different some time ago.  NTP was out of sync with GPS.  We had a CheckMk instance which detected drift between sources in our network.  Turns out there was one or more configs in the Meinberg that enable failover from one source to another, and to ensure GPS and NTP are working together rather than independently.

Maybe your TimeMachines have similar config variabilities.

On 2024-05-10 14:29, Mel Beckman wrote:


We just had two TM1000 TimeMachine brand GPS NTP servers lose clock sync at the same time, in two different cities (LA and Santa Barbara). The  outage lasted about five minutes, during which the NTP servers were responding, but with time that was 1900 seconds out of sync. The devices showed satellite lock on 8 birds (not all the same ones). I've never seen this behavior before with years of NTP clock experience. 

It could be that these inexpensive NTP servers aren't very selective about bogus inputs, as I would have expected them to lose synch in the event of a GPS signal failure. Instead they produced garbage. Our PRTG NTP monitor logged the problem this way:

      
Sensor SNTP (SNTP) ***
Device      10.2.10.90-TimeMachine NTP server (10.2.10.90)   
New Status at 5/10/2024 12:49:52 PM (Pacific Standard Time):
Down    
Last Message:
The target server did not return a valid time. To resolve this issue, use a packet analyzing tool and do a trace of the NTP packets to check if all fields are correctly populated. (code: PE085)

From: NANOG <nanog-bounces+mel=beckman.org@nanog.org> on behalf of John Curran <jcurran@arin.net>
Sent: Friday, May 10, 2024 10:54 AM
To: NANOG <nanog@nanog.org>
Subject: NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic Storm Watch
 

SWPC Issues Its First G4 Watch Since 2005 | NOAA / NWS Space Weather Prediction Center
swpc.noaa.gov
favicon.ico

"Multiple CMEs erupted associated with flare activity from Region 3664 on 07-09 May. These CMEs are expected to merge with potential arrival expected by early May 11 on the UTC day.”

(Low but distinct possibility of effects to radio and transmission systems) 

FYI,
/John

John Curran
President and CEO
American Registry for Internet Numbers