Re: NTP Sync Issue Across Tata (Europe)
On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman <mel@beckman.org> wrote:
Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need. BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits. OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it). Rubens
On 7 Aug 2023, at 12:02, Rubens Kuhl <rubensk@gmail.com> wrote:
On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman <mel@beckman.org> wrote: Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
Well DNS can be cryptographically secured. There really isn’t any good reasons to not sign your zones today. The majority of responses from authoritative servers are validated today so if you sign the responses will be checked. Unfortunately most to those validations still result in insecure instead of secure because people are not signing their zones.
BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
Rubens
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
Well DNS can be cryptographically secured. There really isn’t any good reasons to not sign your zones today. The majority of responses from authoritative servers are validated today so if you sign the responses will be checked. Unfortunately most to those validations still result in insecure instead of secure because people are not signing their zones.
So does NTP, with NTS. https://datatracker.ietf.org/doc/html/rfc8915 Rubens
Diversity from GPS might also be obtained by setting one receiver for GPS and another for Galileo. I think I'd skip GLONASS for now :) On Mon, Aug 7, 2023, 06:09 Rubens Kuhl <rubensk@gmail.com> wrote:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
Well DNS can be cryptographically secured. There really isn’t any good reasons to not sign your zones today. The majority of responses from authoritative servers are validated today so if you sign the responses will be checked. Unfortunately most to those validations still result in insecure instead of secure because people are not signing their zones.
So does NTP, with NTS.
https://datatracker.ietf.org/doc/html/rfc8915
Rubens
GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat. DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust. Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync’d? GPS, I’ll wager. I sense hand-waving :) -mel via cell On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com> wrote: On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation: The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need. BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits. OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it). Rubens
The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals. With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet. In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both. Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... . On Sun, Aug 6, 2023 at 8:36 PM Mel Beckman <mel@beckman.org> wrote:
GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync’d? GPS, I’ll wager.
I sense hand-waving :)
-mel via cell
On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com> wrote:
On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman <mel@beckman.org> wrote:
Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
Rubens
-- - Forrest
I forgot to finish my thought in the third paragraph before hitting send. What I was going to express was that one should choose not only close, trusted, NTP servers, but also perhaps ones from different government agencies, or different sources. Sourcing time from multiple sources not likely to be affected by the same outage and/or attack is good robust design. On Mon, Aug 7, 2023 at 2:39 AM Forrest Christian (List Account) < lists@packetflux.com> wrote:
The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.
With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.
In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.
Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... .
On Sun, Aug 6, 2023 at 8:36 PM Mel Beckman <mel@beckman.org> wrote:
GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync’d? GPS, I’ll wager.
I sense hand-waving :)
-mel via cell
On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com> wrote:
On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman <mel@beckman.org> wrote:
Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
Rubens
-- - Forrest
-- - Forrest
Forrest, GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It’s a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network. -mel On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote: The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals. With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet. In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both. Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... . On Sun, Aug 6, 2023 at 8:36 PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat. DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust. Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync’d? GPS, I’ll wager. I sense hand-waving :) -mel via cell On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote: On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation: The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need. BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits. OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it). Rubens -- - Forrest
Those particular boxes are not cheap. (Yes I know the units you talk about). Note that some of them rely on terrestrial communication of ephermis data to validate the GPS data to further make the time more robust. I was hopefully trying to dispel the seemingly common thread in this discussion that somehow a $400 GPS derived ntp clock is more robust than using the public NTP infrastructure. Without understanding that GPS receivers are also subject to similar attacks as NTP, one comes to the wrong conclusion about how to build a robust timing infrastructure. GPS is subject to DoS and spoofing attacks just like NTP, and as such, it's important to mitigate those as well if you care about time stability. In the end, it's all about risk vs cost to mitigate risk. If you're running a tiny network, deriving your time from the public NTP servers is fine. At the other end of the scale is using exotic hardened solutions which derive their own clock source and can hold over time for days or weeks in the event of a failure or attack. In the middle tends to be a more moderate solution which involves a mix of time transmission methods from a variety of geographically and/or network diverse sources. Taking time from the public trusted ntp servers and adding lower cost GPS receivers at diverse points in your network seems like a good compromise in the middle. That way, only coordinated attacks will be successful. On Mon, Aug 7, 2023, 8:03 AM Mel Beckman <mel@beckman.org> wrote:
Forrest,
GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It’s a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
-mel
On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) < lists@packetflux.com> wrote:
The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.
With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.
In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.
Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... .
On Sun, Aug 6, 2023 at 8:36 PM Mel Beckman <mel@beckman.org> wrote:
GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync’d? GPS, I’ll wager.
I sense hand-waving :)
-mel via cell
On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com> wrote:
On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman <mel@beckman.org> wrote:
Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
Rubens
-- - Forrest
Forrest Christian (List Account) wrote:
In the middle tends to be a more moderate solution which involves a mix of time transmission methods from a variety of geographically and/or network diverse sources. Taking time from the public trusted ntp servers and adding lower cost GPS receivers at diverse points in your network seems like a good compromise in the middle. That way, only coordinated attacks will be successful.
Instead, just rely on atomic clocks operated by you. They are not so expensive (several thousand dollars) and should be accurate enough without adjustment for hundreds of years. There can be no coordinated attacks. They may be remotely accessed through secured NTP. Masataka Ohta
Masataka, To be useful, any atomic clocks you operate must be synchronized to a Stratum Zero time source, such as GPS. Such clocks are useful when you need exceptional accuracy, such as for Building Integrated Timing Service (BITS), but unless they’re synchronized you can’t coordinate time-sensitive activities such as digital certificate validation with anyone else on the Internet. From https://www.gps.gov/applications/timing/<https://www.gps.gov/applications/timing/#:~:text=GPS%20receivers%20decode%20these%20signals,owning%20and%20operating%20atomic%20clocks.> Each GPS satellite contains multiple atomic clocks that contribute very precise time data to the GPS signals. GPS receivers decode these signals, effectively synchronizing each receiver to the atomic clocks. This enables users to determine the time to within 100 billionths of a second, without the cost of owning and operating atomic clocks. Precise time is crucial to a variety of economic activities around the world. Communication systems, electrical power grids, and financial networks all rely on precision timing for synchronization and operational efficiency. The free availability of GPS time has enabled cost savings for companies that depend on precise time and has led to significant advances in capability. -mel via cell On Aug 7, 2023, at 10:04 PM, Masataka Ohta <mohta@necom830.hpcl.titech.ac.jp> wrote: Forrest Christian (List Account) wrote: In the middle tends to be a more moderate solution which involves a mix of time transmission methods from a variety of geographically and/or network diverse sources. Taking time from the public trusted ntp servers and adding lower cost GPS receivers at diverse points in your network seems like a good compromise in the middle. That way, only coordinated attacks will be successful. Instead, just rely on atomic clocks operated by you. They are not so expensive (several thousand dollars) and should be accurate enough without adjustment for hundreds of years. There can be no coordinated attacks. They may be remotely accessed through secured NTP. Masataka Ohta
Mel Beckman wrote:
To be useful, any atomic clocks you operate must be synchronized to a Stratum Zero time source, such as GPS.
Only initially.
Precise time is crucial to a variety of economic activities around the world. Communication systems, electrical power grids, and financial networks all rely on precision timing for synchronization and operational efficiency. The free availability of GPS time has enabled cost savings for companies that depend on precise time and has led to significant advances in capability.
FYI, time difference between two points is not noticeable, that is, does not affect correctness of any distributed algorithm, if the difference is below the communication delay between the points, which means rough synchronization by NTP is good enough. That is an information theoretic version of relativity of simultaneity: https://en.wikipedia.org/wiki/Relativity_of_simultaneity For information theoretic simultaneity, you can consider, instead of light cone, information cone. Masataka Ohta
Depends on how synchronized you need to be. In the context of running airgapped: A rubidium oscillator or Chip Scale Atomic Clock is in the price range you quote. However, these can drift enough that you should occasionally synchronize with a reference time source. This is to ensure continued millisecond accuracy. Of course it all depends on how much drift you'll tolerate, and if you're OK with being within a second, then a rubidium might be ok. Caesium oscillators which have much lower drift are in the $30K-50K range. These would require significantly less frequent synchronization, but are definitely not a few thousand dollars. Note that these are both just oscillators and they need additional support hardware to be able to be queried by NTP. Or stated differently, they still need a NTP server. Yes, there are products out there which integrate everything in one box at an additional cost. On Mon, Aug 7, 2023, 11:02 PM Masataka Ohta < mohta@necom830.hpcl.titech.ac.jp> wrote:
Forrest Christian (List Account) wrote:
In the middle tends to be a more moderate solution which involves a mix of time transmission methods from a variety of geographically and/or network diverse sources. Taking time from the public trusted ntp servers and adding lower cost GPS receivers at diverse points in your network seems like a good compromise in the middle. That way, only coordinated attacks will be successful.
Instead, just rely on atomic clocks operated by you. They are not so expensive (several thousand dollars) and should be accurate enough without adjustment for hundreds of years. There can be no coordinated attacks. They may be remotely accessed through secured NTP.
Masataka Ohta
Forrest Christian (List Account) wrote:
Depends on how synchronized you need to be.
Sure. But, we should be assuming NTP is mostly enough.
A rubidium oscillator or Chip Scale Atomic Clock is in the price range you quote. However, these can drift enough that you should occasionally synchronize with a reference time source. This is to ensure continued millisecond accuracy. Of course it all depends on how much drift you'll tolerate, and if you're OK with being within a second, then a rubidium might be ok.
For millisecond accuracy, Rb clocks do not need any synchronization for centuries. Rb clocks on GPS are a lot more frequently synchronized, because a lot more accuracy is required for positioning (10ns of timing error means 3m of positioning error). Masataka Ohta
Mel Beckman wrote:-
Its a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
Whilst looking forward to being corrected, GPS (even across multiple locations) seems to be a SINGLE source of time. You seem (have I misunderstood?) to be a proponent of using GPS exclusively as the external clock source. Might it be preferable to have a mixture of GPS (perhaps with another GNSS) together with carefully selected Internet-based NTP servers? I recall an incident over here in Jersey (the one they named New Jersey after!) where our primary telco had a substantial time shift on one of their two GPS synced servers. This managed to adjust the clock on enough of their routers that the certificate-based OSPF authentication considered the certificates invalid, and caused a failure of almost their whole network. This is, of course, not to say that GPS is not a very good clock source, but rather to wonder whether more diversity would be preferable than using it as a single source. -- Best wishes, Matthew ------
From: Mel Beckman <mel@beckman.org> To: "Forrest Christian (List Account)" <lists@packetflux.com> Cc: Nanog <nanog@nanog.org> Date: Mon, 7 Aug 2023 14:03:30 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe)
Forrest,
GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. Its a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
-mel
On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:
? The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.
With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.
In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.
Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... .
On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
Your own atomic clocks still have to be synced to a common standard to be useful. To what are they syncd? GPS, Ill wager.
I sense hand-waving :)
-mel via cell
On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote:
?
On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
Rubens
Until the Internet NTP network can be made secure, no. Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I’d like to see the documentation. Using GPS for time sync is simple risk management: the risk of Internet NTP with known, well documented vulnerabilities and many security incidents, versus the risk of some theoretical GPS-based vulnerability, for which mitigations such as geographic diversity are readily available. Sure, you could use Internet NTP as a last resort should GPS fail globally (perhaps due to a theoretical — but conceivable — meteor storm). But that would be a fall-back. I would not mix the systems. -mel
On Aug 8, 2023, at 1:36 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote:
Mel Beckman wrote:-
It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
Whilst looking forward to being corrected, GPS (even across multiple locations) seems to be a SINGLE source of time. You seem (have I misunderstood?) to be a proponent of using GPS exclusively as the external clock source.
Might it be preferable to have a mixture of GPS (perhaps with another GNSS) together with carefully selected Internet-based NTP servers?
I recall an incident over here in Jersey (the one they named New Jersey after!) where our primary telco had a substantial time shift on one of their two GPS synced servers. This managed to adjust the clock on enough of their routers that the certificate-based OSPF authentication considered the certificates invalid, and caused a failure of almost their whole network.
This is, of course, not to say that GPS is not a very good clock source, but rather to wonder whether more diversity would be preferable than using it as a single source.
-- Best wishes, Matthew
------
From: Mel Beckman <mel@beckman.org> To: "Forrest Christian (List Account)" <lists@packetflux.com> Cc: Nanog <nanog@nanog.org> Date: Mon, 7 Aug 2023 14:03:30 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe)
Forrest,
GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
-mel
On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:
? The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.
With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.
In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.
Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... .
On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync'd? GPS, I'll wager.
I sense hand-waving :)
-mel via cell
On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote:
?
On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
Rubens
On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman <mel@beckman.org> wrote:
Until the Internet NTP network can be made secure, no.
Internet NTP can be made secure, it's called NTS. https://developers.cloudflare.com/time-services/nts/ describes it with links to the RFC, and describes one of the many NTP servers that is available with NTS, time.cloudflare.com. I already mentioned 5 others, and there are many more than those 6. Rubens
I’m familiar with NTS, which is pointedly not NTP. That’s like saying “TCP port 80 can be made secure,: just use a VPN!” Perhaps when NTS is widely deployed it will be an option. As the RFC was only approved in 2020, that will probably take a decade. Or more. (I’m talking about you, IPv6 :) Not to mention the complexity or NTS for hardware implementation in network elements, a primary application (https://tinyurl.com/ntsishard). -mel
On Aug 8, 2023, at 8:26 AM, Rubens Kuhl <rubensk@gmail.com> wrote:
On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman <mel@beckman.org> wrote:
Until the Internet NTP network can be made secure, no.
Internet NTP can be made secure, it's called NTS. https://developers.cloudflare.com/time-services/nts/ describes it with links to the RFC, and describes one of the many NTP servers that is available with NTS, time.cloudflare.com. I already mentioned 5 others, and there are many more than those 6.
Rubens
So little deployment that has 3500 occurrences according to shodan.io. With such few choices, It should be hard to find suitable options. Rubens Em ter., 8 de ago. de 2023 13:02, Mel Beckman <mel@beckman.org> escreveu:
I’m familiar with NTS, which is pointedly not NTP. That’s like saying “TCP port 80 can be made secure,: just use a VPN!” Perhaps when NTS is widely deployed it will be an option. As the RFC was only approved in 2020, that will probably take a decade. Or more. (I’m talking about you, IPv6 :) Not to mention the complexity or NTS for hardware implementation in network elements, a primary application (https://tinyurl.com/ntsishard).
-mel
On Aug 8, 2023, at 8:26 AM, Rubens Kuhl <rubensk@gmail.com> wrote:
On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman <mel@beckman.org> wrote:
Until the Internet NTP network can be made secure, no.
Internet NTP can be made secure, it's called NTS. https://developers.cloudflare.com/time-services/nts/ describes it with links to the RFC, and describes one of the many NTP servers that is available with NTS, time.cloudflare.com. I already mentioned 5 others, and there are many more than those 6.
Rubens
Go for it. I’m sure NTS’ complexity clocks lots of hours for expensive consultants :) Me, I’m sticking with GPS.) -mel via cell On Aug 8, 2023, at 11:34 AM, Rubens Kuhl <rubensk@gmail.com> wrote: So little deployment that has 3500 occurrences according to shodan.io<http://shodan.io>. With such few choices, It should be hard to find suitable options. Rubens Em ter., 8 de ago. de 2023 13:02, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> escreveu: I’m familiar with NTS, which is pointedly not NTP. That’s like saying “TCP port 80 can be made secure,: just use a VPN!” Perhaps when NTS is widely deployed it will be an option. As the RFC was only approved in 2020, that will probably take a decade. Or more. (I’m talking about you, IPv6 :) Not to mention the complexity or NTS for hardware implementation in network elements, a primary application (https://tinyurl.com/ntsishard). -mel
On Aug 8, 2023, at 8:26 AM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote:
On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:
Until the Internet NTP network can be made secure, no.
Internet NTP can be made secure, it's called NTS. https://developers.cloudflare.com/time-services/nts/ describes it with links to the RFC, and describes one of the many NTP servers that is available with NTS, time.cloudflare.com<http://time.cloudflare.com>. I already mentioned 5 others, and there are many more than those 6.
Rubens
Mel Beckman wrote:-
Do you have a citation for your Jersey event? I doubt GPS caused the problem, but Id like to see the documentation.
The event took place on the evening of Sunday 12 July 2020, and seems NOT to have been due to an issue caused directly by GPS, but rather to misbehaviour of a GPS NTP server relating to week numbers. Our regulator subsequently issued the following comprehensive document:- https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directio... By way of summary, JT operated two GPS derived NTP servers, with all of their routers were pointing to both. On the evening in question, one of the two reset its clock back to 27 November 2000. Their interior routing protocol used amongst their mesh of routers was IS-IS which was using authentication. The authentication [section 4.19] was described having a "password validity start date" of 01 July 2012. Thus, any routers which had picked up the time from the faulty source no longer had valid IS-IS authentication and were thus isolated. Whilst only 15% of their routers were affected, this was enough to cause an almost total failure in their network, affecting telephony (fixed & mobile) and Internet. For foreign readers (this is NANOG!) "999" calls refer to the emergency services in these parts, where any failures attract the attention of our regulator. The details of why the clock "failed" start at section 4.23, and seem to relate a GPS week number rollover. So, probably not a failure "caused by GPS", rather one caused by poor design (only two clock sources) combined with unsupported and buggy devices. One curious aspect is that some routers followed the "bad" time, which is alluded to in section 4.31. Something not discussed in that report is that JT's email failed during the incident despite its being hosted on Office365. The reason was that the two authoritative DNS servers for jtglobal.com were hosted in Jersey inside their network. As that network was wholly disconnected, there was no DNS and hence no email. Despite my having raised this since with their senior management, their DNS remains hosted in this way:-
matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @ns1.jtibs.net ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462 ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4
;; QUESTION SECTION: ;jtglobal.com. IN NS
;; ANSWER SECTION: jtglobal.com. 60 IN NS ns2.jtibs.net. jtglobal.com. 60 IN NS ns1.jtibs.net.
;; ADDITIONAL SECTION: ns1.jtibs.net. 60 IN A 212.9.0.135 ns2.jtibs.net. 60 IN A 212.9.0.136 ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1 ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2
Rediculously (and again despite my agitation to their management) our government domain gov.je has similar DNS fragility:-
matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249 ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
;; QUESTION SECTION: ;gov.je. IN NS
;; ANSWER SECTION: gov.je. 3600 IN NS ns2.gov.je. gov.je. 3600 IN NS ns1.gov.je.
;; ADDITIONAL SECTION: ns2.gov.je. 3600 IN A 212.9.21.137 ns1.gov.je. 3600 IN A 212.9.21.9
-- Best wishes, Matthew ------
From: Mel Beckman <mel@beckman.org> To: Matthew Richardson <matthew-l@itconsult.co.uk> Cc: Nanog <nanog@nanog.org> Date: Tue, 8 Aug 2023 15:12:29 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe)
Until the Internet NTP network can be made secure, no. Do you have a citation for your Jersey event? I doubt GPS caused the problem, but Id like to see the documentation.
Using GPS for time sync is simple risk management: the risk of Internet NTP with known, well documented vulnerabilities and many security incidents, versus the risk of some theoretical GPS-based vulnerability, for which mitigations such as geographic diversity are readily available. Sure, you could use Internet NTP as a last resort should GPS fail globally (perhaps due to a theoretical but conceivable meteor storm). But that would be a fall-back. I would not mix the systems.
-mel
On Aug 8, 2023, at 1:36 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote:
?Mel Beckman wrote:-
It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
Whilst looking forward to being corrected, GPS (even across multiple locations) seems to be a SINGLE source of time. You seem (have I misunderstood?) to be a proponent of using GPS exclusively as the external clock source.
Might it be preferable to have a mixture of GPS (perhaps with another GNSS) together with carefully selected Internet-based NTP servers?
I recall an incident over here in Jersey (the one they named New Jersey after!) where our primary telco had a substantial time shift on one of their two GPS synced servers. This managed to adjust the clock on enough of their routers that the certificate-based OSPF authentication considered the certificates invalid, and caused a failure of almost their whole network.
This is, of course, not to say that GPS is not a very good clock source, but rather to wonder whether more diversity would be preferable than using it as a single source.
-- Best wishes, Matthew
------
From: Mel Beckman <mel@beckman.org> To: "Forrest Christian (List Account)" <lists@packetflux.com> Cc: Nanog <nanog@nanog.org> Date: Mon, 7 Aug 2023 14:03:30 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe)
Forrest,
GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
-mel
On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:
? The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.
With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.
In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.
Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... .
On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync'd? GPS, I'll wager.
I sense hand-waving :)
-mel via cell
On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote:
?
On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
Rubens
So, probably not a failure "caused by GPS", rather one caused by poor design (only two clock sources) combined with unsupported and buggy devices. -mel beckman On Aug 16, 2023, at 3:51 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote: Mel Beckman wrote:- Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation. The event took place on the evening of Sunday 12 July 2020, and seems NOT to have been due to an issue caused directly by GPS, but rather to misbehaviour of a GPS NTP server relating to week numbers. Our regulator subsequently issued the following comprehensive document:- https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directio... By way of summary, JT operated two GPS derived NTP servers, with all of their routers were pointing to both. On the evening in question, one of the two reset its clock back to 27 November 2000. Their interior routing protocol used amongst their mesh of routers was IS-IS which was using authentication. The authentication [section 4.19] was described having a "password validity start date" of 01 July 2012. Thus, any routers which had picked up the time from the faulty source no longer had valid IS-IS authentication and were thus isolated. Whilst only 15% of their routers were affected, this was enough to cause an almost total failure in their network, affecting telephony (fixed & mobile) and Internet. For foreign readers (this is NANOG!) "999" calls refer to the emergency services in these parts, where any failures attract the attention of our regulator. The details of why the clock "failed" start at section 4.23, and seem to relate a GPS week number rollover. So, probably not a failure "caused by GPS", rather one caused by poor design (only two clock sources) combined with unsupported and buggy devices. One curious aspect is that some routers followed the "bad" time, which is alluded to in section 4.31. Something not discussed in that report is that JT's email failed during the incident despite its being hosted on Office365. The reason was that the two authoritative DNS servers for jtglobal.com were hosted in Jersey inside their network. As that network was wholly disconnected, there was no DNS and hence no email. Despite my having raised this since with their senior management, their DNS remains hosted in this way:- matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @ns1.jtibs.net ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462 ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4 ;; QUESTION SECTION: ;jtglobal.com. IN NS ;; ANSWER SECTION: jtglobal.com. 60 IN NS ns2.jtibs.net. jtglobal.com. 60 IN NS ns1.jtibs.net. ;; ADDITIONAL SECTION: ns1.jtibs.net. 60 IN A 212.9.0.135 ns2.jtibs.net. 60 IN A 212.9.0.136 ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1 ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2 Rediculously (and again despite my agitation to their management) our government domain gov.je has similar DNS fragility:- matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249 ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2 ;; QUESTION SECTION: ;gov.je. IN NS ;; ANSWER SECTION: gov.je. 3600 IN NS ns2.gov.je. gov.je. 3600 IN NS ns1.gov.je. ;; ADDITIONAL SECTION: ns2.gov.je. 3600 IN A 212.9.21.137 ns1.gov.je. 3600 IN A 212.9.21.9 -- Best wishes, Matthew ------ From: Mel Beckman <mel@beckman.org> To: Matthew Richardson <matthew-l@itconsult.co.uk> Cc: Nanog <nanog@nanog.org> Date: Tue, 8 Aug 2023 15:12:29 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe) Until the Internet NTP network can be made secure, no. Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation. Using GPS for time sync is simple risk management: the risk of Internet NTP with known, well documented vulnerabilities and many security incidents, versus the risk of some theoretical GPS-based vulnerability, for which mitigations such as geographic diversity are readily available. Sure, you could use Internet NTP as a last resort should GPS fail globally (perhaps due to a theoretical - but conceivable - meteor storm). But that would be a fall-back. I would not mix the systems. -mel On Aug 8, 2023, at 1:36 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote: ?Mel Beckman wrote:- It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network. Whilst looking forward to being corrected, GPS (even across multiple locations) seems to be a SINGLE source of time. You seem (have I misunderstood?) to be a proponent of using GPS exclusively as the external clock source. Might it be preferable to have a mixture of GPS (perhaps with another GNSS) together with carefully selected Internet-based NTP servers? I recall an incident over here in Jersey (the one they named New Jersey after!) where our primary telco had a substantial time shift on one of their two GPS synced servers. This managed to adjust the clock on enough of their routers that the certificate-based OSPF authentication considered the certificates invalid, and caused a failure of almost their whole network. This is, of course, not to say that GPS is not a very good clock source, but rather to wonder whether more diversity would be preferable than using it as a single source. -- Best wishes, Matthew ------ From: Mel Beckman <mel@beckman.org> To: "Forrest Christian (List Account)" <lists@packetflux.com> Cc: Nanog <nanog@nanog.org> Date: Mon, 7 Aug 2023 14:03:30 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe) Forrest, GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network. -mel On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote: ? The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals. With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet. In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both. Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... . On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat. DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust. Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync'd? GPS, I'll wager. I sense hand-waving :) -mel via cell On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote: ? On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation: The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need. BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits. OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it). Rubens
So, probably not a failure "caused by GPS", rather one caused by poor design (only two clock sources) combined with unsupported and buggy devices.
100% correct. From the PDF : 4.31 JT summarised its findings in relation to the ‘Panic Timer’ on the
Cisco IOS XR NTP Client, namely that: JT’s efforts in understanding the root cause, and mitigation steps to take to avoid future incidents have focused on the Cisco NTP Client behaviour, and notably Cisco’s decision to not implement the ‘Panic Timer’ on their IOS XR operating system. Arguably, whilst the NTP server injected an invalid time into the network, it is the NTP Clients filtering and selection algorithms which are responsible for detecting and disregarding falsetickers, and it was the Cisco NTP Clients failure to appropriately handle this which triggered the network incident. 43 […] Further detailed soak testing, log analysis and debug analysis corroborated that the Cisco IOS XR NTP Client did not implement the ‘Panic Timer’ that would normally cause an NTP Client to ignore an NTP Server exceeding 1000 seconds variance.
On Wed, Aug 16, 2023 at 10:50 AM Mel Beckman <mel@beckman.org> wrote:
So, probably not a failure "caused by GPS", rather one caused by poor design (only two clock sources) combined with unsupported and buggy devices.
-mel beckman
On Aug 16, 2023, at 3:51 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote:
Mel Beckman wrote:-
Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation.
The event took place on the evening of Sunday 12 July 2020, and seems NOT to have been due to an issue caused directly by GPS, but rather to misbehaviour of a GPS NTP server relating to week numbers. Our regulator subsequently issued the following comprehensive document:-
https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directio...
By way of summary, JT operated two GPS derived NTP servers, with all of their routers were pointing to both. On the evening in question, one of the two reset its clock back to 27 November 2000.
Their interior routing protocol used amongst their mesh of routers was IS-IS which was using authentication. The authentication [section 4.19] was described having a "password validity start date" of 01 July 2012. Thus, any routers which had picked up the time from the faulty source no longer had valid IS-IS authentication and were thus isolated.
Whilst only 15% of their routers were affected, this was enough to cause an almost total failure in their network, affecting telephony (fixed & mobile) and Internet. For foreign readers (this is NANOG!) "999" calls refer to the emergency services in these parts, where any failures attract the attention of our regulator.
The details of why the clock "failed" start at section 4.23, and seem to relate a GPS week number rollover.
So, probably not a failure "caused by GPS", rather one caused by poor design (only two clock sources) combined with unsupported and buggy devices.
One curious aspect is that some routers followed the "bad" time, which is alluded to in section 4.31.
Something not discussed in that report is that JT's email failed during the incident despite its being hosted on Office365. The reason was that the two authoritative DNS servers for jtglobal.com were hosted in Jersey inside their network. As that network was wholly disconnected, there was no DNS and hence no email. Despite my having raised this since with their senior management, their DNS remains hosted in this way:-
matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @ ns1.jtibs.net
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4
;; QUESTION SECTION:
;jtglobal.com. IN NS
;; ANSWER SECTION:
jtglobal.com. 60 IN NS ns2.jtibs.net.
jtglobal.com. 60 IN NS ns1.jtibs.net.
;; ADDITIONAL SECTION:
ns1.jtibs.net. 60 IN A 212.9.0.135
ns2.jtibs.net. 60 IN A 212.9.0.136
ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1
ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2
Rediculously (and again despite my agitation to their management) our government domain gov.je has similar DNS fragility:-
matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
;; QUESTION SECTION:
;gov.je. IN NS
;; ANSWER SECTION:
gov.je. 3600 IN NS ns2.gov.je.
gov.je. 3600 IN NS ns1.gov.je.
;; ADDITIONAL SECTION:
ns2.gov.je. 3600 IN A 212.9.21.137
ns1.gov.je. 3600 IN A 212.9.21.9
-- Best wishes, Matthew
------
From: Mel Beckman <mel@beckman.org>
To: Matthew Richardson <matthew-l@itconsult.co.uk>
Cc: Nanog <nanog@nanog.org>
Date: Tue, 8 Aug 2023 15:12:29 +0000
Subject: Re: NTP Sync Issue Across Tata (Europe)
Until the Internet NTP network can be made secure, no. Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation.
Using GPS for time sync is simple risk management: the risk of Internet NTP with known, well documented vulnerabilities and many security incidents, versus the risk of some theoretical GPS-based vulnerability, for which mitigations such as geographic diversity are readily available. Sure, you could use Internet NTP as a last resort should GPS fail globally (perhaps due to a theoretical - but conceivable - meteor storm). But that would be a fall-back. I would not mix the systems.
-mel
On Aug 8, 2023, at 1:36 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote:
?Mel Beckman wrote:-
It's a problem that has received a lot of attention in both NTP and
aviation navigation circles. What is hard to defend against is total signal
suppression via high powered jamming. But that you can do with a
geographically diverse GPS NTP network.
Whilst looking forward to being corrected, GPS (even across multiple
locations) seems to be a SINGLE source of time. You seem (have I
misunderstood?) to be a proponent of using GPS exclusively as the external
clock source.
Might it be preferable to have a mixture of GPS (perhaps with another GNSS)
together with carefully selected Internet-based NTP servers?
I recall an incident over here in Jersey (the one they named New Jersey
after!) where our primary telco had a substantial time shift on one of
their two GPS synced servers. This managed to adjust the clock on enough
of their routers that the certificate-based OSPF authentication considered
the certificates invalid, and caused a failure of almost their whole
network.
This is, of course, not to say that GPS is not a very good clock source,
but rather to wonder whether more diversity would be preferable than using
it as a single source.
--
Best wishes,
Matthew
------
From: Mel Beckman <mel@beckman.org>
To: "Forrest Christian (List Account)" <lists@packetflux.com>
Cc: Nanog <nanog@nanog.org>
Date: Mon, 7 Aug 2023 14:03:30 +0000
Subject: Re: NTP Sync Issue Across Tata (Europe)
Forrest,
GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
-mel
On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) < lists@packetflux.com> wrote:
?
The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.
With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.
In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.
Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... .
On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto: mel@beckman.org>> wrote:
GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync'd? GPS, I'll wager.
I sense hand-waving :)
-mel via cell
On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto: rubensk@gmail.com>> wrote:
?
On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto: mel@beckman.org>> wrote:
Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
Rubens
Thanks for that link. This is jumping out at me though : Their interior routing protocol used amongst their mesh of routers was
IS-IS which was using authentication. The authentication [section 4.19] was described having a "password validity start date" of 01 July 2012. Thus, any routers which had picked up the time from the faulty source no longer had valid IS-IS authentication and were thus isolated.
It's been a while, but last time I remember diving into the IS-IS weeds , the time of the transmitting system wasn't part of a Hello. Is this a Cisco specific option they toss in a TLV? On Wed, Aug 16, 2023 at 9:04 AM Matthew Richardson via NANOG < nanog@nanog.org> wrote:
Mel Beckman wrote:-
Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I’d like to see the documentation.
The event took place on the evening of Sunday 12 July 2020, and seems NOT to have been due to an issue caused directly by GPS, but rather to misbehaviour of a GPS NTP server relating to week numbers. Our regulator subsequently issued the following comprehensive document:-
https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directio...
By way of summary, JT operated two GPS derived NTP servers, with all of their routers were pointing to both. On the evening in question, one of the two reset its clock back to 27 November 2000.
Their interior routing protocol used amongst their mesh of routers was IS-IS which was using authentication. The authentication [section 4.19] was described having a "password validity start date" of 01 July 2012. Thus, any routers which had picked up the time from the faulty source no longer had valid IS-IS authentication and were thus isolated.
Whilst only 15% of their routers were affected, this was enough to cause an almost total failure in their network, affecting telephony (fixed & mobile) and Internet. For foreign readers (this is NANOG!) "999" calls refer to the emergency services in these parts, where any failures attract the attention of our regulator.
The details of why the clock "failed" start at section 4.23, and seem to relate a GPS week number rollover.
So, probably not a failure "caused by GPS", rather one caused by poor design (only two clock sources) combined with unsupported and buggy devices.
One curious aspect is that some routers followed the "bad" time, which is alluded to in section 4.31.
Something not discussed in that report is that JT's email failed during the incident despite its being hosted on Office365. The reason was that the two authoritative DNS servers for jtglobal.com were hosted in Jersey inside their network. As that network was wholly disconnected, there was no DNS and hence no email. Despite my having raised this since with their senior management, their DNS remains hosted in this way:-
matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @ ns1.jtibs.net ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462 ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4
;; QUESTION SECTION: ;jtglobal.com. IN NS
;; ANSWER SECTION: jtglobal.com. 60 IN NS ns2.jtibs.net. jtglobal.com. 60 IN NS ns1.jtibs.net.
;; ADDITIONAL SECTION: ns1.jtibs.net. 60 IN A 212.9.0.135 ns2.jtibs.net. 60 IN A 212.9.0.136 ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1 ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2
Rediculously (and again despite my agitation to their management) our government domain gov.je has similar DNS fragility:-
matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ ns1.gov.je ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249 ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2
;; QUESTION SECTION: ;gov.je. IN NS
;; ANSWER SECTION: gov.je. 3600 IN NS ns2.gov.je. gov.je. 3600 IN NS ns1.gov.je.
;; ADDITIONAL SECTION: ns2.gov.je. 3600 IN A 212.9.21.137 ns1.gov.je. 3600 IN A 212.9.21.9
-- Best wishes, Matthew
------
From: Mel Beckman <mel@beckman.org> To: Matthew Richardson <matthew-l@itconsult.co.uk> Cc: Nanog <nanog@nanog.org> Date: Tue, 8 Aug 2023 15:12:29 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe)
Until the Internet NTP network can be made secure, no. Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I’d like to see the documentation.
Using GPS for time sync is simple risk management: the risk of Internet NTP with known, well documented vulnerabilities and many security incidents, versus the risk of some theoretical GPS-based vulnerability, for which mitigations such as geographic diversity are readily available. Sure, you could use Internet NTP as a last resort should GPS fail globally (perhaps due to a theoretical — but conceivable — meteor storm). But that would be a fall-back. I would not mix the systems.
-mel
On Aug 8, 2023, at 1:36 AM, Matthew Richardson < matthew-l@itconsult.co.uk> wrote:
?Mel Beckman wrote:-
It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
Whilst looking forward to being corrected, GPS (even across multiple locations) seems to be a SINGLE source of time. You seem (have I misunderstood?) to be a proponent of using GPS exclusively as the external clock source.
Might it be preferable to have a mixture of GPS (perhaps with another GNSS) together with carefully selected Internet-based NTP servers?
I recall an incident over here in Jersey (the one they named New Jersey after!) where our primary telco had a substantial time shift on one of their two GPS synced servers. This managed to adjust the clock on enough of their routers that the certificate-based OSPF authentication considered the certificates invalid, and caused a failure of almost their whole network.
This is, of course, not to say that GPS is not a very good clock source, but rather to wonder whether more diversity would be preferable than using it as a single source.
-- Best wishes, Matthew
------
From: Mel Beckman <mel@beckman.org> To: "Forrest Christian (List Account)" <lists@packetflux.com> Cc: Nanog <nanog@nanog.org> Date: Mon, 7 Aug 2023 14:03:30 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe)
Forrest,
GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network.
-mel
On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) < lists@packetflux.com> wrote:
? The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals.
With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet.
In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both.
Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... .
On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto: mel@beckman.org>> wrote: GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync'd? GPS, I'll wager.
I sense hand-waving :)
-mel via cell
On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto: rubensk@gmail.com>> wrote:
?
On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto: mel@beckman.org>> wrote: Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation:
The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need.
BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits.
OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it).
Rubens
So, probably not a failure "caused by GPS", rather one caused by poor design (only two clock sources) combined with unsupported and buggy devices. Interesting software bug, but not really germane to this discussion, other than as a cautionary tale about time distribution architectures. -mel beckman On Aug 16, 2023, at 3:51 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote: Mel Beckman wrote:- Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation. The event took place on the evening of Sunday 12 July 2020, and seems NOT to have been due to an issue caused directly by GPS, but rather to misbehaviour of a GPS NTP server relating to week numbers. Our regulator subsequently issued the following comprehensive document:- https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directio... By way of summary, JT operated two GPS derived NTP servers, with all of their routers were pointing to both. On the evening in question, one of the two reset its clock back to 27 November 2000. Their interior routing protocol used amongst their mesh of routers was IS-IS which was using authentication. The authentication [section 4.19] was described having a "password validity start date" of 01 July 2012. Thus, any routers which had picked up the time from the faulty source no longer had valid IS-IS authentication and were thus isolated. Whilst only 15% of their routers were affected, this was enough to cause an almost total failure in their network, affecting telephony (fixed & mobile) and Internet. For foreign readers (this is NANOG!) "999" calls refer to the emergency services in these parts, where any failures attract the attention of our regulator. The details of why the clock "failed" start at section 4.23, and seem to relate a GPS week number rollover. So, probably not a failure "caused by GPS", rather one caused by poor design (only two clock sources) combined with unsupported and buggy devices. One curious aspect is that some routers followed the "bad" time, which is alluded to in section 4.31. Something not discussed in that report is that JT's email failed during the incident despite its being hosted on Office365. The reason was that the two authoritative DNS servers for jtglobal.com were hosted in Jersey inside their network. As that network was wholly disconnected, there was no DNS and hence no email. Despite my having raised this since with their senior management, their DNS remains hosted in this way:- matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com @ns1.jtibs.net ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462 ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4 ;; QUESTION SECTION: ;jtglobal.com. IN NS ;; ANSWER SECTION: jtglobal.com. 60 IN NS ns2.jtibs.net. jtglobal.com. 60 IN NS ns1.jtibs.net. ;; ADDITIONAL SECTION: ns1.jtibs.net. 60 IN A 212.9.0.135 ns2.jtibs.net. 60 IN A 212.9.0.136 ns1.jtibs.net. 60 IN AAAA 2a02:c28::d1 ns2.jtibs.net. 60 IN AAAA 2a02:c28::d2 Rediculously (and again despite my agitation to their management) our government domain gov.je has similar DNS fragility:- matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249 ;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2 ;; QUESTION SECTION: ;gov.je. IN NS ;; ANSWER SECTION: gov.je. 3600 IN NS ns2.gov.je. gov.je. 3600 IN NS ns1.gov.je. ;; ADDITIONAL SECTION: ns2.gov.je. 3600 IN A 212.9.21.137 ns1.gov.je. 3600 IN A 212.9.21.9 -- Best wishes, Matthew ------ From: Mel Beckman <mel@beckman.org> To: Matthew Richardson <matthew-l@itconsult.co.uk> Cc: Nanog <nanog@nanog.org> Date: Tue, 8 Aug 2023 15:12:29 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe) Until the Internet NTP network can be made secure, no. Do you have a citation for your Jersey event? I doubt GPS caused the problem, but I'd like to see the documentation. Using GPS for time sync is simple risk management: the risk of Internet NTP with known, well documented vulnerabilities and many security incidents, versus the risk of some theoretical GPS-based vulnerability, for which mitigations such as geographic diversity are readily available. Sure, you could use Internet NTP as a last resort should GPS fail globally (perhaps due to a theoretical - but conceivable - meteor storm). But that would be a fall-back. I would not mix the systems. -mel On Aug 8, 2023, at 1:36 AM, Matthew Richardson <matthew-l@itconsult.co.uk> wrote: ?Mel Beckman wrote:- It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network. Whilst looking forward to being corrected, GPS (even across multiple locations) seems to be a SINGLE source of time. You seem (have I misunderstood?) to be a proponent of using GPS exclusively as the external clock source. Might it be preferable to have a mixture of GPS (perhaps with another GNSS) together with carefully selected Internet-based NTP servers? I recall an incident over here in Jersey (the one they named New Jersey after!) where our primary telco had a substantial time shift on one of their two GPS synced servers. This managed to adjust the clock on enough of their routers that the certificate-based OSPF authentication considered the certificates invalid, and caused a failure of almost their whole network. This is, of course, not to say that GPS is not a very good clock source, but rather to wonder whether more diversity would be preferable than using it as a single source. -- Best wishes, Matthew ------ From: Mel Beckman <mel@beckman.org> To: "Forrest Christian (List Account)" <lists@packetflux.com> Cc: Nanog <nanog@nanog.org> Date: Mon, 7 Aug 2023 14:03:30 +0000 Subject: Re: NTP Sync Issue Across Tata (Europe) Forrest, GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but commercial industrial NTP servers have specific anti-spoofing mitigations. There are also antenna diversity strategies that vendors support to ensure the signal being relied upon is coming from the right direction. It's a problem that has received a lot of attention in both NTP and aviation navigation circles. What is hard to defend against is total signal suppression via high powered jamming. But that you can do with a geographically diverse GPS NTP network. -mel On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote: ? The problem with relying exclusively on GPS to do time distribution is the ease with which one can spoof the GPS signals. With a budget of around $1K, not including a laptop, anyone with decent technical skills could convince a typical GPS receiver it was at any position and was at any time in the world. All it takes is a decent directional antenna, some SDR hardware, and depending on the location and directivity of your antenna maybe a smallish amplifier. There is much discussion right now in the PNT (Position, Navigation and Timing) community as to how best to secure the GNSS network, but right now one should consider the data from GPS to be no more trustworthy than some random NTP server on the internet. In order to build a resilient NTP server infrastructure you need multiple sources of time distributed by multiple methods - typically both via satellite (GPS) and by terrestrial (NTP) methods. NTP does a pretty good job of sorting out multiple time servers and discarding sources that are lying. But to do this you need multiple time sources. A common recommendation is to run a couple/few NTP servers which only get time from a GPS receiver and only serve time to a second tier of servers that pull from both those in-house GPS-timed-NTP servers and other trusted NTP servers. I'd recommend selecting the time servers to gain geographic diversity, i.e. poll NIST servers in Maryland and Colorado, and possibly both. Note that NIST will exchange (via mail) a set of keys with you to talk encrypted NTP with you. See https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-auth... . On Sun, Aug 6, 2023 at 8:36?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat. DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust. Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync'd? GPS, I'll wager. I sense hand-waving :) -mel via cell On Aug 6, 2023, at 7:04 PM, Rubens Kuhl <rubensk@gmail.com<mailto:rubensk@gmail.com>> wrote: ? On Sun, Aug 6, 2023 at 8:20?PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: Or one can read recent research papers that thoroughly document the incredible fragility of the existing NTP hierarchy and soberly consider their recommendations for remediation: The paper suggests the compromise of critical infrastructure. So, besides not using NTP, why not stop using DNS ? Just populate a hosts file with all you need. BTW, the stratum-0 source you suggested is known to have been manipulated in the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to bet on that specific state actor not returning to old habits. OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can keep using GPS as well. If GPS goes bananas on timing, that source will just be disregarded (one of the features of the NTP architecture that has been pointed out over and over in this thread and you keep ignoring it). Rubens
On Sun, Aug 6, 2023 at 11:36 PM Mel Beckman <mel@beckman.org> wrote:
GPS Selective Availability did not disrupt the timing chain of GPS, only the ephemeris (position information). But a government-disrupted timebase scenario has never occurred, while hackers are a documented threat.
DNS has DNSSec, which while not deployed as broadly as we might like, at least lets us know which servers we can trust.
NTP has NTS, which all 5 servers I mentioned have, and many more also have.
Your own atomic clocks still have to be synced to a common standard to be useful. To what are they sync’d? GPS, I’ll wager.
Nope, they are synced to the reference atomic clock of National Observatory, pretty similar to USNO but local. Rubens
participants (8)
-
Dorn Hetzel -
Forrest Christian (List Account) -
Mark Andrews -
Masataka Ohta -
Matthew Richardson -
Mel Beckman -
Rubens Kuhl -
Tom Beecher