On 8/5/23 18:30, Matthew McGehrin wrote:

Hi Mark.

Wouldn't a local server be more optimal?

IE:

server 0.nl.pool.ntp.org server 1.nl.pool.ntp.org server 2.nl.pool.ntp.org server 3.nl.pool.ntp.org

I have a bunch of servers all over Europe I'd prefer not to touch if this is just a transient issue.

Or for Africa server 0.za.pool.ntp.org server 1.za.pool.ntp.org server 2.za.pool.ntp.org server 3.za.pool.ntp.org

I was speaking about Europe. There is no problem in Africa. Mark.

On 8/5/23 19:51, Andreas Ott wrote:

See for yourself how his pool server scores at https://www.ntppool.org/scores/197.224.66.40

I am not sure why it would be inserted into DNS answers for a worldwide pool like 0.freebsd as it clearly does have connectivity issues from some of the pool project's own sensors.

Many thanks, Andreas. I'll take this up with the FreeBSD folk. Mark.

On 8/5/23 20:17, Chris Adams wrote:

It's the NTP pool people you need to talk to - the .freebsd. bit is just a vendored entry into the pool (more for load tracking and management).

Yes, Andreas clarified in unicast. Will do. Thanks. Mark.

On Sat, Aug 5, 2023 at 12:26 PM Mel Beckman <mel@beckman.org> wrote:

You might consider setting up your own GPS-based NTP network.

GPS time is monitored (and when necessary, adjusted) from the U.S. Naval Observatory Master Clock, which is -the- authoritative time source for the United States. The USNO also provides an NTP time source from the same master clock: https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-Naval-Observatory... You -should not- just point your servers there, but it's useful to point a few servers each at one of them in order to serve as your network stratum 2 sources that keep the rest of your machines in sync with each other. That last point is key. You don't want your servers in sync with random Internet time sources. You want them in sync with each other. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/

* mel@beckman.org (Mel Beckman) [Sun 06 Aug 2023, 04:26 CEST]:

if you can eliminate such security problems for $400, I say it’s cheap at twice the price.

You must be unfamiliar with the prices neutral colocation facilities charge for roof access. -- Niels.

William, Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive. That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it. -mel

On Aug 6, 2023, at 10:53 AM, William Herrin <bill@herrin.us> wrote:

On Sat, Aug 5, 2023 at 7:24 PM Mel Beckman <mel@beckman.org> wrote:

...

That still leaves you open to NTP attacks. The USNO accuracy and monitoring is worthless if you suffer, for example, an NTP DDoS attack.

Hi Mel,

From what I can tell, a fairly simple firewall policy of allow UDP 123 from known NTP clients and established connections (I sent them a UDP packet recently) stops every one of those attacks (that's actually an NTP attack and not something else like a DNS attack) except for upstream address hijack that happens to coincide with your system boot. And it still depends on the attacker executing an additional sophisticated attack to do more than cause you a denial of service.

The links you sent are very interesting, at least in an academic sense, but they don't cause me to be unduly concerned about employing NTP.

...

if you can eliminate such security problems for $400, I say it’s cheap at twice the price.

Except you can't. Redundancy is required for any critical service. At the $400 price point, your approach has multiple single-points-of-failure. The device itself of course. Your ability to receive continuous non-jammed GPS signals at the location where you're able to place an antenna. And in your plan you'll need one of these in every discontiguous network where you have equipment since you're not doing NTP over the Internet.

Not to mention the operations cost. Keeping track of a six inch brick with a wall wart and an antenna installed at a remote site is... not entirely abnormal but it's a one-off that consumes manpower.

And then you're only vulnerable to the litany of Internet attacks which don't involve NTP. Yay!

Don't get me wrong: the Time Machines TM1000A you recommended looks like a cool little device well worth checking into. As a supplement to Internet NTP, not a replacement.

Regards, Bill Herrin

-- William Herrin bill@herrin.us https://bill.herrin.us/

In a nutshell, no. Refer to my prior cites for detailed explanations. For a list of real-world attack incidents, see https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#<https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#:~:text=NTP%20server%20misuse%20and%20abuse%20covers%20a%20number%20of%20practices,the%20NTP%20rules%20of%20engagement.> -mel On Aug 6, 2023, at 12:03 PM, Royce Williams <royce@techsolvency.com> wrote:  Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering a reasonable mitigation for this, as designed? Royce On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: William, Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive. That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it.

On Sun, Aug 6, 2023 at 1:19 PM Royce Williams <royce@techsolvency.com> wrote:

Wouldn't a robust implementation of peering - say, seven peers, with the NTP algorithm handily selecting a subset to peer with for each cycle - require an attacker to know and overwhelm not just one, but a majority of the peer IPs?

Hi Royce, More or less, yes. There are some corner cases where that may not be true, but in terms of designing a useful attack, the attacker has to be able to figure out which NTP servers you're talking to, flood you with enough packets forged from all of them that the forged packet arrives ahead of the real one, generally no more than 10s of milliseconds behind the sent packet that only happens once every 15 minutes or so. And then it has to move you off real time gradually enough for the NTP daemon not to decide the peer has become a false ticker. It's like DNS cache poisoning but a few orders of magnitude harder.

This is *not* to say that I'm advocating against maintaining local stratum 0s as part of a proper operator-grade NTP mesh. (My definition of "robust implementation" would include both local stratum 0 and a healthy serving of Internet stratum 1s). I'm just suggesting "don't peer with public servers" seems draconian given the deliberate design choices of the protocol.

For this audience, this would recommend a tiered system where multiple mixed-stratum internal stratrum 0+1s would peer with each other, and maintain internal-facing consensus for all other downstream internal devices - such that the vast majority of internal peers would indeed not be talking to the public Internet (but the "parent" peers would, and have the mesh and mix that I describe).

Well, it's not really one-size-fits-all. Some systems consist of a well defined network containing routers and servers with clear boundaries between self and Internet. Your approach would work well there. Some systems consist of equipment scattered at various data centers, interconnected by the Internet itself. Implementing a GPS time receiver at every one could get very expensive very fast. Standard security rules apply: don't spend more protecting yourself than the risk-cost. Risk is threat times vulnerability. Given the complexity of the attack, you have to consider whether you're enough of a target (threat) for anyone to bother. Some systems are heavily virtualized, scattered over many vendors and locations. You could -try- to track down a "secured" NTP source from the vendor at each location, but that way lies configuration insanity. Some systems are air-gapped from the Internet. You're going to get time from GPS or the cellular phone network. GPS devices like the one Mel pointed out are probably cheaper and more accurate. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/

This entirely discounts the fact that bcp-38 and bcp-84 which, more or less, eliminate this "problem space" entirely. I find it hard to believe ntp reflection is actually a problem in the year 2023, assuming you're not running a ridiculously old ntp client and have taken really simple steps to protect your network. On Sun, Aug 6, 2023, 15:42 Mel Beckman <mel@beckman.org> wrote:

In a nutshell, no. Refer to my prior cites for detailed explanations. For a list of real-world attack incidents, see

https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse# <https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#:~:text=NTP%20server%20misuse%20and%20abuse%20covers%20a%20number%20of%20practices,the%20NTP%20rules%20of%20engagement.>

-mel

On Aug 6, 2023, at 12:03 PM, Royce Williams <royce@techsolvency.com> wrote:

 Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering a reasonable mitigation for this, as designed?

Royce

On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman <mel@beckman.org> wrote:

...

William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it.

A carefully selected set of stratum 0 sources for a set of stratum 1 servers is the heart of good NTP source design. With at least four “local” stratum 1 servers, Dr. Mills algorithm is excellent at distinguishing truechimers from falsetickers and providing a reliable source of monotonic time. DOS is a separate problem. My NTP network deployment experience for a major auto manufacturer, among others, is in agreement with William Herin. A GPS NTP source is a valid Stratum 0 source, but relying on a single instance for local time is not exceedingly better than querying time.apple.com <http://time.apple.com/> or a similar source. - James R. Cutler

William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These flaws make it trivial to spoof NTP packets, and many firewalls have no specific protection against this. in one attack the malefactor simply fires a continuous stream of NTP packets with invalid time at your firewall. When your NTP client queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex remediation. Why should people bother being exposed to the risk at all? Simply avoid Internet-routed NTP. there are many solutions, as I’ve already described. Having suffered through such attacks more than once, I can say from personal experience that you don’t want to risk it.

-mel

...

On Aug 6, 2023, at 10:53 AM, William Herrin <bill@herrin.us> wrote:

On Sat, Aug 5, 2023 at 7:24 PM Mel Beckman <mel@beckman.org> wrote:

...

That still leaves you open to NTP attacks. The USNO accuracy and monitoring is worthless if you suffer, for example, an NTP DDoS attack.

Hi Mel,

From what I can tell, a fairly simple firewall policy of allow UDP 123 from known NTP clients and established connections (I sent them a UDP packet recently) stops every one of those attacks (that's actually an NTP attack and not something else like a DNS attack) except for upstream address hijack that happens to coincide with your system boot. And it still depends on the attacker executing an additional sophisticated attack to do more than cause you a denial of service.

The links you sent are very interesting, at least in an academic sense, but they don't cause me to be unduly concerned about employing NTP.

...

if you can eliminate such security problems for $400, I say it’s cheap at twice the price.

Except you can't. Redundancy is required for any critical service. At the $400 price point, your approach has multiple single-points-of-failure. The device itself of course. Your ability to receive continuous non-jammed GPS signals at the location where you're able to place an antenna. And in your plan you'll need one of these in every discontiguous network where you have equipment since you're not doing NTP over the Internet.

Not to mention the operations cost. Keeping track of a six inch brick with a wall wart and an antenna installed at a remote site is... not entirely abnormal but it's a one-off that consumes manpower.

And then you're only vulnerable to the litany of Internet attacks which don't involve NTP. Yay!

Don't get me wrong: the Time Machines TM1000A you recommended looks like a cool little device well worth checking into. As a supplement to Internet NTP, not a replacement.

Regards, Bill Herrin

-- William Herrin bill@herrin.us https://bill.herrin.us/

"We use these exclusively in data centers" How well does GPS work inside the datacenter? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mel Beckman" <mel@beckman.org> To: "Mark Tinka" <mark@tinka.africa> Cc: nanog@nanog.org Sent: Saturday, August 5, 2023 2:26:37 PM Subject: Re: NTP Sync Issue Across Tata (Europe) Mark, You might consider setting up your own GPS-based NTP network. Commercial Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as little as $400. Or you can roll your own using a Raspberry Pi or similar nano computer with a GPS module and antenna. We use these exclusively in data centers now rather than depending on Internet NTP servers, primarily for security, because financial transactions in e-commerce can be sensitive to false time information. There are also a variety of NTP-based Internet attacks, so if you can block NTP at your border you’ve eliminated another attack surface. -mel via cell

On Aug 5, 2023, at 11:22 AM, Mark Tinka <mark@tinka.africa> wrote:

...

On 8/5/23 20:17, Chris Adams wrote:

It's the NTP pool people you need to talk to - the .freebsd. bit is just a vendored entry into the pool (more for load tracking and management).

Yes, Andreas clarified in unicast. Will do. Thanks.

Mark.

My facility experience ranges from prohibited to infeasible. I must not be in the right facilities. Yes, many radio platforms have GPS for timing. Some expose it for external time and timing purposes, some do not. Naturally, they do have a pretty good view of the sky. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mel Beckman" <mel@beckman.org> To: "Mike Hammett" <nanog@ics-il.net> Cc: nanog@nanog.org, "Mark Tinka" <mark@tinka.africa> Sent: Tuesday, August 8, 2023 10:05:55 AM Subject: Re: NTP Sync Issue Across Tata (Europe) It works fine, and is an industry standard. you have to mount the GPS antenna near a window with sky visibility, or on the roof. Many point-to-point microwave radios have GPS built in to obtain accurate timing for transmission multiplexing. -mel On Aug 8, 2023, at 7:16 AM, Mike Hammett <nanog@ics-il.net> wrote: <blockquote> "We use these exclusively in data centers" How well does GPS work inside the datacenter? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mel Beckman" <mel@beckman.org> To: "Mark Tinka" <mark@tinka.africa> Cc: nanog@nanog.org Sent: Saturday, August 5, 2023 2:26:37 PM Subject: Re: NTP Sync Issue Across Tata (Europe) Mark, You might consider setting up your own GPS-based NTP network. Commercial Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as little as $400. Or you can roll your own using a Raspberry Pi or similar nano computer with a GPS module and antenna. We use these exclusively in data centers now rather than depending on Internet NTP servers, primarily for security, because financial transactions in e-commerce can be sensitive to false time information. There are also a variety of NTP-based Internet attacks, so if you can block NTP at your border you’ve eliminated another attack surface. -mel via cell

On Aug 5, 2023, at 11:22 AM, Mark Tinka <mark@tinka.africa> wrote:

...

On 8/5/23 20:17, Chris Adams wrote:

It's the NTP pool people you need to talk to - the .freebsd. bit is just a vendored entry into the pool (more for load tracking and management).

Yes, Andreas clarified in unicast. Will do. Thanks.

Mark.

</blockquote>

Frontier COs is the easiest example.No antennas and no time service. Yes, they provide BITS, but BITS isn't time. I was also speaking specifically about installing GPS antennas in viable places, not using a facility-provided GPS or NTP service. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mel Beckman" <mel@beckman.org> To: "Mike Hammett" <nanog@ics-il.net> Cc: nanog@nanog.org, "Mark Tinka" <mark@tinka.africa> Sent: Tuesday, August 8, 2023 10:36:46 AM Subject: Re: NTP Sync Issue Across Tata (Europe) I’d be interested in an example of a Colo that does NOT provide GPS-based NTP even if they don’t let tenants install their own. I’ve never, ever seen one. -mel On Aug 8, 2023, at 8:20 AM, Mike Hammett <nanog@ics-il.net> wrote: <blockquote> My facility experience ranges from prohibited to infeasible. I must not be in the right facilities. Yes, many radio platforms have GPS for timing. Some expose it for external time and timing purposes, some do not. Naturally, they do have a pretty good view of the sky. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mel Beckman" <mel@beckman.org> To: "Mike Hammett" <nanog@ics-il.net> Cc: nanog@nanog.org, "Mark Tinka" <mark@tinka.africa> Sent: Tuesday, August 8, 2023 10:05:55 AM Subject: Re: NTP Sync Issue Across Tata (Europe) It works fine, and is an industry standard. you have to mount the GPS antenna near a window with sky visibility, or on the roof. Many point-to-point microwave radios have GPS built in to obtain accurate timing for transmission multiplexing. -mel <blockquote> On Aug 8, 2023, at 7:16 AM, Mike Hammett <nanog@ics-il.net> wrote: </blockquote> <blockquote> "We use these exclusively in data centers" How well does GPS work inside the datacenter? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mel Beckman" <mel@beckman.org> To: "Mark Tinka" <mark@tinka.africa> Cc: nanog@nanog.org Sent: Saturday, August 5, 2023 2:26:37 PM Subject: Re: NTP Sync Issue Across Tata (Europe) Mark, You might consider setting up your own GPS-based NTP network. Commercial Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as little as $400. Or you can roll your own using a Raspberry Pi or similar nano computer with a GPS module and antenna. We use these exclusively in data centers now rather than depending on Internet NTP servers, primarily for security, because financial transactions in e-commerce can be sensitive to false time information. There are also a variety of NTP-based Internet attacks, so if you can block NTP at your border you’ve eliminated another attack surface. -mel via cell

On Aug 5, 2023, at 11:22 AM, Mark Tinka <mark@tinka.africa> wrote:

...

On 8/5/23 20:17, Chris Adams wrote:

It's the NTP pool people you need to talk to - the .freebsd. bit is just a vendored entry into the pool (more for load tracking and management).

Yes, Andreas clarified in unicast. Will do. Thanks.

Mark.

</blockquote> </blockquote>

I was also speaking specifically about installing GPS antennas in viable places, not using a facility-provided GPS or NTP service.

Am I confused? Getting the time over a multi-gigabit Internet from a national time standard agency such as NIST (or your local country's equivalent) should produce far better accuracy and stability than relying on locally received GPS signals. GPS uses very weak radio signals which are regularly spoofed by all sorts of bad actors: https://www.gps.gov/spectrum/jamming/ for all sorts of reasons (like misleading drone navigation): https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incident Depending on satnav systems creates a large single point of failure for worldwide civilian infrastructure. Jamming GPS with subtly fake time data near big data centers seems like an easy move that would cause all sorts of distributed algorithms to start failing in unusual ways. And in a more serious wartime attack, many or most GPS satellites themselves would be destroyed or disabled. Yet digital radio modulations like FT8 or DMR rely on tight time synchronization among different transmitters. So do many modern cellphone modulations -- not to mention distributed database sync algorithms. Depending on any of these for emergency communications when their time comes from GPS, is a recipe for having no communications during wars or cyber-wars in which GPS satellites are attacked or jammed. See a longer explanation here: https://www.ardc.net/apply/grants/2020-grants/grant-ntpsec/ I suspect that even today, if you rely on civilian GPS time near the US White House, Pentagon, or other military targets like bases, you will discover "anomalies" in the local radio GPS data, compared to what you get from an authenticated time standard over NTP. How reliable is civilian GPS time in Ukraine these days? John

When GPS is working, time transmission with accuracies of under 1 microsecond is common. This is especially true if the GPS integrates some sort of disciplined oscillator. Note that this is in excess of what NTPd running on a typical OS can reliably retransmit. BUT.. if I was to choose only one protocol, it would be NTP, not GPS, because of all of the reasons you mention. I find it distressing that sites are relying on GPS only. I suspect that this a failure to assign proper risk to using GPS. It's particularly odd when one considers that adding NTP time sources are essentially free and improve robustness and reliability greatly. NTP is not without it's risks but the most common server implementation is specifically designed to be able to discard time sources which are not telling the truth, provided the server is given enough valid time sources. Even if a spoofed or misconfigured server is giving the wrong time, NTPd will be able to ignore those errant time sources. When configured with numerous network time sources and a GPS source, NTPd will determine what the correct time should be, and then will use the higher accuracy GPS source to improve the overall accuracy. This is more or less automatic since the latency to the GPS time source will be essentially zero when compared to a typical network source. However, if the GPS source starts lying about the time, NTPd will start ignoring it as a potential time source even with the lower latency. Without having non-GPS sources in your configuration, this essentially free protection against GPS spoofing is no longer available since it has nothing to compare it to. If your network is large enough that you could install multiple GPS receivers in diverse locations, then I'd configure all of the NTPd servers to pull from all of the GPS receivers. That way you gain additional redundancy. I'd still not drop the public trusted NTP servers though. On Tue, Aug 8, 2023, 2:58 PM John Gilmore <gnu@toad.com> wrote:

...

I was also speaking specifically about installing GPS antennas in viable places, not using a facility-provided GPS or NTP service.

Am I confused? Getting the time over a multi-gigabit Internet from a national time standard agency such as NIST (or your local country's equivalent) should produce far better accuracy and stability than relying on locally received GPS signals. GPS uses very weak radio signals which are regularly spoofed by all sorts of bad actors:

https://www.gps.gov/spectrum/jamming/

for all sorts of reasons (like misleading drone navigation):

https://en.wikipedia.org/wiki/Iran%E2%80%93U.S._RQ-170_incident

Depending on satnav systems creates a large single point of failure for worldwide civilian infrastructure.

Jamming GPS with subtly fake time data near big data centers seems like an easy move that would cause all sorts of distributed algorithms to start failing in unusual ways. And in a more serious wartime attack, many or most GPS satellites themselves would be destroyed or disabled. Yet digital radio modulations like FT8 or DMR rely on tight time synchronization among different transmitters. So do many modern cellphone modulations -- not to mention distributed database sync algorithms. Depending on any of these for emergency communications when their time comes from GPS, is a recipe for having no communications during wars or cyber-wars in which GPS satellites are attacked or jammed. See a longer explanation here:

https://www.ardc.net/apply/grants/2020-grants/grant-ntpsec/

I suspect that even today, if you rely on civilian GPS time near the US White House, Pentagon, or other military targets like bases, you will discover "anomalies" in the local radio GPS data, compared to what you get from an authenticated time standard over NTP. How reliable is civilian GPS time in Ukraine these days?

John

On 8/9/23 09:29, Seth Mattinen via NANOG wrote:

I liked having a WWVB receiver in my mix, but all the hardware appliances (at least those offering OCXO or Rubidium oscillator options) seem to have rejected it in favor of GPS only. I can only conclude that either vendors think options like WWVB are a dead end or there's no demand for GPS alternatives.

Both GPS and WWVB are over-the-air. There has been concern expressed of a bad actor spoofing or jamming GPS. Comparatively speaking, jamming or spoofing WWVB is a trivial joke. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

While GPS spoofing is technically possible, all the extant spoofing only tampers with the ephemeris (satellite position) data, not the timing stream. That's because hackers have been aiming at navigation, and may not have expressed interest in GPS tampering when NTP tampering is so easy 🙂 To spoof GPS clocks, a hacker has to know where the antennas are, and get above them in order to inject a signal with the right directionality. Commercial GPS clock vendors have implemented various anti-spoofing measures that, for example, only accept signals from a certain cone of visibility, which faces up. They have other measures too, some of which exploit geographic diversity, so if you can have two or more GPS clocks in different hub sites, the clocks will reject spoofing signals. This seems like a much easier defense than deploying secure NTP (NTS), which adds a huge amount of complexity. At least one researcher has shown that poluting the existing public NTP pool with enough bogus servers to seriously impact network time is trivial (I cited their paper in an earlier post on this thread). A well funded state actor could be laying the framework for such an attack as we speak, lying in wait until an opportunity to disrupt Internet NTP globally. -mel ________________________________ From: NANOG <nanog-bounces+mel=beckman.org@nanog.org> on behalf of Jay Hennigan <jay@west.net> Sent: Wednesday, August 9, 2023 10:58 AM To: nanog@nanog.org <nanog@nanog.org> Subject: Re: NTP Sync Issue Across Tata (Europe) On 8/9/23 09:29, Seth Mattinen via NANOG wrote:

I liked having a WWVB receiver in my mix, but all the hardware appliances (at least those offering OCXO or Rubidium oscillator options) seem to have rejected it in favor of GPS only. I can only conclude that either vendors think options like WWVB are a dead end or there's no demand for GPS alternatives.

Both GPS and WWVB are over-the-air. There has been concern expressed of a bad actor spoofing or jamming GPS. Comparatively speaking, jamming or spoofing WWVB is a trivial joke. -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

----- Original Message -----

From: "Forrest Christian (List Account)" <lists@packetflux.com>

Let me address your points: [ ... ] Let's assume you have a typical GPS-derived NTP server using a typical commercially available timing GNSS module. To convince that receiver that it was a different time, I'd need to have an SDR that would operate in the GPS band. These are widely available for under $500. You'd also need a laptop and a download of a GPS simulator from GitLab. With a total investment of $500 (assuming I already have a laptop), I now have a system that can generate a GPS signal to convince your GPS receiver that it's any time at all. If you're a tech neophyte, there are youtube videos on how to do this.

All I need to do now is add appropriate antennas and/or amplifiers to overcome the official GNSS signals. As you pointed out, depending on the location and directivity of your antenna, this is either trivial or becomes slightly more difficult. If I can see your antenna, it becomes a lot cheaper as I just need a relatively low-powered amplifier and a highly directional antenna. If I can't see your antenna, I would opt for a higher-power amplifier and a less directional transmit antenna to blanket a wide area with the spoofed signal.

If I'm trying to get time out of a NAVSTAR (yes, I know, shut up) receiver, it can see like 8-20 birds, right? Is there not some voting and such inside such a receiver? Just letting it see one 'bird' with spoofed time doesn't seem like it ought to work, to me; what don't I know? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

Gotcha. The Bad Guys are smarter than me. :-) Cheers, -- jra ----- Original Message -----

From: "Forrest Christian (List Account)" <lists@packetflux.com> To: "jra" <jra@baylink.com> Cc: "nanog list" <nanog@nanog.org> Sent: Sunday, August 13, 2023 8:06:30 PM Subject: Re: NTP Sync Issue Across Tata (Europe)

If I'm spoofing time, I'm going to produce an entire constellation of satellites. That is, I'm going to provide a signal which looks like all of the satellites in view providing their timing signals on whatever time I want your GPS receiver to think it is. All I have to do is ensure that your receiver receives my signal loud enough that it thinks the real satellites are noise, and my signal is the real one.

This isn't that hard to accomplish, especially since there are youtube videos showing you how.

On Sun, Aug 13, 2023 at 6:03 PM Jay R. Ashworth <jra@baylink.com> wrote:

...

----- Original Message -----

...

From: "Forrest Christian (List Account)" <lists@packetflux.com>

...

Let me address your points: [ ... ] Let's assume you have a typical GPS-derived NTP server using a typical commercially available timing GNSS module. To convince that receiver that it was a different time, I'd need to have an SDR that would operate in the GPS band. These are widely available for under $500. You'd also need a laptop and a download of a GPS simulator from GitLab. With a total investment of $500 (assuming I already have a laptop), I now have a system that can generate a GPS signal to convince your GPS receiver that it's any time at all. If you're a tech neophyte, there are youtube videos on how to do this.

All I need to do now is add appropriate antennas and/or amplifiers to overcome the official GNSS signals. As you pointed out, depending on the location and directivity of your antenna, this is either trivial or becomes slightly more difficult. If I can see your antenna, it becomes a lot cheaper as I just need a relatively low-powered amplifier and a highly directional antenna. If I can't see your antenna, I would opt for a higher-power amplifier and a less directional transmit antenna to blanket a wide area with the spoofed signal.

If I'm trying to get time out of a NAVSTAR (yes, I know, shut up) receiver, it can see like 8-20 birds, right? Is there not some voting and such inside such a receiver? Just letting it see one 'bird' with spoofed time doesn't seem like it ought to work, to me; what don't I know?

Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

-- - Forrest

-- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

On 8/9/23 3:25 PM, Forrest Christian (List Account) wrote:

Note that NIST operates a pool of 24 time servers for public use.  These are spread across four different locations in two different states.  My understanding is that they all get their time directly from the official NIST clocks without GPS or NTP being involved.

I used to jump through all the hoops for that but honestly I like the appliances better (they are also PTP grandmaster clocks). I can always disable the GPS inputs if any of the doom and gloom actually comes to pass. ~Seth

The recommendation tends to be the following: 1) Run your GPS-derived NTP appliances, but DO NOT point end-user clients at it. 2) Run a set of internal NTPd servers, and configure them to pull time from all of your GPS-derived NTP servers, AND trusted public NTP servers 3) Point your clients at the internal NTPd servers. Note that it's not entirely unreasonable to go out and buy numerous GPS appliances, deploy them at multiple locations, and point your NTPd servers at those. With enough sites, your NTPd server will skip over any defective NTP appliance. At some point, using publicly available NTP sources is redundant unless one wants to mitigate away the risks behind failure of the GPS system itself. What I'm advocating against is the seemingly common practice to go buy an off-the-shelf lower-cost GPS-NTP appliance (under $1K or so), stick an antenna in a window or maybe on the rooftop, and point all your devices at that device. This is asking for a failure for reasons I've covered previously. Robust time needs multiple time sources in order to mitigate against any one source being spoofed or jammed. The more time sources you incorporate into your solution, the less likely it is that any one of them going haywire would cause a timing failure. On Wed, Aug 9, 2023 at 6:12 PM Mel Beckman <mel@beckman.org> wrote:

Seth,

My point exactly. Use GPS as primary, and have anti-PS back up, and if you want automatic fail over, do that in an intermediate server on your site that makes a conscious test and decision to fail over to regular NTP

-mel via cell

...

On Aug 9, 2023, at 5:01 PM, Seth Mattinen via NANOG <nanog@nanog.org> wrote:

On 8/9/23 3:25 PM, Forrest Christian (List Account) wrote:

...

Note that NIST operates a pool of 24 time servers for public use. These are spread across four different locations in two different states. My understanding is that they all get their time directly from the official NIST clocks without GPS or NTP being involved.

I used to jump through all the hoops for that but honestly I like the appliances better (they are also PTP grandmaster clocks). I can always disable the GPS inputs if any of the doom and gloom actually comes to pass.

~Seth

-- - Forrest

The NIST time servers do NOT get their time from GPS. NIST, like several government standards agencies and other research labs around the globe, run an ensemble of high precision atomic clocks. In the case of NIST, they use the ensemble to produce the timescale UTC(NIST) at their Boulder, Colorado campus. In addition, they produce secondary copies of the timescale at Fort Collins and Maryland. The time transfer from Boulder to these locations use various techniques, but not traditional "get your time from GPS" methods. The closest they come is that the Maryland location uses GNSS common view time transfer where the phase difference between the UTC(NIST) realization at each site is measured against the signal from a single GPS satellite that both sites can see in the sky at the same time. The GPS signal is only used as a reference... think start button on a stopwatch. If both sides have the same delay from a specific feature in the GPS signal then you can be sure they are synchronized with each other. This is also just a measurement, and a scientist makes the decision about whether the timescale needs to be adjusted to be kept within specs. These are physical realizations of UTC... that is, a phase-aligned 1PPS pulse and a high precision clock signal. These realizations are used to directly drive the NIST NTP servers at each location. GPS is not involved. Note that this realization of UTC is different than what you get from GPS. GPS gets its time from the naval observatory which has it's own ensemble of clocks which produce UTC(USNO). These two timescales are within a few ns of each other, also verified with GNSS common view technology, so one can consider them the same for most purposes. Note that a similar process is used to derive UTC(NICT) in Japan. Pointing a NTP server at ntp.nict.jp would result in receiving time that was produced independently by NICT, and was not transmitted by GPS. There are various simplifications I've made to the above description so there are a few places the description leaves out intermediate steps. As far as a rubidium clock goes, I'd much rather see it disciplined regularly to a GPS time source, but that comes from the fact that I like my 1PPS to be within a microsecond or so of UTC due to the precision I need in the lab. If I let either of my lab rubidiums free run for more than a few days, I'm typically off UTC by more than that amount. But that level of accuracy isn't typically needed for computer time so I will concede that you could get away with freerunning for an extended period. Not hundreds of years on a rubidium. But a while. Assuming the original calibration was done correctly. Note that some of the high end appliances I'm referring to just use GPS over days and weeks to discipline a precision oscillator (sometimes rubidium) which is essentially an automatic calibrating version of what you're proposing. On Fri, Aug 11, 2023, 3:33 AM Masataka Ohta < mohta@necom830.hpcl.titech.ac.jp> wrote:

Forrest Christian (List Account) wrote:

...

The recommendation tends to be the following:

1) Run your GPS-derived NTP appliances, but DO NOT point end-user clients at it. 2) Run a set of internal NTPd servers, and configure them to pull time from all of your GPS-derived NTP servers, AND trusted public NTP servers 3) Point your clients at the internal NTPd servers.

That is not a very good recommendation. See below.

...

At some point, using publicly available NTP sources is redundant unless one wants to mitigate away the risks behind failure of the GPS system itself.

Your assumption that public NTP servers were not GPS-derived NTP servers is just wrong.

...

What I'm advocating against is the seemingly common practice to go buy an off-the-shelf lower-cost GPS-NTP appliance (under $1K or so), stick an antenna in a window or maybe on the rooftop, and point all your devices at that device.

Relying on a local expensive GPS appliance does not improve security so much and is the worst thing to do.

But, additionally relying on remote servers (including those provided by NIST) is subject to DOS attacks.

As such, the ultimate (a little expensive) solution is to have your own Rb clocks locally.

Masataka Ohta

Forrest Christian (List Account) <lists@packetflux.com> wrote:

...

...

At some point, using publicly available NTP sources is redundant unless one wants to mitigate away the risks behind failure of the GPS system itself.

On Fri, Aug 11, 2023, 3:33 AM Masataka Ohta wrote:

...

Your assumption that public NTP servers were not GPS-derived NTP servers is just wrong.

Subsequent conversation has shown that you are both right here. Yes, many public NTP servers ARE using GPS-derived time. Yes, some public NTP servers ARE NOT using GPS-derived time. Up to this point, popular public NTP pools have not made these distinctions readily configurable, though. For sites that need to run even during a war, or a similar situation that is likely to disrupt or distort GPS, they might like to have access to NTP servers that are completely independent from GPS. At one point I proposed that some big NTP server pools be segregated by names, to distinguish between GPS-derived time and national-standard derived time. For example, two domain names could be e.g.: fromnist.pool.tick.tock fromgps.pool.tick.tock If you wanted particular redundancy, say because you have a local GPS clock and you want a non-GPS check on that, you'd use fromnist.pool.tick.tock (or fromnict.pool.tick.tock for the Japanese timebase, etc). (If you were agnostic about where your times comes from, you would just use a generic domain name like vendorname.pool.tick.tock.) An automated tool could periodically trace that the stratum 0 source currently being used by each node in each of the pools is actually the same as advertised in its domain name. Alerting any difference to the relevant system administrators would allow those clocks to continue running with a backup timebase, while making it more likely that some human would work to restore their access to the correct stratum 0 source. So far this is just an idea. John PS: When we say "GPS", do we really mean any GNSS (global navigation satellite system)? There are now four such systems that have global coverage, plus regionals. While they attempt to coordinate their time-bases and reference-frames, they are using different hardware and systems, and are under different administration, so there are some differences in the clock values returned by each GNSS. These differences and discontinuties have ranged up to 100ns in normal operation, and higher amounts in the past. See: Nicolini, Luca; Caporali, Alessandro (9 January 2018). "Investigation on Reference Frames and Time Systems in Multi-GNSS". Remote Sensing. 10 (2): 80. doi:10.3390/rs10010080. https://www.mdpi.com/2072-4292/10/1/80

I've responded in bits and pieces to this thread and haven't done an excellent job expressing my overall opinion. This is probably because my initial goal was to point out that GPS-transmitted time is no less subject to being attacked than your garden variety NTP-transmitted time. Since this thread has evolved, I'd like to describe my overall position to be a bit clearer. To start, we need a somewhat simplified version of how UTC is created so I can refer to it later: Across the globe, approximately 85 research and standards institutions run a set of freestanding atomic clocks that contribute to UTC. The number of atomic clocks across all these institutions totals around 450. Each institution also produces a version of UTC based on its own set of atomic clocks. In the international timekeeping world, this is designated as UTC(Laboratory), where Laboratory is replaced with the abbreviation for the lab producing that version of UTC. So UTC(NIST) is the version that NIST produces at Boulder, Colorado, NICT produces UTC(NICT) in Tokyo, and so on. Because no clock is perfectly accurate, all of these versions of UTC drift in relation to each other, and you could have significant differences in time between different labs. As a result, there has to be a way to synchronize them. Each month, the standards organization BIPM collects relative time measurements and other statistics from each institution described above. This data is then used to determine the actual value of UTC. BIPM then produces a report detailing each organization's difference from the correct representation of UTC. Each institution uses this data to adjust its UTC representation, and the cycle repeats the next month. In this way, all of the representations of UTC end up being pretty close to each other. The document BIPM produces is titled "Circular T." The most recent version indicates that most of the significant standards institutions maintain a UTC version that differs by less than 10ns from the official version of UTC. Note that 10ns is far more accurate than we need for NTP, so most of the UTC representations can be considered identical as far as this discussion goes. Still, it is essential to realize that UTC(NIST) is generated separately from UTC(USNO) or other UTC implementations. For example, a UTC(NIST) failure should not cause UTC(USNO) to fail as they utilize separate hardware and systems. Each of these versions of UTC is also disseminated in various ways. UTC(NIST) goes out via the "WWV" radio stations, NTP, and other esoteric methods. GPS primarily distributes UTC(USNO), which is also available directly via NTP. UTC(SU) is the timescale for GLONASS. And so on. So, back to NTP and the accuracy required: Most end users (people running everyday web applications or streaming video or similar) don't need precisely synchronized time. The most sensitive application I'm aware of in this space is likely TOTP, which often needs time on the server and time on the client (or hardware key) within 90 seconds of each other. In addition, having NTP time fail usually isn't the end of the world for these users. The best way to synchronize their computers (including desktop and server systems) to UTC is to point their computer time synchronization service (whatever that is) at pool.ntp.org, time.windows.com, their ISP's time server, or similar. Or, with modern OS'es, you can leave the time configured to whatever server the OS manufacturer preconfigured. As an aside, one should note that historically windows ticked at 15ms or so, so trying to synchronize most windows closer than 15ms was futile. On the other hand, large ISPs or other service providers (including content providers) see real benefits to having systems synchronized to fractions of seconds of UTC. Comparing logs and traces becomes much easier when you know that something logged at 10:02:23.1 on one device came before something logged at 10:02:23.5 on another. Various server-to-server protocols and software implementations need time to be synchronized to sub-second intervals since they rely on timestamps to determine the latest copy of data, and so on. In addition, as an ISP, you'll often provide time services to downstream customers who demand more accuracy and reliability than is strictly necessary. As a result, one wants to ensure that all time servers are synchronized within some reasonable standard of accuracy. Within 100ms is acceptable for most applications but a goal of under 50ms is better. If you have local GPS receivers, times down to around 1ms is achievable with careful design. Beyond that, you're chasing unnecessary accuracy. Note that loss of precision is somewhat cumulative here - running a time server synchronized to within 100ms will ensure that no client can be synchronized to better than within 100ms from that server. Generally, you'll want your time server to be synchronized much better than needed to avoid the time server being the limiting factor. In a perfect world with no bad actors and where all links ran perfectly, one could set up an NTP server that pulled from pool.ntp.org or used GPS and essentially acted as a proxy. Unfortunately, we don't live in this world. So one has to ask how you build a system that meets at least the following goals: * Synchronized to UTC within 50ms, with lower being better. * Not subject to a reasonable set of attacks (typical DoS attacks, RF signal attacks, spoofing, etc). * Able to be run by typical network operations staff In addition, an ideal server setup would be made up of redundant servers in case one piece of hardware fails. I will ignore this part, as it's usually just setting up multiple copies of the same thing. The two most straightforward options are using a GPS-based NTP appliance or installing an NTP server and pointing it at pool.ntp.org. Under normal circumstances, both options will be synchronized to UTC with enough accuracy for most applications, and both are easy to run by typical network operations staff. This assumes reasonably consistent network latency in the NTP case and a good sky view in the GPS case. The GPS-based appliance is, however, subject to spoofing or jamming, as I've discussed earlier. The NTP server is at the mercy of the quality of the servers it picked from pool.ntp.org and is also subject to various outside attacks (spoofing, etc.). One must decide how critical time is to them before deciding whether this option is valid. The other end of the scale is the "develop your own offline version of UTC using atomic clocks" methodology. This fixes the attack issue but introduces several others. The main one is that you are now relying on the clock's accuracy. Admittedly rubidium and especially cesium clocks tend to be sufficiently reliable and stable. However, one has to ensure the frequency is accurate initially and stays that way. You must also wire the clock to an NTP Server and calibrate the initial UTC offset. If the clock goes haywire or is less accurate than is required, your in-house version of UTC will drift in relation to real UTC. This means you may need 2 or 3 or more atomic clocks to be sufficiently reliable. You'll then need to regularly take an average, compare it to UTC, and adjust if it's drifted too much. This quickly becomes more of a science project than something you want network operations staff to deal with on an ongoing basis. To be clear: If you need robust time not subject to outside forces and have or can obtain the skill set to pull this off internally, I won't argue that this is a bad option. However, I feel this isn't the type of service most providers want to run internally. So, looking at some middle-ground options that trade a bit of robustness for ease of use is reasonable. My lowest cost preference has always been to use a set of in-house NTP servers pointed at a carefully curated collection of NTP servers. Your curation strategy should depend on network connectivity, the reliability of the time sources, etc. In North America, picking one or two NIST servers from each NIST location is a good starting point. That is one or two from each of Maryland, Fort Collins, Boulder, and the University of Colorado. One may want to add some servers from other timekeeping organizations (such as USNO). Note that there is one commonality: These time servers are run by organizations listed in circular T as contributing to UTC, and the servers are tied to the atomic clocks. That way, we ensure that the servers are not subject to inaccuracies caused by time transfer from an authoritative source for UTC. What is left is any potential attack on the time transfer over NTP itself. I would argue that with a curated list of enough NTP servers, this risk can be pushed down to where it is low enough for many use cases. A lot will depend on the quantity and quality of NTP servers you select and the robustness of the network path to those servers. If the packets between your NTP server and the NTP servers you choose traverse a relatively secure and short path with plenty of bandwidth, and the paths to differing NTP servers are diverse, many attacks will become harder to implement. In addition, the more NTP servers you add, the more likely it is that NTP will be able to correctly pick the servers providing the correct time, even if an attacker is successfully spoofing one or more sources. In some cases it may make sense to add additional servers which are run by third parties if it gains additional robustness based on network architecture. This is especially true if you're closely connected network-wise with the third party and they run a good quality NTP service as well. As I've mentioned, a good middle-of-the-road solution is adding various sources of time derived via GPS. Note I said, "to add." Start with the carefully curated NTP server set, then install one or more GPS-based NTP Servers polled by your NTP server. Adding these GPS time sources to your NTP servers does three things: First, it provides another source of time NTP can use to determine the correct time. Second, we're now using a different time transmission method with different vulnerabilities. And finally, it will significantly improve the accuracy of the time the NTP server produces as NTPd will generally prefer it to do the final trimming to UTC. The strength of the combination of both terrestrial transmitted time via NTP and the precision of rf-transmitted GPS time ensures that time is both correct and precise. There are still attack vectors here, but as you add more time sources, the complexity of pulling off a successful attack increases. This is especially true if you can monitor the NTP server for signs of stress, such as time servers that are not telling the correct time or GPS signals which are inconsistent with the NTP-derived time. A successful attack would require simultaneous NTP (network) and GPS (rf) attacks. Other options or blends of options are also possible. With a reasonably large network, putting enough GPS receivers into place would significantly reduce the possibility of a spoofer or jammer taking out your entire GPS infrastructure. Reducing or eliminating external NTP time sources might be reasonable in that case. The theory is that attacking GPS receivers at one location is easy. Doing it at dozens simultaneously is much more difficult. To use an exaggeration to make a point: If you had 100 different GPS receivers spread across 100 widely geographically diverse locations, and all of your NTP servers were able to poll all of them for time, the chances that an attacker would be able to take out or spoof enough GPS receivers to make a difference would be close to zero. Your failure point becomes UTC(USNO) and the GPS constellation itself. The same argument would apply to NTP servers regarding quantity and diversity. Other options involve adding additional technologies. For example, some appliances use GPS to discipline (adjust) an internal atomic clock. Once the atomic clock is locked to UTC, the GPS can fail for extended periods without affecting NTP output. In addition, some of these will filter updates from the GPS based on the appliance's internal atomic time. That way, a spoofer would be ignored, jammers would have to continue for hours or days, and so on. Of course, these solutions' reliability depends on the implementation quality. If I had the budget to implement something like this in a network, I'd likely scatter a few of these around the network and then still use garden variety NTPd servers which would be pointed at these appliances. I might even consider buying solutions from multiple vendors to ensure a bug in one implementation was filtered out and ignored. I can't cover every option here, but balancing security, cost, operational complexity, and application needs is the key. Some solutions are cheap and easy but not robust. Some are highly robust but expensive and not easy. Somewhere in the middle is probably where most real implementations should lie. Now, to address a couple of specific items: 1) Additional GPS and commercial time distribution systems will likely improve reliability. However, only GPS and GALILEO are available for free in the US. I'm ignoring GLONASS for various legal and political reasons. GALILEO is a valid option but it lives in the same band as GPS, so jamming GPS will usually also jam GALILEO. Utilizing GNSS receivers that use the civilian signals in the newer bands would also help. Some commercial solutions are available that don't require GNSS, but they're relatively new and not as commonly available as one would like. 2) For running my own time servers in a service-provider environment, I'd rather specifically designate the exact NTP server I want to utilize and not rely on a third party to give me a pool of servers. It's more about ensuring the server I use is running a trusted server, and if I delegate the server selection, I lose this ability. On the other hand, where I'm not running a NTP server that is critical for many clients, I'll just point it at pool.ntp.org, or north-america.pool.ntp.org and skip all of the recommendations that I've made above. I would be cautious about requesting pool.ntp.org add entries for "stratum of server" or "origin of time" as this seems like it would tend to overload the stratum one servers in the pool with people "optimizing" their configuration to use only stratum one servers. Remember that pool.ntp.org is generally intended as an end-user-device service, and providing methods that end users can bypass the robustness that a fully distributed pool will provide is probably not a great idea. 3) This all should hopefully sort itself out over the next few years. GPS and GALILEO are flying new birds that have changes designed to improve attack resilience by using cryptography to ensure authentic transmissions (which may rely on ground transmission of cryptographic keys). NTP already supports manual cryptographic keys that work, but NTS is a pain in the rear. Hopefully, NTPv5 will have a better security mechanism. Other, more secure, time sources are on the horizon as the cybersecurity crowd is aware of the issues. And finally, as a sort of a tl;dr; Summary: Each operator needs to decide how critical time is to their network and pick a solution that works for them and fits the organization's budget. Some operators might point everything at pool.ntp.org and not run their own servers. Others might run their own time lab and use that time to provide NTP time and precision time and frequency via various methods. Most will be somewhere in between. But regardless of which you choose, please be aware that GPS isn't 100% secure, and neither is NTP. If attack resilience matters to you, you should think about all of the attack vectors and design something that is robust enough to meet your use case.

We're going to have to somewhat disagree here... I may not have been 100% clear about what I see as the most common risks for GPS. The reason I suggest that NTP risks and GPS risks are similar is not primarily due to intentional time injection hacks (although that is a risk). Instead, it's related to GPS failure modes, and the increased commonality of GPS jamming causing those failures. I will 100% concede that NTP carries far more spoofing or intentional DoS risks. But GPS is far more likely to suffer a failure in the absence of a bad actor than NTP. The reason for this is that the GPS signal is incredibly weak, and it's incredibly easy to break GPS reception. Good antenna placement and antennas that try to reject terrestrial signals help but don't always prevent the failures from happening. Because GPS is used more and more to track objects and people, people who don't want to be tracked are starting to buy and use jammers. In addition, it's becoming increasingly common for gamers to spoof their GPS location (and, as a result, time) via GPS injection. So the kid down the street trying to cheat at pokemon go or the truck driver not wanting to get in trouble for speeding may unintentionally cause your time server to quit working correctly. Not to mention the random piece of electronic gear which malfunctions and spews noise across the GPS band. So, yes, I will 100% agree with you that NTP carries more intentional hacking risk. But I'm going to argue that GPS carries a significantly higher risk of a jamming-related failure. Without good statistics, it's hard to tell which is more prevalent. I see a lot more GPS failures from my viewpoint, but I also have to talk to customers who are having precision timing issues due to GPS failures. My intuitive feeling is that in the absence of bad actors, NTP is significantly more reliable than GPS. In the presence of remote bad actors, I'll grant that NTP is 100% the loser here. When everything is working, GPS will provide better time. Adding a holdover oscillator to GPS does help in marginal situations, but doesn't resolve all of the GPS issues. In those situations where time is not critical, either NTP or GPS is a good solution, and it largely comes down to which you prefer. I deal with way too many antennas so I'd rather just harden a NTP server. You might deal with way too many hackers getting in your systems so you might prefer relying on a GPS antenna. Either way, most of the time you're going to get decent time service. We could go into a lot of details about how each system can fail, but for non-time critical applications I'm not sure either would come out a clear winner. I know you believe GPS does, and I believe that it isn't 100% clear which one is better for those "just want time that works most of the time" applications. We could argue all day about this and we won't get anywhere beyond us disagreeing about this. Once you get to more time-critical apps where actual budget is going to be expended on ensuring reliable NTP services are available 24x7, then neither a default configuration NTP server nor a single GPS receiver will provide reliable time. Selecting servers and hardening firewalls to limit the likelihood of time injection can work wonders on NTP robustness. GPS works too if you provide enough GPS timing sources that multiple locations would have to be jammed at once. Providing a mix of these is even better. If I was to go GPS-only I'd probably try to ensure a minimum of 3 different GPS receivers at 3 different locations, with internal NTP servers pulling from each of the GPS-connected NTP servers. 5 would even be better. An even more robust option would be to go with 5 GPS receivers and 2 or 3 NTP-connected stratum 1 time sources. In this last case, you could spoof ALL of the NTP servers and the GPS would still be in control. You could also have signal failures at 3 of the GPS sites and the NTP connections would provide redundant time sources. Only with GPS failures at multiple sites AND NTP failures or spoofing happening at the same time would one have an issue where the NTP servers could possibly fail to receive correct time. On Mon, Aug 14, 2023 at 2:00 AM Mel Beckman <mel@beckman.org> wrote:

Forrest,

I think you’re gilding the lilly. My original recommendation was to use GPS as primary, for its superior accuracy and resistance to attack, and have anti-GPS back up. If you want automatic fail over, do that in an intermediate server on your site that makes a conscious test and decision to fail over to Internet NTP.

You’re mistaken to say that the vulnerability of GPS is remotely comparable to the vulnerability of Internet-based NTP. To interfere with a GPS-derived clock, an attacker has to physically be present. That’s a huge expense — and risk — that hackers are really not interested in undertaking. They would much rather sit in Russia or China and attack NTP servers remotely using any of the several attack methodologies I’ve cited previously.

So curate Internet NTP or not (personally, that seems like just another thing to monitor and maintain), but make GPS your primary time standard. You’re much better off staying air-gapped from Internet NTP until you detect a GPS failure. All the other machinations are pointless while GPS is working, because GPS gives you by far the best accuracy and security for the buck. Like I said, spend $400 on a commercial GPS time server and timing problems are solved. Or use facility-provided GPS if you can’t get an antenna up.

-mel

On Aug 14, 2023, at 12:10 AM, Forrest Christian (List Account) < lists@packetflux.com> wrote:

 I've responded in bits and pieces to this thread and haven't done an excellent job expressing my overall opinion. This is probably because my initial goal was to point out that GPS-transmitted time is no less subject to being attacked than your garden variety NTP-transmitted time. Since this thread has evolved, I'd like to describe my overall position to be a bit clearer.

To start, we need a somewhat simplified version of how UTC is created so I can refer to it later:

Across the globe, approximately 85 research and standards institutions run a set of freestanding atomic clocks that contribute to UTC. The number of atomic clocks across all these institutions totals around 450. Each institution also produces a version of UTC based on its own set of atomic clocks. In the international timekeeping world, this is designated as UTC(Laboratory), where Laboratory is replaced with the abbreviation for the lab producing that version of UTC. So UTC(NIST) is the version that NIST produces at Boulder, Colorado, NICT produces UTC(NICT) in Tokyo, and so on.

Because no clock is perfectly accurate, all of these versions of UTC drift in relation to each other, and you could have significant differences in time between different labs. As a result, there has to be a way to synchronize them. Each month, the standards organization BIPM collects relative time measurements and other statistics from each institution described above. This data is then used to determine the actual value of UTC. BIPM then produces a report detailing each organization's difference from the correct representation of UTC. Each institution uses this data to adjust its UTC representation, and the cycle repeats the next month. In this way, all of the representations of UTC end up being pretty close to each other. The document BIPM produces is titled "Circular T." The most recent version indicates that most of the significant standards institutions maintain a UTC version that differs by less than 10ns from the official version of UTC.

Note that 10ns is far more accurate than we need for NTP, so most of the UTC representations can be considered identical as far as this discussion goes. Still, it is essential to realize that UTC(NIST) is generated separately from UTC(USNO) or other UTC implementations. For example, a UTC(NIST) failure should not cause UTC(USNO) to fail as they utilize separate hardware and systems.

Each of these versions of UTC is also disseminated in various ways. UTC(NIST) goes out via the "WWV" radio stations, NTP, and other esoteric methods. GPS primarily distributes UTC(USNO), which is also available directly via NTP. UTC(SU) is the timescale for GLONASS. And so on.

So, back to NTP and the accuracy required:

Most end users (people running everyday web applications or streaming video or similar) don't need precisely synchronized time. The most sensitive application I'm aware of in this space is likely TOTP, which often needs time on the server and time on the client (or hardware key) within 90 seconds of each other. In addition, having NTP time fail usually isn't the end of the world for these users. The best way to synchronize their computers (including desktop and server systems) to UTC is to point their computer time synchronization service (whatever that is) at pool.ntp.org, time.windows.com, their ISP's time server, or similar. Or, with modern OS'es, you can leave the time configured to whatever server the OS manufacturer preconfigured. As an aside, one should note that historically windows ticked at 15ms or so, so trying to synchronize most windows closer than 15ms was futile.

On the other hand, large ISPs or other service providers (including content providers) see real benefits to having systems synchronized to fractions of seconds of UTC. Comparing logs and traces becomes much easier when you know that something logged at 10:02:23.1 on one device came before something logged at 10:02:23.5 on another. Various server-to-server protocols and software implementations need time to be synchronized to sub-second intervals since they rely on timestamps to determine the latest copy of data, and so on. In addition, as an ISP, you'll often provide time services to downstream customers who demand more accuracy and reliability than is strictly necessary.

As a result, one wants to ensure that all time servers are synchronized within some reasonable standard of accuracy. Within 100ms is acceptable for most applications but a goal of under 50ms is better. If you have local GPS receivers, times down to around 1ms is achievable with careful design. Beyond that, you're chasing unnecessary accuracy. Note that loss of precision is somewhat cumulative here - running a time server synchronized to within 100ms will ensure that no client can be synchronized to better than within 100ms from that server. Generally, you'll want your time server to be synchronized much better than needed to avoid the time server being the limiting factor.

In a perfect world with no bad actors and where all links ran perfectly, one could set up an NTP server that pulled from pool.ntp.org or used GPS and essentially acted as a proxy. Unfortunately, we don't live in this world. So one has to ask how you build a system that meets at least the following goals:

* Synchronized to UTC within 50ms, with lower being better. * Not subject to a reasonable set of attacks (typical DoS attacks, RF signal attacks, spoofing, etc). * Able to be run by typical network operations staff

In addition, an ideal server setup would be made up of redundant servers in case one piece of hardware fails. I will ignore this part, as it's usually just setting up multiple copies of the same thing.

The two most straightforward options are using a GPS-based NTP appliance or installing an NTP server and pointing it at pool.ntp.org. Under normal circumstances, both options will be synchronized to UTC with enough accuracy for most applications, and both are easy to run by typical network operations staff. This assumes reasonably consistent network latency in the NTP case and a good sky view in the GPS case. The GPS-based appliance is, however, subject to spoofing or jamming, as I've discussed earlier. The NTP server is at the mercy of the quality of the servers it picked from pool.ntp.org and is also subject to various outside attacks (spoofing, etc.). One must decide how critical time is to them before deciding whether this option is valid.

The other end of the scale is the "develop your own offline version of UTC using atomic clocks" methodology. This fixes the attack issue but introduces several others. The main one is that you are now relying on the clock's accuracy. Admittedly rubidium and especially cesium clocks tend to be sufficiently reliable and stable. However, one has to ensure the frequency is accurate initially and stays that way. You must also wire the clock to an NTP Server and calibrate the initial UTC offset. If the clock goes haywire or is less accurate than is required, your in-house version of UTC will drift in relation to real UTC. This means you may need 2 or 3 or more atomic clocks to be sufficiently reliable. You'll then need to regularly take an average, compare it to UTC, and adjust if it's drifted too much. This quickly becomes more of a science project than something you want network operations staff to deal with on an ongoing basis. To be clear: If you need robust time not subject to outside forces and have or can obtain the skill set to pull this off internally, I won't argue that this is a bad option. However, I feel this isn't the type of service most providers want to run internally.

So, looking at some middle-ground options that trade a bit of robustness for ease of use is reasonable.

My lowest cost preference has always been to use a set of in-house NTP servers pointed at a carefully curated collection of NTP servers. Your curation strategy should depend on network connectivity, the reliability of the time sources, etc. In North America, picking one or two NIST servers from each NIST location is a good starting point. That is one or two from each of Maryland, Fort Collins, Boulder, and the University of Colorado. One may want to add some servers from other timekeeping organizations (such as USNO). Note that there is one commonality: These time servers are run by organizations listed in circular T as contributing to UTC, and the servers are tied to the atomic clocks. That way, we ensure that the servers are not subject to inaccuracies caused by time transfer from an authoritative source for UTC. What is left is any potential attack on the time transfer over NTP itself. I would argue that with a curated list of enough NTP servers, this risk can be pushed down to where it is low enough for many use cases. A lot will depend on the quantity and quality of NTP servers you select and the robustness of the network path to those servers. If the packets between your NTP server and the NTP servers you choose traverse a relatively secure and short path with plenty of bandwidth, and the paths to differing NTP servers are diverse, many attacks will become harder to implement. In addition, the more NTP servers you add, the more likely it is that NTP will be able to correctly pick the servers providing the correct time, even if an attacker is successfully spoofing one or more sources. In some cases it may make sense to add additional servers which are run by third parties if it gains additional robustness based on network architecture. This is especially true if you're closely connected network-wise with the third party and they run a good quality NTP service as well.

As I've mentioned, a good middle-of-the-road solution is adding various sources of time derived via GPS. Note I said, "to add." Start with the carefully curated NTP server set, then install one or more GPS-based NTP Servers polled by your NTP server. Adding these GPS time sources to your NTP servers does three things: First, it provides another source of time NTP can use to determine the correct time. Second, we're now using a different time transmission method with different vulnerabilities. And finally, it will significantly improve the accuracy of the time the NTP server produces as NTPd will generally prefer it to do the final trimming to UTC. The strength of the combination of both terrestrial transmitted time via NTP and the precision of rf-transmitted GPS time ensures that time is both correct and precise. There are still attack vectors here, but as you add more time sources, the complexity of pulling off a successful attack increases. This is especially true if you can monitor the NTP server for signs of stress, such as time servers that are not telling the correct time or GPS signals which are inconsistent with the NTP-derived time. A successful attack would require simultaneous NTP (network) and GPS (rf) attacks.

Other options or blends of options are also possible. With a reasonably large network, putting enough GPS receivers into place would significantly reduce the possibility of a spoofer or jammer taking out your entire GPS infrastructure. Reducing or eliminating external NTP time sources might be reasonable in that case. The theory is that attacking GPS receivers at one location is easy. Doing it at dozens simultaneously is much more difficult. To use an exaggeration to make a point: If you had 100 different GPS receivers spread across 100 widely geographically diverse locations, and all of your NTP servers were able to poll all of them for time, the chances that an attacker would be able to take out or spoof enough GPS receivers to make a difference would be close to zero. Your failure point becomes UTC(USNO) and the GPS constellation itself. The same argument would apply to NTP servers regarding quantity and diversity.

Other options involve adding additional technologies. For example, some appliances use GPS to discipline (adjust) an internal atomic clock. Once the atomic clock is locked to UTC, the GPS can fail for extended periods without affecting NTP output. In addition, some of these will filter updates from the GPS based on the appliance's internal atomic time. That way, a spoofer would be ignored, jammers would have to continue for hours or days, and so on. Of course, these solutions' reliability depends on the implementation quality. If I had the budget to implement something like this in a network, I'd likely scatter a few of these around the network and then still use garden variety NTPd servers which would be pointed at these appliances. I might even consider buying solutions from multiple vendors to ensure a bug in one implementation was filtered out and ignored.

I can't cover every option here, but balancing security, cost, operational complexity, and application needs is the key. Some solutions are cheap and easy but not robust. Some are highly robust but expensive and not easy. Somewhere in the middle is probably where most real implementations should lie.

Now, to address a couple of specific items:

1) Additional GPS and commercial time distribution systems will likely improve reliability. However, only GPS and GALILEO are available for free in the US. I'm ignoring GLONASS for various legal and political reasons. GALILEO is a valid option but it lives in the same band as GPS, so jamming GPS will usually also jam GALILEO. Utilizing GNSS receivers that use the civilian signals in the newer bands would also help. Some commercial solutions are available that don't require GNSS, but they're relatively new and not as commonly available as one would like.

2) For running my own time servers in a service-provider environment, I'd rather specifically designate the exact NTP server I want to utilize and not rely on a third party to give me a pool of servers. It's more about ensuring the server I use is running a trusted server, and if I delegate the server selection, I lose this ability. On the other hand, where I'm not running a NTP server that is critical for many clients, I'll just point it at pool.ntp.org, or north-america.pool.ntp.org and skip all of the recommendations that I've made above. I would be cautious about requesting pool.ntp.org add entries for "stratum of server" or "origin of time" as this seems like it would tend to overload the stratum one servers in the pool with people "optimizing" their configuration to use only stratum one servers. Remember that pool.ntp.org is generally intended as an end-user-device service, and providing methods that end users can bypass the robustness that a fully distributed pool will provide is probably not a great idea.

3) This all should hopefully sort itself out over the next few years. GPS and GALILEO are flying new birds that have changes designed to improve attack resilience by using cryptography to ensure authentic transmissions (which may rely on ground transmission of cryptographic keys). NTP already supports manual cryptographic keys that work, but NTS is a pain in the rear. Hopefully, NTPv5 will have a better security mechanism. Other, more secure, time sources are on the horizon as the cybersecurity crowd is aware of the issues.

And finally, as a sort of a tl;dr; Summary: Each operator needs to decide how critical time is to their network and pick a solution that works for them and fits the organization's budget. Some operators might point everything at pool.ntp.org and not run their own servers. Others might run their own time lab and use that time to provide NTP time and precision time and frequency via various methods. Most will be somewhere in between. But regardless of which you choose, please be aware that GPS isn't 100% secure, and neither is NTP. If attack resilience matters to you, you should think about all of the attack vectors and design something that is robust enough to meet your use case.

-- - Forrest

On Aug 14, 2023, at 3:07 AM, Forrest Christian (List Account) <lists@packetflux.com> wrote:

I've responded in bits and pieces to this thread and haven't done an excellent job expressing my overall opinion. This is probably because my initial goal was to point out that GPS-transmitted time is no less subject to being attacked than your garden variety NTP-transmitted time. Since this thread has evolved, I'd like to describe my overall position to be a bit clearer.

<SNIP/>

And finally, as a sort of a tl;dr; Summary: Each operator needs to decide how critical time is to their network and pick a solution that works for them and fits the organization's budget. Some operators might point everything at pool.ntp.org <http://pool.ntp.org/> and not run their own servers. Others might run their own time lab and use that time to provide NTP time and precision time and frequency via various methods. Most will be somewhere in between. But regardless of which you choose, please be aware that GPS isn't 100% secure, and neither is NTP. If attack resilience matters to you, you should think about all of the attack vectors and design something that is robust enough to meet your use case.

This has been an interesting thread. I consider Forrest Christian’s note to be most cogent. Much of the GPS vs Internet sourcing arguments can probably be found in NANOG archives from many years ago. The threat list is longer now, but the problem of providing Time Service is still the same. Twenty-five or so years ago my design process for providing Network Time Service to a large company intranet started with the business requirements for time service. The Management practice of “Not in my cost center” was fundamental to NOT attempting GPS-based deployment. The internal enterprise network provided a set of geographically distributed Stratum 2 servers having carefully firewalled access to a similar set of Stratum 1 servers with Internet access. The Stratum 0 server set list included NIST, USNO, and other similar sources distributed globally. The magic of Dr. Mills algorithm made truechimers of the intranet NTP server set which did serve well for the lifetime of the company. - James R. Cutler

" As such, the ultimate (a little expensive) solution is to have your own Rb clocks locally." Yeah, that's a reasonable course of action for most networks. *sigh* ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Masataka Ohta" <mohta@necom830.hpcl.titech.ac.jp> To: nanog@nanog.org Sent: Friday, August 11, 2023 4:33:20 AM Subject: Re: NTP Sync Issue Across Tata (Europe) Forrest Christian (List Account) wrote:

The recommendation tends to be the following:

1) Run your GPS-derived NTP appliances, but DO NOT point end-user clients at it. 2) Run a set of internal NTPd servers, and configure them to pull time from all of your GPS-derived NTP servers, AND trusted public NTP servers 3) Point your clients at the internal NTPd servers.

That is not a very good recommendation. See below.

At some point, using publicly available NTP sources is redundant unless one wants to mitigate away the risks behind failure of the GPS system itself.

Your assumption that public NTP servers were not GPS-derived NTP servers is just wrong.

What I'm advocating against is the seemingly common practice to go buy an off-the-shelf lower-cost GPS-NTP appliance (under $1K or so), stick an antenna in a window or maybe on the rooftop, and point all your devices at that device.

Relying on a local expensive GPS appliance does not improve security so much and is the worst thing to do. But, additionally relying on remote servers (including those provided by NIST) is subject to DOS attacks. As such, the ultimate (a little expensive) solution is to have your own Rb clocks locally. Masataka Ohta

John Gilmore wrote:

...

I was also speaking specifically about installing GPS antennas in viable places, not using a facility-provided GPS or NTP service.

Am I confused? Getting the time over a multi-gigabit Internet from a national time standard agency such as NIST (or your local country's equivalent) should produce far better accuracy and stability than relying on locally received GPS signals.

When the (wrong) question is "how to build a stratum 1 server?", that can not be an answer.

GPS uses very weak radio signals which are regularly spoofed by all sorts of bad actors:

The question, seemingly, is not "how to build a secure stratum 1 server?". BTW, the proper question should be "how to obtain secure time?". Masataka Ohta

On 8/7/23 11:04, Giovane C. M. Moura via NANOG wrote:

TL;DR: I'd guess your NTP Server IP address is geolocated to Mauritius. The Mauritius zone[0] on the pool has only one server, so you'll only see this one. To fix it, use europe.pool.ntp.org (_do not_ use pool.ntp.org).

So the Anycast address our devices use internally to find the closest NTP server is geo-mapped to MU. However, the physical server is geo-mapped to the specific countries in Europe, e.g., GB, NL, FR, DE, e.t.c. Unless the geo data ntp.org are using is inconsistent, I'd imagine the servers should be mapped to a European pool, since the physical address from which the server queries the pool is geo-mapped locally, for this specific reason. Mark.