I would second the idea of using your own GPS appliance if possible. On May 9, 2016 11:08 PM, "Mel Beckman" <mel@beckman.org> wrote:

NTP has vulnerabilities that make it generally unsuitable for provider networks. I strongly recommend getting a GPS-based time server. These are as cheap as $300. Here is one I use quite a bit:

http://www.amazon.com/TM1000A-GPS-Network-Time-Server/dp/B002RC3Q4Q

You’ll have a stratum 1 clock on site. Hard to beat.

-mel

On May 9, 2016, at 8:01 PM, b f <freetexwatson@gmail.com<mailto: freetexwatson@gmail.com>> wrote:

Hello List,

In search of stable, disparate stratum 1 NTP sources.

Looking for anyone’s advice/experiences (good/bad/ugly/weird) using NIST’s NTP servers per: http://tf.nist.gov/tf-cgi/servers.cgi

We tried using “time.nist.gov<http://time.nist.gov>” which returns varying round-robin addresses (as the link says), but Cisco IOS resolved the FQDN and embedded the numeric address in the “ntp server” config statement.

After letting the new server config go through a few days of update cycles, the drift, offset and reachability stats are not anywhere as good as what the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.

I would greatly appreciate and feedback / advice, etc.

Thanks!!!

Ed

NTP has vulnerabilities, so using an external source opens your networks and infrastructure to disruptions. Going with an internal GPS/GLONASS/RADIO based S1 allows you to restrict incoming traffic and not rely on volunteers or external entities (which may undergo maintenance or budget issues). My preference is more so something akin to the GLN180PEX (I am not affiliated or paid to endorse this product). It allows you to use commodity hardware (like a decommissioned 1U or several preferably) and creation of ones own reliable internal time source(s). Introducing black boxes into a production (revenue generation or expected services by paying customers) environment is undesirable.

From there setting up NTPd, Chronyd, and PTPd is up to you.

Relying on satellites may seem like just another external reliance, but the next life is proposing a design life of 12 years..... On Mon, May 9, 2016 at 11:12 PM, Majdi S. Abbas <msa@latt.net> wrote:

On Tue, May 10, 2016 at 03:08:16AM +0000, Mel Beckman wrote:

...

NTP has vulnerabilities that make it generally unsuitable for provider networks. I strongly recommend getting a GPS-based time server. These are as cheap as $300. Here is one I use quite a bit:

So how does this stop from distributing time to their customers via NTP?

GPS doesn't save the protocol, in particular where the S1 clocks involved are embedded devices with rather coarse clocks and timestamping.

--msa

-- Miano, Steven M. http://stevenmiano.com

On Tue, May 10, 2016 at 06:48:52AM -0400, Steven Miano <mianosm@gmail.com> wrote a message of 41 lines which said:

Going with an internal GPS/GLONASS/RADIO based S1 allows you to restrict incoming traffic and not rely on volunteers or external entities (which may undergo maintenance or budget issues).

You mean the GPS network is not managed by an external entity? With budget issues? http://www.schriever.af.mil/GPS

On Tue, May 10, 2016 at 10:52:28AM -0400, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> wrote a message of 37 lines which said:

Note that they *do* have motivation to keep it working, simply because so much of their *own* gear (from gear for individual soldiers all the way to strategic bombers and aircraft carriers) wants a working GPS signal...

Yes, but they may switch it off for civilian use (by going encrypted, for instance) at any time, if it is better for *their* operations.

Tue, May 10, 2016 at 04:59:02PM +0200, Stephane Bortzmeyer wrote:

On Tue, May 10, 2016 at 10:52:28AM -0400, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> wrote a message of 37 lines which said:

...

Note that they *do* have motivation to keep it working, simply because so much of their *own* gear (from gear for individual soldiers all the way to strategic bombers and aircraft carriers) wants a working GPS signal...

Yes, but they may switch it off for civilian use (by going encrypted, for instance) at any time, if it is better for *their* operations.

Civilian signals are already degraded in terms of accuracy (without assisted GPS) and military ones are encrypted (but see [1]). The primary reason for encryption is, by the way, to address spoofing issues for the mil people themselves, but it is also good not to arm the potential enemy with the precise coordinates or to be able to do spoofing for them. And since many civilian, but nonetheless, vital orgs are using civilian parts, it could be hard to shut it off without affecting some parts of the infrastructure. Not that nobody wants that, but it will give an unneeded resonance and could lead to the terrible mess. [1] http://www.gps.gov/technical/codeless/ -- Eygene Ryabinkin, National Research Centre "Kurchatov Institute" Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.

Jean-Francois Mezei <jfmezei_nanog@vaxination.ca> wrote:

Today, if someone were to jam the GPS signal in an areas in USA, you'd likely hear about large number of car accidents in the news before noticing your systems canMt get time from the GPS-NTP and went to a backup ip address (nist etc).

The USA and the UK governments regularly perform GPS jamming tests, but they do so in remote areas. See http://www.navcen.uscg.gov/?pageName=gpsServiceInterruptions http://stakeholders.ofcom.org.uk/spectrum/gps-jamming-exercises/ (Dunno if other governments have similar exercises.) Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ - I xn--zr8h punycode Lundy, Fastnet, Irish Sea, South Shannon: Northerly or northeasterly 4 or 5, occasionally 6 except in south Shannon. Slight or moderate. Mainly fair. Good, occasionally poor at first in south Lundy.

On 5/10/2016 at 10:30 AM, "Chuck Church" <chuckchurch@gmail.com> wrote:

It doesn't really. Granted there are a lot of CVEs coming out for NTP the last year or so. But I just don't think there are that many attacks on it. It's just not worth the effort. Changing time on devices is more an annoyance than anything, and doesn't necessarily get you into a device. Sure you can hide your tracks a little by altering time in logs and altering it back, but that's more of an in-depth nation-state kind of attack, not going to be a script kiddie kind of thing. Just follow the best practices for verifying packet sources and NTP security itself, and you should be ok.

Chuck

I would argue that the fact the NTP can, and has been, be used in DDoS amplification attacks is a serious concern for using protocol going forward. allan

Yo Chuck! On Tue, 10 May 2016 10:29:35 -0400 "Chuck Church" <chuckchurch@gmail.com> wrote:

Changing time on devices is more an annoyance than anything, and doesn't necessarily get you into a device.

So, you are not worried about getting DoS'ed? How about you set the time on your server ahead by 5 years. Got any idea what would happen? Most of your passwords would expire. All your SSL certs would expire. All your TOTPs, like Google Authenticator would fail. All your IPSEC tunnels would drop, and refuse to restart. Many of your cron jobs would got nuts, possibly deleting all your logs. Much of your DNSSEC would expire. Many of your backups would be deleted since they 'expired'. Until recently, setting your iPhone to 1 Jan 1970 would brick it. I'm sure there are many more examples, but likely you can no longer log in, via SSH or HTTPS, and your iPhone is dead. I think any of those would qualify as more than an annoyance. RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588

-----Original Message----- From: Gary E. Miller [mailto:gem@rellim.com] Sent: Tuesday, May 10, 2016 3:58 PM To: Chuck Church <chuckchurch@gmail.com> Cc: 'Majdi S. Abbas' <msa@latt.net>; nanog@nanog.org Subject: Re: NIST NTP servers Yo Chuck! On Tue, 10 May 2016 10:29:35 -0400 "Chuck Church" <chuckchurch@gmail.com> wrote:

Changing time on devices is more an annoyance than anything, and doesn't necessarily get you into a device.

So, you are not worried about getting DoS'ed? How about you set the time on your server ahead by 5 years. Got any idea what would happen? Most of your passwords would expire. All your SSL certs would expire. All your TOTPs, like Google Authenticator would fail. All your IPSEC tunnels would drop, and refuse to restart. Many of your cron jobs would got nuts, possibly deleting all your logs. Much of your DNSSEC would expire. Many of your backups would be deleted since they 'expired'. Until recently, setting your iPhone to 1 Jan 1970 would brick it. I'm sure there are many more examples, but likely you can no longer log in, via SSH or HTTPS, and your iPhone is dead. I think any of those would qualify as more than an annoyance. RGDS GARY ---------------------------------------------------------------------------- ---------------------------------------------------------------- Ok, annoyance might have been a little light on the severity wording. Still, modifying all your incoming NTP packets from all your sources to actually get your NTP servers to agree on a bad time is tricky. That is assuming you've got multiple links, multiple sources from multiple organizations (more than 4), they're all authenticated, etc. Even if a criminal was to do all that damage you listed, it still probably doesn't result in obtaining sensitive data or money that would be the main motivators for such extreme hacking. If I had an iPhone, perhaps I'd worry about that as well. But fortunately, not an issue ;) Chuck

Accurate time to the millisecond is pretty much essential for any network troubleshooting. Say you want to diagnose a SIP problem. You collect transaction logs from both phones, the VoIP gateway, and the PBX. Now you try to merge them to derive the sequence of events. You NEED millisecond accuracy. But more importantly, Gary is right about the risks. I’ve had several customers receive major NTP DoS attacks using forged source addresses. In today’s Internet, there is very little source address verification (despite several mechanisms being proposed). Everyone relies on the originating network preventing spoofing, but thousands of ISPs — particularly overseas — do not do spoof checks. And the issues of NTP pollution are even more dangerous. As Gary notes, changing dates is a risk. A big enough change (say 30 days) would be catastrophic to most accounting systems. A big leap — a year or more — could expire software license and disable all kinds of encryption. We haven’t even discussed multi-stage attacks, where NTP is used to disrupt systems at multiple points, and then the attacker storms in and takes over unnoticed during the confusion. All because of misplaced trust in a tiny UDP packet that can worm its way into your network from anywhere on the Internet. I say you’re crazy if you don’t run a GPS-based NTP server, especially given that they cost as little as $300 for very solid gear. Heck, get two or three! -mel

On May 10, 2016, at 12:58 PM, Gary E. Miller <gem@rellim.com> wrote:

Yo Chuck!

On Tue, 10 May 2016 10:29:35 -0400 "Chuck Church" <chuckchurch@gmail.com> wrote:

...

Changing time on devices is more an annoyance than anything, and doesn't necessarily get you into a device.

So, you are not worried about getting DoS'ed?

How about you set the time on your server ahead by 5 years. Got any idea what would happen?

Most of your passwords would expire.

All your SSL certs would expire.

All your TOTPs, like Google Authenticator would fail.

All your IPSEC tunnels would drop, and refuse to restart.

Many of your cron jobs would got nuts, possibly deleting all your logs.

Much of your DNSSEC would expire.

Many of your backups would be deleted since they 'expired'.

Until recently, setting your iPhone to 1 Jan 1970 would brick it.

I'm sure there are many more examples, but likely you can no longer log in, via SSH or HTTPS, and your iPhone is dead. I think any of those would qualify as more than an annoyance.

RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588

I hope your receivers aren't all from a single source. I was in Iraq when this ( http://dailycaller.com/2010/06/01/glitch-shows-how-much-us-military-relies-o... ) happened, which meant I had no GPS guided indirect fire assets for 2 weeks. On Wed, May 11, 2016 at 8:31 AM, Leo Bicknell <bicknell@ufp.org> wrote:

In a message written on Tue, May 10, 2016 at 08:23:04PM +0000, Mel Beckman wrote:

...

All because of misplaced trust in a tiny UDP packet that can worm its way into your network from anywhere on the Internet.

I say you’re crazy if you don’t run a GPS-based NTP server, especially given that they cost as little as $300 for very solid gear. Heck, get two or three!

You're replacing one single point of failure with another.

Personally, my network gets NTP from 14 stratum 1 sources right now. You, and the hacker, do not know which ones. You have to guess at least 8 to get me to move to your "hacked" time. Good luck.

Redundancy is the solution, not a new single point of failure. GPS can be part of the redundancy, not a sole solution.

-- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/

----- Original Message -----

From: "Mel Beckman" <mel@beckman.org>

Read deeper into the thread and you'll find where I sourced inexpensive RF-based NTP servers using CDMA, GSM, and even WWV. All radically different technologies that are unlikely to have common failure modes. But yes, buying different brands can't hurt either.

CDMA and GSM are false diversity: both network types nodes *get their time* from GPS, so far as I know. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

No, many cell carriers run their own completely independent timing networks. I support some head-ends where they have rubidium clocks and a T1-delivered time source. They do reference GPS, and many cell sites have GPS as a backup clock (you can see their conical antennas on the very top of the tower). But most cellular providers are very protective of their time sources. They’re also typically clocking SONET networks too, which requires BITS. -mel JAshworth said:

CDMA and GSM are false diversity: both network types nodes *get their time* from GPS, so far as I know.

On May 11, 2016, at 10:54 AM, Valdis.Kletnieks@vt.edu wrote:

On Wed, 11 May 2016 15:36:34 -0000, "Jay R. Ashworth" said:

...

CDMA and GSM are false diversity: both network types nodes *get their time* from GPS, so far as I know.

I'll make the fairly reasonable assumption that most readers of this list have networks that span multiple buildings.

If somebody is managing to figure out that you have a GPS in Building 37, and a GPS-based CDMA up on the corner of Building 3, and the *other* 4 clocks at other locations and getting close enough to all of them at the same time to conduct a successful spoofing attack, all just to move your time source a few seconds off....

... then the fact that GPS is spoofable is probably *NOT* your biggest security problem.

On 2016-05-11 10:30, Mel Beckman wrote:

Read deeper into the thread and you'll find where I sourced inexpensive RF-based NTP servers using CDMA, GSM, and even WWV.

For shortwave, you would need to calculate propagation delay between transmitter and receiver. (does signal reach via line of sight, bounce against ionosphere ?). Since CDMA is dead outside the USA and drying in USA, I wouldn't rely on that. If GSM towers rely on a GPS receiver on the tower and those towers are near enough to your location (< 30km), then chances are that blocked GPS signals at your location would also jam the signals at the GSM antenna. And if you are setup to be totally autonomous in case of power failures, you need to know whether the GSM antenna you are relying on is also on permanent power backup or only has autonomy of a few hours.

In a message written on Wed, May 11, 2016 at 09:00:54AM -0500, Josh Reynolds wrote:

I hope your receivers aren't all from a single source.

I have 4 each ACTS, GPS, and CDMA in my list, agumented with a pair of PTP. Amazingly right now all but 3 are within 2 microsconds of each other, and those three outliers are 10 and 35 microseconds off. That's pretty impressive! I didn't have to buy any of them, because various trustable entities run those infrastructures. Some of the trustable entites are the same ones that send the time up to the GPS satellites. :) -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/

A note on using a Raspberry Pi as a NTP server. In my limited home lab testing the RPi server had enough instability that Internet time sources were always preferred by my workstation after ntpd had been running for a while. Presumably this was due to the RPi's clock frequency drifting. At some point I will look at it again. If you do want to build your own Stratum 1 server you might want to glance at: https://github.com/meekj/ntp/blob/master/jon_meek_ntp_poster2009a.pdf and the references there. I had hoped to use the very low cost RPi Stratum 1 servers at $DAY_JOB, but the test device was clearly not up to the job. At some point I hope to revisit this and do some more testing like I did for that poster. I'll add in a CDMA server and a dedicated WWVB receiver. Jon

On 05/11/2016 09:46 PM, Josh Reynolds wrote:

maybe try [setting up an NTP server] with an odroid?

... I have several ODroid C2's, and the first thing to note about them is that there is no RTC at all. Also, the oscillator is just a garden-variety non-temperature-compensated quartz crystal, and not necessarily a very precise one, either (precise quartz oscillators can cost more than the whole ODroid board costs). The XU4 and other ODroid devices make nice single-board ARM computers, but have pretty ratty oscillator precision. You really have to have at least a temperature compensated quartz crystal oscillator (TCXO) to even begin to think about an NTP server, for anything but the most rudimentary of timing. Ovenized quartz oscillators (OCXO) and rubidium standards are the next step up, and most reasonably good GPS-disciplined clocks have at least an ovenized quartz oscillator module (the Agilent Z3816 and kin are of this type).

On 05/13/2016 10:38 AM, Mel Beckman wrote:

You make it sound like TXCOs are rare, but they're actually quite common in most single board computers. True, you're probably not gonna find them in the $35 cellular-based SBCs, but since these temperature compensated oscillators cost less than a dollar each in quantity, they're quite common in most industrial species for well under $100.

Correct, they're not rare in the industrial line (for that matter you can get TCXO-equipped RTL-SDR dongles, but that's not NTP-related). Something like a Transko TFC or TX-P or similar is enough for reasonable timing for basic purposes, and they're not expensive. They're also not a stock item on the consumer-level SBC's either. I looked at one of our half-dozen ODroid C2's, and the main processor clock, X3, is under the heatsink, so I can't see what part is being used. X1 and X2 are outside, and it doesn't appear that they are TCXO modules, although I didn't use a magnifier to check the part number and might have made an error. The Nicegear DS3231 RTC has a TCXO, and might be the best low-cost choice at $12 (need to have an RPi, ODroid, or similar on which to mount it). It's not that TCXO's are rare or expensive, it's that they're not often considered to be important to accuracy in many circles.

An Ovenized XCO is absolutely not required for IT-grade NTP servers.

No, but it is for my purposes here. But, as I said in my post:

You really have to have at least a temperature compensated quartz crystal oscillator (TCXO) to even begin to think about an NTP server, for anything but the most rudimentary of timing. Ovenized quartz oscillators (OCXO) and rubidium standards are the next step up, ...

I was just saying that OCXO and Rb are just the next step up if you would like more stability, that's all. For 'within a second' on a GPS-disciplined clock (even without the 1PPS signal) you wouldn't necessarily need TXCO. But that's what I meant by 'anything but the most rudimentary of timing.' Rudimentary is down to the millisecond in my environment. PTP takes you to the next level, and beyond that you don't use network timing but put a dedicated time distribution network running IRIG-B or similar. But that is beyond the scope of a typical IT NTP server's needs.....

"Either method needs the specs" should read "Either method meets the specs." -mel beckman

On May 13, 2016, at 1:39 PM, Mel Beckman <mel@beckman.org> wrote:

Lamar,

Because you need microsecond-level time accuracy (which is beyond NTP's capabilities) you'll requires an adjunct protocol, such as PPS, to get that. For continued NTP delivery despite periodic GPS signal loss, then you need an OCXO internal clock.

But anyone satisfied with NTP's millisecond time accuracy at worst needs a $1 temperature-compensated internal clock. Either method needs the specs for a Stratum 1 time source on a local network.

As you point out, the hobbyist SBCs can't deliver even basic clock accuracy.

But another key consideration beyond accuracy is the reliability of a server's GPS constellation view. If you can lose GPS sync for an hour or more (not uncommon in terrain-locked locations), the NTP time will go free-running and could drift quite a bit. You need an OCXO to minimize that drift to acceptable levels.

But I see that the price premium for an OCXO clock is only $100 to $200 on low-cost (I.e., ~$500) commercial NTP servers. So buyers need only make a minor cost adjustment to have very good, and inexpensive, COTS NTP performance and reliability.

-mel beckman

...

...

On May 13, 2016, at 9:30 AM, Lamar Owen <lowen@pari.edu> wrote:

On 05/13/2016 10:38 AM, Mel Beckman wrote: You make it sound like TXCOs are rare, but they're actually quite common in most single board computers. True, you're probably not gonna find them in the $35 cellular-based SBCs, but since these temperature compensated oscillators cost less than a dollar each in quantity, they're quite common in most industrial species for well under $100.

Correct, they're not rare in the industrial line (for that matter you can get TCXO-equipped RTL-SDR dongles, but that's not NTP-related). Something like a Transko TFC or TX-P or similar is enough for reasonable timing for basic purposes, and they're not expensive. They're also not a stock item on the consumer-level SBC's either. I looked at one of our half-dozen ODroid C2's, and the main processor clock, X3, is under the heatsink, so I can't see what part is being used. X1 and X2 are outside, and it doesn't appear that they are TCXO modules, although I didn't use a magnifier to check the part number and might have made an error.

The Nicegear DS3231 RTC has a TCXO, and might be the best low-cost choice at $12 (need to have an RPi, ODroid, or similar on which to mount it). It's not that TCXO's are rare or expensive, it's that they're not often considered to be important to accuracy in many circles.

...

An Ovenized XCO is absolutely not required for IT-grade NTP servers.

No, but it is for my purposes here. But, as I said in my post:

...

You really have to have at least a temperature compensated quartz crystal oscillator (TCXO) to even begin to think about an NTP server, for anything but the most rudimentary of timing. Ovenized quartz oscillators (OCXO) and rubidium standards are the next step up, ...

I was just saying that OCXO and Rb are just the next step up if you would like more stability, that's all. For 'within a second' on a GPS-disciplined clock (even without the 1PPS signal) you wouldn't necessarily need TXCO. But that's what I meant by 'anything but the most rudimentary of timing.' Rudimentary is down to the millisecond in my environment. PTP takes you to the next level, and beyond that you don't use network timing but put a dedicated time distribution network running IRIG-B or similar. But that is beyond the scope of a typical IT NTP server's needs.....

On 2016-05-13 14:12, Lamar Owen wrote:

On 05/11/2016 09:46 PM, Josh Reynolds wrote:

...

maybe try [setting up an NTP server] with an odroid?

...

You really have to have at least a temperature compensated quartz crystal oscillator (TCXO) to even begin to think about an NTP server, for anything but the most rudimentary of timing.

There are WWVB clocks that try to sync nightly. Many of them don't even have a second indicator, but they give reliable time to the minute. NTP is a lot better than this as it continuously disciplines the clock instead of just lining it up once a day, but we're talking about doing this over the internet where we measure latency in milliseconds. If you're working down at the picosecond level you will probably not be using NTP to distribute your clock signal. Running an NTP client against pool servers is a lot better than not running it at all, but running it against a fancy local server with a GPSDO hooked up to it is only marginally better than the pool servers. It all depends on what you want to do but a cheap ARM or Intel Atom computer works well for an NTP server (remember millisecond level accuracy). If you can afford to build a secure bunker with armed guards and redundant everything for your time server that's good, but a few RPi style computers with GPS hats are almost as good, and you can buy a lot of them for very little money.. -Laszlo

Since we are on the subject, I would strongly recommend that you don't run NTP on Linux 2.2.13, since its especially vulnerable to our IPv4 fragmentation attack. "SunOS" also seems vulnerable, but I am not 100% sure what systems that say they are "SunOS" actually are. These OS will fragment packets to 64 bytes, and are vulnerable to frag attacks using "tiny" fragments. See Section VI of our paper: https://eprint.iacr.org/2015/1020.pdf You can also test your OS here (scroll to the bottom). http://www.cs.bu.edu/~goldbe/NTPattack.html On Fri, May 13, 2016 at 10:46 AM, Chuck Anderson <cra@wpi.edu> wrote:

On Fri, May 13, 2016 at 10:12:49AM -0400, Lamar Owen wrote:

...

On 05/11/2016 09:46 PM, Josh Reynolds wrote:

...

maybe try [setting up an NTP server] with an odroid?

...

I have several ODroid C2's, and the first thing to note about them is that there is no RTC at all. Also, the oscillator is just a garden-variety non-temperature-compensated quartz crystal, and not necessarily a very precise one, either (precise quartz oscillators can cost more than the whole ODroid board costs). The XU4 and other ODroid devices make nice single-board ARM computers, but have pretty ratty oscillator precision.

You really have to have at least a temperature compensated quartz crystal oscillator (TCXO) to even begin to think about an NTP server, for anything but the most rudimentary of timing. Ovenized quartz oscillators (OCXO) and rubidium standards are the next step up, and most reasonably good GPS-disciplined clocks have at least an ovenized quartz oscillator module (the Agilent Z3816 and kin are of this type).

Does anyone know of any COTS NTP servers that are based on non-ancient Linux kernel versions? In 2012 we bought new GPS/CDMA NTP servers with OCXO that are based on Linux 2.4, but they are fiddly as you can imagine with such an ancient software stack.

What would people recommend for NTP server hardware/software?

-- Sharon Goldberg Computer Science, Boston University http://www.cs.bu.edu/~goldbe

Once upon a time, John Souvestre <john@souvestre.com> said:

The Enhanced WWVB signal has better range and more accuracy, but I don't know if any receivers are available yet.

I know it's supposed to have better range and signal quality, but I thought accuracy was about the same. The variables that affect accuracy are mostly external to the signal itself (propagation delay affected by atmospheric conditions). At one point, they were going to put a second transmitter closer to the east coast, and it was going to be at the Army's Redstone Arsenal, next to Huntsville, AL, where I live; I probably could have put a receiver in a steel box and still had good signal! NASA vetoed it though. -- Chris Adams <cma@cmadams.net>

-----Original Message-----

From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Leo Bicknell Sent: Wednesday, May 11, 2016 9:31 AM To: nanog@nanog.org Subject: Re: NIST NTP servers

Personally, my network gets NTP from 14 stratum 1 sources right now. You, and the hacker, do not know which ones. You have to guess at least 8 to get me to move to your "hacked" time. Good luck.

Redundancy is the solution, not a new single point of failure. GPS can be part of the redundancy, not a sole solution.

This seems like the most reasonable advise. If this truly becomes a concern, I would think IPS vendors could implement signatures to look for bad time. Lots of ways to do this - look for a difference between the IPS realtime and NTP status versus the incoming packets. - look for duplicate NTP responses, or responses that weren't requested - duplicate responses, but with differing TTLs, which might hint at one being spoofed. Chuck

On 5/10/2016 11:22 AM, Leo Bicknell wrote:

In a message written on Mon, May 09, 2016 at 11:01:23PM -0400, b f wrote:

...

In search of stable, disparate stratum 1 NTP sources.

http://wpollock.com/AUnix2/NTPstratum1PublicServers.htm

...

We tried using “time.nist.gov” which returns varying round-robin addresses (as the link says), but Cisco IOS resolved the FQDN and embedded the numeric address in the “ntp server” config statement.

Depending on your hardware platform your Cisco Router is likely not a great NTP server. IOS is not designed for hyper-accuracy.

...

After letting the new server config go through a few days of update cycles, the drift, offset and reachability stats are not anywhere as good as what the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.

The correct answer here is to run multiple NTP servers in your network. ... [snip]

I think the correct answer here starts with a question --- what level of time accuracy is required for the local NTP server(s)? Which then begs the question, what level of accuracy is needed for the clients? A shop with a client need for nanosecond accuracy begs for an entirely different solution set than a shop where a millisecond of accuracy is needed on the clients, and still a different solution set that a shop where "a few milliseconds either way" is quite OK.

Leo Bicknell writes:

...

The correct answer here is to run multiple NTP servers in your network. And by servers I mean real servers, with good quality oscellators on the motherboard. Then configure them to talk to _many_ sources. You need 4 sources of time minimum to redundantly detect false tickers. If you're serious about it then find ~10 Stratum 1 sources (ideally authenticated and from trusted entities),

Byzantine General's problem. With 3 sources you can detect *1* false ticker. But if one of those becomes unreachable you only have 2 time sources. Dilemma. With 4 sources you run the risk of 2 going one way, and 2 going another way. This happened to several folks recently, when some sites put NTP servers on the 'net that offered leap-smeared time. That's really a different problem where one of the effects is that it causes "time islands".

one of which could be GPS as several have suggested. You'll then have high quality false ticker rejection.

For extra points, use GPS receivers from different manufacturers, using whatever "variety" you can get for all of the components involved. Are you mounting each GPS receiver inside a coffee can to prevent drive-by jamming? Are the cables properly grounded? Using gas discharge tubes? Periodically tested/inspected? How much fun do you want to have thinking about all of these cases?

Configure all of your devices to get NTP from the servers you run using authentication.

Yes, and properly monitor your ntpd instances. -- Harlan Stenn <stenn@ntp.org> http://networktimefoundation.org - be a member!

Yo Jared! On Tue, 10 May 2016 16:29:26 -0400 Jared Mauch <jared@puck.nether.net> wrote:

If you’re using Redhat based systems consider using chrony instead, even the new beta fedora 24 uses 4.2.6 derived code vs 4.2.8

Or, new but under heavy development: NTPsec : https://www.ntpsec.org/ It is a fork of classic NTPD, but was not vulnerable to most of the recent NTPD CVEs. RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588

Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance? IT guy: He changed our clocks. Boss: How did he do that? IT guy: We have an opening in our firewall that permits time clock packets to come from anywhere in the world, under certain conditions. Boss: Why didn’t you block that? IT guy: Well, we filtered to only accept clock settings from a trusted source, but the hacker lied and pretended to be that protected source. Boss: I thought encryption was supposed to prevent that. IT guy: Time clock packets aren’t encrypted. There is no standard for that. Boss: Not even a password? IT guy: Yes, there is a sophisticated authentication mechanism, but it doesn’t work. Boss: So how could we have prevented this? IT guy: We could have purchased our own time server synchronized to the U.S. Department of Standards atomic clock via Global Positioning System satellites using a special antenna. Then we wouldn’t need time from the Internet. Boss: That sounds expensive. How much are we talking? IT guy: $300 Boss: You’re fired. On May 10, 2016, at 1:51 PM, Jared Mauch <jared@puck.nether.net<mailto:jared@puck.nether.net>> wrote: On May 10, 2016, at 4:40 PM, Gary E. Miller <gem@rellim.com<mailto:gem@rellim.com>> wrote: Yo Jared! Yo, Gary! On Tue, 10 May 2016 16:29:26 -0400 Jared Mauch <jared@puck.nether.net<mailto:jared@puck.nether.net>> wrote: If you’re using Redhat based systems consider using chrony instead, even the new beta fedora 24 uses 4.2.6 derived code vs 4.2.8 Or, new but under heavy development: NTPsec : https://www.ntpsec.org/ It is a fork of classic NTPD, but was not vulnerable to most of the recent NTPD CVEs. Yeah, there are some issues here in how the NTP community has implemented solutions without discussing with each other through the community splits. The NTPWG at IETF has been in a bit of stasis for years now because the various aspects of how it works, and those who present sometimes don’t output in the most organized fashion requiring a lot of effort on the receiver. There’s also a very narrow universe of people who actually care about the implementations and details, with people like Majdi, Harlan and Miroslav understanding the needs more than I’ve seen anyone from the ntpsec/cisco funded side grasp the nuances of. As a general statement, we are well served by having diverse and robust implementations, but as we’ve seen in the (mostly) router space that NANOG community cares about.. there are far more BGP implementations than NTP. This isn’t good if the community wants to move to a model of certificate based routing and the dependent infrastructure is weak. I would suggest moving parts of this discussion to either the NTP Pool or the NTPWG mailing lists. - jared

Andreas, Most data centers will require a remotely positioned NTP server, which is actually easier and cheaper than a remotely located active GPS antenna. I have placed the $300 commercial NTP servers in an environmental box on the roof, powering t by PoE, without problems. You don't need a redundant network either, nor redundant power. Just plunk down two of these gizmos, or as I suggested elsewhere, deploy one or more CDMA, GSM, or WWV-based clocks, for as much local infrastructure and signal source diversity as you like (I sourced some of these units earlier in the thread, all well less than $1K each. You pay more for diversity, but you get more too. In response to the several DIYers on this thread: Anyone who thinks they're building a raspberry pi-based GPS NTP server for just $150 is kidding themselves. They forgot to include their labor, which is worth far more than the $300 for a commercial unit. The same goes for people who complain about even the minimal $300 price, forgetting that a commercial product has to pay for marketing, support, and make a profit. External NTP has two kinds of vulnerabilities: the ones we know, and the ones we don't. The ones we know are serious enough the pat the ones we don't should be considered with respect. Maybe diversity in Internet sources is a cure, maybe it isn't. But diverse RF sources is demonstrably more secure than the Internet. My point stands: secure external RF NTP sources are so plentiful that relying on Internet NTP is just plain crazy. -mel beckman

On May 11, 2016, at 7:12 AM, Andreas Ott <andreas@naund.org> wrote:

Hi,

...

Boss: That sounds expensive. How much are we talking? IT guy: $300

Beware!

Over the past year we made engineering samples to deploy to datacenters. The goal was to use GPS and PPS to discipline ntpd appliances and serve as stratum 1 to other NTP distribution servers without the $5k price tag of commercial NTP 1RU gear. We also deliberately not pursued the path of running antenna coax from the roof to a receiver, as that is not an option in all the datacenters where we need to deploy.

These appliances were built for lesss than $150 as

(a) Raspberry-Pi with GPS receiver board

(b) Garmin 18(x) LVC with DB-9 to an "older whitebox server"

In my experience, most locations inside datacenters where you have good power and network connectivity have not "good enough" GPS signal reception due to servers emitting lots of RF noise in the range 1-2 GHz on the L-band. A brand new suite in the datacenter had OK GPS quality, but once we added 20+ racks with 40 servers each, GPS reception was pretty much gone. This hardware was an active antenna with less than 6 feet of cabling routed to the top of the network cabling rack. Most smartphones can run an app to show you the GPS signal on the phone, just walk around your datacenter and compare the signal.

The only workable solution was to move the GPS clock to a location where it had good GPS signal but neither redundant network nor conditioned power (aka. "on my desk near a south facing window"). It also works pretty well "in my garage".

In places where GPS reception is good, you can achieve 10E-06 seconds accuracy over time even with cheap hardware. If you chose to run the DB-9 NMEA0183 and GPS as "serial port passthrough" to a VM on a Hypervisor you can still get better than 10E-03 seconds accuracy.

-andreas -- Andreas Ott (Time-Nut) K6OTT +1.408.431.8727 andreas@naund.org

I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now. My point is, when the fix is so cheap, why put up with this risk at all? -mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams <cma@cmadams.net> wrote:

Once upon a time, Mel Beckman <mel@beckman.org> said:

...

Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy.

First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe Random Hacker isn't going to have a high chance of guessing the servers your system is using.

Second, he'd have to guess at least three to "win".

Third, at best, he'd only be able to change your clocks a little; the common software won't step the clock more than IIRC 15 minutes. Yes, that can cause problems, but not the catastrophes of years in the future or Jan 1, 1970 mentioned in this thread.

Is it possible to cause problems? Yes. Is it a practical attack? I'm not so sure, and I haven't seen proof to the contrary. -- Chris Adams <cma@cmadams.net>

Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return? The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted. So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum. Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. 6. All core time systems should be monitored by your security team or SOC. One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?". Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, May 10, 2016 at 9:59 PM, Mel Beckman <mel@beckman.org> wrote:

I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

...

On May 10, 2016, at 5:18 PM, Chris Adams <cma@cmadams.net> wrote:

Once upon a time, Mel Beckman <mel@beckman.org> said:

...

Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy.

First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe Random Hacker isn't going to have a high chance of guessing the servers your system is using.

Second, he'd have to guess at least three to "win".

Third, at best, he'd only be able to change your clocks a little; the common software won't step the clock more than IIRC 15 minutes. Yes, that can cause problems, but not the catastrophes of years in the future or Jan 1, 1970 mentioned in this thread.

Is it possible to cause problems? Yes. Is it a practical attack? I'm not so sure, and I haven't seen proof to the contrary. -- Chris Adams <cma@cmadams.net>

Regarding Roland’s reference to time and position spoofing via a hacked GPS signal, the hacker has to get physical line of sight to the victim’s antenna in order to succeed with this attack. That’s likely within a few blocks, if not within a few feet. And a rooftop antenna might require a drone attack. And how does the drone get guidance without a reliable GPS signal? :) Eric, I agree that sometimes a site can’t get a GPS signal, but in my experience designing data centers, that’s still pretty rare. Many NTP systems use an active GPS antenna that can be hundreds of feet away. But you can always put the $300 NTP server in an outdoor enclosure and power it with PoE. There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet: http://www.beaglesoft.com/celsynhowworks.htm And their $400 WWV-based Stratum 1 time server: http://www.beaglesoft.com/radsynreceiver.htm So if you want non-Internet clock diversity, you can have clock diversity. You just have to pay for it. -mel On May 10, 2016, at 9:18 PM, Eric Kuhnke <eric.kuhnke@gmail.com<mailto:eric.kuhnke@gmail.com>> wrote: For quite some time, in debian the default configuration for the ntpd.conf that ships with the package for the ntpd is to poll from four different, semi-randomly assigned DNS pool based sources. I believe the same is true for redhat/centos. In the event that one out of four sources is wildly wrong the ntpd will ignore it. If people have routers/networking equipment inside their network that only supports retrieving ntp from one IP address (or hostname) and have manually configured it to request time from a single external source, not their own internal ntpd that is <10ms away, bad things could definitely happen. It is worthwhile to have both polling from external sources via IP as well as GPS sync. Many locations in a network have no hope of getting a GPS signal or putting an antenna with a clear view to the sky, but may be on a network segment that is <4ms away from many other nodes where you can colocate a 1U box and GPS antenna. On Tue, May 10, 2016 at 9:05 PM, Joe Klein <jsklein@gmail.com<mailto:jsklein@gmail.com>> wrote: Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return? The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted. So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum. Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. 6. All core time systems should be monitored by your security team or SOC. One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?". Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, May 10, 2016 at 9:59 PM, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now. My point is, when the fix is so cheap, why put up with this risk at all? -mel beckman On May 10, 2016, at 5:18 PM, Chris Adams <cma@cmadams.net<mailto:cma@cmadams.net>> wrote: Once upon a time, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> said: Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance? IT guy: He changed our clocks. So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy. First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe Random Hacker isn't going to have a high chance of guessing the servers your system is using. Second, he'd have to guess at least three to "win". Third, at best, he'd only be able to change your clocks a little; the common software won't step the clock more than IIRC 15 minutes. Yes, that can cause problems, but not the catastrophes of years in the future or Jan 1, 1970 mentioned in this thread. Is it possible to cause problems? Yes. Is it a practical attack? I'm not so sure, and I haven't seen proof to the contrary. -- Chris Adams <cma@cmadams.net<mailto:cma@cmadams.net>>

Building a S1 system with RaspberryPis would not fly in most of the corporate/enterprise environments I've worked in (random 'appliances', non-uniformity, and lack of support are all glaring issues). Get a PCIe card with a BNC connector and dual power supplies for life in a data center. For home/hobby use a Garmin 18x LVC and any spare compute is a great project: http://www.catb.org/gpsd/gpsd-time-service-howto.html On Wed, May 11, 2016 at 6:47 AM, Dovid Bender <dovid@telecurve.com> wrote:

What about something like this? http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html Has anyone used a Pi to create their own server?

On Wed, May 11, 2016 at 3:24 AM, Mel Beckman <mel@beckman.org> wrote:

...

Regarding Roland’s reference to time and position spoofing via a hacked GPS signal, the hacker has to get physical line of sight to the victim’s antenna in order to succeed with this attack. That’s likely within a few blocks, if not within a few feet. And a rooftop antenna might require a drone attack. And how does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my experience designing data centers, that’s still pretty rare. Many NTP systems use an active GPS antenna that can be hundreds of feet away. But you can always put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You just have to pay for it.

-mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke <eric.kuhnke@gmail.com<mailto: eric.kuhnke@gmail.com>> wrote:

For quite some time, in debian the default configuration for the ntpd.conf that ships with the package for the ntpd is to poll from four different, semi-randomly assigned DNS pool based sources. I believe the same is true for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will ignore it.

If people have routers/networking equipment inside their network that only supports retrieving ntp from one IP address (or hostname) and have manually configured it to request time from a single external source, not their own internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well as GPS sync. Many locations in a network have no hope of getting a GPS signal or putting an antenna with a clear view to the sky, but may be on a network segment that is <4ms away from many other nodes where you can colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein <jsklein@gmail.com<mailto: jsklein@gmail.com>> wrote:

Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum.

Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. 6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?".

Joe Klein "Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman <mel@beckman.org<mailto: mel@beckman.org>> wrote:

I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams <cma@cmadams.net<mailto: cma@cmadams.net>> wrote:

Once upon a time, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> said: Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy.

First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe Random Hacker isn't going to have a high chance of guessing the servers your system is using.

Second, he'd have to guess at least three to "win".

Third, at best, he'd only be able to change your clocks a little; the common software won't step the clock more than IIRC 15 minutes. Yes, that can cause problems, but not the catastrophes of years in the future or Jan 1, 1970 mentioned in this thread.

Is it possible to cause problems? Yes. Is it a practical attack? I'm not so sure, and I haven't seen proof to the contrary. -- Chris Adams <cma@cmadams.net<mailto:cma@cmadams.net>>

-- Miano, Steven M. http://stevenmiano.com

[...] but I would also have doubts over running anything business critical on a RP2.

We use them as reverse terminal servers, for dhcp/tftp bootstrapping other machines, and soon, NTP. They are absolutely rock solid. There's something to be said for "no moving parts inside." --lyndon

But would you not need to actually spend three times $300 to get a good redundant solution? While we are there, why not go all the way and get a rubidium standard with GPS sync? Anyone know of a (relatively) cheap solution with NTP output? https://en.wikipedia.org/wiki/Rubidium_standard Regards, Baldur On 2016-05-11 09:24, Mel Beckman wrote:

Regarding Roland’s reference to time and position spoofing via a hacked GPS signal, the hacker has to get physical line of sight to the victim’s antenna in order to succeed with this attack. That’s likely within a few blocks, if not within a few feet. And a rooftop antenna might require a drone attack. And how does the drone get guidance without a reliable GPS signal? :)

Eric, I agree that sometimes a site can’t get a GPS signal, but in my experience designing data centers, that’s still pretty rare. Many NTP systems use an active GPS antenna that can be hundreds of feet away. But you can always put the $300 NTP server in an outdoor enclosure and power it with PoE.

There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet:

http://www.beaglesoft.com/celsynhowworks.htm

And their $400 WWV-based Stratum 1 time server:

http://www.beaglesoft.com/radsynreceiver.htm

So if you want non-Internet clock diversity, you can have clock diversity. You just have to pay for it.

-mel

On May 10, 2016, at 9:18 PM, Eric Kuhnke <eric.kuhnke@gmail.com<mailto:eric.kuhnke@gmail.com>> wrote:

For quite some time, in debian the default configuration for the ntpd.conf that ships with the package for the ntpd is to poll from four different, semi-randomly assigned DNS pool based sources. I believe the same is true for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will ignore it.

If people have routers/networking equipment inside their network that only supports retrieving ntp from one IP address (or hostname) and have manually configured it to request time from a single external source, not their own internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well as GPS sync. Many locations in a network have no hope of getting a GPS signal or putting an antenna with a clear view to the sky, but may be on a network segment that is <4ms away from many other nodes where you can colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein <jsklein@gmail.com<mailto:jsklein@gmail.com>> wrote:

Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum.

Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. 6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?".

Joe Klein "Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote:

I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all?

-mel beckman

On May 10, 2016, at 5:18 PM, Chris Adams <cma@cmadams.net<mailto:cma@cmadams.net>> wrote:

Once upon a time, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> said: Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance?

IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy.

First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe Random Hacker isn't going to have a high chance of guessing the servers your system is using.

Second, he'd have to guess at least three to "win".

Third, at best, he'd only be able to change your clocks a little; the common software won't step the clock more than IIRC 15 minutes. Yes, that can cause problems, but not the catastrophes of years in the future or Jan 1, 1970 mentioned in this thread.

Is it possible to cause problems? Yes. Is it a practical attack? I'm not so sure, and I haven't seen proof to the contrary. -- Chris Adams <cma@cmadams.net<mailto:cma@cmadams.net>>

Regarding Roland’s reference to time and position spoofing via a hacked GPS signal, the hacker has to get physical line of sight to the victim’s antenna in order to succeed with this attack. That’s likely within a few blocks, if not within a few feet. And a rooftop antenna might require a drone attack. And how does the drone get guidance without a reliable GPS signal? :) Eric, I agree that sometimes a site can’t get a GPS signal, but in my experience designing data centers, that’s still pretty rare. Many NTP systems use an active GPS antenna that can be hundreds of feet away. But you can always put the $300 NTP server in an outdoor enclosure and power it with PoE. There’s always CDMA, GSM, and even WWV for a less-accurate plan B time source. Here’s a somewhat pricey ($700) CDMA gizmo I haven’t investigated yet: http://www.beaglesoft.com/celsynhowworks.htm And their $400 WWV-based Stratum 1 time server: http://www.beaglesoft.com/radsynreceiver.htm So if you want non-Internet clock diversity, you can have clock diversity. You just have to pay for it. -mel On May 10, 2016, at 9:18 PM, Eric Kuhnke <eric.kuhnke@gmail.com<mailto:eric.kuhnke@gmail.com>> wrote: For quite some time, in debian the default configuration for the ntpd.conf that ships with the package for the ntpd is to poll from four different, semi-randomly assigned DNS pool based sources. I believe the same is true for redhat/centos. In the event that one out of four sources is wildly wrong the ntpd will ignore it. If people have routers/networking equipment inside their network that only supports retrieving ntp from one IP address (or hostname) and have manually configured it to request time from a single external source, not their own internal ntpd that is <10ms away, bad things could definitely happen. It is worthwhile to have both polling from external sources via IP as well as GPS sync. Many locations in a network have no hope of getting a GPS signal or putting an antenna with a clear view to the sky, but may be on a network segment that is <4ms away from many other nodes where you can colocate a 1U box and GPS antenna. On Tue, May 10, 2016 at 9:05 PM, Joe Klein <jsklein@gmail.com<mailto:jsklein@gmail.com>> wrote: Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return? The reasons were not fully explained, but the impact was global. Routers, switches, power grids, phone systems, certificates, encryption, Kerberos, logging and any tightly coupled transaction systems were impacted. So I began doing 'security research' on the topic (don't confuse me with joe hacker), and discovered both interesting and terrifying issues, which I will not disclose on an open forum. Needless to say, my suggestions are: 1. Configure a trusted time source and good time stratum architecture for your organization. 2. When identifying your source of time, the majority of the technologies can be DDOS'ed, spoofed or MITM, so consider using redundant sources and authentication. 3. For distribution of time information inside your organization, ensure your critical systems (Encryption, PKI, transactions, etc) are using your redundant sources and authentication. 4. Operating systems, programming languages, libraries, and applications are sensitive to time changes and can fail in unexpected ways. Test them before it's too late. 5. Disallow internal system to seek NTP from other sources beyond your edge routers. 6. All core time systems should be monitored by your security team or SOC. One question, is this a topic anyone would find interested at a future NANOG? Something like "Hacking and Defending time?". Joe Klein "Inveniam viam aut faciam" PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8 On Tue, May 10, 2016 at 9:59 PM, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: I don't pretend to know all the ways a hacker can find out what nap servers a company uses, but I can envision a virus that could do that once behind a firewall. Every ntp response lists the current reference ntp server in the next higher stratum. There are many ways that process could harvest all ntp servers over time, and then pass the public IP back to a mother ship controller. It could be going on right now. My point is, when the fix is so cheap, why put up with this risk at all? -mel beckman On May 10, 2016, at 5:18 PM, Chris Adams <cma@cmadams.net<mailto:cma@cmadams.net>> wrote: Once upon a time, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> said: Boss: So how did a hacker get in and crash our accounting server, break our VPNs, and kill our network performance? IT guy: He changed our clocks. So, this has been repeated several times (with how bad things will go if your clocks get changed by years). It isn't that easy. First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe Random Hacker isn't going to have a high chance of guessing the servers your system is using. Second, he'd have to guess at least three to "win". Third, at best, he'd only be able to change your clocks a little; the common software won't step the clock more than IIRC 15 minutes. Yes, that can cause problems, but not the catastrophes of years in the future or Jan 1, 1970 mentioned in this thread. Is it possible to cause problems? Yes. Is it a practical attack? I'm not so sure, and I haven't seen proof to the contrary. -- Chris Adams <cma@cmadams.net<mailto:cma@cmadams.net>>

On 05/11/2016 12:05 AM, Joe Klein wrote:

Is this group aware of the incident with tock.usno.navy.mil & tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12 years for the period of one hour, then return?

... I remember it like it was only four years ago.... oh, wait.... We have multiple sync sources ourselves, with a Symmetricom (formerly Datum) SSU2000 setup with a cesium PRS, a rubidium secondary, and an ovenized quartz for tertiary oscillators. SSU2000 architecture is separate control and data planes, with time-sync on a different interface from the LAN-facing NTP NIC. And the control plane is firewalled off from the main LAN. An Agilent (now Symmetricom) Z3816 is secondary. PC and SBC (RasPi, etc) oscillators are just not accurate enough for Stratum 1 standards; at best stratum 3 or 4, even when directly GPS-disciplined (stratum is NOT just a synonym for 'level' as a particular stratum really has stability, precision, and accuracy requirements). WWV plus GPS; GPS as you may or may not be aware, is spoofable and is not as accurate as one might want. Neither is WWV. Good reference for time-nuts is, well, the 'time-nuts@febo.com' mailing list. (We're a radio astronomy observatory; accurate time and frequency standards are a must here, especially as position accuracy of radio telescopes approach tens of arcseconds).

Well, if you really want to learn about the NTP servers a target is using you can always just sent them a regular NTP timing query (mode 3) and just read off the IP address in the reference ID field of the response (mode 4). Reference ID reveals the target that the client is sync'd to. If you do this over and over as the client changes the servers it sync's to, you learn all the servers. Or if you are really keen you can use our "kiss-of-death" attack to learn all the servers a client is willing to take time from. See sections V.B-V.C of our paper. https://eprint.iacr.org/2015/1020.pdf Sharon On Wed, May 11, 2016 at 3:07 PM, Florian Weimer <fw@deneb.enyo.de> wrote:

* Chris Adams:

...

First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe Random Hacker isn't going to have a high chance of guessing the servers your system is using.

A determined attacker will just run servers in the official pool.

-- Sharon Goldberg Computer Science, Boston University http://www.cs.bu.edu/~goldbe

Sharon Goldberg writes:

Well, if you really want to learn about the NTP servers a target is using you can always just sent them a regular NTP timing query (mode 3) and just read off the IP address in the reference ID field of the response (mode 4).

Unless the server is an IPv6 server. This trick only works for IPv4. And we have a fix for all of this that will be out soon. -- Harlan Stenn <stenn@ntp.org> http://networktimefoundation.org - be a member!

On Wed, 11 May 2016 21:07:21 +0200, Florian Weimer said:

* Chris Adams:

...

First, out of the box, if you use the public pool servers (default config), you'll typically get 4 random (more or less) servers from the pool. There are a bunch, so Joe Random Hacker isn't going to have a high chance of guessing the servers your system is using.

A determined attacker will just run servers in the official pool.

Such attacks have allegedly been attempted against Tor by certain very well funded adversaries. Thus my statement that if you're seeing that scale attack on your time sources, the fact that your time source is being attacked is the *least* of your problems...

On Wed, 11 May 2016 17:23:31 -0700, Eric Kuhnke said:

average of $150/mo x 500 = $75,000

Id worry more about the fact that somebody is willing to spend $75K/mo to attack me than the fact that it might be possible to wiggle my time base a bit. At that point, you *really* have to worry about other, more productive attacks, such as social engineering (including spear phishing and bribery), or obtaining a 0-day against something in your network. Remember that the FBI has basically admitted that the most effective means of uncloaking a Tor user has been browser 0-days, even though decloaking by pwning a majority of the servers is possible.... (Seriously - if *you* had $75K/mo to attack somebody, what would *you* spend it on?)

On Wed, May 11, 2016 at 03:24:43PM +0000, Jay R. Ashworth wrote:

We're all aware this project is underway, right?

https://www.ntpsec.org/

Despite the name, I'm not aware of any significant protocol changes. It's just a recent fork of the reference implementation minus the refclocks, which isn't particularly helpful if you /don't/ trust network time sources. Long term, be looking at NTS: https://datatracker.ietf.org/doc/draft-ietf-ntp-network-time-security/ In the meanwhile, I'd recommend something along the following lines: - Several nearby upstream servers configured per time server, per site (As diversely as possible.) - Diverse reference clocks (I run everything from WWV to GPS here.) providing authenticated time to your servers. - That all your time servers in all sites be configured in an authenticated full mesh of symmetric peers, allowing the other sites to provide time to a site that has lost its upstream servers or for whatever reason does not trust them at the moment. And of course, ensure any hosts whose clocks you care about are talking to at least a few of these, and preferably several. I know the common case configuration is either default/ntp-pool, or "we have two time servers in this site and everything just chimes from them," but neither is that great of a configuration. --msa

On 5/11/2016 11:24 AM, Jay R. Ashworth wrote:

----- Original Message -----

...

From: "Jared Mauch" <jared@puck.nether.net>

...

...

Yes, and properly monitor your ntpd instances.

And upgrade them.

Some software distributors don’t ship modern software. if you are using a distribution packaged ntpd it’s likely old and difficult to determine its lineage due to how it’s packaged.

If you’re using Redhat based systems consider using chrony instead, even the new beta fedora 24 uses 4.2.6 derived code vs 4.2.8

We're all aware this project is underway, right?

https://www.ntpsec.org/

Speaking of nascent time projects http://nwtime.org/projects/ntimed/ So far, I like the overall architecture of the client/slave/master task differentiation. I've played around with the client in a test environment and ~it seems to work OK~, but it looks like the project is still a bit in the future.