----- Original Message -----

From: "Mark Milhollan" <mlm@pixelgate.net>

Generally speaking, you'll need at least 3 sources if you want stablity.

My usual practice is to set up two in house servers, each of which talks to: time.windows.com time.apple.com and one of the NIST servers 0.us.pool.ntp.org 1.us.pool.ntp.org 2.us.pool.ntp.org and each other. And then point everyone in house to both of them, assuming they accept multiple server names. But I am young, and not much travelled. :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

On Fri, Feb 7, 2014 at 5:35 AM, Saku Ytti <saku@ytti.fi> wrote:

On (2014-02-06 21:14 -0500), Jay Ashworth wrote:

...

My usual practice is to set up two in house servers, each of which talks to: Two is worst possible amount of NTP servers to have. Either one fails and your timing is wrong, because you cannot vote false ticker. And chance of either of two failing is higher than one specific of them.

+1 to having at least 3 NTP servers. Because complete outage is only one kind of failure. Don't forget poor performance due to high latency, or Server X emitting corrupted or inaccurate data -- -JH

On 2/7/2014 3:35 AM, Saku Ytti wrote:

On (2014-02-06 21:14 -0500), Jay Ashworth wrote:

...

My usual practice is to set up two in house servers, each of which talks to:

And then point everyone in house to both of them, assuming they accept multiple server names. Two is worst possible amount of NTP servers to have. Either one fails and your timing is wrong, because you cannot vote false ticker. And chance of either of two failing is higher than one specific of them.

"A man with a watch knows what time it is. A man with two watches is never sure."

On Feb 7, 2014, at 10:56 AM, Matthew Huff <mhuff@ox.com> wrote:

Working in the financial world, the best practices is to have 4 ntp servers (if not using PTP).

1) You need 3 to determine the correct time (and detect bad tickers) 2) If you lose 1 of the 3 above, then you no longer can determine the correct time 3) Therefore with 4, you have redundancy.

We have two Symmetricom Stratum 1 time servers synced via GPS with Rubidium oscillators, and two RHEL 6 servers running ntpd for our 4 servers.

Having a number of NTP servers will help you detect false tickers which may be critical. If you want something that is "cheap" as in you for your home, I can recommend this: ~$350 w/ antenna, etc.. http://www.netburnerstore.com/product_p/pk70ex-ntp.htm You can get the whole thing going quickly. Majdi has also had good luck with this unit (perhaps he wants to chime-in, heh pun unintended) regarding a few other devices. If you ask politely off-list, I will point you at where one of these is that you can talk to (in Dallas at the Infomart for your low-latency config). - Jared

Raspberry Pi ------------------- This unfortunately doest give you trusted time. It gives you David's Raspberry Pi with an Adafruit Ultimate GPS breakout board which is a waste of time if you need an evidence grade of time service. It also means you assemble it and run it yourself. If you need access to NTP - we can handle that ------------------------------------------------------------------- As to how to get NTP into your networks - why screw around??? What do you need - your own VLAN into the back of the switch hosting the NIST ITS server... yeah no problem. Go to the source and join USTiming.ORG and use our landing switch to cross connect your network into a VLAN type management network bringing NIST ITS services to the perimeter of your network - poof - no DDoS, and hey you get to work with us to expand the availability of the services across the US so its a win-win. We have them spread out through the US under USTiming and are looking for more sites that are telco hotels in particular - so if you have space and want to host is in a balance-of-trade type deal let us know. Todd On 2/7/2014 12:32 PM, Anthony Williams wrote:

With a quick and easy mod, another option for $35 is a Sure Electronics GPS board.

GPS: http://www.sureelectronics.net/goods.php?id=99

Mod: http://www.satsignal.eu/ntp/Sure-GPS.htm

-Alby

On 2/7/2014 1:14 PM, Jared Mauch wrote:

...

Having a number of NTP servers will help you detect false tickers which may be critical.

If you want something that is "cheap" as in you for your home, I can recommend this: ~$350 w/ antenna, etc..

-- ------------- Personal Email - Disclaimers Apply

On Fri, Feb 07, 2014 at 01:14:09PM -0500, Jared Mauch wrote:

If you want something that is "cheap" as in you for your home, I can recommend this: ~$350 w/ antenna, etc..

http://www.netburnerstore.com/product_p/pk70ex-ntp.htm

You can get the whole thing going quickly. Majdi has also had good luck with this unit (perhaps he wants to chime-in, heh pun unintended) regarding a few other devices.

The Netburner NTP sample app works well enough for basic home use, although I get better timing performance out of a fleet of hand modified Soekrii. I've been modifying NET4801s to include internal Motorola Oncore timing receivers (this is a tight fit, but doable, in the factory cases), or to break out their second serial port for connections to external reference clocks. (I have one connected to a TrueTime TL-3 to use WWV as a backup to GPS, but it can also be a travelling GPS NTP server with, say, a Garmin GPS18lvc connected.) You can make your own sub-$150 NTP server -- I'll spare the list the details, but those that are interested should see: http://puck.nether.net/~majdi/ntp/ Feedback is appreciated -- I've only spent about an hour on this doc, and it assumes a lot of familiarity with FreeBSD. I will try to flesh it out more as I have time. Cheers, --msa

----- Original Message -----

From: "Saku Ytti" <saku@ytti.fi>

On (2014-02-06 21:14 -0500), Jay Ashworth wrote:

...

My usual practice is to set up two in house servers, each of which talks to:

And then point everyone in house to both of them, assuming they accept multiple server names.

Two is worst possible amount of NTP servers to have. Either one fails and your timing is wrong, because you cannot vote false ticker. And chance of either of two failing is higher than one specific of them.

Fair point. In practice, it never bit me because nearly everything that wanted NTP would only accept one server name (being windows) and the things that *did* take more than one, I generally pointed to both internals, and something outside the firewall as well. In the architecture I described, though, is it really true that the odds of the common types of failure are higher than with only one? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

Best practice is five. =) I don't remember if it's in FAQ on ntp.org or in David Mills' book. Your local clock is kind of gullible "push-over" which will "vote" for the "party" providing most reasonable data. The algorithm would filter out insane sources which run too far from the rest and then group sane sources into 2 "parties" - your clock will follow the one where runners are closer to each other. That is why uneven number of trustworthy sources at least at start is required. With 2 sources you will blindly follow the one which is closer to your own clock. You're also having the the risk to degrade into this situation when you lose 1 out of 3 sources. Four is again 2:2 and only with five you have a good chance to start disciplining your clock into the right direction at the right pace, so when 1 source is lost you (most probably) won't run into insanity. On Sun, Feb 9, 2014 at 9:03 AM, Saku Ytti <saku@ytti.fi> wrote:

On (2014-02-08 19:43 -0500), Jay Ashworth wrote:

...

In the architecture I described, though, is it really true that the odds of the common types of failure are higher than with only one?

I think so, lets assume arbitrarily that probability of NTP server not starting to give incorrect time is 99% over 1 year time. Then either of two servers not giving incorrect time is 0.99**2 i.e. 98%, so two NTP servers would be 1% point more likely to give incorrect time than one over 1 year time.

Obviously the chance of working is more than 99% maybe it's something like 99.999%? And is that really typical failure-mode or is typical failure-mode complete loss of connectivity? Two NTP servers would protect from this, single not. However loss-of-connectivity minor impact on clients, wrong time has major impact of client. Maybe if loss-of-connectivity is fixed in somewhat short period of time, single NTP always win, if loss-of-connectivity is fixed typically in very long period of time, single NTP loses.

I don't really have exact data, but best practice is >2. Matthew said 4, which gives the advantage that in single failure you are still operating redundantly and do not have urgency to fix, with 3 in single failure another failure must not occur before it is fixed. I think 3 is enough, networks are typically designed to handle 1 arbitrary failure at the same time and 2 arbitrary failures in most networks, when chosen correctly, will cause SLA breaking faults (Cheaper to pay SLA compensations than to recover from any 2 failures). But NTP servers are cheap, so if you want to be robust and recover from n false tickers, have 3+n.

-- ++ytti

Unfortunately I don't have the book handy. May be I am wrong too. Just checked and 4 looks to be a valid solution for 1 falseticker according to Byzantine Generals' Problem. On Sun, Feb 9, 2014 at 10:03 PM, Saku Ytti <saku@ytti.fi> wrote:

On (2014-02-09 21:08 +0100), Andriy Bilous wrote:

...

Best practice is five. =) I don't remember if it's in FAQ on ntp.org or in David Mills' book. Your local clock is kind of gullible "push-over" which will "vote" for the "party" providing most reasonable data. The algorithm would filter out insane sources which run too far from the rest and then group sane sources into 2 "parties" - your clock will follow the one where runners are closer to each other. That is why uneven number of trustworthy sources at least at start is required. With 2 sources you will blindly follow the one which is closer to your own clock. You're also having the the risk to degrade into this situation when you lose 1 out of 3 sources. Four is again 2:2 and only with five you have a good chance to start disciplining your clock into the right direction at the right pace, so when 1 source is lost you (most probably) won't run into insanity.

I'm having bit difficulties understanding the issue with 4.

Is the implication that you have two groups which all agree with each other reasonably well, but do not agree between the groups. Which would mean that 4 cannot handle situation where 2 develop problem where they agree with each other but are wrong. But even in that case, you'd still recover from 1 of them being wrong. So

3 = correct time, no redundancy 4 = correct time, 1 can fail 5 = correct time, 2 can fail and so forth?

But not sure here, just stabbing in the dark. For the fun of it, threw email to Mills, if he replies, I'll patch it back here.

-- ++ytti

On (2014-02-09 15:16 -0500), Jay Ashworth wrote:

...

Then either of two servers not giving incorrect time is 0.99**2 i.e. 98%, so two NTP servers would be 1% point more likely to give incorrect time than one over 1 year time.

That's only true if the two devices have common failure modes, though, is it not?

No, we can assume arbitrary fault which causes NTP to output bad time. With two NTP servers it's more likely that any one of them will start doing that than with one alone. And if any of the two start doing it, you don't know which one. -- ++ytti

On (2014-02-09 15:45 -0500), Jay Ashworth wrote:

If I'm locked to 2 coherent upstreams and one goes insane, I'm going to know which one it is, because the other one will still match what I already have running, no?

Or do I understand NTP less well than I think?

I don't think you can reasonably tell which of the two is the false ticker. Andriy says your PC would blindly follow one who is in more agreement with your local lock, and PC's have terrible oscillators (I don't know why, 5EUR would buy LOT better oscillator). -- ++ytti

On Sun, Feb 9, 2014 at 2:45 PM, Jay Ashworth <jra@baylink.com> wrote: [snip]

If I'm locked to 2 coherent upstreams and one goes insane, I'm going to know which one it is, because the other one will still match what I already have running, no?

The question should be how assured is the reliability of the clocks of the 2 upstream servers. I think I am pretty happy with the concept of having two local centralized NTP servers, used by various servers in an environment ---- some SNTP some NTP, each of the local centralized NTP servers using 5 external time sources. These external time sources need to be periodically checked, to ensure the central NTP servers continue to synchronize with them, and that they continue to be accurate. So the pair of NTP servers is not redundant in the sense that the time is allowed to be wrong, but they are resilient in the sense of being configured, so their own clock should always be correct, unless there is a once in 100 years failure scenario. Each of the local servers, then has two NTP peers as time source, and the local clock discipline, except for virtual machines: which should use just the two NTP servers. A local pair of NTP servers are not "redundant" in the sense of being able to survive a catastrophic software bug in NTP; the local time sources should be redundant to survive the more highly frequent condition of temporary total failure of a local NTP server.

Or do I understand NTP less well than I think?

Cheres, -- jra

-- -JH