What sort of products are people using to provide timing services to third parties in datacenters? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "James Harrison" <james@talkunafraid.co.uk> To: nanog@nanog.org Sent: Wednesday, May 1, 2019 5:27:38 PM Subject: Re: NTP question On 01/05/2019 20:29, Job Snijders wrote:

The trick is to order a spot on the roof of the datacenter, have the facility staff place the antenna there, and run a cable to the NTP server in your rack.

Some DCs also offer GPS antenna feeds fed from a splitter, though it's important to get the total cable length from the antenna to your receiver so you can set your propagation delay offset accordingly. I've also been in facilities that distribute IRIG and 10MHz references so you can feed a reference directly, but that's fairly rare. It's worth asking what your facilities can provide, in either case. Many DCs don't want a dozen GPS antennae cluttering the roof up but are happy to provide the service from one they look after (for a cost, of course). If you have external facilities, of course, so long as you can run PTP/1588 back from them, you can always host your clocks there and distribute to 1588 masters in the DC. -- Cheers, James Harrison

thank you guys, looks like GPS based NTP is the way to go. On Wed, May 1, 2019 at 3:36 PM Bryan Fields <Bryan@bryanfields.net> wrote:

On 5/1/19 3:22 PM, Mehmet Akcin wrote:

...

hey there Nanog,

I am trying to buy a GPS based NTP server like this one

https://timemachinescorp.com/product/gps-time-server-tm1000a/

but I will be placing this inside a data center, do these need an actual view of a sky to be able to get signal or will they work fine inside a data center building?

You will need a clear view to the sky for at least the antenna.

Most GPS "antennas" are an antenna and Low Noise Amplifier (LNA) which is powered via 5-12v on the coax. This sets the noise figure and gain of the system, so you can run 50-100' of RG6 coax if needed. You'll need a F to sma adapter for this unit it looks like. Don't worry about the impedance mismatch, 50 to 75 ohm is not horrid, the RG-174 thin cable has more loss in 10' than 100' of RG6.

You will not want to use the low gain puck antenna, but rather get a proper grounded/mounted/weatherproofed antenna such as the ubiquitous 26 dBi Quadrifilar Helix antenna. https://www.ebay.com/itm/192899151132

-- Bryan Fields

727-409-1194 - Voice http://bryanfields.net

Hi, On Wed, May 01, 2019 at 02:01:44PM -0600, Brielle Bruns wrote:

If you can't get a good spot for an antenna, you could be on the lookout for a CDMA NTP clock.

CDMA service is about to be retired in several places, please check in your area before you install a "new" CDMA based time server. C.f. https://www.verizonwireless.com/support/knowledge-base-218813/ I looked into the same thing and decided not to go with CDMA. A simple check inside a (datacenter) building is to use one of the GPS smart phone apps that display you number of Sats and signal strength then walk around where you would place the NTP server appliance. Beware of server CPUs and memory making RF noise in the same frequency spectrum of 1.2 - 2 GHz, completely blanking out any GPS indoors. I concur that installing an amplified roof-top antenna and running coax to your receiver is the best option. -andreas -- Andreas Ott K6OTT +1.408.431.8727 andreas@naund.org

On 5/1/2019 2:50 PM, Andreas Ott wrote:

...

If you can't get a good spot for an antenna, you could be on the lookout for a CDMA NTP clock. CDMA service is about to be retired in several places, please check in your area before you install a "new" CDMA based time server. C.f.https://www.verizonwireless.com/support/knowledge-base-218813/

I looked into the same thing and decided not to go with CDMA.

There's actually a few other CDMA networks in our area (Boise) besides Verizon, so it wouldn't hurt to look. I seem to remember Sprint is planning to go to 2021? There also appears to be a few smaller independent CDMA networks around as well. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org

Kinda sucks all the good 'backup' methods of time keeping are dwindling. I've got a WWVB clock as well that I'd love to get hooked into my main NTP server, but I worry they're going to finally kill that off in the next year or so. LORAN C clocks still have potential to work well too... High accuracy time keeping is a fun hobby. :) On 5/1/2019 3:05 PM, Mike Hammett wrote:

I looked before at who had spectrum allocations in the frequencies my boxes supported. I then used Cell Mapper to figure out what technology was deployed on that frequency. IIRC, both US Cellular and Verizon had basic CDMA running in my area on those channels. Sprint was running LTE and 1x Advanced (or something like that), so probably wouldn't have worked out. If Verizon is dropping theirs, then depending on only one company seems a bit unwise....  which means I gotta find some kind of solution by then. *sigh*

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com

Midwest-IX http://www.midwest-ix.com

------------------------------------------------------------------------ *From: *"Brielle Bruns" <bruns@2mbit.com> *To: *nanog@nanog.org *Sent: *Wednesday, May 1, 2019 3:58:57 PM *Subject: *Re: NTP question

On 5/1/2019 2:50 PM, Andreas Ott wrote:

...

...

If you can't get a good spot for an antenna, you could be on the lookout for a CDMA NTP clock. CDMA service is about to be retired in several places, please check in your area before you install a "new" CDMA based time server. C.f.https://www.verizonwireless.com/support/knowledge-base-218813/

I looked into the same thing and decided not to go with CDMA.

There's actually a few other CDMA networks in our area (Boise) besides Verizon, so it wouldn't hurt to look.  I seem to remember Sprint is planning to go to 2021?  There also appears to be a few smaller independent CDMA networks around as well.

-- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org    /     http://www.ahbl.org

-- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org

On Wed, May 01, 2019 at 02:35:58PM -0700, Harlan Stenn wrote:

- Why do folks want to have one or more NTP server masters that have at least 1 refclock on them in a data center, instead of having their data center NTP server masters that only get time over the internet?

I had that discussion before with the QSA for a compliance audit, pointing to requirement "10.4.3 Time settings are received from industry-accepted time sources" and "verify that the time server(s) accept time updates from specific, industry-accepted external sources (to prevent a malicious individual from changing the clock)" in the PCI-DSS document. He non-jokingly suggested "why don't you use pool.ntp.org?", not really realizing how many servers are in fact just someone's PC behind a cable modem in their home, which negated the "do I trust the time I am receiving?". My immediate answer was "we could use NIST servers", but the easiest way out of this is "we operate our own NTP appliance with a GPS receiver" and provide that as evidence. Don't get me wrong, I support pool.ntp.org by operating and contributing servers to it, but it is not deemed good enough if you need traceability of your NTP time source(s), even though the pool will only admit members above a certain quality threshold.

- What % of data center operators provide time servers in their data centers for their tenants (or for the general public)?

My $employer does that in our datacenters and points of presence for our customers. -andreas -- Andreas Ott K6OTT +1.408.431.8727 andreas@naund.org

for our PCI-DSS audit, the rational for at least -one- local source, instead of depending on pool.ntp.org, was "backhoe fade". it was worth the $135 for an NTP source using GPS. the cable run up the elevator shaft for the antenna works without needing OSHPD permits. We are very happy with the result. /Wm On Wed, May 1, 2019 at 3:01 PM Andreas Ott <andreas@naund.org> wrote:

On Wed, May 01, 2019 at 02:35:58PM -0700, Harlan Stenn wrote:

...

- Why do folks want to have one or more NTP server masters that have at least 1 refclock on them in a data center, instead of having their data center NTP server masters that only get time over the internet?

I had that discussion before with the QSA for a compliance audit, pointing to requirement "10.4.3 Time settings are received from industry-accepted time sources" and "verify that the time server(s) accept time updates from specific, industry-accepted external sources (to prevent a malicious individual from changing the clock)" in the PCI-DSS document. He non-jokingly suggested "why don't you use pool.ntp.org?", not really realizing how many servers are in fact just someone's PC behind a cable modem in their home, which negated the "do I trust the time I am receiving?". My immediate answer was "we could use NIST servers", but the easiest way out of this is "we operate our own NTP appliance with a GPS receiver" and provide that as evidence.

Don't get me wrong, I support pool.ntp.org by operating and contributing servers to it, but it is not deemed good enough if you need traceability of your NTP time source(s), even though the pool will only admit members above a certain quality threshold.

...

- What % of data center operators provide time servers in their data centers for their tenants (or for the general public)?

My $employer does that in our datacenters and points of presence for our customers.

-andreas -- Andreas Ott K6OTT +1.408.431.8727 andreas@naund.org

On Thu, 02 May 2019 08:59:19 -0400, Tom Beecher said:

Passes the backhoe test, but might have an issue with the Die Hard Elevator Shaft Fight Scene checks.

If your data center is suffering from both backhoe face and a Die Hard Fight Scene, the *real* question is whether you're going to care about NTP when the Halon dumps and the emergency power interlock shuts down all your hardware... In other words, you got bigger problems. :)

On 5/2/19 8:03 AM, Kain, Rebecca (.) wrote:

Or the fbi shuts off the power grid

Na. Battery backup and generators with days ~> weeks worth of fuel. }:-) -- Grant. . . . unix || die

On 5/2/19 9:32 AM, Alain Hebert wrote:

Unless the Firemen turn your roof generator off because someone in the street yelled fire =D

The firemen & women that I've had the pleasure of working with did have more brains than that. Despite their reputation of brute force, they do think. -- Grant. . . . unix || die

Accurate timing is also often required for telco gear. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Harlan Stenn" <stenn@nwtime.org> To: nanog@nanog.org Sent: Wednesday, May 1, 2019 4:35:58 PM Subject: Re: NTP question So I gotta ask, just as a reality check: - Why do folks want to have one or more NTP server masters that have at least 1 refclock on them in a data center, instead of having their data center NTP server masters that only get time over the internet? - What % of data center operators provide time servers in their data centers for their tenants (or for the general public)? -- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!

On Wed, May 1, 2019, at 18:46, Brandon Martin wrote:

Think about what might happen if you lost time sync as a result of the incident causing said connectivity outage. Depending on your time sources available, you might see rapid drift or, worst case, lose your time reference entirely as a result of equipment restarts, etc. GPS, as long as you have a good view of the sky, provides extremely accurate "lights out" time info, both absolute and relative, from a single source with no (mostly) strings attached for that purpose.

Properly deployed NTP should calibrate the local hardware clocks to prevent drift even during connectivity outages. (I'm talking both the low resolution hardware clocks used for timing across power cycles and reboots, and the oscillators used while the OS is running). While most computer hardware is temperature sensitive, if your datacenter is suddenly changing temperature enough to cause clock drift, well, you have bigger problems. :) I admit that this is an anecdote, but in our environment, I find that our GPSDO loses its GPS signal due to weather more often than we lose our connections to internet NTP servers. On the other hand, we once had a site-wide Kerberos authentication outage because all of our Windows clients were using some windows NTP client that by default used two NTP sources owned by the software developer; when they both suddenly stepped by 20 minutes, Kerberos locked everyone out. Time is hard :) -- Harald Koch chk@pobox.com

On 5/1/2019 4:17 PM, Brandon Martin wrote:

On 5/1/19 7:03 PM, Harald Koch wrote:

...

Properly deployed NTP should calibrate the local hardware clocks to prevent drift even during connectivity outages. (I'm talking both the low resolution hardware clocks used for timing across power cycles and reboots, and the oscillators used while the OS is running). While most computer hardware is temperature sensitive, if your datacenter is suddenly changing temperature enough to cause clock drift, well, you have bigger problems.:)

For sure, sudden loss of time "shouldn't" happen, but having a local refclock is comparatively cheap insurance against it in many deployments.

BCP these days is "orphan mode", not "local refclock".

I've seen things like this when there's a sudden power loss across a small site e.g. a remote PoP.  Think a loss of utility power and UPS fails to transfer for some unanticipated reason.  Everything will come back up when either the utility power comes back or generator spins up, but it will all be hard reset.  Depending on your NTP implementation, the local hardware clock may not be particularly accurate.  Even good implementations often lack the necessary hardware capabilities to trim the low-resolution hardware reference and have to resort to simply flushing the time to hardware every so often.

Relative inaccuracies of a few seconds are pretty normal in that kind of situation in my experience.  Putting everything together from logs where there's an unknown time offset of a few seconds after the fact can be tough.  Then again, maybe you don't care in this example case since the cause of the problem is proximate - the frigging UPS didn't do its job.  More complex scenarios might be easily envisioned, though.

Now, obviously you've still got an issue of the fact that the GPS refclk will take a while to lock and start serving time, but at least you've potentially got known-good time info before you start bringing higher-level network protocols up (and can purposely delay until you do, if desired) which is potentially impossible if your only source of time is the network itself.

Ah, this is the dance with "have enough sources of time"... -- Harlan Stenn, Network Time Foundation http://nwtime.org - be a Member!

On Wednesday, 1 May, 2019 15:36, Harlan Stenn <stenn@nwtime.org> wrote:

So I gotta ask, just as a reality check:

- Why do folks want to have one or more NTP server masters that have at least 1 refclock on them in a data center, instead of having their data center NTP server masters that only get time over the internet?

That entirely depends on what you need the time for. For example, in a Continuous Control environment you really do not care about the accuracy of the time -- just like a printer will not suddenly fail to print documents with dates in them because of Y2K, the printer neither cares nor knows what time it is. What you may care about, however, is that all your Distributed Control and Outboard Systems have the SAME TIME and that that time, relative to each other, is closely synchronized. This has a huge impact when comparing log events from one system to another. What is important is that they all have the same time, and that they all drift together. If you have one such installation, then you really do not care about the "accuracy" of the time. However if you have multiple such installations then you want them all to have the same time (if you will be comparing logs between them, for example). At some point it becomes "cheaper" to spend thousands of dollars per site to have a single Stratum 0 timesource (for example, the GPS system) at each site (and thus comparable time stamps) than it is to pay someone to go though the rigamarole of computing offsets and slew rates between sites to be able to do accurate comparison. And if you communicate any of that info to outsiders then being able to say "my log timestamps are accurate to +/- 10 nanoseconds so it must be you who is farked up" (and be able to prove it) has immense value. --- The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.

On Wed, May 1, 2019 at 9:56 PM William Herrin <bill@herrin.us> wrote:

On Wed, May 1, 2019 at 5:48 PM Keith Medcalf <kmedcalf@dessus.com> wrote:

...

If you have one such installation, then you really do not care about the "accuracy" of the time. However if you have multiple such installations then you want them all to have the same time (if you will be comparing logs between them, for example). At some point it becomes "cheaper" to spend thousands of dollars per site to have a single Stratum 0 timesource (for example, the GPS system) at each site (and thus comparable time stamps) than it is to pay someone to go though the rigamarole of computing offsets and slew rates between sites to be able to do accurate comparison. And if you communicate any of that info to outsiders then being able to say "my log timestamps are accurate to +/- 10 nanoseconds so it must be you who is farked up" (and be able to prove it) has immense value.

If your network is air gapped from the Internet then sure. If it's not, you can run NTP against a reasonably reliable set of time sources (not random picks from Pool) and be able to say, "my log timestamps are accurate to +/- 10 milliseconds so it must be you who is farked up." While my milliseconds loses the pecking order contest, it's just as good for practical purposes and a whole lot less expensive.

And while time source stability is a good criteria, the most important NTP criteria is path latency symmetry between directions. It's better to have a path that is 100 ms of 1-way latency both ways than a path that is 1 ms one way, 100 ms the other way. Rubens

Hi Keith, On 5/1/19 6:17 PM, Keith Medcalf wrote:

...

If your network is air gapped from the Internet then sure. If it's not, you can run NTP against a reasonably reliable set of time sources (not random picks from Pool) and be able to say, "my log timestamps are accurate to +/- 10 milliseconds so it must be you who is farked up." While my milliseconds loses the pecking order contest, it's just as good for practical purposes and a whole lot less expensive.

You mean something like this, which is relatively easy to achieve:

============================================================================== offset -0.000009, frequency -0.823, time_const 30, watchdog 238 synchronised to NTP server (192.5.41.40) at stratum 2 time correct to within 12 ms polling server every 1024 s ============================================================================== remote refid st t when poll reach delay offset jitter ============================================================================== +clock.sjc.he.ne .CDMA. 1 u 287 1024 377 64.313 0.337 0.867 -tock.usnogps.na .IRIG. 1 u 5 1024 377 103.080 -2.097 0.316 -tick.usnogps.na .IRIG. 1 u 806 1024 377 103.053 -2.328 0.363 +india.colorado. .NIST. 1 u 270 1024 377 41.214 -0.159 0.113 +time-b-b.nist.g .NIST. 1 u 984 1024 377 42.609 0.200 0.045 +time-c-b.nist.g .NIST. 1 u 180 1024 377 42.563 0.201 0.064 +time-a-b.nist.g .NIST. 1 u 163 1024 377 42.639 0.137 0.032 *192.5.41.40 .PTP. 1 u 235 1024 377 12.756 -0.388 12.479 -192.5.41.41 .IRIG. 1 u 312 1024 377 13.575 -1.172 2.425 LOCAL(0) .LOCL. 10 l - 64 0 0.000 0.000 0.000 ------------------------------------------------------------------------------ pll offset: -8.474e-06 s pll frequency: -0.823 ppm maximum error: 0.123149 s estimated error: 0.000122 s status: 2001 pll nano pll time constant: 10 precision: 1e-09 s frequency tolerance: 500 ppm ==============================================================================

That all looks great except for the LOCAL clock at S10. In the event you lose connectivity to the outside, this system will jump from S2 to S10. Depending on the setup of your other systems, groups of them will go sailing off in their own directions. http://support.ntp.org/bin/view/Support/OrphanMode is the better solution. If you cannot do that for some reason, please see the "Dual Time Servers" case at http://support.ntp.org/bin/view/Support/UndisciplinedLocalClock . -- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!

Stephen, LOL. That’s not a real problem with today’s microprocessors. The TM1000A, for example: “...is capable of serving 135+ synchronizations per second. That provides support for over 120,000+ devices updating every 15 minutes on the network.” As for ARP traffic deluges, if that’s happening on your LAN, you have bigger problems :) -mel

On May 1, 2019, at 6:21 PM, Stephen Satchell <list@satchell.net> wrote:

One word of caution when using a low-priced NTP appliance: your network activity could overwhelm the TCP/IP stack of the poor thing, especially if you want to sync your entire shop to it. In the case of the networks I set up, I set up a VLAN specific to the NTP appliance and to the two servers that sync up with it. Everything else in the network is configured to talk to the two servers, but NOT on the three-device "NTP Appliance VLAN".

NOTE: Don't depend on the appliance to provide VLAN capability; use a configuration in a connected switch. How you wire from the appliance to a port on your network leaves you with a lot of options to reach a window with good satellite visibility, as CAT 5 at 10 megabits/s can extend a long way successfully. Watch your cable dress, particularly splices and runs against metal. (Or through rooms with MRI machines -- I'm not joking.)

The two servers in question also sync up with NTP servers in the cloud using whatever baseband or VLANs (other than the "NTP VLAN") you configure. Ditto clients using the two servers as time sources.

The goal here is to minimize the amount of traffic in the "NTP Appliance VLAN". What killed one installation I did was the huge amount of ARP traffic that the appliance had to discard; it wasn't up to the deluge.

Learn from my mistakes.

On Wed, May 1, 2019 at 7:03 PM Harlan Stenn <stenn@nwtime.org> wrote:

It's not clear to me that there's anything *wrong* with using the pool, especially if you're using our 'pool' directive in your config file.

The one time I relied on the pool I lost sync a year later when all three servers the configuration picked withdrew time services and the still-running ntp client didn't return to the names to find new ones. Wonderful if that's fixed now but the pool folks argued just as strongly for using it back then. Also, telling the security auditor that you have no idea who supplies your time source is pretty much a non-starter. You can convince them of a lot of things but you can't convince them it's OK to have no idea where critical services come from. That's what's wrong with the pool. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Dirtside Systems ......... Web: <http://www.dirtside.com/>

On May 2, 2019, at 10:59 AM, William Herrin <bill@herrin.us> wrote:

On Wed, May 1, 2019 at 7:03 PM Harlan Stenn <stenn@nwtime.org <mailto:stenn@nwtime.org>> wrote: It's not clear to me that there's anything *wrong* with using the pool, especially if you're using our 'pool' directive in your config file.

The one time I relied on the pool I lost sync a year later when all three servers the configuration picked withdrew time services and the still-running ntp client didn't return to the names to find new ones. Wonderful if that's fixed now but the pool folks argued just as strongly for using it back then.

Also, telling the security auditor that you have no idea who supplies your time source is pretty much a non-starter. You can convince them of a lot of things but you can't convince them it's OK to have no idea where critical services come from.

That's what's wrong with the pool.

Regards, Bill Herrin

-- William Herrin ................ herrin@dirtside.com <mailto:herrin@dirtside.com> bill@herrin.us <mailto:bill@herrin.us> Dirtside Systems ......... Web: <http://www.dirtside.com/ <http://www.dirtside.com/>>

I have only ever used the pool as a supplement to other servers. Here is a snippet from ntp.conf that was found in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard.’ * # External Time Synchronization Source Servers # server tick.usno.navy.mil # open access server time.apple.com # open access server Time1.Stupi.SE # open access server ntps1-0.uni-erlangen.de # open access server 0.pool.ntp.org # open access server 1.pool.ntp.org # open access server 2.pool.ntp.org # open access server nist1-nj2-ustiming.org # open access server nist1-chi-ustiming.org # open access server nist1-pa-ustiming.org # open access # I have not kept up with pool changes since then. *Apologies to Douglas Adams

On May 2, 2019, at 2:44 PM, Harlan Stenn <stenn@nwtime.org> wrote:

On 5/2/2019 9:13 AM, James R Cutler wrote:

...

...

On May 2, 2019, at 10:59 AM, William Herrin <bill@herrin.us <mailto:bill@herrin.us>> wrote:

On Wed, May 1, 2019 at 7:03 PM Harlan Stenn <stenn@nwtime.org <mailto:stenn@nwtime.org>> wrote:

It's not clear to me that there's anything *wrong* with using the pool, especially if you're using our 'pool' directive in your config file.

The one time I relied on the pool I lost sync a year later when all three servers the configuration picked withdrew time services and the still-running ntp client didn't return to the names to find new ones. Wonderful if that's fixed now but the pool folks argued just as strongly for using it back then.

Also, telling the security auditor that you have no idea who supplies your time source is pretty much a non-starter. You can convince them of a lot of things but you can't convince them it's OK to have no idea where critical services come from.

That's what's wrong with the pool.

Regards, Bill Herrin

-- William Herrin ................ herrin@dirtside.com <mailto:herrin@dirtside.com> bill@herrin.us <mailto:bill@herrin.us> Dirtside Systems ......... Web: <http://www.dirtside.com/>

I have only ever used the pool as a supplement to other servers. Here is a snippet from ntp.conf that was found in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard.’ *

#External Time Synchronization Source Servers # servertick.usno.navy.mil# open access servertime.apple.com <http://time.apple.com># open access serverTime1.Stupi.SE# open access serverntps1-0.uni-erlangen.de <http://ntps1-0.uni-erlangen.de># open access server0.pool.ntp.org <http://0.pool.ntp.org># open access server1.pool.ntp.org <http://1.pool.ntp.org># open access server2.pool.ntp.org <http://2.pool.ntp.org># open access

I recommend you replace the above 3 lines with:

pool CC.pool.ntp.org

where CC is an appropriate country code or region.

H --

...

servernist1-nj2-ustiming.org <http://nist1-nj2-ustiming.org># open access servernist1-chi-ustiming.org <http://nist1-chi-ustiming.org># open access servernist1-pa-ustiming.org <http://nist1-pa-ustiming.org># open access #

I have not kept up with pool changes since then.

*Apologies to Douglas Adams

-- Harlan Stenn, Network Time Foundation http://nwtime.org - be a Member!

Harlan, That is good advice. Company($dayjob) no longer exists, but I will remember your advice next time I configure 4 or more Mac minis as an NTP peer group in my home office lab — I let the last configuration lapse as keeping up with Apple hardware and macOS changes was challenge enough and I no longer supported Network Time Services for any $dayjob or client. The only other note is that, for Company($dayjob), I obtained explicit permission from each of a set of globally distributed time services (not shown above). I recommend that any new NTP peer group be configured with as diverse a set of servers as possible, not limited to just pool and not limited to a single connection type. Thank you. Jim - James R. Cutler James.cutler@consultant.com GPG keys: hkps://hkps.pool.sks-keyservers.net

Brielle Bruns <bruns@2mbit.com>:

I've got a WWVB clock as well that I'd love to get hooked into my main NTP server, but I worry they're going to finally kill that off in the next year or so.

Alas, your WWVB clock is probably already almost useless except as a wall decoration. The modulation of the subsecond part of the WWVB signal changed in 2012. If your clock is older than that, the best it can still do is pick up the low-precision per-second tick. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

On 5/1/2019 6:12 PM, Mike Hammett wrote:

Anyone know of a solution that doesn't require an external antenna, is NEBS compliant, and has T1-type outputs for me to hook into my Metaswitch gear?

You forgot 'world peace' in there too. :) -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org

On Thu, 2 May 2019, Carsten Bormann wrote:

Why don’t data centers provide a GPS signal along with power and air conditioning? Installing a distribution amplifier for 1.5 GHz is not rocket science.

(Or an Ethernet with IEEE1588 precise time, but that is probably asking too much.)

They should :-) I tried to include time (i.e. Buiding Integrated Timing System) as part of the basic data center services (hvac, power, access control, etc) when I worked at Equinix many, many years ago. Your data center operator can install its GPS (or other time source) antennas, drive the building master clock, and distribute time to customers using several different protocols. For folks with firewall/security concerns, the building master clock can drive non-Internet protocols (IRIG-B, IEE1588 PTP, etc) or connections in addition to NTP. You can still have your own NTP server. The difference is instead of a GPS antenna connection, your clock box uses the BITS connection as one of the time sources. Unfortunately, I was ahead of my time and customers (and sales people) didn't really understand the advantages. Yes, the DC operator can screw up the BITS just like the DC operator can screw up the power, hvac and access control systems. Everyone wanted a separate GPS antenna, and the sales people made more commission selling space on the antenna platform :-(

On May 1, 2019, at 16:53, Mel Beckman <mel@beckman.org> wrote:

It’s hard to consider messing with signal converters and pricey remotely-powered active antennas when you can solve the problem for $300. :)

As I said, it really depends on your requirements and expectations. :-) For my “normal” use cases there hasn’t been room for a lot of stuff between “well run NTP server with networked time source” and “server with fancy clocks and frequency input”. Though, on the topic of unusual requirements there are a bunch of contributors to the NTP Pool using this curious device that can do line rate NTP responses (100Mbps, but still): https://store.uputronics.com/index.php?route=product/product&product_id=92 Ask

On 5/1/19 4:53 PM, Mel Beckman wrote:

Ask,

But with a small compact server like the DC-powered TimeMachines Inc unit, which costs something like $300, you simply put the server where the visibility is and connect back to the nearest Ethernet port in your network, up to 300’ away, or virtually any distance with fiber transceivers. We’ve installed these in Cantex boxes on a windy, rainy tenth-story rooftop in upstate NY and it runs flawlessly, warmed by its own internal heat at sub-zero temps, and perfectly happy at ambient temps of 110F.

It’s hard to consider messing with signal converters and pricey remotely-powered active antennas when you can solve the problem for $300. :)

I sure hope you have ntpd set up to peer or get time with enough other servers. H --

-mel

...

On May 1, 2019, at 4:44 PM, Ask Bjørn Hansen <ask@develooper.com> wrote:

...

On May 1, 2019, at 12:22, Mehmet Akcin <mehmet@akcin.net> wrote:

I am trying to buy a GPS based NTP server like this one

https://timemachinescorp.com/product/gps-time-server-tm1000a/

but I will be placing this inside a data center, do these need an actual view of a sky to be able to get signal or will they work fine inside a data center building? if you have any other hardware requirements to be able to provide stable time service for hundreds of customers, please let me know.

[ with my hobby-hat on … ]

tl;dr: if any of the below is too much work, just run reasonably well monitored NTP server syncing from other NTP servers. If you want more than that, you need to see the sky. Don’t do the CDMA thing.

Depending on your requirements having the antenna in the window may or may not be satisfactory. If it’s fine you probably could just have done a regular NTP server in the first place. For long swaths of the day you might not see too many satellites which will add to the uncertainty of the signal.

Meinberg’s GPS antenna has a bit more smarts which helps it work on up to 300 meters on RG58 or 700 meters on RG213. (They also have products that use regular L1 antennas with the limitations Bryan mentioned).

https://www.meinbergglobal.com/english/products/gps-antenna-converter.htm

They also have a multi-mode fiber box to have the antenna be up to 2km from the box or 20km with their single mode fiber box, if you have fiber to somewhere else where you can see the sky and place an antenna.

It will be more than the one you linked to, but their systems are very reasonably priced, too. For “hundreds of customers” whatever is the smallest/cheapest box they have will work fine. Even their smallest models have decent oscillators (for keeping the ticks accurate between GPS signals).

The Meinberg time server products (I am guessing all of them, but I’m not sure) also have a mode where they poll an upstream NTP server aggressively and then steer the oscillator after it. I haven’t used it in production, but it worked a lot better than it sounded like it would. (In other words, even without GPS it’s a better time server than most systems).

Ask

-- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!

Yo Mel! On Thu, 2 May 2019 02:54:25 +0000 Mel Beckman <mel@beckman.org> wrote:

Why? The GPS NTP Server is Stratum-1. If it fails computer clocks will freewheel for hours or days before losing significant time, during which period you can simply order a replacement unit. If that isn’t fast enough, buy two $300 boxes. The “consensus” issue is moot, since a GPS server gets a consensus of clock time from the GPS satellite constellation.

I guess you slept through GPS Week Roll Over day last April 6th? Some GPS went nuts, others did not. Many 777 and 787 were grounded that weekend for software updates to their expensive Honeywell GPS. I'll spare you the many more examples that hapened. Not nice when yoar clock rolls back to 1999, or forward to 2035. RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin

Yo Mel! On Thu, 2 May 2019 03:30:03 +0000 Mel Beckman <mel@beckman.org> wrote:

I’m also an FAA licensed A&P mechanic, and have worked for airlines in fleet maintenance. Air carriers have extremely thorough systems reviews, by law, through the Airworthiness Directive program, which started identifying 2019 GPS rollover vulnerabilities in ... 2009! Nobody was surprised. If any GPS systems “went nuts”, it was through the incompetence and negligence of their owners.

How many GPS owners happen to have $30,000 GPS simulators to check their $300 GPS/NTP servers? Some of mine did, most did not. Seems to me the negligence is in the GPS manufacturer that failed to notify their customers. To be fair, Avidyne and Telit did notify their customers, but not with a fix or enough lead time to swap out the units. RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin

For those wondering what a GPS certification letter for the rollover bug looks like, here’s Garmin’s. Note the phrase “for many years, Garmin has anticipated and prepared for this event...”: Garmin GPS Week Number Rollover Statement What is the GPS Week Number Rollover (WNRO)? The GPS system is world renowned for its ability to provide accurate and reliable positioning and timing information worldwide. The GPS satellites transmit to users the date and time accurate to nanoseconds. However, back in 1980, when the GPS system first began to keep track of time, the date and time was represented by a counter that could only count forward to a maximum of 1024 weeks, or about 19.7 years. After 1024 weeks had elapsed, this counter “rolled over” to zero, and GPS time started counting forward again. This first rollover occurred in August of 1999. The second rollover will occur on April 6, 2019. Is My Device Affected? For many years, Garmin has anticipated and prepared for this event. Regardless, Garmin has been performing exhaustive testing of current and legacy devices to determine if they will be affected by the GPS week number rollover. Our testing shows the vast majority of Garmin GPS devices will handle the WNRO without issues. What is the Effect of a GPS Week Number Rollover Issue? For GPS devices that are affected, after the rollover occurs, an incorrect date and time will be displayed. This incorrect time will also be used to timestamp track logs, compute sunrise and sunset, and other functions that rely upon the correct date and time. However, the positioning accuracy will not be affected. The device will continue to deliver the same positioning performance as before the rollover. -mel On May 1, 2019, at 8:56 PM, Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: Gary, Gary, Gary, You don’t need a $30,000 GPS simulator to verify if a GPS product in your inventory has the rollover bug. You simply ask the supplier to certify that they don’t have the rollover bug. They use their _$100,000_ GPS simulator If needed, but usually it’s done with a trivial code review. If the supplier can’t provide such a certification, then they are no longer a supplier. This tends to persuade them to certify. If you as an air carrier (or any other critical GPS consumer) fail to ask for such a certification in time to field a replacement, that’s your fault. You might not be aware, but zero US air carriers had any unplanned downtime from the GPS rollover. I can’t say the same thing for certain Asian air carriers :) -mel via cell On May 1, 2019, at 8:39 PM, Gary E. Miller <gem@rellim.com<mailto:gem@rellim.com>> wrote: Yo Mel! On Thu, 2 May 2019 03:30:03 +0000 Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: I’m also an FAA licensed A&P mechanic, and have worked for airlines in fleet maintenance. Air carriers have extremely thorough systems reviews, by law, through the Airworthiness Directive program, which started identifying 2019 GPS rollover vulnerabilities in ... 2009! Nobody was surprised. If any GPS systems “went nuts”, it was through the incompetence and negligence of their owners. How many GPS owners happen to have $30,000 GPS simulators to check their $300 GPS/NTP servers? Some of mine did, most did not. Seems to me the negligence is in the GPS manufacturer that failed to notify their customers. To be fair, Avidyne and Telit did notify their customers, but not with a fix or enough lead time to swap out the units. RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com<mailto:gem@rellim.com> Tel:+1 541 382 8588 Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin

I can tell you how the GPS server behaves when it loses it signal: it stops giving out verified time and lapses into Stratum-“goners” mode. But today’s RTP chips don’t start losing seconds-per-day when they are free running. Typically they might lose ten seconds per week on cheap systems. That’s of little concern if you have two GPS clocks. But wait. What is the GPS constellation goes down? THEN we have bigger problems :) It’s possible to over-think the clock problem, just as it’s possible to overthink RAID storage protection. Sometimes a manual restore from backup is just fine. -mel

On May 1, 2019, at 8:13 PM, Harlan Stenn <stenn@nwtime.org> wrote:

...

On 5/1/19 7:54 PM, Mel Beckman wrote: Harlan,

Why? The GPS NTP Server is Stratum-1. If it fails computer clocks will freewheel for hours or days before losing significant time, during which period you can simply order a replacement unit. If that isn’t fast enough, buy two $300 boxes. The “consensus” issue is moot, since a GPS server gets a consensus of clock time from the GPS satellite constellation.

The “enough NTP peers” you speak of are simply not necessary.

You might be right about the GPS server. It depends on how your $300 box behaves if it loses the GPS signal.

The consensus issue isn't about the number of satellites the GPS receiver sees, it's about the number of time sources your NTP servers see.

H --

...

-mel via cell

...

On May 1, 2019, at 6:49 PM, Harlan Stenn <stenn@nwtime.org> wrote:

...

On 5/1/19 4:53 PM, Mel Beckman wrote: Ask,

But with a small compact server like the DC-powered TimeMachines Inc unit, which costs something like $300, you simply put the server where the visibility is and connect back to the nearest Ethernet port in your network, up to 300’ away, or virtually any distance with fiber transceivers. We’ve installed these in Cantex boxes on a windy, rainy tenth-story rooftop in upstate NY and it runs flawlessly, warmed by its own internal heat at sub-zero temps, and perfectly happy at ambient temps of 110F.

It’s hard to consider messing with signal converters and pricey remotely-powered active antennas when you can solve the problem for $300. :)

I sure hope you have ntpd set up to peer or get time with enough other servers.

H --

...

-mel

...

On May 1, 2019, at 4:44 PM, Ask Bjørn Hansen <ask@develooper.com> wrote:

...

On May 1, 2019, at 12:22, Mehmet Akcin <mehmet@akcin.net> wrote:

I am trying to buy a GPS based NTP server like this one

https://timemachinescorp.com/product/gps-time-server-tm1000a/

but I will be placing this inside a data center, do these need an actual view of a sky to be able to get signal or will they work fine inside a data center building? if you have any other hardware requirements to be able to provide stable time service for hundreds of customers, please let me know.

[ with my hobby-hat on … ]

tl;dr: if any of the below is too much work, just run reasonably well monitored NTP server syncing from other NTP servers. If you want more than that, you need to see the sky. Don’t do the CDMA thing.

Depending on your requirements having the antenna in the window may or may not be satisfactory. If it’s fine you probably could just have done a regular NTP server in the first place. For long swaths of the day you might not see too many satellites which will add to the uncertainty of the signal.

Meinberg’s GPS antenna has a bit more smarts which helps it work on up to 300 meters on RG58 or 700 meters on RG213. (They also have products that use regular L1 antennas with the limitations Bryan mentioned).

https://www.meinbergglobal.com/english/products/gps-antenna-converter.htm

They also have a multi-mode fiber box to have the antenna be up to 2km from the box or 20km with their single mode fiber box, if you have fiber to somewhere else where you can see the sky and place an antenna.

It will be more than the one you linked to, but their systems are very reasonably priced, too. For “hundreds of customers” whatever is the smallest/cheapest box they have will work fine. Even their smallest models have decent oscillators (for keeping the ticks accurate between GPS signals).

The Meinberg time server products (I am guessing all of them, but I’m not sure) also have a mode where they poll an upstream NTP server aggressively and then steer the oscillator after it. I haven’t used it in production, but it worked a lot better than it sounded like it would. (In other words, even without GPS it’s a better time server than most systems).

Ask

-- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!

-- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!

I’m talking about _my_ GPS server. I have no idea what you’ve cobbled up :) -mel

On May 1, 2019, at 8:41 PM, Gary E. Miller <gem@rellim.com> wrote:

Yo Mel!

On Thu, 2 May 2019 03:35:31 +0000 Mel Beckman <mel@beckman.org> wrote:

...

I can tell you how the GPS server behaves when it loses it signal: it stops giving out verified time and lapses into Stratum-“goners” mode.

I happen to have a few GPS in my lab that do not agree with your statement. I'll spare this list the details...

RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588

Veritas liberabit vos. -- Quid est veritas? "If you can’t measure it, you can’t improve it." - Lord Kelvin

Bill, I did say _today’s_ RTP chips :) Although as a Mac user with multiple types, many not Internet-connected, I’ve never seen any lose minutes per day. You might have a dead clock battery. -mel On May 2, 2019, at 7:57 AM, William Herrin <bill@herrin.us<mailto:bill@herrin.us>> wrote: On Wed, May 1, 2019 at 8:35 PM Mel Beckman <mel@beckman.org<mailto:mel@beckman.org>> wrote: I can tell you how the GPS server behaves when it loses it signal: it stops giving out verified time and lapses into Stratum-“goners” mode. But today’s RTP chips don’t start losing seconds-per-day when they are free running. Typically they might lose ten seconds per week on cheap systems. That’s of little concern if you have two GPS clocks. The macbook my employer issued gains about 20 minutes a day when not synced. Easier to not replace it because oh look, the drive is soldered to the motherboard. I've taken to calling it my crapbook. Really disappointed with the quality out of Apple lately. -Bill -- William Herrin ................ herrin@dirtside.com<mailto:herrin@dirtside.com> bill@herrin.us<mailto:bill@herrin.us> Dirtside Systems ......... Web: <http://www.dirtside.com/>

well, if they all go down, here is my backup clock. On Fri, May 3, 2019 at 10:04 AM Seth Mattinen <sethm@rollernet.us> wrote:

...

But wait. What is the GPS constellation goes down? THEN we have bigger

On 5/1/19 8:35 PM, Mel Beckman wrote: problems

For timing if we lose the WWV stations and CDMA, then it seems the diversity plan is going to be a combination of US GPS, Galileo, and GLONASS disciplined sources.

I'd like to give a plug for Symetricom products like the Time Provider 1100. I used these in my previous life at a half dozen sites. They function as ntp servers and peer with each other over a network. In addition (and most important to me) they provided BITS clocks to our optical gear and pbx's. Very reliable and you could waste all sorts of money by equipping them with 1 or 2 oscillators, rubidium if you liked. The antenna needed a clear view of the sky and we mounted these at roof level to avoid lightning. They were heated to avoid icing. Good stuff, never had an issue with rollovers, software was upgradable. Sent from my android device. -----Original Message----- From: "Ask Bjørn Hansen" <ask@develooper.com> To: Mehmet Akcin <mehmet@akcin.net> Cc: nanog <nanog@nanog.org> Sent: Wed, 01 May 2019 19:43 Subject: Re: NTP question

On May 1, 2019, at 12:22, Mehmet Akcin <mehmet@akcin.net> wrote:

I am trying to buy a GPS based NTP server like this one

https://timemachinescorp.com/product/gps-time-server-tm1000a/

but I will be placing this inside a data center, do these need an actual view of a sky to be able to get signal or will they work fine inside a data center building? if you have any other hardware requirements to be able to provide stable time service for hundreds of customers, please let me know.

[ with my hobby-hat on … ] tl;dr: if any of the below is too much work, just run reasonably well monitored NTP server syncing from other NTP servers. If you want more than that, you need to see the sky. Don’t do the CDMA thing. Depending on your requirements having the antenna in the window may or may not be satisfactory. If it’s fine you probably could just have done a regular NTP server in the first place. For long swaths of the day you might not see too many satellites which will add to the uncertainty of the signal. Meinberg’s GPS antenna has a bit more smarts which helps it work on up to 300 meters on RG58 or 700 meters on RG213. (They also have products that use regular L1 antennas with the limitations Bryan mentioned). https://www.meinbergglobal.com/english/products/gps-antenna-converter.htm They also have a multi-mode fiber box to have the antenna be up to 2km from the box or 20km with their single mode fiber box, if you have fiber to somewhere else where you can see the sky and place an antenna. It will be more than the one you linked to, but their systems are very reasonably priced, too. For “hundreds of customers” whatever is the smallest/cheapest box they have will work fine. Even their smallest models have decent oscillators (for keeping the ticks accurate between GPS signals). The Meinberg time server products (I am guessing all of them, but I’m not sure) also have a mode where they poll an upstream NTP server aggressively and then steer the oscillator after it. I haven’t used it in production, but it worked a lot better than it sounded like it would. (In other words, even without GPS it’s a better time server than most systems). Ask

On 5/1/19 5:39 PM, William Herrin wrote:

On Wed, May 1, 2019 at 12:23 PM Mehmet Akcin <mehmet@akcin.net> wrote:

...

I am trying to buy a GPS based NTP server like this one

https://timemachinescorp.com/product/gps-time-server-tm1000a/

but I will be placing this inside a data center, do these need an actual view of a sky to be able to get signal or will they work fine inside a data center building? if you have any other hardware requirements to be able to provide stable time service for hundreds of customers, please let me know.

You buy a powered GPS antenna for it. Which antenna depends on the cable length and type. The amplifier in the antenna amplifies the signal just enough to overcome the cable loss between the antenna and the receiver. Nice thick cables lose less signal. Dinky thin ones are easier to work with.

You sure you need a GPS NTP server? You understand that if you do, you need two for reliability right, and probably at geographically diverse locations? If you're not on an air-gapped network, consider syncing a couple head-end NTP servers against tick and tock (.usno.navy.mil, the naval observatory) and not worrying about it. One less piece of equipment to manage, update, secure, etc.

Two is not a great number. If they disagree, there is no majority clique to be found. Also, there is something to be said for using different models/vendors for the time sources. If you only have the same model from one vendor and there is a bug, you can lose all your time sources at once. The GPS week rollover happens every ~19.7 years, and when that problem hits is a function of the firmware and a manufacturing date put in the firmware. These problems can be mitigated if you have "enough" time sources for your internal NTP servers and you peer with enough other, possibly your, servers.

Regards, Bill Herrin

-- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!

What about GPS, GLONASS, Galileo, etc.? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "James R Cutler" <james.cutler@consultant.com> To: "Harlan Stenn" <stenn@nwtime.org> Cc: nanog@nanog.org Sent: Wednesday, May 1, 2019 8:55:51 PM Subject: Re: NTP question On May 1, 2019, at 9:45 PM, Harlan Stenn < stenn@nwtime.org > wrote: On 5/1/19 5:39 PM, William Herrin wrote: <blockquote> On Wed, May 1, 2019 at 12:23 PM Mehmet Akcin < mehmet@akcin.net > wrote: <blockquote> I am trying to buy a GPS based NTP server like this one https://timemachinescorp.com/product/gps-time-server-tm1000a/ but I will be placing this inside a data center, do these need an actual view of a sky to be able to get signal or will they work fine inside a data center building? if you have any other hardware requirements to be able to provide stable time service for hundreds of customers, please let me know. You buy a powered GPS antenna for it. Which antenna depends on the cable length and type. The amplifier in the antenna amplifies the signal just enough to overcome the cable loss between the antenna and the receiver. Nice thick cables lose less signal. Dinky thin ones are easier to work with. You sure you need a GPS NTP server? You understand that if you do, you need two for reliability right, and probably at geographically diverse locations? If you're not on an air-gapped network, consider syncing a couple head-end NTP servers against tick and tock (.usno.navy.mil, the naval observatory) and not worrying about it. One less piece of equipment to manage, update, secure, etc. </blockquote> Two is not a great number. If they disagree, there is no majority clique to be found. Also, there is something to be said for using different models/vendors for the time sources. If you only have the same model from one vendor and there is a bug, you can lose all your time sources at once. The GPS week rollover happens every ~19.7 years, and when that problem hits is a function of the firmware and a manufacturing date put in the firmware. These problems can be mitigated if you have "enough" time sources for your internal NTP servers and you peer with enough other, possibly your, servers. <blockquote> Regards, Bill Herrin </blockquote> -- Harlan Stenn < stenn@nwtime.org > http://networktimefoundation.org - be a member! </blockquote> To amplify the points made by Harlan Stenn: Four is a better number locally for ntpd instances. As for different models/vendors for the time sources, I consider the GPS constellation as one vendor so I add multiple internet-connected sources as well to my ntp.conf instances. James R. Cutler James.cutler@consultant.com GPG keys: hkps://hkps.pool.sks-keyservers.net <blockquote> </blockquote>