Cost-effectivenesss of highly-accurate clocks for NTP
Mel Beckman <mel@beckman.org>:
Yes, this is a topic on which I have some well-developed judgments due to having collected (and, where practical, tested) a pretty comprehensive set of figures on components of the NTP error budget. I've even invented some hardware that simplifies the problem. The background to my findings is laid out in my "Introduction to Time Service" HOWTO: http://www.catb.org/gpsd/time-service-intro.html I find that an effective way to learn my way into a new application domain is to first do knowledge capture on the assumptions its experts are using and then document those. "Introduction to Time Service" was written to do that and as a white paper for my project management. Criticism and corrections are, of course, welcome. In order to discuss the value of accurate clocks intelligently, we need to split apart two issues: accuracy and availability. Of course we want the most accurate time our networks can deliver; we also want to hedge against sporadic or systemic failure of single time sources. The most important simplification of either issue is that clock accuracy worth paying for is bounded both by user expectations and the noise floor defined by our network jitter. According to RFC 5095 expected accuracy of NTP time is "several tens of milliseconds." User expectations seem to evolved to on the close order of 10ms. I think it's not by coincidence this is pretty close to the jitter in ping times I see when I bounce ICMP off a well-provisioned site like (say) google.com through my Verizon FIOS connection. It's good rule-of-thumb engineering that if you want to be metrologically accurate you should pay for precision an order of magnitude below your feature size *and not more than that*. Thus, with a feature size of 10ms the economic sweet spot is a clock with accuracy about 1ms. One reason discussions of how to budget for WAN timeservice clocks has tended to become heated in the past is that nothing inexpensive hit this sweet spot. The world was largely divided between cheap time sources with too much jitter (e.g. GPS in-band data with a wander of 100ms or worse) and expensive high-precision clocks designed for PTP over Ethernet that deliver three or more orders of magnitude than WAN time service can actually use. However...also use the 1PPS signal your GPS engine ships (top of UTC second accurate to about 50ns) and the picture changes completely. With that over RS232 your delivered accuracy rises to single-digit numbers of microseconds, which is two orders of magnitude below what you need for your 1ms goal. Now we have a historical problem, though: RS232 and the handshake lines you could use to signal 1PPS are dying, being replaced by USB. which doesn't normally bring 1PPS out to where the timeserver OS can see it. In 2012, nearly three years before being recruited for NTPsec, I solved this problem as part of my work on GPSD. The key to this solution is an obscure feature of USB, and a one-wire patch to the bog-standard design for generic USB that exploits it. Technical details on request, but what it comes down to is that with this one weird trick(!) you can mass-produce primary time sources with a jitter bounded by the USB polling interval for about $20 a pop. The USB 1 polling interval is 1ms. Bingo. We're done. If we're only weighting accuracy and not availability, a USB GPS is as much clock as you need for WAN timeservice *provided it exposes 1PPS*. These devices exist, because I designed them and found a shop in Shenzhen to build them. They're called the Navisys GR-601W, GR-701W, and GR-801W. (A viable, only skightly more expensive alternative is to mate a GPS daughterboard to a hackerboard like the Raspberry Pi and run NTP service on that. I'll have much, much more to say about that in a future post.) Of course, now we have to talk about availability. GPS sometimes loses lock. There are sporadic and systemic availability risks due to jamming and system failures like the 2012 hiccup, and extreme scenarios like a giant meteorite hitting GPSS ground control in Colorado. What you should be willing to pay for a hedge against this is proportional to your 95% confidence estimate of the maximum outage interval. At the low end, this is simple; put your antenna on a mast that guarantees unobstructed skyview. At the high end it's effectively impossible, in that anything that takes down GNSS and Galileo permanently (giant meteor impact, war in space, collapse of the U.S. and Europe) is likely to be in the you-have-much-bigger- problems than-inaccurate-time department. Traditionally dedicated time-source hardware like rubidium-oscillator GPSDOs is sold on accuracy, but for WAN time service their real draw is long holdover time with lower frequency drift that you get from the cheap, non-temperature-compensated quartz crystals in your PC. There is room for debate about how much holdover you should pay for, but you'll at least be thinking more clearly about the problem if you recognize that you *should not* buy expensive hardware for accuracy. For WAN time service, in that price range, you're wither buying holdover and knowing you're doing so or wasting your money. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a> Everything you know is wrong. But some of it is a useful first approximation.
Den 13. maj 2016 21.40 skrev "Eric S. Raymond" <esr@thyrsus.com>:
Ok how many hours or days of holdover can you expect from quartz, temperature compensated quartz or Rubidium? Should we calculate holdover as time until drift is more than 1 millisecond, 10 ms or more for NTP applications? I am thinking that many available datacenter locations will have poor GPS signal so we can expect signal loss to be common. Some weather patterns might even cause extended GPS signal loss. Regards Baldur
On 13 May 2016 at 23:01, Baldur Norddahl <baldur.norddahl@gmail.com> wrote:
I found some data points here: https://en.wikipedia.org/wiki/Crystal_oven Assuming that acceptable drift is 10 milliseconds due that being the expected accuracy from NTP. The common crystal oscillator can be as bad as 1E-4 => holdover time is 2 minutes. TCXO is listed as 1E-6 => holdover time is 3 hours. OCXO is listed as multiple values, I will use 1E-7 => holdover time is 1 day. Rubidium is listed as 1E-9 => 3 months Caesium is listed as 1E-11 => 30 years Hydrogen Maser 1E-15 => 300 millennia I clearly need three of those maser things for my network. Regards, Baldur
Baldur Norddahl <baldur.norddahl@gmail.com>:
Ok how many hours or days of holdover can you expect from quartz, temperature compensated quartz or Rubidium?
Sorry, I don't have those drift figures handy. I'm a programmer, not a large-site sysadmin - I've never had purchase authority with a budget large enough to buy a rubidium-oscillator GPSDO or any other device with holdover. Better to ask Mel Beckman or someone else with field experience.
If you want to go super-accurate, 1ms. If you want to go cheap, on sampling-theory grounds I'd say you want to vary your drift threshold from 1 to 5ms (half the expected precision of WAN time, think of it as the Nyquist rate) and look for a knee in the cost curve.
Weather won't do it, usually. Rain and fog and clouds are transparent to GPS frequencies. Standing water directly on an antenna can cause some attenuation, but with any serious GPS engine made more recently than 5 years ago I would be extremely surprised if that lost it lock. The newer ones handle down to 30 feet in ocean water on robot submarines. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
I have deployed rubidium-disciplined clocks at cellular carriers, where money is no object (look at your cellphone bill), typically for $20K-$100K for redundant clocks. The holdover time on these is supposed to be days, but we've never tested that. I think I'd better. But a more critical deployment of rubidium clocks is in cash-strapped public safety institutions, such as local police dispatch centers. Timing is crucial for the squad car communication systems, which these days are all digital, based on wireless T1/T3 trunks to remote repeaters. The clocking on these trunks can't drift much before voice communication fails due to repeater outages. The telecom gear has OXCO clocks that can provide a few hours holdover. A rubidium clock onsite provides coverage for longer outages. In these installations I have tested GPS outages of up to a week without enough clock drift to knock out the Tx links. I've never gone longer than that though, so I don't know the actual breaking point. But if you lose that rubidium clock, things go south in a less than a day. The cash-strapping comes into play when municipal bean counters eyeball the rubidium clock(s) and start thinking they can get by with a cheaper substitute. The upshot is that there are many real-world situations where expensive clock discipline is needed. But IT isn't, I don't think, one of them, with the exception of private SONET networks (fast disappearing in the face of metro Ethernet). -mel beckman
Mel Beckman <mel@beckman.org>:
Thank you, that was very interesting information. I'm not used to thinking of IT as a relatively low-challenge environment! You're implicitly suggesting there might be a technical case for replacing these T1/T3 trunks with some kind of VOIP provisioning less dependent on accurate time synch. Do you think that's true? -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
On 5/15/16 10:05 AM, Eric S. Raymond wrote:
APCO and TETRA trunked radio are mature systems, they do carry data, but are somewhat lower bandwidth. Being TDM they are dependent on accurate clocks. LTE systems are used or envisioned being used for high bandwidth applications.
Joe and Eric, It's frustrating how far public safety technology lags behind what Industry can actually deliver. It's the same in aviation. Institutions are slow to adopt new tech due to fears about reliability, and and unwillingness to take any risk at all. So PS and aviation capabilities lag horribly. This is why commercial pilots, tired of waiting on the FAA, are buying their own tablets and running non-certified navigation tools. And police officers use cellular data connection with VPN to query wants and warrants databases. -mel beckman
On 05/15/2016 01:05 PM, Eric S. Raymond wrote:
I'm not used to thinking of IT as a relatively low-challenge environment!
I actually changed careers from broadcast engineering to IT to lower my stress level and 'personal bandwidth challenge.' And, yes, it worked. In my case, I'm doing IT for radio and optical astronomy, and at least the timing aspect is higher-challenge that most IT environments.
While I know the question was directed at Mel specifically, I'll just say from the point of view of a T1 voice trunk customer that I hope to never see it go to a VoIP solution. VoIP codecs can have some serious latency issues; I already notice the round-trip delay if I try to carry on a conversation between our internal VoIP system and someone on a cell phone. And this is before codec artifacting (and cascaded codec scrambling) is counted. Can we please keep straight μ-law (A-law if relevant) lossless DS0 PCM timeslices for trunklines so we get at least one less lossy codec cascade? Or have you never experimented with what happens when you cascade G.722 with G.729 with G.726 and then G.711 and back? Calls become mangled gibberish. I would find it interesting to see how many carriers are still doing large amounts of SONET, as that is the biggest use-case for high-stability timing.
Lamar, Although VoIP has different technical challenges than TDM, they are all surmountable. Usually VoIP problems result from poor network design (e.g., mixed traffic with no QoS, Internet transport with no guarantees, etc). Public safety networks are generally well designed, don’t use the Internet, and support a single type of traffic: audio streams. I’ve done demonstrations of VoIP that works well in this environment, and you get the advantages of IP routing for redundancy, as opposed to clunky T1 failover mechanisms usually based on hardware switches. But public safety customers are not swayed. TDM works, and it’s the gold standard. They don’t want to change, and you can’t make them. They only see risk, no reward. BTW, in the TDM model, remote data and audio are usually two different systems, which is probably a good idea for redundancy: you might lose audio or data, but rarely both. In any event, I’m only proposing that public safety upgrade their audio systems to VoIP (or cellular-style private GSM, which is in essence VoIP). But nobody is willing. -mel
Give this guy a look, 1RMU form factor, GPS with Rubidium holdover (7 days) if you need it very inexpensive, http://www.fibrolan.com/FibroLAN/SendFile.asp?DBID=1&LNGID=1&GID=3979 or http://www.fibrolan.com/FibroLAN/SendFile.asp?DBID=1&LNGID=1&GID=3978 if you have a SYC-E source or If you just need highly accurate NTP without the holdover, you can do it with FBSD, and a GPS device. http://www.satsignal.eu/ntp/FreeBSD-GPS-PPS.htm http://www.brandywinecomm.com/46-products/bus-level-plug-in-boards/212-pcie-... -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mel Beckman Sent: Monday, May 16, 2016 11:27 AM To: Lamar Owen Cc: nanog@nanog.org Subject: Re: Cost-effectivenesss of highly-accurate clocks for NTP Lamar, Although VoIP has different technical challenges than TDM, they are all surmountable. Usually VoIP problems result from poor network design (e.g., mixed traffic with no QoS, Internet transport with no guarantees, etc). Public safety networks are generally well designed, don’t use the Internet, and support a single type of traffic: audio streams. I’ve done demonstrations of VoIP that works well in this environment, and you get the advantages of IP routing for redundancy, as opposed to clunky T1 failover mechanisms usually based on hardware switches. But public safety customers are not swayed. TDM works, and it’s the gold standard. They don’t want to change, and you can’t make them. They only see risk, no reward. BTW, in the TDM model, remote data and audio are usually two different systems, which is probably a good idea for redundancy: you might lose audio or data, but rarely both. In any event, I’m only proposing that public safety upgrade their audio systems to VoIP (or cellular-style private GSM, which is in essence VoIP). But nobody is willing. -mel
Subject: Re: Cost-effectivenesss of highly-accurate clocks for NTP Date: Sun, May 15, 2016 at 03:21:02PM +0000 Quoting Mel Beckman (mel@beckman.org):
The upshot is that there are many real-world situations where expensive clock discipline is needed. But IT isn't, I don't think, one of them, with the exception of private SONET networks (fast disappearing in the face of metro Ethernet).
Pro audio is moving to Ethernet (they talk about it, Ethernet, as either "RJ45" or "Internet"...) and sometimes even to IP in a fairly rapid pace. If you think the IP implementations in IoT devices are naîve, wait until you've seen what passes for broadcast quality network engineering. Shoving digital audio samples in raw Ethernet frames is at least 20 years old, but the last perhaps 5 years has seen some progress in actually using IP to carry audio streams. (this is close-to-realtime audio, not file transfers, btw.) A lot of audio is sent using codecs like Opus, with SIP as signalling. That works quite nicely. We've got a smartphone app to do that, for instance. But, this is all mostly floating in terms of absolute sampling frequency. Digital audio needs a clock to work. In the simple home stereo case, this is taken care of by listening to the pace samples arrive at, and using that. But as soon as you are mixing two sources, they need to be in tune. Something needs to decide what to use as master. In the smartphone case, we simply buffer some 20-100ms of audio and start playing back, using our own clock. Then we hope the interview is over before the buffer is overflowing or drained. Which mostly works. Inside facilities, when we use the SIP-signalled streams, we usually can rely on a separate clock distribution. In our specific case, we've bought country-wide clock distribution that gives us the same sample clock in all facilities. (Digital TV is mostly built as single frequency networks, which requires syntonous (at least) transmitters. Thus, it today is quite easy to find providers of frequency in the broadcast business.) Now, the Audio Engineering Society has published AES67 which in essence is multichannel, multicast RTP audio (L48 mediatype, ie. linear 48KHz 24-bit) synchronized by PTP, also multicast. Now, bear in mind that I wrote _synchronized_, not _syntonized_. Up to now, the only thing that mattered to keep track of was frequency. Since one of the big reasons for AES67 is distributing sound to several different loudspeakers that can be heard by one listener simultaneously, the prime example being a stereo pair of active loudspeakers with one network jack on each, _phase_ matters, as well as absolute time. (Mostly, telco synchronization mentions absolute time as phase.) This application requires absolute time, since a mono sound in our stereo example needs to play back _at the same time_ from both speakers. Or it ceases to be a mono sound, instead becoming a sound that is offset in the soundstage by delaying it. Most classical stereo recordings are mono in terms of level, but not in terms of the time domain; since they derive all spatial info from time, not gain. Like we humans do. The usual test case is to buy a PTP-aware switch, a PTP Grand Master, steered by <something> and build a small LAN, test that Vendor A and Vendor B can send audio between themselves via this simple network and call it a day. That is a nice lab setup. Also very far from what needs to be built in order to solve the actual production cases. But, to try to return to "relevant for NANOG", there are actual products requiring microsecond precision being sold. And used. And we've found that those products don't have a very good holdover. On ranty days I usually accuse them of having hotglued an Ethernet adapter onto the old TDM-based audio devices and sent them out to customers with a prayer and instructions to build an overengineered network to make certain that PTP always is delivered with zero IPDV. A lot of strange things are getting network connectors these days. Not all of them are content with a http connection to some cloud provider. -- Måns Nilsson primary/secondary/besserwisser/machina MN-1334-RIPE +46 705 989668 The PILLSBURY DOUGHBOY is CRYING for an END to BURT REYNOLDS movies!!
On 05/15/2016 03:16 PM, Måns Nilsson wrote:
Close to realtime is a true statement. Using an IP STL (studio-transmitter link) has enough latency that the announcer can no longer use the air signal as a monitor. And the security side of things is a pretty serious issue; just ask a major IP STL appliance vendor about the recent hijacking of some of their customers' IP STL devices.... yeah, a random intruder on the internet hijacked several radio stations' IP STL's and began broadcasting their content over the radio. Not pretty. I advise any of my remaining broadcast clients that if they are going to an IP STL that they put in a dedicated point to point IP link without publicly routable IP addresses. Digital audio for broadcast STL's is old tech; we were doing G.722/G.723 over switched-56 in the early 90's. But using a public-facing internet connection with no firewalling for an IP STL appliance like the Barix boxes and the Tieline boxes and similar? That borders on networking malpractice.
Valdis mentioned the scariest thing.... the scariest thing I've seen recently? Windows NT 3.5 being used for a transmitter control system, within the past five years. I will agree with Valdis on the scary aspects of the public safety communications Mel mentioned. Thanks, Mel, for the educational post.
On 05/13/2016 03:39 PM, Eric S. Raymond wrote:
Eric, Thanks for the pointers; nice information. A cheap way to get a WAN frequency standard is to use a WAN that is delivered over something derived from the telco's synchronous network; a POS on an OC3 with the clock set to network has an exceptionally stable frequency standard available. Less expensive, get a voice T1 trunk delivered (robbed-bit signaled will typically be less expensive than PRI) and grab clock from that; tarriffs for RBS T1/fractional T1 around here at least are less than an analog POTS line). Very stable. The plesiochronous digital hierarchy on copper or synchronous digital hierarchy/SONET on fiber have cesium clocks behind them, and you can get that stability by doing clock recovery on those WAN circuits. Back when this was the most common WAN technology frequency standards were there for the taking; Ethernet, on the other hand, not so much. But a nice catch on using the isochronous nature of USB. Cheap webcams also take advantage of the isochronous transfer mode. Do note that isochronous is often not supported in USB-passthrough for virtualization, though. But you shouldn't use a VM to do timing, either. :-) Now I'm looking for myself one of those Navisys devices you mentioned..... do any of them have external antenna inputs, say on an SMA connector (MCX is in my experience just too fragile) with a bias tee to drive phantom to an active antenna? The quick search I did seemed to indicate that the three you mentioned are self-contained with their own smart antenna. External antenna input would be required here, where we use timing-grade GPS antennas to feed our Z3816's. But for straight 1PPS and GPS timecode, dealing with the Z3816's complexity is overkill. Thanks again for the info; looking forward to seeing how NTPsec develops.
Bruce Simpson <bms@fastmail.net>:
Nobody makes GPSes with even USB 2 or 3 yet, and it is unlikely to happen for a long time. Cost reasons - USB GPSes are cheap consumer-grade hardware and the manufacturers care about fractions of a cent on the BOM. -- <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
In a message written on Fri, May 13, 2016 at 03:39:27PM -0400, Eric S. Raymond wrote:
For a typical site, there are two distinct desires from the same NTP process. First, syncronization with the rest of the world, which is generally over the WAN and the topic that was well discussed in your post. I agree completely that the largest factor is WAN jitter. The other is local syncronization, insuring multiple servers have the same time for log analysis purposes and such. This typically takes place across a lightly loaded ethernet fabric, perhaps with small latency (across a compus). Jitter here is much, much smaller. Does the limitation of accuracy remain jitter in the low jitter LAN/Campus enviornment, or does that expose other issues like the quality of oscellators, OS scheduling, etc? Or perhaps another way, is it possible to get say 10's or 100's of nanosecond accuracy in the lan/campus? -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
For a typical site, there are two distinct desires from the same NTP
First, synchronization with the rest of the world, which is generally over
-----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Leo Bicknell Sent: Monday, May 16, 2016 10:28 AM To: nanog@nanog.org Subject: Re: Cost-effectivenesss of highly-accurate clocks for NTP process. the WAN and the topic that was well discussed in your post.
I agree completely that the largest factor is WAN jitter.
Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
I was just thing about this WAN jitter issue myself. I'm wondering how many folks put NTP traffic in priority queues? At least for devices in your managed IP ranges. Seems like that would improve jitter. Would like to hear about others doing this successfully prior to suggesting it for our network. Chuck
participants (13)
-
Baldur Norddahl -
Bruce Simpson -
Chuck Church -
Eric S. Raymond -
esr＠thyrsus.com
-
Jameson, Daniel -
joel jaeggli -
Lamar Owen -
Leo Bicknell -
Mel Beckman -
Måns Nilsson -
sthaug＠nethelp.no -
Valdis.Kletnieks＠vt.edu