Hey guys: I wanted to open up this question regarding NTP server. I recalled someone had created a posting of this quite awhile back.
From a service provider/ISP standpoint, does anyone think that having a local NTP server is really necessary?
I've asked some of my fellow engineers at work and many of them gives me the same response, "Can't we just use free ones out on the internet?" 1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate? 2) If you do have a local NTP server, is it only for local internal use, or do you provide this NTP server to your clients as an added service? 3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server? Thoughts? Thanks in advance, and this list is such a valuable wealth of resource.... Brandon
On 10/24/10 9:34 AM, Brandon Kim wrote:
I wanted to open up this question regarding NTP server. I recalled someone had created a posting of this quite awhile back.
From a service provider/ISP standpoint, does anyone think that having a local NTP server is really necessary?
It may not be necessary, but it certainly is not a bad thing. Not having to depend on third parties for a service is a good thing.
I've asked some of my fellow engineers at work and many of them gives me the same response, "Can't we just use free ones out on the internet?"
1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate?
Perfectly accurate is very helpful when trying to associate several incidents going on at the same time or when trying to figure out the timeline leading up to why a machine had a kernel panic, for example.
2) If you do have a local NTP server, is it only for local internal use, or do you provide this NTP server to your clients as an added service?
Our master stratum 1 GPS clock only has ipv6 access to the outside world. Our two 'public' ntp servers can talk directly to it over ipv4 or ipv6, and those are are publicly available via ipv4 or ipv6.
3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server?
If the stratum 1 becomes unavailable (its 500 miles away on a different network), the two public NTP servers are peered with one another, and both have a different outside third-party NTP server to sync with (may it be an upstream provider's ntp server, or one of the pool ones from ntp.org). Never had a problem with this setup, and its worked rather well. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
Looks like you have a pretty good setup. What vendor equipment are you using? You can let me know offline so it doesn't sound like you're advertising them....
Date: Sun, 24 Oct 2010 11:03:18 -0600 From: bruns@2mbit.com To: nanog@nanog.org Subject: Re: NTP Server
On 10/24/10 9:34 AM, Brandon Kim wrote:
I wanted to open up this question regarding NTP server. I recalled someone had created a posting of this quite awhile back.
From a service provider/ISP standpoint, does anyone think that having a local NTP server is really necessary?
It may not be necessary, but it certainly is not a bad thing. Not having to depend on third parties for a service is a good thing.
I've asked some of my fellow engineers at work and many of them gives me the same response, "Can't we just use free ones out on the internet?"
1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate?
Perfectly accurate is very helpful when trying to associate several incidents going on at the same time or when trying to figure out the timeline leading up to why a machine had a kernel panic, for example.
2) If you do have a local NTP server, is it only for local internal use, or do you provide this NTP server to your clients as an added service?
Our master stratum 1 GPS clock only has ipv6 access to the outside world. Our two 'public' ntp servers can talk directly to it over ipv4 or ipv6, and those are are publicly available via ipv4 or ipv6.
3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server?
If the stratum 1 becomes unavailable (its 500 miles away on a different network), the two public NTP servers are peered with one another, and both have a different outside third-party NTP server to sync with (may it be an upstream provider's ntp server, or one of the pool ones from ntp.org).
Never had a problem with this setup, and its worked rather well.
-- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate?
what is "perfectly accurate?" perfection is not very realistic. to what use do you put these logs? what precision and jitter are required for that use? imiho, if you are just comparing router and server log files, run off public. if you are trying to do fine-grained measurement, you are going to invest a lot in clock and propagation research.
2) If you do have a local NTP server, is it only for local internal use, or do you provide this NTP server to your clients as an added service?
i would generally let customers chime off routers which are strat 2 or 3. if a customer has other needs, then they can deal. if they are really concerned, they should not bet on me anyway.
3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server?
again, depends on your needs. randy
More than likely, it's more important that all your machines are synced accurately in time to each other, vs. a wider sync range that's statistically closer to the 'real' value. -Jack Carrozzo On Sun, Oct 24, 2010 at 1:09 PM, Randy Bush <randy@psg.com> wrote:
1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate?
what is "perfectly accurate?" perfection is not very realistic. to what use do you put these logs? what precision and jitter are required for that use?
imiho, if you are just comparing router and server log files, run off public. if you are trying to do fine-grained measurement, you are going to invest a lot in clock and propagation research.
2) If you do have a local NTP server, is it only for local internal use, or do you provide this NTP server to your clients as an added service?
i would generally let customers chime off routers which are strat 2 or 3. if a customer has other needs, then they can deal. if they are really concerned, they should not bet on me anyway.
3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server?
again, depends on your needs.
randy
Just for log purposes and possibly providing it to our clients as an added service at no charge of course. I don't see us needing to get very granular in the details of the times on the logs....
Date: Sun, 24 Oct 2010 10:09:25 -0700 From: randy@psg.com To: brandon.kim@brandontek.com CC: nanog@nanog.org Subject: Re: NTP Server
1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate?
what is "perfectly accurate?" perfection is not very realistic. to what use do you put these logs? what precision and jitter are required for that use?
imiho, if you are just comparing router and server log files, run off public. if you are trying to do fine-grained measurement, you are going to invest a lot in clock and propagation research.
2) If you do have a local NTP server, is it only for local internal use, or do you provide this NTP server to your clients as an added service?
i would generally let customers chime off routers which are strat 2 or 3. if a customer has other needs, then they can deal. if they are really concerned, they should not bet on me anyway.
3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server?
again, depends on your needs.
randy
On Oct 24, 2010, at 1:09 PM, Randy Bush wrote:
1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate?
what is "perfectly accurate?" perfection is not very realistic. to what use do you put these logs? what precision and jitter are required for that use?
imiho, if you are just comparing router and server log files, run off public. if you are trying to do fine-grained measurement, you are going to invest a lot in clock and propagation research.
As one of the aforementioned "time-nuts", I'd strongly second Randy's recommendation. It's hard to find a middle ground in timing: Most of the network-accessible stratum {1, 2} clocks are good enough for many uses. If you find yourself needing really precise time with good guarantees, you're not just talking about buying one GPS unit -- you can easily go down a rathole of finding multiple units with good holdover. (And if you don't need that, then ask yourself why public isn't good enough). Possible very reasonable answers include needing to do one-way delay measurements; others include wanting to depend on time for authentication protocols or other protocols and not have an external dependency (assuming you're not high-value enough for someone to try to spoof GPS at you). The problem is that once you have a timing device or two, you've added to the set of crap you have to manage and monitor. I use a lot of CDMA-based time receivers so that I can throw them in machine rooms with no sky access, and every year or two, I have to go upgrade a lot of firmware because some cellular company has changed their protocols. I find a lot of cellular base stations that keep the wrong time (suggesting that their GPS-based time sync is fubared in some way). Yadda, yadda. Nothing is free. -Dave
i would generally let customers chime off routers which are strat 2 or 3. if a customer has other needs, then they can deal. if they are really concerned, they should not bet on me anyway.
3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server?
I agree. Someone downstream from you who is *really* concerned about time can either do it themselves or pay you to do it. If you are just date stamping logfiles, you are probably better off running a few servers that sync up externally to one of the free pools (http://www.pool.ntp.org/en/ ) in your region (they generally round-robin IPs to several different servers) and sync your internal stuff to your internal servers. The main reason for that is that the "free" servers won't remain "free" if every single individual host on the Internet is hitting them. By running your own internal servers a stratum down you offload that traffic from the public servers and preserve that resource. NTP is a great candidate for v4 anycast, too, so you can have a common configuration at all your locations if you want.
On Sun, 24 Oct 2010, George Bonser wrote:
The main reason for that is that the "free" servers won't remain "free" if every single individual host on the Internet is hitting them. By running your own internal servers a stratum down you offload that traffic from the public servers and preserve that resource. NTP is a great candidate for v4 anycast, too, so you can have a common configuration at all your locations if you want.
It sure would be nice if datacenter facilities offered an independent NTP time source as a benefit for hosting with them. It would also be great if ISPs would offer this on the local network as well for their customers, as likely they are already have one in several regions. time.windows.com and time.apple.com are also fine, though I'm not sure either has published their NTP source, whether it is a device or they are simply using the same ntp.org pool as many of us. I've never had a problem with the public NTP sources, but as George said, "free" may not always be "free." Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
From: Peter Beckman Sent: Sunday, October 24, 2010 11:33 AM To: North American Network Operators Group Subject: RE: NTP Server
On Sun, 24 Oct 2010, George Bonser wrote: It sure would be nice if datacenter facilities offered an independent NTP time source as a benefit for hosting with them.
One provider I worked with in the past used to offer it but stopped because customers apparently had a wide variety of expectations on what that should be. It turned into a complaint generator when people would demand stratum 1 or if one of the servers was down for a bit so they just shut it off. It provided no benefit as far as they were concerned and it cost them time, effort, and power to provide it. They reasoned that the customer could provide their own time servers and own the issues themselves. That is probably an exercise in properly setting expectations at the start, though. Provide a stratum 3 service and tell people that one of the sources is subject to disappearing from time to time as a function of regular maintenance and folks should be fine with that or do it themselves.
On Sun, 24 Oct 2010, George Bonser wrote:
The main reason for that is that the "free" servers won't remain "free" if every single individual host on the Internet is hitting them. By running your own internal servers a stratum down you offload that traffic from the public servers and preserve that resource. NTP is a great candidate for v4 anycast, too, so you can have a common configuration at all your locations if you want.
It sure would be nice if datacenter facilities offered an independent NTP time source as a benefit for hosting with them. It would also be great if ISPs would offer this on the local network as well for their customers, as likely they are already have one in several regions.
time.windows.com and time.apple.com are also fine, though I'm not sure either has published their NTP source, whether it is a device or they are simply using the same ntp.org pool as many of us.
I've never had a problem with the public NTP sources, but as George said, "free" may not always be "free."
That's particularly true given what some of the free servers have been made to endure. For example, Netgear caused UW Madison a ton of trouble with a defective product that caused a traffic flood: http://pages.cs.wisc.edu/~plonka/netgear-sntp/ NTP is not that hard to provide. Set up four servers. If you only care about relatively stable time, they probably need only be stratum two, this is easy, just go and sync each one with two different stratum one servers, monitor them, and tell customers that it's a free service you make a reasonable attempt to keep running accurate to the second, and that your goal is to keep three operational at any time. Your internal servers can then run NTP to sync with those servers; the use of four will make the failure of up to two (one offline, one with the wrong time, for example) fairly tolerable. Some customers will not care to listen to instructions to sync to four clocks. You have to consider how to make their failure to listen to you to be their own problem. Four is, IMHO, the best number of servers to have. They do not need to be fast or modern machines. You can use something cheap like a pile of old Intel ISP1100's (~40-50 watts each) which might even be doing something else like DNS, monitoring, etc. if you have to. Speaking of which, if anyone is in need of some nice Intel ISP1100's, we are retiring some. Great low power platform for basic services like NTP, proc speeds up to 1GHz, memory up to 1GB, two PCI slots, serial console capable, etc. Available fairly cheap. Great for things like NTP, DNS, we've got one for our FTP archive, Asterisk PBX, etc. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On 25/10/2010 15:56, Joe Greco wrote:
Four is, IMHO, the best number of servers to have. They do not need to be fast or modern machines.
They do need to have a somewhat unbroken internal clock.
That's a good point.
This tends to mean that running ntp on a VM is not generally a good idea.
Running it on a busy host of any kind is not generally a good idea. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
In a message written on Sun, Oct 24, 2010 at 11:34:12AM -0400, Brandon Kim wrote:
From a service provider/ISP standpoint, does anyone think that having a local NTP server is really necessary?
Do you provide NTP to your customers? If you do there is probably an obligation there to make a reasonable effort to have accurate times. I'm not sure relying on random servers across the internet rises to that standard. I think you should have at least four clocks getting time not from the internet to compare. For instance, for a couple of thousand dollars you can get a Symmetricom appliance that will do GPS timing with analog dial backup to NIST. That gives you two non-internet sources at relatively low cost and low effort. Deploy four in different POP's and you have redundancy on your own network, and can market that you provide high quality NTP to your customers. It's nearly fire and forget, and a check for alarms from the box and make sure you watch for patches, that's about it. If you don't offer NTP to your customers whatever you need for your own internal logging is fine. Generally as long as they all sync to the same set of servers they will be accurate to each other, so you can compare times across servers. Set up 4 NTP servers, let them sync to the outside world, let all of your internal boxes sync to them. Notice in both cases I said deploy 4. If you understand the protocol, and in particular the decision process that really is the minimum number to have high quality NTP. Syncing everything to one or two NTP servers really doesn't work so well. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Leo Bicknell <bicknell@ufp.org> writes:
For instance, for a couple of thousand dollars you can get a Symmetricom appliance that will do GPS timing with analog dial backup to NIST. That gives you two non-internet sources at relatively low cost and low effort. Deploy four in different POP's and you have redundancy on your own network, and can market that you provide high quality NTP to your customers. It's nearly fire and forget, and a check for alarms from the box and make sure you watch for patches, that's about it. ... Notice in both cases I said deploy 4. If you understand the protocol, and in particular the decision process that really is the minimum number to have high quality NTP. Syncing everything to one or two NTP servers really doesn't work so well.
You can deploy four, which is the appropriate minimum number to deploy if you're doing it in-house, but four of the same brand and model does not protect you against *other* failure modes, like the problem we all experienced with TrueTime almost 9 years ago. A brief review is here: http://groups.google.com/group/comp.protocols.time.ntp/msg/5f4e774dccf34c47 Not only is it wise to have more than one chipset in play (I have Motorola and Garmin here), but it is good to have time sources from more than one place. Sure, the odds of the GPS C/A code getting it wrong on a global scale are pretty small and if it happens will create an enormous news event... Here in the future, we've taken an enormous step backwards in terms of precision time sources. Here, I only have GPS and WWVB as sources, and WWVB is not a 24-hour source (a better antenna might help this after I move, but the signal strength is not particularly good here on the east coast). Remember GOES? It's gone. LORAN? Canceled and shut down. GLONASS is fully restored to service as of last month after a bad multi-year post-Soviet hit, but good luck finding commodity-priced chipsets or reasonably priced NTP appliances that talk to it. It looks like Duke Nukem Forever may finally ship next year, but until it does I'll continue to draw unfavorable comparisons between it and Galileo. In answer to the original question, running a small constellation (four is the right number) of local stratum 2 servers in each datacenter is a no-brainer. A strong case can be made for running your own stratum 1 servers. They do not have to be on the same subnet as has been suggested (and in fact, you don't want that kind of non-redundancy as a general rule), but NTP really does want the path to the server to be symmetric, which is a big argument in favor of your own inside your network. The folks at NRC in Canada will do cryptographically authenticated NTP with you for an annual fee. I have no idea if there is something similar available from NIST in the US, but if they do I sure hope it doesn't go over the same links as time-a and time-b - from my location anyway, those two get tossed out as falsetickers on weekday afternoon due to too much jitter. -r
On Mon, 25 Oct 2010, Robert E. Seastrom wrote:
The folks at NRC in Canada will do cryptographically authenticated NTP with you for an annual fee. I have no idea if there is something
Robert, Thanks for the shout. NRC does do this, more info here: http://www.nrc-cnrc.gc.ca/eng/services/inms/time-services/network-time.html You can use the services as well for non-auth. I should also point out to folks on this list that the NRC NTP servers have renumbered, but I still see quite a bit of traffic from what appears to be ISP infrastructure looking for the old addresses. wfms
On Sun, 24 Oct 2010 11:34:12 -0400 Brandon Kim <brandon.kim@brandontek.com> wrote:
I wanted to open up this question regarding NTP server. I recalled someone had created a posting of this quite awhile back.
From a service provider/ISP standpoint, does anyone think that having a local NTP server is really necessary?
It's not strictly necessary, but I think any serious and reasonably-sized ISP should probably have their own set of time sources. This thread might be useful to review for some suggestions, but in particular Michael's comments are relevant: <http://listserv.educause.edu/cgi-bin/wa.exe?A2=ind0809&L=SECURITY&T=0&F=&S=&P=102171>
1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate? 2) If you do have a local NTP server, is it only for local internal use, or do you provide this NTP server to your clients as an added service? 3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server?
The "perfect accuracy" of log files might be hard to justify and quantify. I'd say it's more about having your own trustworthy and reliable source that you can ensure is operational, reachable and correct. That said, it is perfectly fine and probably useful to use external sources in addition to your own for backup and time redundancy in your design. You probably don't need to provide time to your customers unless you have a good reason to do so or they've been asking, which I'd find surprising these days for new installations. The default Microsoft time service and the pool.ntp.org servers probably work fine for the majority of end users. We have some NTP configuration templates here if it helps any: <http://www.team-cymru.org/ReadingRoom/Templates/> John
1) How necessary do you believe in local NTP servers? Do you really need th= e logs to be perfectly accurate? 2) If you do have a local NTP server=2C is it only for local internal use= =2C or do you provide this NTP server to your clients as an added service? 3) If you do have a local NTP server=2C do you have a standby local NTP ser= ver or do you use the internet as your standby server? Thoughts?
How do you knew that your local NTP server knew what time it is? (for sure) -P
On 24/10/10 5:44 PM, Peter Lothberg wrote:
How do you knew that your local NTP server knew what time it is? (for sure)
By polling as many stratum 1 and 2 time servers as possible. Having your own stratum 2 server(s) beats nebulous NTP servers out in the big bad Internet every time. Regards, Ben
On Mon, Oct 25, 2010 at 02:51:24AM +1100, Ben McGinnes wrote:
How do you knew that your local NTP server knew what time it is? (for sure)
By polling as many stratum 1 and 2 time servers as possible. Having your own stratum 2 server(s) beats nebulous NTP servers out in the big bad Internet every time.
For those you care about that: http://leapsecond.com/time-nuts.htm
On 25/10/10 2:55 AM, Eugen Leitl wrote:
For those you care about that:
Wow ... that's a lot more effort than I'm willing to put in on a time server. Regards, Ben
I guess what I'm trying to understand is, is having your own NTP server just a luxury? I personally would like to have my own, I just need to pitch its advantages to my company. Unless everyone here on the NANOG group clearly spells it out to me that it's a luxury. I can see it as an added service/benefit though to our customers.....
Date: Sun, 24 Oct 2010 17:55:22 +0200 From: eugen@leitl.org To: nanog@nanog.org Subject: Re: NTP Server
On Mon, Oct 25, 2010 at 02:51:24AM +1100, Ben McGinnes wrote:
How do you knew that your local NTP server knew what time it is? (for sure)
By polling as many stratum 1 and 2 time servers as possible. Having your own stratum 2 server(s) beats nebulous NTP servers out in the big bad Internet every time.
For those you care about that:
Time Service is more complicated than just having a single NTP server. But it can be useful and is not really a luxury. Two primary reasons for local time service are to reliably serve a network that is relatively or completely isolated from the general internet, and, to provide a local time source for "dumb" clients that is closer (less jitter) in network terms. Other reasons can include policy (everything in the network uses the same identical time service), policy (the time service is locally controlled), operational simplicity (the routers don't need to run NTP), and, separation of functions/operational responsibility (your run your servers, they run the backbone, I tell you the time. Implementing a local time service is actually fairly simple, but fewer than four servers is wasted effort. I can't explain in just a few words how the servers interact and compute delays and jitter to come to an "accurate" time. Take my word or ask David Mills for all that. Implementation of an internet-referenced time service involves the following: 1. Select a set of stratum one servers - pick open access servers or get permission to use limited access servers. Four to six should do. 2. Select a set local hosts on your network - DNS servers, for example. These should be well distributed. Four to six should do. The actual NTP load is small compared to DNS queries. 3. Configure the local hosts as peers using the stratum one set as servers. Use crypto authentication if you feel the need. 4. Add NTP monitoring to your network management process. 5. Advertise the local time servers to your network - DHCP, word of mouth, configuration requirements, configuration scripts, standard builds, etc. It is simple enough to do for a five node home network. It is almost that simple for a network with hundreds of thousands of client nodes. I've done both. On Oct 24, 2010, at 12:29 PM, Brandon Kim wrote:
I guess what I'm trying to understand is, is having your own NTP server just a luxury?
I personally would like to have my own, I just need to pitch its advantages to my company. Unless everyone here on the NANOG group clearly spells it out to me that it's a luxury.
I can see it as an added service/benefit though to our customers.....
Date: Sun, 24 Oct 2010 17:55:22 +0200 From: eugen@leitl.org To: nanog@nanog.org Subject: Re: NTP Server
On Mon, Oct 25, 2010 at 02:51:24AM +1100, Ben McGinnes wrote:
How do you knew that your local NTP server knew what time it is? (for sure)
By polling as many stratum 1 and 2 time servers as possible. Having your own stratum 2 server(s) beats nebulous NTP servers out in the big bad Internet every time.
For those you care about that:
=
James R. Cutler james.cutler@consultant.com
James -- Well said. I was going to submit the exact same thing. This is what we we do at my company and it works extremely well - we only use three stratum-1 time servers, and three internal servers to go get the time from the three externals, via a one-to-one correspondence. Once all three internals have acquired the time from the three stratum-1 clocks, they all poll each other for the average. every host in the network is pointed to one of the three internals. On Sun, Oct 24, 2010 at 1:12 PM, Cutler James R <james.cutler@consultant.com
wrote:
Time Service is more complicated than just having a single NTP server. But it can be useful and is not really a luxury.
Two primary reasons for local time service are to reliably serve a network that is relatively or completely isolated from the general internet, and, to provide a local time source for "dumb" clients that is closer (less jitter) in network terms. Other reasons can include policy (everything in the network uses the same identical time service), policy (the time service is locally controlled), operational simplicity (the routers don't need to run NTP), and, separation of functions/operational responsibility (your run your servers, they run the backbone, I tell you the time.
Implementing a local time service is actually fairly simple, but fewer than four servers is wasted effort. I can't explain in just a few words how the servers interact and compute delays and jitter to come to an "accurate" time. Take my word or ask David Mills for all that.
Implementation of an internet-referenced time service involves the following: 1. Select a set of stratum one servers - pick open access servers or get permission to use limited access servers. Four to six should do. 2. Select a set local hosts on your network - DNS servers, for example. These should be well distributed. Four to six should do. The actual NTP load is small compared to DNS queries. 3. Configure the local hosts as peers using the stratum one set as servers. Use crypto authentication if you feel the need. 4. Add NTP monitoring to your network management process. 5. Advertise the local time servers to your network - DHCP, word of mouth, configuration requirements, configuration scripts, standard builds, etc.
It is simple enough to do for a five node home network. It is almost that simple for a network with hundreds of thousands of client nodes. I've done both.
On Oct 24, 2010, at 12:29 PM, Brandon Kim wrote:
I guess what I'm trying to understand is, is having your own NTP server
just a luxury?
I personally would like to have my own, I just need to pitch its
advantages to my company. Unless everyone here on the NANOG group
clearly spells it out to me that it's a luxury.
I can see it as an added service/benefit though to our customers.....
Date: Sun, 24 Oct 2010 17:55:22 +0200 From: eugen@leitl.org To: nanog@nanog.org Subject: Re: NTP Server
On Mon, Oct 25, 2010 at 02:51:24AM +1100, Ben McGinnes wrote:
How do you knew that your local NTP server knew what time it is? (for sure)
By polling as many stratum 1 and 2 time servers as possible. Having your own stratum 2 server(s) beats nebulous NTP servers out in the big bad Internet every time.
For those you care about that:
=
James R. Cutler james.cutler@consultant.com
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
On 10/24/2010 7:37 PM, Peter Lothberg wrote:
acquired the time from the three stratum-1 clocks, they all poll each other for the average.
How many clocks/servers do you need to average from to knew that you are within say 1ms of UTC(nist)?
What type of evidence model do you need to prove this with? - The NIST servers located around the US are mostly operated out of people like our operations (we have seven of them now and Atlanta coming online in about three weeks as well.) NTP has some foibles most are probably unaware of - that is it must have three (3) competent sources defined so that it can vote. We like to also say all three voices need to be coming from the same subnet so that the network latency and other physical aspects which control the policy-implementation are reliable as well. If you take one server from multiple sites you will be stuck with multiple network latency overhead factors polluting the resolution and certainty in the 'small bits' of your time-attestation. The real issue is how you prove the time-setting took. Or better yet - that you allow Applications to make their own NTP queries of reference time servers - that's really where the rubber meets the road in time-centric trust models. Todd Glassey
-P
-- //----------------------------------------------------------------- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
On 10/24/2010 12:29 PM, Brandon Kim wrote:
I guess what I'm trying to understand is, is having your own NTP server just a luxury?
I personally would like to have my own, I just need to pitch its advantages to my company. Unless everyone here on the NANOG group clearly spells it out to me that it's a luxury.
I can see it as an added service/benefit though to our customers.....
We have one internally because we use private IP'S on some of our own equipment for security reasons and those systems are unable to poll an external NTP server on the Internet. Plus some of our equipment only accepts a single NTP server and in the past we occasionally found external NTP servers to not be up, at least with our own server we know if it's accessible or not. As for pitching one to your company, not sure why that's an issue...talking about 500K app that can run on $50 pc with Linux from ebay Bret
On Sun, Oct 24, 2010 at 10:44 AM, Peter Lothberg <roll@stupi.se> wrote:
1) How necessary do you believe in local NTP servers? Do you really need th= e logs to be perfectly accurate? 2) If you do have a local NTP server=2C is it only for local internal use= =2C or do you provide this NTP server to your clients as an added service? 3) If you do have a local NTP server=2C do you have a standby local NTP ser= ver or do you use the internet as your standby server? Thoughts?
How do you knew that your local NTP server knew what time it is? (for sure)
this question is a trap.
On 10/24/10 10:20 AM, Christopher Morrow wrote:
On Sun, Oct 24, 2010 at 10:44 AM, Peter Lothberg <roll@stupi.se> wrote:
1) How necessary do you believe in local NTP servers? Do you really need th= e logs to be perfectly accurate? 2) If you do have a local NTP server=2C is it only for local internal use= =2C or do you provide this NTP server to your clients as an added service? 3) If you do have a local NTP server=2C do you have a standby local NTP ser= ver or do you use the internet as your standby server? Thoughts?
How do you knew that your local NTP server knew what time it is? (for sure)
this question is a trap.
a man with one watch knows what time it is, a man with two is never sure.
On Sun, Oct 24, 2010 at 1:24 PM, Joel Jaeggli <joelja@bogus.com> wrote:
On 10/24/10 10:20 AM, Christopher Morrow wrote:
On Sun, Oct 24, 2010 at 10:44 AM, Peter Lothberg <roll@stupi.se> wrote:
How do you knew that your local NTP server knew what time it is? (for sure)
this question is a trap.
a man with one watch knows what time it is, a man with two is never sure.
how about a man with 7?
On 24 Oct 2010, at 18:28, Christopher Morrow <morrowc.lists@gmail.com> wrote:
On Sun, Oct 24, 2010 at 1:24 PM, Joel Jaeggli <joelja@bogus.com> wrote:
On 10/24/10 10:20 AM, Christopher Morrow wrote:
On Sun, Oct 24, 2010 at 10:44 AM, Peter Lothberg <roll@stupi.se> wrote:
How do you knew that your local NTP server knew what time it is? (for sure)
this question is a trap.
a man with one watch knows what time it is, a man with two is never sure.
how about a man with 7?
He calibrates them against each other to find out which ones run fast and which slow. Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/
On Sun, 24 Oct 2010, Christopher Morrow wrote:
How do you knew that your local NTP server knew what time it is? (for sure)
this question is a trap.
Quite. We had 2 HP 5071s,(+ several GPS standards) and at the time being the definition of a second, either could be correct at any time. When I took a moment to think about it, and discovered both standards were correct, and yet disagreed, my brain fell out. -- Steven Hill "Women and cats will do as they please, and men and dogs should relax and get used to the idea." - Robert A. Heinlein
On 10/24/2010 2:14 PM, Steven Hill wrote:
On Sun, 24 Oct 2010, Christopher Morrow wrote:
How do you knew that your local NTP server knew what time it is? (for sure)
Because you got the time service from an authoritative source who did the rest of the work to make sure that the NTP evidence practice worked. This means you are a partner of the time-provider and you do both client and peering type time-service events. It also means you address the issues of certainty and the need to say absolutely what time it is in the US or where ever at any given instance. Let me be blunt... The NTP Evidence Model was designed for a totally unrelated purpose and doesn't make the needed hurdle today. In fact there are a number of related NTP Services necessary to create a complete service model and not all of them are compatible with running a NTP Service as a daemon unless you can do time-queries through NTPDATE or other tools on separate ports. Todd Glassey
this question is a trap.
Quite.
We had 2 HP 5071s,(+ several GPS standards) and at the time being the definition of a second, either could be correct at any time. When I took a moment to think about it, and discovered both standards were correct, and yet disagreed, my brain fell out.
-- //----------------------------------------------------------------- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
On Sun, 24 Oct 2010, Brandon Kim wrote:
1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate? 2) If you do have a local NTP server, is it only for local internal use, or do you provide this NTP server to your clients as an added service? 3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server?
First terminology. What do you mean by a local NTP server? Almost any Cisco/Juniper router, Unix server and some recent Windows servers have NTP server software and can synchronize clocks in your network. So you may already have a NTP server capable device. You just need to configure it, and give it a good source of time. It would be a Stratum 2 or greater NTP server because the good source of time is another NTP server. Left to itself, NTP is pretty good at keeping clocks in arbitrary networks synchronized with each other. But most people are also interested in synchronizing clocks with some official time source. The Network Time Protocol doesn't really have the notion of a "standby" server. It uses multiple time sources together, and works best with about four time sources. But for many end-systems, the Simple Network Time Protocol with a single time source may be sufficient. If you are in a regulated industry (stock broker, electric utility, 9-1-1 answering point, etc) there are specific time and frequency standards you must follow. On the other hand, are you are asking about a local clock receiver (radio, satellite, etc) for a stratum 1 NTP server? Clock receivers are getting cheaper, the problem is usually the antenna location. Or on the third hand, are you asking about local primary reference clock (caesium, rubium, etc) for a stratum 1 NTP server? These are still relatively expensive up to extremely expensive. Or on the fourth hand, are you a time scientist working to improve international time standards. If you are one of these folks, you already know. Most major ISPs use NTP across their router backbone, and incidently provide it to their customers. The local ISP router connected to your circuit probably has NTP enabled. Required accuracy is in the eye of the beholder. NASDAQ requires brokers to have their clocks synchronized within 3 seconds of UTC(NIST). 9-1-1 centers are required to have their clocks synchronized within 0.5 seconds of UTC. Kerberos/Active Directory requires clocks to be synchronized within 5 minutes of each other. If your log files have a resolution of 1 second, you probably won't see much benefit of sub-second clock precision or accuracy. If you are conducting distributed measurements with sub-microsecond resolution, you probably will want something more.
Hi Sean: By local I meant in-house, on-site in our datacenter. As far as what applications could use our NTP service, I would leave that up to each client and what they are running. For my own personal purposes, it would just be for log purposes. (error logs, syslogs, etc etc) I have heard that routers don't make good NTP servers since they weren't designed to keep track of time. This, I have read from a Cisco source. Can't remember where though. Or maybe they were just referring to older less powerful routers like 2500 series... Brandon
Date: Sun, 24 Oct 2010 14:42:24 -0400 From: sean@donelan.com To: nanog@nanog.org Subject: Re: NTP Server
On Sun, 24 Oct 2010, Brandon Kim wrote:
1) How necessary do you believe in local NTP servers? Do you really need the logs to be perfectly accurate? 2) If you do have a local NTP server, is it only for local internal use, or do you provide this NTP server to your clients as an added service? 3) If you do have a local NTP server, do you have a standby local NTP server or do you use the internet as your standby server?
First terminology. What do you mean by a local NTP server?
Almost any Cisco/Juniper router, Unix server and some recent Windows servers have NTP server software and can synchronize clocks in your network. So you may already have a NTP server capable device. You just need to configure it, and give it a good source of time. It would be a Stratum 2 or greater NTP server because the good source of time is another NTP server. Left to itself, NTP is pretty good at keeping clocks in arbitrary networks synchronized with each other. But most people are also interested in synchronizing clocks with some official time source.
The Network Time Protocol doesn't really have the notion of a "standby" server. It uses multiple time sources together, and works best with about four time sources. But for many end-systems, the Simple Network Time Protocol with a single time source may be sufficient.
If you are in a regulated industry (stock broker, electric utility, 9-1-1 answering point, etc) there are specific time and frequency standards you must follow.
On the other hand, are you are asking about a local clock receiver (radio, satellite, etc) for a stratum 1 NTP server? Clock receivers are getting cheaper, the problem is usually the antenna location.
Or on the third hand, are you asking about local primary reference clock (caesium, rubium, etc) for a stratum 1 NTP server? These are still relatively expensive up to extremely expensive.
Or on the fourth hand, are you a time scientist working to improve international time standards. If you are one of these folks, you already know.
Most major ISPs use NTP across their router backbone, and incidently provide it to their customers. The local ISP router connected to your circuit probably has NTP enabled.
Required accuracy is in the eye of the beholder. NASDAQ requires brokers to have their clocks synchronized within 3 seconds of UTC(NIST). 9-1-1 centers are required to have their clocks synchronized within 0.5 seconds of UTC. Kerberos/Active Directory requires clocks to be synchronized within 5 minutes of each other.
If your log files have a resolution of 1 second, you probably won't see much benefit of sub-second clock precision or accuracy. If you are conducting distributed measurements with sub-microsecond resolution, you probably will want something more.
Regarding leap seconds: A modern OS kernel using the NTP daemon to control time will always experience monotonic time. Negative leap seconds should result in the local clock slowing slightly until the local time matches the NTP-derived time. This is in strong contrast to what can happen when ntpdate or similar queries are used to adjust system time, as in the following example from a popular CPE log where the local CPU clock tended to run fast: Oct 24 04:27:59 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 05:28:00 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 06:28:00 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted -1 seconds). Oct 24 07:28:01 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 08:28:01 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted -1 seconds). Oct 24 09:28:02 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted -1 seconds). Oct 24 10:28:03 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 11:28:04 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 12:28:04 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 13:28:05 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 14:28:05 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 15:28:05 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted -1 seconds). Oct 24 16:28:06 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted -1 seconds). Oct 24 17:28:07 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 18:28:08 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted -1 seconds). Oct 24 19:28:08 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted +0 seconds). Oct 24 20:28:09 192.168.1.1 HomeN ntp: Clock synchronized to network time server time.apple.com (adjusted -1 seconds). Regarding local time servers: This is the situation in which local clients, at least those with UNIX or UNIX-like OS can take advantage of ntpd and local time servers to have consistent and monotonic time across your network with a measure of insulation from external vagaries. Yes, I have run ntpd on Windows systems, but have no quotable experience with the current Windows version (Windows 7). James R. Cutler james.cutler@consultant.com
On Sun, 24 Oct 2010, Brandon Kim wrote:
By local I meant in-house, on-site in our datacenter.
What do you think it means to have a NTP server in-house, on-site in your datacenter? There all many different levels of NTP servers. Putting some free software on a spare computer, and synchronizing it to a few public NTP servers on the Internet? Or buying a $5,000 specialized NTP hardware device (or more if you want backups) and installing an external antenna to pick up a radio reference clock source from a satellite or radio station? If you already provide DNS/DHCP and other services in your datacenter, its usually not that much effort to add the NTP service. In many cases, the software is already part of the base operating system package, or easily added to most modern systems. But in most cases, NTP seems to be treated as an unsupported service. If it works, great. If it doesn't, don't complain. If the person who cared about NTP leaves, no one else even knows it exists. If you need traceable time, or have some other regulatory requirement, its going to be more work. My point is there isn't one answer.
On Sun, 24 Oct 2010 20:15:56 -0400 Brandon Kim <brandon.kim@brandontek.com> wrote:
I have heard that routers don't make good NTP servers since they weren't designed to keep track of time. This, I have read from a Cisco source. Can't remember where though. Or maybe they were just referring to older less powerful routers like 2500 series...
I've implemented a two separate sets of stratum-2 services in two different academic networks using retired Cisco router gear. As far as I know they are still operating fine, providing time primarily to the institution's other infrastructure and server devices. They only provide time services. I had done some rudimentary stress testing and benchmarking at the time. They performed sufficiently. The advantage of this setup was that they were simple to setup, easy to manage and cheap to replace with the same retired gear we had an abundance of. Maybe the clock resolution isn't as precise as some other hardware, but for the purposes I had used it for it seemed fine. I doubt the underlying code has changed all that much, but at one time David Mills gave his stamp of approval: <http://groups.google.com/group/comp.protocols.time.ntp/msg/1afed797bf898dd0?dmode=source> On another note, contrary to another's position, I'd advocate not implementing public NTP service along with your DNS infrastrucure if at all possible. Co-mingling of critical network services such as naming, routing and time not only with themselves, but also with other less critical network infrastructure subsystems (e.g. web, mail) should generally be avoided in all, but the most resource constrained environments. John
On Sun, Oct 24, 2010 at 8:34 AM, Brandon Kim <brandon.kim@brandontek.com> wrote:
Hey guys:
I wanted to open up this question regarding NTP server. I recalled someone had created a posting of this quite awhile back.
From a service provider/ISP standpoint, does anyone think that having a local NTP server is really necessary?
I've asked some of my fellow engineers at work and many of them gives me the same response, "Can't we just use free ones out on the internet?"
Depends on how much you trust other people. NTP can potentially be used as a DoS vector by your upstream clocks, if you're not running your own. I've seen 50,000 servers panic in the blink of an eye when the NTP source issued a leap second, and the kernel wasn't patched to handle it properly; and that's a forward leap second. Nobody's tested reverse leap seconds yet; who knows what would happen to your hosts if your upstream NTP servers decided to issue a reverse leap second towards you? Granted, if you choose enough diverse upstream clocks, that becomes more difficult for someone to exploit; but it's not impossible, and you can't count on keeping your upstream clock sources secret, given the bidirectional communication that can take place between NTP servers. *shrug* It's cheap enough to run your own clock sources, once you're above a certain size, and it's one less potential attack vector from the outside; why wouldn't you want to secure your edge against it? Matt
On Oct 24, 2010, at 4:48 PM, Matthew Petach wrote:
On Sun, Oct 24, 2010 at 8:34 AM, Brandon Kim <brandon.kim@brandontek.com> wrote:
Hey guys:
I wanted to open up this question regarding NTP server. I recalled someone had created a posting of this quite awhile back.
From a service provider/ISP standpoint, does anyone think that having a local NTP server is really necessary?
I've asked some of my fellow engineers at work and many of them gives me the same response, "Can't we just use free ones out on the internet?"
Depends on how much you trust other people. NTP can potentially be used as a DoS vector by your upstream clocks, if you're not running your own.
I've seen 50,000 servers panic in the blink of an eye when the NTP source issued a leap second, and the kernel wasn't patched to handle it properly; and that's a forward leap second. Nobody's tested reverse leap seconds yet; who knows what would happen to your hosts if your upstream NTP servers decided to issue a reverse leap second towards you?
Negative leap seconds are certainly possible, and 20 years ago (when I was working for the USNO Directorate of Time) I thought that the currents down in the core might be going to give us a few; I have often wondered how many systems would choke on this. Regards Marshall
Granted, if you choose enough diverse upstream clocks, that becomes more difficult for someone to exploit; but it's not impossible, and you can't count on keeping your upstream clock sources secret, given the bidirectional communication that can take place between NTP servers.
*shrug* It's cheap enough to run your own clock sources, once you're above a certain size, and it's one less potential attack vector from the outside; why wouldn't you want to secure your edge against it?
Matt
On Oct 25, 2010, at 3:48 AM, Matthew Petach wrote:
NTP can potentially be used as a DoS vector by your upstream clocks, if you're not running your own.
+1 Also, if you experience a network partition event for any reason (DDoS attack, backhoe attack, et. al.) which disrupts communications between your network and the one(s) on the Internet where the public ntp servers you're using live, the accuracy of your time-hack becomes a concern just at the moment when you need it the most for combinatorial analysis of multiple forms of telemetry. And of course, time services for your infrastructure/services/apps ought to run across your DCN, anyways, which should be kept isolated from your production network (you don't want to rely upon proxies to enable something as critical as time service, IMHO). As Sean pointed out, all your routers from modern vendors are ntp-capable, and getting a couple of radio cards for servers to sync with WWVB isn't very expensive, assuming you can plug into an aerial which gets good reception: <http://www.nist.gov/pml/div688/grp40/wwvb.cfm> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.
Nowadays is not that difficult to get off the shelf solutions for different applications. Just to point to one: http://www.symmetricom.com/products/ntp-servers/ntp-network-appliances/ Regards Jorge
On Mon, 25 Oct 2010, Dobbins, Roland wrote:>
On Oct 25, 2010, at 3:48 AM, Matthew Petach wrote:
NTP can potentially be used as a DoS vector by your upstream clocks, if you're not running your own. +1
Also, if you experience a network partition event for any reason (DDoS attack, backhoe attack, et. al.) which disrupts communications between your network and the one(s) on the Internet where the public ntp servers you're using live, the accuracy of your time-hack becomes a concern just at the moment when you need it the most for combinatorial analysis of multiple forms of telemetry.
Modern versions of NTP have a relatively long polling interval once the clock is stable. Unless you are already using specialized timing hardware, your tolorance of the clock drift on off-the-shelf computers and routers is not going to be an immediate issue during short-term or even medium-term network problems. Any clock source can have an indeterminate outage. Generally the longer the hold time, the more expensive the clock hardware.
And of course, time services for your infrastructure/services/apps ought to run across your DCN, anyways, which should be kept isolated from your production network (you don't want to rely upon proxies to enable something as critical as time service, IMHO).
NTP started on Fuzzball routers. Its very light-weight on any hardware. There are lots of reasons not to have customers accessing your infrastructure devices. Lots of NTP queries can overload any device. Although your infrastructure devices should still have synchronized clocks with the rest of your infrastructure. If you have an enterprise network dependent on firewalls, another pin-hole through the firewall for NTP port 123 is also an another opportunity for mischief. There are lots of different ways to measure time. But I've noticed some people seem to create extreme Rube Goldberg contraptions. Figure out what precision and accuracy you really need. Time is always just an estimate.
participants (29)
-
Ben McGinnes -
Brandon Kim -
Bret Clark -
Brielle Bruns -
Christopher Morrow -
Cutler James R -
David Andersen -
Dobbins, Roland -
Eugen Leitl -
George Bonser -
Jack Carrozzo -
Joe Greco -
Joel Jaeggli -
John Kristoff -
Jorge Amodio -
Leo Bicknell -
Marshall Eubanks -
Matthew Petach -
Nick Hilliard -
Peter Beckman -
Peter Lothberg -
Randy Bush -
Robert E. Seastrom -
Sean Donelan -
Steven Fischer -
Steven Hill -
todd glassey -
Tony Finch -
William F. Maton Sotomayor