On Mon, Jun 30, 2014 at 7:27 PM, Majdi S. Abbas <msa@latt.net> wrote:
On Mon, Jun 30, 2014 at 05:33:52PM -0700, Tim Heckman wrote:
I just was alerted to one of the systems I managed having a time skew greater than 100ms from NTP sources. Upon further investigation it seemed that the time was off by almost exactly 1 second.
Looking back over our NTP monitoring, it would appear that this system had a large time adjust at approximately 00:00 UTC:
Okay. Do you have any logging configured (peerstats, etc?) for ntpd?
Our systems all have loopstats and peerstats logging enabled. I have those log files available if interested. However, when I searched over the files I wasn't able to find anything that seemed to indicate this was the peer who told the system to introduce a leap second. That said, I might just not know what to look for in the logs.
A few of our systems did alert early this morning, indicating they were going to be receiving a leap second today. However, I was unable to determine the exact cause for NTP believing a leap second should be added. And after some time a few of the systems were no longer indicating that a leap second would be introduced.
This can happen if a server is either passing along a leap notification that it received, or is configured to use a leapseconds file that is incorrect.
Correct, I was hoping to determine which peer it was so I can reach out to them to make sure this doesn't bleed in to the pool at the end of the year. I was also more-or-less curious how wide-spread of an issue this was, but I'm starting to think I may have been the only person to catch it in the act. :)
This specific system is hosted in AWS US-WEST-2C and uses the 0.amazon.pool.ntp.org pool.
0 is just one server in the pool (whichever you draw by rotation); is this the only server you have configured?
We use 0.amazon.pool.ntp.org, 1.amazon.pool.ntp.org, and 2.amazon.pool.ntp.org. As with the other widely-used pool hostnames, each of these is a round-robin DNS entry with 4 hosts and a TTL of 150s.
--msa
Thank you for getting back to me. Cheers! -Tim