Just a reminder, at midnight UTC there's a leap second added to most time systems. Some time systems will stop the clock at 23:59:59.999999 for 1 second, some will display 23:59:60 for a second. Since the last leap second (1998), "leap second aware" time keeping systems(NTP, GPS, etc) have become much more prevalent, so it's much more likely this time that applications and NTP sync'ed devices will see a leap second happen(rather than have them manually corrected later, or not corrected at all). But, I'm not too sure how well everyone has tested applications and devices for how they handle the clock stopping for a second OR an "invalid" time of 23:59:60. If anyone sees anything die at 00:00:00UTC I'd be interested to know. -- Kevin
On Dec 31, 2005, at 11:58 AM, Kevin Day wrote:
Just a reminder, at midnight UTC there's a leap second added to most time systems.
I've had quite a few replies already off list, that I'll sum up: 1) ISO 8601 specifically defines ":60" as a valid representation for seconds, just for this event. If your app breaks, it's probably not obeying the relevant RFC. 2) Anything using the obvious implementation of regular expressions to check for a valid time is probably going to fail "23:59:60", leading to events not being logged, or transient weirdness for that second. One example I thought of after a few minutes is SpamAssassin, who will flag a mail header: Date: Sat, 31 Dec 2005 23:59:60 +0000 As "INVALID_DATE", bumping up its spam score by 1.7 to 2.0(by default), because of the regexp it uses which contains "[0-5][0-9]" 3) Two people have noted that their Juniper routers spat out some strange ntpd messages earlier today, such as: Dec 31 06:26:46 hostname xntpd[66479]: time reset 0.999681 s Dec 31 06:48:11 hostname xntpd[66479]: time reset -0.437785 s (local time set to America/Chicago) 4) You can't set your clock to 23:59:60 manually, which makes testing what's going to happen difficult. 5) The "leap second pending" bit that's supposed to have been set by now in NTP isn't being set by some clock sources, leading some to wonder what they're planning on doing at midnight.
Last NTP spam: I'm by no means an NTP expert, if anyone else is, please pipe up. About 30 minutes before the leap second should have occurred, several of our systems reported "xntpd[13742]: time reset 0.958385 s", which was really strange. They moved the wrong direction, and they did it early. Shortly after, those systems lost ntp association and began drifting. About 10 minutes after midnight all have regained sync. I wasn't checking things that early to see why, it's possible some of our NTP sources started disagreeing on what the correct time was, and would also match what other people have reported off-list, going back as far as 18 hours before midnight. Several public NTP sources are now indicating a "leap second alarm" (setting the leap bits to 11), which will cause most NTP clients to rule them out as a source. ntp-2.gw.uiuc.edu is an example: 130.126.24.44: Server dropped: Leap not in sync server 130.126.24.44, port 123 stratum 2, precision -19, leap 11, trust 000 refid [128.174.38.133], delay 0.03357, dispersion 0.00049 According to ntpdate, its clock seems to have stopped about 5 minutes before midnight, and hasn't yet recovered. Other NTP servers haven't cleared their "today is a leap second day" bit, which they should have by now. Some NTP implementations rule out servers that don't agree with what their "master" server thinks the leap second bits should be. My reading of the NTP spec says that at 00:00:00 the leap bits should have been returned to zero. Attempting to sync from one of these servers will produce a "Next leap second occurs at 00:00:00.000 UTC Sun Jan 01 2006" message, but that should be harmless as long as they correct themselves a while before midnight. Still others have their clocks off by a significant amount(10+ minutes) and think they're still in sync, but since I started typing this email, they all have corrected themselves. While I can't say anything broke on our network as a result of the leap second, a good percentage of our gear lost NTP sync or had some kind of NTP problem around midnight UTC. You may want to check your NTP status at some point, in case something drifted quite a way off and won't step itself back now because the difference is too great. -- Kevin
Kevin Day wrote:
Last NTP spam:
I'm by no means an NTP expert, if anyone else is, please pipe up.
About 30 minutes before the leap second should have occurred, several of our systems reported "xntpd[13742]: time reset 0.958385 s", which was really strange. They moved the wrong direction, and they did it early. Shortly after, those systems lost ntp association and began drifting. About 10 minutes after midnight all have regained sync. I wasn't checking things that early to see why, it's possible some of our NTP sources started disagreeing on what the correct time was, and would also match what other people have reported off-list, going back as far as 18 hours before midnight.
Several public NTP sources are now indicating a "leap second alarm" (setting the leap bits to 11), which will cause most NTP clients to rule them out as a source. ntp-2.gw.uiuc.edu is an example:
130.126.24.44: Server dropped: Leap not in sync server 130.126.24.44, port 123 stratum 2, precision -19, leap 11, trust 000 refid [128.174.38.133], delay 0.03357, dispersion 0.00049
According to ntpdate, its clock seems to have stopped about 5 minutes before midnight, and hasn't yet recovered.
Other NTP servers haven't cleared their "today is a leap second day" bit, which they should have by now. Some NTP implementations rule out servers that don't agree with what their "master" server thinks the leap second bits should be. My reading of the NTP spec says that at 00:00:00 the leap bits should have been returned to zero. Attempting to sync from one of these servers will produce a "Next leap second occurs at 00:00:00.000 UTC Sun Jan 01 2006" message, but that should be harmless as long as they correct themselves a while before midnight.
Still others have their clocks off by a significant amount(10+ minutes) and think they're still in sync, but since I started typing this email, they all have corrected themselves.
While I can't say anything broke on our network as a result of the leap second, a good percentage of our gear lost NTP sync or had some kind of NTP problem around midnight UTC. You may want to check your NTP status at some point, in case something drifted quite a way off and won't step itself back now because the difference is too great.
-- Kevin
There is at least one stratum-1 server here on the West coast that my NTP says is now off by 1 second. Several stratum-2 are synced to it and are now off also. So checking servers might be a good idea Roy Engehausen
On Sat, Dec 31, 2005 at 05:06:59PM -0800, Roy wrote:
Kevin Day wrote:
Last NTP spam:
I'm by no means an NTP expert, if anyone else is, please pipe up.
About 30 minutes before the leap second should have occurred, several of our systems reported "xntpd[13742]: time reset 0.958385 s", which was really strange. They moved the wrong direction, and they did it early. Shortly after, those systems lost ntp association and began drifting. About 10 minutes after midnight all have regained sync. I wasn't checking things that early to see why, it's possible some of our NTP sources started disagreeing on what the correct time was, and would also match what other people have reported off-list, going back as far as 18 hours before midnight.
Several public NTP sources are now indicating a "leap second alarm" (setting the leap bits to 11), which will cause most NTP clients to rule them out as a source. ntp-2.gw.uiuc.edu is an example:
130.126.24.44: Server dropped: Leap not in sync server 130.126.24.44, port 123 stratum 2, precision -19, leap 11, trust 000 refid [128.174.38.133], delay 0.03357, dispersion 0.00049
According to ntpdate, its clock seems to have stopped about 5 minutes before midnight, and hasn't yet recovered.
Other NTP servers haven't cleared their "today is a leap second day" bit, which they should have by now. Some NTP implementations rule out servers that don't agree with what their "master" server thinks the leap second bits should be. My reading of the NTP spec says that at 00:00:00 the leap bits should have been returned to zero. Attempting to sync from one of these servers will produce a "Next leap second occurs at 00:00:00.000 UTC Sun Jan 01 2006" message, but that should be harmless as long as they correct themselves a while before midnight.
Still others have their clocks off by a significant amount(10+ minutes) and think they're still in sync, but since I started typing this email, they all have corrected themselves.
While I can't say anything broke on our network as a result of the leap second, a good percentage of our gear lost NTP sync or had some kind of NTP problem around midnight UTC. You may want to check your NTP status at some point, in case something drifted quite a way off and won't step itself back now because the difference is too great.
-- Kevin
There is at least one stratum-1 server here on the West coast that my NTP says is now off by 1 second. Several stratum-2 are synced to it and are now off also. So checking servers might be a good idea
Are they a GPS sync (or do you know?) The GPS system doesn't really handle the leap second situation the same as others, there is a little blurb here that talks about it: http://gpsinformation.net/main/gpstime.htm I know that I saw my GPS output the following: @051231235958 @051231235959 @060101000000 @060101000000 @060101000001 @060101000002 @060101000003 Which is mostly correct, it should have really read 5960 instead. I saw my el-cheapo clocks drift by a second at midnight utc. I suggest changing your clock sources to something more reliable if you're seeing folks that are not diligent stratum-1 sources. I suggest this as a source: http://www.stupi.se/Time/ I'm kinda curious what CDMA network clocks said around this time or if they just drifted, i'm sure someone here put their cmda phone in debug and watched it. - jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Once upon a time, Kevin Day <toasty@dragondata.com> said:
While I can't say anything broke on our network as a result of the leap second, a good percentage of our gear lost NTP sync or had some kind of NTP problem around midnight UTC. You may want to check your NTP status at some point, in case something drifted quite a way off and won't step itself back now because the difference is too great.
I watched my Tru64 5.1B and Linux 2.2-2.6 servers (NTP wasn't running on my Solaris 9 server accidentally) and Juniper and Cisco gear, and they all stayed in sync. The Linux systems logged: Dec 31 17:59:59 kosh kernel: Clock: inserting leap second 23:59:60 UTC 17:59:59 lasted 2 seconds in local time (since the normal time zone data doesn't pass through leap seconds). I saw the following from JUNOS: Dec 31 18:00:00 hsvrouter /kernel: microuptime() went backwards (8144133.847075 -> 8144132.881330) Tru64 and Cisco didn't log anything. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
--On December 31, 2005 6:57:45 PM -0600 Kevin Day <toasty@dragondata.com> wrote:
While I can't say anything broke on our network as a result of the leap second, a good percentage of our gear lost NTP sync or had some kind of NTP problem around midnight UTC. You may want to check your NTP status at some point, in case something drifted quite a way off and won't step itself back now because the difference is too great.
We've Nagios monitoring a majority of our NTP devices. Around the appropriate time I got a pretty big flurry of ntp sync warnings, took about half an hour for everything to get in sync. Everything looks normal as of right now (has been for a while). I hadn't thought to turn off the alarms even though I was aware of the leap. That resulted in a lot of notifications going out to our on-call people.
Interesting. ntp-2.gw.uiuc.edu is an old Cisco box running 12.0(27). I wasn't around for the leap second, so I don't have any information about what happened, and its log is completely silent. FWIW, its clock and NTP process appear fine now. It's synced off our GPS clock, which may be a part of the problem. ntp-2>show ntp sta Clock is synchronized, stratum 2, reference is 128.174.38.133 nominal freq is 250.0000 Hz, actual freq is 249.9982 Hz, precision is 2**19 reference time is C7642455.CAE491EF (16:14:45.792 CST Mon Jan 2 2006) clock offset is 0.0594 msec, root delay is 3.57 msec root dispersion is 6.30 msec, peer dispersion is 0.12 msec ntp-2>show ntp asso address ref clock st when poll reach delay offset disp +~130.126.24.24 128.174.38.133 2 192 1024 336 6.4 -0.15 4.0 +~192.5.41.40 .USNO. 1 477 1024 377 32.5 -2.24 1.5 *~128.174.38.133 .PPS. 1 442 1024 377 3.6 0.06 0.1 +~130.126.24.53 128.174.38.133 2 623 1024 377 8.5 1.20 3.1 * master (synced), # master (unsynced), + selected, - candidate, ~ configured /cvk On Dec 31, 2005, at 6:57p, Kevin Day wrote:
Several public NTP sources are now indicating a "leap second alarm" (setting the leap bits to 11), which will cause most NTP clients to rule them out as a source. ntp-2.gw.uiuc.edu is an example:
130.126.24.44: Server dropped: Leap not in sync server 130.126.24.44, port 123 stratum 2, precision -19, leap 11, trust 000 refid [128.174.38.133], delay 0.03357, dispersion 0.00049
According to ntpdate, its clock seems to have stopped about 5 minutes before midnight, and hasn't yet recovered.
The most comprehensive supplier information I have found so far. http://www.ntp-systems.com/leapsecond/ Colin Johnston HHT IT
On Dec 31, 2005, at 11:58 AM, Kevin Day wrote:
Just a reminder, at midnight UTC there's a leap second added to most time systems.
Some time systems will stop the clock at 23:59:59.999999 for 1 second, some will display 23:59:60 for a second.
Since the last leap second (1998), "leap second aware" time keeping systems(NTP, GPS, etc) have become much more prevalent, so it's much more likely this time that applications and NTP sync'ed devices will see a leap second happen(rather than have them manually corrected later, or not corrected at all). But, I'm not too sure how well everyone has tested applications and devices for how they handle the clock stopping for a second OR an "invalid" time of 23:59:60.
If anyone sees anything die at 00:00:00UTC I'd be interested to know.
-- Kevin
My Juniper seems to be aware: xxx@juniper> show ntp status status=4694 leap_add_sec, sync_ntp, 9 events, event_peer/strat_chg, version="ntpd 4.1.0-a Thu Jul 14 23:46:40 GMT 2005 (1)", processor="i386", system="JUNOS7.3R1.5", leap=01, stratum=3, precision=-30, rootdelay=40.669, rootdispersion=49.522, peer=35302, refid=ntpx.xxx.xxx, reftime=c7615c1c.65f78359 Sat, Dec 31 2005 13:35:56.398, poll=10, clock=c7615d8e.d66d698f Sat, Dec 31 2005 13:42:06.837, state=4, offset=-2.649, frequency=73.810, jitter=5.194, stability=0.024 note the leap=01 and leap_add_sec
Cisco seems happy as well. (adding leap second, leap second occurs at), etc.
sh ntp status Clock is synchronized (adding leap second), stratum 2, reference is xxx.xxx.xxx.xxx nominal freq is 250.0000 Hz, actual freq is 249.9975 Hz, precision is 2**18 reference time is C7617E39.3A0B3D8C (17:01:29.226 EST Sat Dec 31 2005) clock offset is 0.5332 msec, root delay is 5.11 msec root dispersion is 7.72 msec, peer dispersion is 7.14 msec Leap second occurs at C7619A00.00000000 (19:00:00.000 EST Sat Dec 31 2005)
Happy New Year! Deepak Gerry Boudreaux wrote:
On Dec 31, 2005, at 11:58 AM, Kevin Day wrote:
Just a reminder, at midnight UTC there's a leap second added to most time systems.
Some time systems will stop the clock at 23:59:59.999999 for 1 second, some will display 23:59:60 for a second.
Since the last leap second (1998), "leap second aware" time keeping systems(NTP, GPS, etc) have become much more prevalent, so it's much more likely this time that applications and NTP sync'ed devices will see a leap second happen(rather than have them manually corrected later, or not corrected at all). But, I'm not too sure how well everyone has tested applications and devices for how they handle the clock stopping for a second OR an "invalid" time of 23:59:60.
If anyone sees anything die at 00:00:00UTC I'd be interested to know.
-- Kevin
My Juniper seems to be aware:
xxx@juniper> show ntp status status=4694 leap_add_sec, sync_ntp, 9 events, event_peer/strat_chg, version="ntpd 4.1.0-a Thu Jul 14 23:46:40 GMT 2005 (1)", processor="i386", system="JUNOS7.3R1.5", leap=01, stratum=3, precision=-30, rootdelay=40.669, rootdispersion=49.522, peer=35302, refid=ntpx.xxx.xxx, reftime=c7615c1c.65f78359 Sat, Dec 31 2005 13:35:56.398, poll=10, clock=c7615d8e.d66d698f Sat, Dec 31 2005 13:42:06.837, state=4, offset=-2.649, frequency=73.810, jitter=5.194, stability=0.024
note the leap=01 and leap_add_sec
On (2005-12-31 17:18 -0500), Deepak Jain wrote: Linux seemed to survive happily too [ytti@foo ~]% dmesg|tail -n1 Clock: inserting leap second 23:59:60 UTC Curious though that not so many people who've I asked got these messages, only explanation I can think of is that their NTP peers weren't telling it. Without much NTP clues, could someone explain what steps NTP takes to protect itself from attackers spoofing packets and forcing you to leap? Probably sane implementation would restrict leaping to happen only at 23:59:59, so you'd drift maximum of 1s/d? But crappy implementation might allow you to leap every second. This http://www.ntp.org/ntpfaq/NTP-s-config-adv.htm#AEN2950 just tells Most users of NTP do not need authentication as the protocol contains several filters against bad time. So I guess it's pretty implementation spesific how the input is sanitized.
Cisco seems happy as well. (adding leap second, leap second occurs at), etc.
sh ntp status Clock is synchronized (adding leap second), stratum 2, reference is xxx.xxx.xxx.xxx nominal freq is 250.0000 Hz, actual freq is 249.9975 Hz, precision is 2**18 reference time is C7617E39.3A0B3D8C (17:01:29.226 EST Sat Dec 31 2005) clock offset is 0.5332 msec, root delay is 5.11 msec root dispersion is 7.72 msec, peer dispersion is 7.14 msec Leap second occurs at C7619A00.00000000 (19:00:00.000 EST Sat Dec 31 2005)
Happy New Year!
Deepak
Gerry Boudreaux wrote:
On Dec 31, 2005, at 11:58 AM, Kevin Day wrote:
Just a reminder, at midnight UTC there's a leap second added to most time systems.
Some time systems will stop the clock at 23:59:59.999999 for 1 second, some will display 23:59:60 for a second.
Since the last leap second (1998), "leap second aware" time keeping systems(NTP, GPS, etc) have become much more prevalent, so it's much more likely this time that applications and NTP sync'ed devices will see a leap second happen(rather than have them manually corrected later, or not corrected at all). But, I'm not too sure how well everyone has tested applications and devices for how they handle the clock stopping for a second OR an "invalid" time of 23:59:60.
If anyone sees anything die at 00:00:00UTC I'd be interested to know.
-- Kevin
My Juniper seems to be aware:
xxx@juniper> show ntp status status=4694 leap_add_sec, sync_ntp, 9 events, event_peer/strat_chg, version="ntpd 4.1.0-a Thu Jul 14 23:46:40 GMT 2005 (1)", processor="i386", system="JUNOS7.3R1.5", leap=01, stratum=3, precision=-30, rootdelay=40.669, rootdispersion=49.522, peer=35302, refid=ntpx.xxx.xxx, reftime=c7615c1c.65f78359 Sat, Dec 31 2005 13:35:56.398, poll=10, clock=c7615d8e.d66d698f Sat, Dec 31 2005 13:42:06.837, state=4, offset=-2.649, frequency=73.810, jitter=5.194, stability=0.024
note the leap=01 and leap_add_sec
-- ++ytti
participants (10)
-
Charley Kline -
Chris Adams -
Colin Johnston -
Deepak Jain -
Gerry Boudreaux -
Jared Mauch -
Kevin Day -
Michael Loftis -
Roy -
Saku Ytti