On Tue, Mar 17, 2009 at 1:07 AM, Ask Bjørn Hansen <ask@develooper.com>wrote:

On Dec 31, 2008, at 15:28, Kevin Oberman wrote:

We use CDMA clocks and last leap second it took weeks for all of the

...

cell sites to adjust the last one. As a result, I have set all of our clocks for manual leap second and set them to adjust tonight at midnight (UTC).I'll take a look in about 35 minutes and see how it worked.

Chiming in a little late here ...

Oh, quiet. After all, what's 6.5 million seconds or so between friends? -j -- Jamie Rishaw // .com.arpa@j <- reverse it. ish. [Impressive C-level Title Here], arpa / arpa labs

On Tue, 17 Mar 2009 08:06:51 PDT, Kevin Oberman said:

Routers as ntp servers. Yuck! Routers route well, but they treat time as a low priority job and jitter on Cisco routers is simply terrible. Junipers do better, but are still a poor time server.

They may suck for being a Stratum-1/2 server, but even the most jittery Cisco is still far and away good enough to serve up a ntpdate so that an end-user PC-class machine is in the right minute.

As long as the end-user is made aware that the accuracy of said NTP clock is +/- 30.000 seconds (or whatever jitter might exist). Seems kind of ridiculous to use an NTP source that is, for many purposes, wildly inaccurate. For my purposes, wildly is more than +/- 0.1 seconds. Trying to troubleshoot a problem, network or server, where the timestamps on each server/router/device vary inconsistently, is like walking on broken fluorescent bulbs -- painful and dangerous to one's health.

Not being a time geek, since Cisco's were called out for being wild jitter-mongers... how much jitter are we talking about? Clock is synchronized, stratum 2, xxxxxxxxxxxxxxxxxxxxxxxx nominal freq is 250.0000 Hz, actual freq is 249.9989 Hz, precision is 2**18 reference time is CD6A7CD4.45A9BB00 (19:47:32.272 UTC Tue Mar 17 2009) clock offset is 2.0581 msec, root delay is 29.62 msec root dispersion is 6.81 msec, peer dispersion is 3.30 msec Are we talking about +/- 30 seconds, or a problem bounded by +/- 30 msec? Deepak Jain AiNET

Not being a time geek, since Cisco's were called out for being wild jitter-mongers... how much jitter are we talking about?

Clock is synchronized, stratum 2, xxxxxxxxxxxxxxxxxxxxxxxx nominal freq is 250.0000 Hz, actual freq is 249.9989 Hz, precision is 2**18 reference time is CD6A7CD4.45A9BB00 (19:47:32.272 UTC Tue Mar 17 2009) clock offset is 2.0581 msec, root delay is 29.62 msec root dispersion is 6.81 msec, peer dispersion is 3.30 msec

Are we talking about +/- 30 seconds, or a problem bounded by +/- 30 msec?

I've actually been gathering some statistics on this using Munin (http://munin.projects.linpro.no/) on my linux server. There's currently 10 ntp servers being monitored and one of them is a 7600-series Cisco, which is handling quite a bit of traffic (CPU load around 20%). Here are the Munin graphs for it http://dx.fi/alt/ntp/7600.png (times in Finnish time, UTC+2). In comparison, here are the same graphs for time1.mikes.fi (a stratum-2 clock provided by the Finnish Centre for metrology and accreditation) http://dx.fi/alt/ntp/time1.mikes.fi.png and for Netnods stratum-1 clock in Stockholm http://dx.fi/alt/ntp/ntp1.sth.netnod.se.png Best regards, -- Tero Toikkanen Nebula Oy

This begs the question - how the heck do timekeepers and politicians get away with last minute time changes? Surely there's -some- pushback from technology related interest groups to try and get more than four weeks warning? :) Adrian On Mon, Jan 05, 2009, Frank Bulk wrote:

A report from a DHCP/DNS appliance vendor here: ==================== Several customers have reported a complete lock-up of their Proteus system around the beginning of January 1st 2009. We believe that we have traced this to a problem in the underlying kernel and NTP and the handling of the date change associated with 2008 being a Leap Year and therefore having 366 days.

Several conditions must be met to trigger this problem: 1. The Proteus was originally installed as v2.1.x or earlier. 2. NTP is enabled as a client with 2 or more external source servers defined. 3. There is a discrepancy in the times reported back by these other NTP servers.

There is no correction available at this time, and the resolution is to power cycle the system, after which it will run fine.

If you experienced a similar problem at the indicated time, please submit a trouble ticket so that we can confirm that this occurred on your system. ====================

I don't know what the underlying OS is.

Frank

-----Original Message----- From: Kevin Day [mailto:toasty@dragondata.com] Sent: Wednesday, December 31, 2008 4:42 PM To: NANOG list Subject: Leap second tonight

Just a reminder that there's a leap second tonight.

Last time I watched for what happened on 01/01/2006, there was a little bit of chaos: http://markmail.org/message/cpoj3jw5onzhhjkr?q=%22kevin+day%22+leap+second+r eminder+nanog&page=1&refer=cnkxb3iv7sls5axu

I've been told that some of the causes of these problems are fixed on any reasonably recent ntp distribution, but just in case, you might wanna keep an eye out if you're seeing any weirdness. The worst damage I'd heard from anyone after that event was their clock being significantly off for several hours.

-- Kevin

-- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -

On Mon, Jan 05, 2009, Nick Hilliard wrote:

Notice for the leap second was issued on July 4 2008.

http://hpiers.obspm.fr/iers/bul/bulc/bulletinc.36

Wow, how'd I miss that, I wonder? :) I'm just angry at the jack moves pulled by last minute timezone changes back in Australia, and the massively stupid repercussions seen throughout chunks of IT (incl. network auditing setups I had to poke at the time.) I'll add "handling second == 60" to the list of things I should check for in my code. Thanks. :) Adrian

On 05/01/2009 6:01, "Nick Hilliard" <nick@foobar.org> wrote: [...]

But seriously. Leap seconds occur every couple of years, either on July 30th and Dec 31. Sometimes both. And sometimes every consecutive year for a couple of years on the run.

It's theoretically possible for leap seconds to be introduced at the end of March and September. It's never happened but it might, so I suppose software should allow for the possibility. Regards, Leo

On Tue, Jan 06, 2009 at 01:30:51AM +0900, Adrian Chadd wrote:

This begs the question - how the heck do timekeepers and politicians get away with last minute time changes?

Surely there's -some- pushback from technology related interest groups to try and get more than four weeks warning? :)

Try six months. NTP itself sets the leap indicator by 28 days prior to the leap and clears it before the end of the following day, so in theory the appliance itself had at least 4 weeks notice and the rest of us had an additional five months. IERS announces a pending leap second six months in advance. The announcement for this one was dated July 4th. System vendors have only had 37 years since the first leap second to figure this out; please be patient. However, I can't excuse them for bugs surrounding the final day of a leap year. The Julian calendar is not exactly a new phenomenon. --msa

On Jan 5, 2009, at 11:30 AM, Adrian Chadd wrote:

This begs the question - how the heck do timekeepers and politicians get away with last minute time changes?

Surely there's -some- pushback from technology related interest groups to try and get more than four weeks warning? :)

Having been involved in the leap second business, I can tell you that Daniel Gambis strictly follows the rules, which are Bulletin C is mailed every six months, either to announce a time step in UTC or to confirm that there will be no time step at the next possible date. If you want more lead time warning, pay attention to the LOD graph in http://hpiers.obspm.fr/eop-pc/ The long term LOD offset is about 1 msec now. That means that every day, Earth time and atomic time will drift off by 1 msec. Since there are 1000 msec in the second, and since the rule is that a leap second is chosen when the difference (UT1−UTC) approaches 0.9 seconds, projected out to the next period, and since the strong preference is to have leap seconds in January, you can generally figure out what will happen before Daniel announces it. For example, in one year the offset should be ~ 400 msec, so I will informally predict another leap second in January, 2011, not 2010. Keep watching that graph. Anyone who is dealing with Leap Second code should keep in mind that negative leap seconds (i.e., no second # 59, instead of an extra second called 60) are a distinct possibility. It all depends on the "weather" at the core mantle boundary - note that the LOD offset was almost 3 msec not too long ago. Regards Marshall

Adrian

On Mon, Jan 05, 2009, Frank Bulk wrote:

...

A report from a DHCP/DNS appliance vendor here: ==================== Several customers have reported a complete lock-up of their Proteus system around the beginning of January 1st 2009. We believe that we have traced this to a problem in the underlying kernel and NTP and the handling of the date change associated with 2008 being a Leap Year and therefore having 366 days.

Several conditions must be met to trigger this problem: 1. The Proteus was originally installed as v2.1.x or earlier. 2. NTP is enabled as a client with 2 or more external source servers defined. 3. There is a discrepancy in the times reported back by these other NTP servers.

There is no correction available at this time, and the resolution is to power cycle the system, after which it will run fine.

If you experienced a similar problem at the indicated time, please submit a trouble ticket so that we can confirm that this occurred on your system. ====================

I don't know what the underlying OS is.

Frank

-----Original Message----- From: Kevin Day [mailto:toasty@dragondata.com] Sent: Wednesday, December 31, 2008 4:42 PM To: NANOG list Subject: Leap second tonight

Just a reminder that there's a leap second tonight.

Last time I watched for what happened on 01/01/2006, there was a little bit of chaos: http://markmail.org/message/cpoj3jw5onzhhjkr?q=%22kevin+day%22+leap+second+r eminder+nanog&page=1&refer=cnkxb3iv7sls5axu

I've been told that some of the causes of these problems are fixed on any reasonably recent ntp distribution, but just in case, you might wanna keep an eye out if you're seeing any weirdness. The worst damage I'd heard from anyone after that event was their clock being significantly off for several hours.

-- Kevin

-- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -

...

...

On 1/5/2009 at 1:19 PM, Peter Beckman <beckman@angryox.com> wrote: I've gleened from this thread that:

* everyone uses UTC, or should, because UTC is a uniform time scale, except for those leap seconds

Local time is totally appropriate in some circumstances, but it is pretty much always defined as just an offset to UTC. UT1 is important when you are doing astronomical observations or depend on such things.

* UTC is sourced from the frequence of a radio emission from cesium atoms which are extremely constant

There is nothing too special about cesium. It's properties and the properties of the particular transition are convenient from an engineering standpoint.

* UTC can get out of whack with the rotation of the earth around the sun, because our rotation is not uniform, but is calculated rather than measured (well, sort of)

No its not about years, that's what February 29th is for, it's about days. As the Earth rotates (there's a "rotation" versus "revolution" nomenclature), the sun appears to move around it daily. This is the solar day. At a certain point, at a certain location, the sun is at the highest it will be in the sky for that rotation. This is solar noon. The time between two consecutive noons at any location is not constant mostly due to the Earth's orbit being elliptical and the tilt of the Earth's axis. But for any place on Earth, the mean solar day, the average of the solar day over the year, is the same. If the Earth was a solid sphere rotating at a constant speed, that would be the end of the story. But it's not and for a variety of reasons, the rotation of the Earth changes with time (mostly slows). This causes the mean solar day to get longer.

* UTC is TAI plus leap seconds. In 1972, when leap seconds were first introduced, UTC was TAI - 10 seconds. UTC is now TAI - 34 seconds. TAI ticks exactly as fast as UTC, ignoring leap second adjustments. * UTC is slower than UT1 by about 1ms per day.

That's a very tricky sentence. I think that the mean solar day right now is about 86400.002 s long. The mean solar day was actually exactly 86400 s long sometime around 1820.

* On 12/31/2008 UTC was (-) 591.8ms behind UT1. On 1/1/2008 UTC was 407.1ms ahead of UT1. * Leap seconds are applied to UTC every few years to remain in line with UT1, the time based on the rotation of the earth around the sun.

A time based on the rotation of the Earth.(period) As mentioned above, it doesn't really have anything directly to do with Earth's location in its orbit.

* GMT is used to imply UT1, but sometimes UTC, but really GMT is just massively confusing and you shouldn't use it, either in conversation or in your servers/routers, because nobody is really sure without reading a lot of documentation what GMT means for each manufacturer/OS/software. * When writing code regarding dates and times, know that any year may have 366 days, and any minute may have 61 seconds.

Also remember that an administrator or automated clock recalibration may take you forward or back in time any arbitrary amount.

* When in doubt, Dr. Daniel Gambis is always right.

Very, very few should not defer to his judgement on such matters.

On Mon, Jan 5, 2009 at 4:19 PM, Peter Beckman <beckman@angryox.com> wrote:

* UTC can get out of whack with the rotation of the earth around the sun, because our rotation is not uniform, but is calculated rather than measured (well, sort of)

As Crist Clark points out, leap seconds are about the Earth's rotation about its own axis, not its revolution (orbit) around the sun. Atomic clocks are much more consistent and uniform than the Earth's rotation. If we don't correct for the differences somehow, eventually wall clock time would get out of sync with the day/night cycle. In $LARGE_NUMBER years, 1:00 PM would be in the middle of the night. Earth's orbit and the Julian calendar have the same problem, which is why we have leap days. Otherwise, the calendar would get out of sync with the seasons.

* When writing code regarding dates and times, know that any year may have 366 days, and any minute may have 61 seconds.

I wouldn't even define it that narrowly. We might end up with 62 or 63 or 57 seconds in a minute, or 364 or 367 days in a year. Internally, store everything using a fixed-unit offset (e.g., traditional Unix time, i.e., seconds from 1 Jan 1970). Use OS provided facilities to translate to and from human-friendly representations, and thus make it the OS's problem. (If the OS is your problem... sucks to be you. You'll need lots of tables of historic, idiosyncratic clock/calendar changes to get it right.) For program timers (timeouts, etc.), don't use wall clock time at all, since the wall clock may be wrong, and the admin may fix it, yielding time travel. Most OSes provide something like a "seconds since boot" value, which should always monotonically increase (unless you're running Windows 95, heh), regardless of what the admin is doing to confuse matters. From the other side of the coin: As an admin, avoid advancing the wall clock in large steps, or going backwards at all. If you must do either, do it in single-user mode, or whatever your platform's equivalent is. Don't assume the programmers got it right. Another programmer tip: When storing dates and times in a file, database, etc., and you have to care about multiple timezones: Store at least three of UTC, local time, calculated difference, logical local timezone. The extra information lets one figure out after-the-fact what the timezone tables on the system said when the user entered the information. When the gov'mint monkeys with the time zones, there's always a lag between official announcement and local implementation. Without knowing what the time zone tables said, it can be hard to know if you should apply a correction later. -- Ben