We have 5 NTP server: 2 x stratum 1 rubidium oscillator time servers with GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to external internet based NTP stratum 1 servers. We monitor our NTP environment closely, and over the last 10+ years, normally all of our NTP servers are sync'ed within +/- 2 msec. Starting last Friday, we started seeing some remote NTP servers with GPS reference consistently offset by 10 msec. Any one else seeing this? ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669
On Mon, Jul 13, 2015 at 01:17:01PM +0000, Matthew Huff <mhuff@ox.com> wrote a message of 14 lines which said:
We have 5 NTP server: 2 x stratum 1 rubidium oscillator time servers with GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to external internet based NTP stratum 1 servers. We monitor our NTP environment closely, and over the last 10+ years, normally all of our NTP servers are sync'ed within +/- 2 msec. Starting last Friday, we started seeing some remote NTP servers with GPS reference consistently offset by 10 msec.
I have no idea but I just wanted to remind people that, for a few months, RIPE Atlas probes have been able to do NTP queries <https://atlas.ripe.net/docs/data_struct/#v4660_ntp> so it may be a cool way to monitor NTP servers from many points.
Depending on how exactly you have these servers configured with relation to one another, small variations from one single source can be augmented down the line. https://en.wikipedia.org/wiki/Propagation_of_uncertainty On Mon, Jul 13, 2015 at 8:17 AM, Matthew Huff <mhuff@ox.com> wrote:
We have 5 NTP server: 2 x stratum 1 rubidium oscillator time servers with GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to external internet based NTP stratum 1 servers. We monitor our NTP environment closely, and over the last 10+ years, normally all of our NTP servers are sync'ed within +/- 2 msec. Starting last Friday, we started seeing some remote NTP servers with GPS reference consistently offset by 10 msec.
Any one else seeing this?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669
I have had a consistent 10ms offset on a set of servers for the last 5 years. After extensive one-way tracing, it turns out there is a 20ms asymmetry "within" the Seattle Westin colo between HE & Comcast, causing all the IPv6 peers appearing over the HE tunnel to be 10ms offset from everything else. There may be other instances of indirect peering causing a static asymmetric path delay, and NTP will report that as an offset of half of the difference. Tony
-----Original Message----- From: NANOG [mailto:nanog-bounces+alh-ietf=tndh.net@nanog.org] On Behalf Of Rafael Possamai Sent: Thursday, July 16, 2015 8:53 AM To: Matthew Huff Cc: nanog@nanog.org Subject: Re: Speaking of NTP...
Depending on how exactly you have these servers configured with relation to one another, small variations from one single source can be augmented down the line.
https://en.wikipedia.org/wiki/Propagation_of_uncertainty
On Mon, Jul 13, 2015 at 8:17 AM, Matthew Huff <mhuff@ox.com> wrote:
We have 5 NTP server: 2 x stratum 1 rubidium oscillator time servers with GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to external internet based NTP stratum 1 servers. We monitor our NTP environment closely, and over the last 10+ years, normally all of our NTP servers are sync'ed within +/- 2 msec. Starting last Friday, we started seeing some remote NTP +servers with GPS reference consistently offset by 10 msec.
Any one else seeing this?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669
Thanks. We have always had a few outliers, but we have never had a large number of external NTP have a consistent offset, and not one as big as 10ms. Something changed last Friday, probably at some peering point that caused the issue. Maybe a symmetric path got created to route around some outage. Maybe some MPLS circuit got introduced into the mix that hides the underlying path/latency. Glad to know someone else has seen something like this. Our 3 NTP servers that sync from external sources have at least 5 upstream stratum 1 servers and are peered to each other . They have settled on a sense of time that is good within +/i 1 msec of our strata 1 clocks, so all is good, but it was a stage occurrence after we had been good for so long. Each of our servers are clients of our 2 x strata 1 servers and 3 x strata 2 NTP servers. They all look good now.
On Jul 16, 2015, at 3:24 PM, Tony Hain <alh-ietf@tndh.net> wrote:
I have had a consistent 10ms offset on a set of servers for the last 5 years. After extensive one-way tracing, it turns out there is a 20ms asymmetry "within" the Seattle Westin colo between HE & Comcast, causing all the IPv6 peers appearing over the HE tunnel to be 10ms offset from everything else. There may be other instances of indirect peering causing a static asymmetric path delay, and NTP will report that as an offset of half of the difference.
Tony
-----Original Message----- From: NANOG [mailto:nanog-bounces+alh-ietf=tndh.net@nanog.org] On Behalf Of Rafael Possamai Sent: Thursday, July 16, 2015 8:53 AM To: Matthew Huff Cc: nanog@nanog.org Subject: Re: Speaking of NTP...
Depending on how exactly you have these servers configured with relation to one another, small variations from one single source can be augmented down the line.
https://en.wikipedia.org/wiki/Propagation_of_uncertainty
On Mon, Jul 13, 2015 at 8:17 AM, Matthew Huff <mhuff@ox.com> wrote:
We have 5 NTP server: 2 x stratum 1 rubidium oscillator time servers with GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to external internet based NTP stratum 1 servers. We monitor our NTP environment closely, and over the last 10+ years, normally all of our NTP servers are sync'ed within +/- 2 msec. Starting last Friday, we started seeing some remote NTP +servers with GPS reference consistently offset by 10 msec.
Any one else seeing this?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-694-5669
participants (4)
-
Matthew Huff -
Rafael Possamai -
Stephane Bortzmeyer -
Tony Hain