Re: Recent NTP pool traffic increase
Yo All! Someone on nanog was reporrting on the new NTP mystery. He suggested doing a dump similar to this: # tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:" And I do indeed get odd results. Some on my local network... This is from a chronyd host to an ntpsec host. I monitor them both continuously and both seem to be keeping good time. 17:36:11.369329 IP (tos 0x0, ttl 64, id 21405, offset 0, flags [DF], proto UDP ( 17), length 76) 204.17.205.7.50937 > 204.17.205.27.123: [udp sum ok] NTPv4, length 48 Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecifi ed), poll 6 (64s), precision 32 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 3691013707.207257069 (2016/12/17 17:35:07) Receive Timestamp: 276521666.321684728 (2044/11/11 10:02:42) Transmit Timestamp: 3684123061.899235956 (2016/09/29 00:31:01) Originator - Receive Timestamp: +880475255.114427658 Originator - Transmit Timestamp: -6890645.308021113 That 'Receive Timestamp' is strange. Here is another one from the same chronyd host, to another ntpsec host: 17:36:23.395415 IP (tos 0x0, ttl 64, id 3599, offset 0, flags [DF], proto UDP (1 7), length 76) 204.17.205.7.33551 > 204.17.205.1.123: [udp sum ok] NTPv4, length 48 Client, Leap indicator: clock unsynchronized (192), Stratum 0 (unspecifi ed), poll 6 (64s), precision 32 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 3691013718.824150890 (2016/12/17 17:35:18) Receive Timestamp: 1779216017.648483479 (2092/06/24 18:08:33) Transmit Timestamp: 1405803137.064633429 (2080/08/24 20:20:33) Originator - Receive Timestamp: -1911797701.175667410 Originator - Transmit Timestamp: +2009756714.240482539 Note both the 'Receive Timestamp' and 'Transmit Timestamp' are both strange. All three hosts have GPS for local time. Here is one from a laptop, running chrony, that has not GPS: 17:36:52.643814 IP (tos 0x0, ttl 64, id 24624, offset 0, flags [DF], proto UDP ( 17), length 76) 204.17.205.21.41485 > 204.17.205.8.123: [udp sum ok] NTPv4, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 6 (64s), pre cision 32 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 3691013747.797479298 (2016/12/17 17:35:47) Receive Timestamp: 317494016.811980062 (2046/02/28 15:15:12) Transmit Timestamp: 127487236.597620268 (2040/02/21 11:35:32) Originator - Receive Timestamp: +921447565.014500764 Originator - Transmit Timestamp: +731440784.800140969 I have only seen this oddity from chronyd hosts... RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
Yo All! On Sat, 17 Dec 2016 17:54:55 -0800 "Gary E. Miller" <gem@rellim.com> wrote:
# tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"
And I do indeed get odd results. Some on my local network...
To follow up on my own post, so this can be promply laid to rest. After some discussion at NTPsec. It seems that chronyd takes a lot of 'creative license' with RFC 5905 (NTPv4). But it is not malicious, just 'odd', and not new. So, nothing see here, back to the hunt for the real cause of the new NTP traffic. RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
I also have a similar experience with an increased load. I'm running a pretty basic Linode VPS and I had to fine tune a few things in order to deal with the increased traffic. I can clearly see a date around the 14-15 where my traffic increases to 3-4 times the usual amounts. I did a quick dump and in 60 seconds I was hit by slightly over 190K IPs http://i.imgur.com/mygYINk.png Weird stuff Laurent On 12/17/2016 10:25 PM, Gary E. Miller wrote:
Yo All!
On Sat, 17 Dec 2016 17:54:55 -0800 "Gary E. Miller" <gem@rellim.com> wrote:
# tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"
And I do indeed get odd results. Some on my local network... To follow up on my own post, so this can be promply laid to rest.
After some discussion at NTPsec. It seems that chronyd takes a lot of 'creative license' with RFC 5905 (NTPv4). But it is not malicious, just 'odd', and not new.
So, nothing see here, back to the hunt for the real cause of the new NTP traffic.
RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
My WAG is that the one plus updated firmeware on that day and they baked in the pool. Complete WAG, but time and distributed sources including wireless networks On Mon, Dec 19, 2016 at 10:30 AM Laurent Dumont <admin@coldnorthadmin.com> wrote:
I also have a similar experience with an increased load.
I'm running a pretty basic Linode VPS and I had to fine tune a few
things in order to deal with the increased traffic. I can clearly see a
date around the 14-15 where my traffic increases to 3-4 times the usual
amounts.
I did a quick dump and in 60 seconds I was hit by slightly over 190K IPs
http://i.imgur.com/mygYINk.png
Weird stuff
Laurent
On 12/17/2016 10:25 PM, Gary E. Miller wrote:
Yo All!
On Sat, 17 Dec 2016 17:54:55 -0800
"Gary E. Miller" <gem@rellim.com> wrote:
# tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"
And I do indeed get odd results. Some on my local network...
To follow up on my own post, so this can be promply laid to rest.
After some discussion at NTPsec. It seems that chronyd takes a lot
of 'creative license' with RFC 5905 (NTPv4). But it is not malicious,
just 'odd', and not new.
So, nothing see here, back to the hunt for the real cause of the new
NTP traffic.
RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
gem@rellim.com Tel:+1 541 382 8588
I noticed now many customers using tp-links reported issues with internet connection. Analyzing internet traffic, i noticed that tp-link seems excessively requesting ntp from those ip addresses, and not trying others:
192.5.41.40.123: NTPv3, Client, length 48 192.5.41.41.123: NTPv3, Client, length 48 133.100.9.2.123: NTPv3, Client, length 48
I'm asking customer to make photo of device, to retrieve model and revision, and checking other customers as well, if they are abusing same servers. On 2016-12-19 20:33, Ca By wrote:
My WAG is that the one plus updated firmeware on that day and they baked in the pool.
Complete WAG, but time and distributed sources including wireless networks
On Mon, Dec 19, 2016 at 10:30 AM Laurent Dumont <admin@coldnorthadmin.com> wrote:
I also have a similar experience with an increased load.
I'm running a pretty basic Linode VPS and I had to fine tune a few
things in order to deal with the increased traffic. I can clearly see a
date around the 14-15 where my traffic increases to 3-4 times the usual
amounts.
I did a quick dump and in 60 seconds I was hit by slightly over 190K IPs
http://i.imgur.com/mygYINk.png
Weird stuff
Laurent
On 12/17/2016 10:25 PM, Gary E. Miller wrote:
Yo All!
On Sat, 17 Dec 2016 17:54:55 -0800
"Gary E. Miller" <gem@rellim.com> wrote:
# tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"
And I do indeed get odd results. Some on my local network...
To follow up on my own post, so this can be promply laid to rest.
After some discussion at NTPsec. It seems that chronyd takes a lot
of 'creative license' with RFC 5905 (NTPv4). But it is not malicious,
just 'odd', and not new.
So, nothing see here, back to the hunt for the real cause of the new
NTP traffic.
RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
gem@rellim.com Tel:+1 541 382 8588
Many sorry! Update, seems illiterate in english (worse than me, hehe) customer was not precise about model of router, while he reported issue. I noticed now many customers using specific models of routers reported issues with internet connection. Analyzing internet traffic, i noticed that this routers seems excessively requesting ntp from those ip addresses, and not trying others:
192.5.41.40.123: NTPv3, Client, length 48 192.5.41.41.123: NTPv3, Client, length 48 133.100.9.2.123: NTPv3, Client, length 48
I'm asking customer to make photo of device, to retrieve model and revision, and checking other customers as well, if they are abusing same servers. There is definitely pattern, that all of them are using just this 3 hardcoded servers. Problem is that many customers are changing mac of router, so i cannot clearly identify vendor by first mac nibbles. He sent me 2 photos, one of them LB-Link (mac vendor lookup 20:f4:1b says Shenzhen Bilian electronic CO.,LTD), another is Tenda (c8:3a:35 is Tenda). If it is necessary i can investigate further. On 2016-12-19 20:33, Ca By wrote:
My WAG is that the one plus updated firmeware on that day and they baked in the pool.
Complete WAG, but time and distributed sources including wireless networks
On Mon, Dec 19, 2016 at 10:30 AM Laurent Dumont <admin@coldnorthadmin.com> wrote:
I also have a similar experience with an increased load.
I'm running a pretty basic Linode VPS and I had to fine tune a few
things in order to deal with the increased traffic. I can clearly see a
date around the 14-15 where my traffic increases to 3-4 times the usual
amounts.
I did a quick dump and in 60 seconds I was hit by slightly over 190K IPs
http://i.imgur.com/mygYINk.png
Weird stuff
Laurent
On 12/17/2016 10:25 PM, Gary E. Miller wrote:
Yo All!
On Sat, 17 Dec 2016 17:54:55 -0800
"Gary E. Miller" <gem@rellim.com> wrote:
# tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"
And I do indeed get odd results. Some on my local network...
To follow up on my own post, so this can be promply laid to rest.
After some discussion at NTPsec. It seems that chronyd takes a lot
of 'creative license' with RFC 5905 (NTPv4). But it is not malicious,
just 'odd', and not new.
So, nothing see here, back to the hunt for the real cause of the new
NTP traffic.
RGDS
GARY
---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
gem@rellim.com Tel:+1 541 382 8588
On 2016-12-19 11:29 AM, Laurent Dumont wrote:
I also have a similar experience with an increased load.
I'm running a pretty basic Linode VPS and I had to fine tune a few things in order to deal with the increased traffic. I can clearly see a date around the 14-15 where my traffic increases to 3-4 times the usual amounts.
From a source network point of view we see devices come online and hit ~35 unique NTP servers within a few seconds. I'll try to see if I can track down what type of devices they are.
I did a quick dump and in 60 seconds I was hit by slightly over 190K IPs
http://i.imgur.com/mygYINk.png
Weird stuff
Laurent
On 12/17/2016 10:25 PM, Gary E. Miller wrote:
Yo All!
On Sat, 17 Dec 2016 17:54:55 -0800 "Gary E. Miller" <gem@rellim.com> wrote:
# tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"
And I do indeed get odd results. Some on my local network... To follow up on my own post, so this can be promply laid to rest.
After some discussion at NTPsec. It seems that chronyd takes a lot of 'creative license' with RFC 5905 (NTPv4). But it is not malicious, just 'odd', and not new.
So, nothing see here, back to the hunt for the real cause of the new NTP traffic.
RGDS GARY ---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
On 2016-12-19 12:52 PM, David wrote:
On 2016-12-19 11:29 AM, Laurent Dumont wrote:
I also have a similar experience with an increased load.
I'm running a pretty basic Linode VPS and I had to fine tune a few things in order to deal with the increased traffic. I can clearly see a date around the 14-15 where my traffic increases to 3-4 times the usual amounts.
From a source network point of view we see devices come online and hit ~35 unique NTP servers within a few seconds.
I'll try to see if I can track down what type of devices they are.
I found devices doing lookups for all of these at the same time {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
Quoting David <opendak@shaw.ca>:
I found devices doing lookups for all of these at the same time {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
I'm very interested to find out what devices these are. This would explain why places like New Zealand are getting massive amounts of NTP traffic from North America.
On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
I found devices doing lookups for all of these at the same time {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
Thanks, David. That perfectly matches the list of servers used by older versions of the ios-ntp library[1][2], which would point toward some iPhone app being the source of the traffic. [1] https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0... [2] https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff... -- Jan Tore Morken
On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
I found devices doing lookups for all of these at the same time {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
Thanks, David. That perfectly matches the list of servers used by older versions of the ios-ntp library[1][2], which would point toward some iPhone app being the source of the traffic.
[1] https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0... [2] https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff...
That would make sense - I see a lot of iCloud related lookups from these hosts as well. Also, app.snapchat.com generally seems to follow just after the NTP pool DNS lookups. I don't have an iPhone to test that though. Thanks,
the new Mario app perhaps? :) ____________ Justin Paine Head of Trust & Safety Cloudflare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D On Mon, Dec 19, 2016 at 1:12 PM, David <opendak@shaw.ca> wrote:
On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
I found devices doing lookups for all of these at the same time
{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa,europe}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
Thanks, David. That perfectly matches the list of servers used by older versions of the ios-ntp library[1][2], which would point toward some iPhone app being the source of the traffic.
[1] https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0... [2] https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff...
That would make sense - I see a lot of iCloud related lookups from these hosts as well.
Also, app.snapchat.com generally seems to follow just after the NTP pool DNS lookups. I don't have an iPhone to test that though.
Thanks,
Quoting David <opendak@shaw.ca>:
On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
I found devices doing lookups for all of these at the same time {0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
Thanks, David. That perfectly matches the list of servers used by older versions of the ios-ntp library[1][2], which would point toward some iPhone app being the source of the traffic.
[1] https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0... [2] https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff...
That would make sense - I see a lot of iCloud related lookups from these hosts as well.
Also, app.snapchat.com generally seems to follow just after the NTP pool DNS lookups. I don't have an iPhone to test that though.
Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs. Anyone have a contact at Snapchat?
replying off list. ____________ Justin Paine Head of Trust & Safety Cloudflare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown <dan-nanog@drown.org> wrote:
Quoting David <opendak@shaw.ca>:
On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
I found devices doing lookups for all of these at the same time
{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
Thanks, David. That perfectly matches the list of servers used by older versions of the ios-ntp library[1][2], which would point toward some iPhone app being the source of the traffic.
[1] https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0... [2] https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff...
That would make sense - I see a lot of iCloud related lookups from these hosts as well.
Also, app.snapchat.com generally seems to follow just after the NTP pool DNS lookups. I don't have an iPhone to test that though.
Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs.
Anyone have a contact at Snapchat?
If anything comes from this, I'd love to hear about it. As a student in the field, this is the kind of stuff I live for! ;) Pretty awesome to see the chain of events after seeing a post on the [pool] list! Laurent On 12/19/2016 05:12 PM, Justin Paine via NANOG wrote:
replying off list.
____________ Justin Paine Head of Trust & Safety Cloudflare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown <dan-nanog@drown.org> wrote:
Quoting David <opendak@shaw.ca>:
On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
I found devices doing lookups for all of these at the same time
{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
Thanks, David. That perfectly matches the list of servers used by older versions of the ios-ntp library[1][2], which would point toward some iPhone app being the source of the traffic.
[1] https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0... [2] https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff...
That would make sense - I see a lot of iCloud related lookups from these hosts as well.
Also, app.snapchat.com generally seems to follow just after the NTP pool DNS lookups. I don't have an iPhone to test that though.
Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs.
Anyone have a contact at Snapchat?
We - at Snap - were forwarded this thread just a few hours ago and are investigating. Please email me should you still be looking for a contact for Snapchat. Thank you, Jad On Mon, Dec 19, 2016 at 9:18 PM, Laurent Dumont <admin@coldnorthadmin.com> wrote:
If anything comes from this, I'd love to hear about it. As a student in the field, this is the kind of stuff I live for! ;)
Pretty awesome to see the chain of events after seeing a post on the [pool] list!
Laurent
On 12/19/2016 05:12 PM, Justin Paine via NANOG wrote:
replying off list.
____________ Justin Paine Head of Trust & Safety Cloudflare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown <dan-nanog@drown.org> wrote:
Quoting David <opendak@shaw.ca>:
On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
I found devices doing lookups for all of these at the same time
{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}. pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
Thanks, David. That perfectly matches the list of servers used by older versions of the ios-ntp library[1][2], which would point toward some iPhone app being the source of the traffic.
[1] https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0 c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts [2] https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9d ec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122
That would make sense - I see a lot of iCloud related lookups from these hosts as well.
Also, app.snapchat.com generally seems to follow just after the NTP pool DNS lookups. I don't have an iPhone to test that though.
Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs.
Anyone have a contact at Snapchat?
Immediately after being notified that our latest iOS release was causing problems with NTP traffic, we started working to disable the offending code in v9.45. We submitted a new mobile release to the Apple App Store earlier this morning for their review, which should disable these NTP requests. We are hoping Apple will be able to review this release in time before the holiday break, and we have stressed its urgency. When the release does get approved, we should very quickly begin to see a decrease in NTP traffic from our app as users start upgrading to the new release. We deeply regret this situation, and we will post an update here once we hear back from Apple. We are also open to any suggestions on how we can help with the present traffic. On Mon, Dec 19, 2016 at 9:27 PM, Jad Boutros <jad@snap.com> wrote:
We - at Snap - were forwarded this thread just a few hours ago and are investigating. Please email me should you still be looking for a contact for Snapchat.
Thank you, Jad
On Mon, Dec 19, 2016 at 9:18 PM, Laurent Dumont <admin@coldnorthadmin.com> wrote:
If anything comes from this, I'd love to hear about it. As a student in the field, this is the kind of stuff I live for! ;)
Pretty awesome to see the chain of events after seeing a post on the [pool] list!
Laurent
On 12/19/2016 05:12 PM, Justin Paine via NANOG wrote:
replying off list.
____________ Justin Paine Head of Trust & Safety Cloudflare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown <dan-nanog@drown.org> wrote:
Quoting David <opendak@shaw.ca>:
On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
> I found devices doing lookups for all of these at the same time > > {0,0.uk,0.us,asia,europe,north-america,south-america,oceania > ,africa}.pool.ntp.org > and then it proceeds to use everything returned, which explains why > everyone is seeing an increase. >
Thanks, David. That perfectly matches the list of servers used by older versions of the ios-ntp library[1][2], which would point toward some iPhone app being the source of the traffic.
[1] https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0 c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts [2] https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9d ec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122
That would make sense - I see a lot of iCloud related lookups from these hosts as well.
Also, app.snapchat.com generally seems to follow just after the NTP pool DNS lookups. I don't have an iPhone to test that though.
Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs.
Anyone have a contact at Snapchat?
Mostly out of curiosity, what was the reason for the change in the Snapchat code, and what plans does Snap have for whatever reason the NTP change was put in place? Beckman On Tue, 20 Dec 2016, Jad Boutros via NANOG wrote:
Immediately after being notified that our latest iOS release was causing problems with NTP traffic, we started working to disable the offending code in v9.45. We submitted a new mobile release to the Apple App Store earlier this morning for their review, which should disable these NTP requests. We are hoping Apple will be able to review this release in time before the holiday break, and we have stressed its urgency. When the release does get approved, we should very quickly begin to see a decrease in NTP traffic from our app as users start upgrading to the new release.
We deeply regret this situation, and we will post an update here once we hear back from Apple. We are also open to any suggestions on how we can help with the present traffic.
On Mon, Dec 19, 2016 at 9:27 PM, Jad Boutros <jad@snap.com> wrote:
We - at Snap - were forwarded this thread just a few hours ago and are investigating. Please email me should you still be looking for a contact for Snapchat.
Thank you, Jad
On Mon, Dec 19, 2016 at 9:18 PM, Laurent Dumont <admin@coldnorthadmin.com> wrote:
If anything comes from this, I'd love to hear about it. As a student in the field, this is the kind of stuff I live for! ;)
Pretty awesome to see the chain of events after seeing a post on the [pool] list!
Laurent
On 12/19/2016 05:12 PM, Justin Paine via NANOG wrote:
replying off list.
____________ Justin Paine Head of Trust & Safety Cloudflare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
On Mon, Dec 19, 2016 at 1:49 PM, Dan Drown <dan-nanog@drown.org> wrote:
Quoting David <opendak@shaw.ca>:
On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
> On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote: > >> I found devices doing lookups for all of these at the same time >> >> {0,0.uk,0.us,asia,europe,north-america,south-america,oceania >> ,africa}.pool.ntp.org >> and then it proceeds to use everything returned, which explains why >> everyone is seeing an increase. >> > > Thanks, David. That perfectly matches the list of servers used by > older versions of the ios-ntp library[1][2], which would point toward > some iPhone app being the source of the traffic. > > [1] > https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0 > c976dd4aaeed37e0564/ios-ntp-rez/ntp.hosts > [2] > https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9d > ec9da5183e283eff9f2/ios-ntp-lib/NetworkClock.m#L122 > > That would make sense - I see a lot of iCloud related lookups from these hosts as well.
Also, app.snapchat.com generally seems to follow just after the NTP pool DNS lookups. I don't have an iPhone to test that though.
Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs.
Anyone have a contact at Snapchat?
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
Mostly out of curiosity, what was the reason for the change in the Snapchat code, and what plans does Snap have for whatever reason the NTP change was put in place?
From other comments in the thread, it sounds like the app was simply linked against a broken version of a library....
Yo Valdis.Kletnieks@vt.edu! On Tue, 20 Dec 2016 20:20:48 -0500 Valdis.Kletnieks@vt.edu wrote:
On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
Mostly out of curiosity, what was the reason for the change in the Snapchat code, and what plans does Snap have for whatever reason the NTP change was put in place?
From other comments in the thread, it sounds like the app was simply linked against a broken version of a library....
But why is a chat app doing NTP at all? it should rely on the OS, or a specialized app, to keep local time accurate. RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
This was my thought actually, Apple does offer some time services as part of the OS but it’s becoming common with larger / more popular apps to provide some of these services internally. Look at the FB app for example, there are a lot of “system” things they do themselves due to the ability to control specifics. Users don’t want to have to install a second “specialised app” for this either. With regard to an ephemeral chat app requiring time sync, I can think of quite a few use cases and mechanisms in the app that might require time services. - Tim
On 21 Dec. 2016, at 9:26 am, Gary E. Miller <gem@rellim.com> wrote:
Yo Valdis.Kletnieks@vt.edu!
On Tue, 20 Dec 2016 20:20:48 -0500 Valdis.Kletnieks@vt.edu wrote:
On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
Mostly out of curiosity, what was the reason for the change in the Snapchat code, and what plans does Snap have for whatever reason the NTP change was put in place?
From other comments in the thread, it sounds like the app was simply linked against a broken version of a library....
But why is a chat app doing NTP at all? it should rely on the OS, or a specialized app, to keep local time accurate.
RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
Perhaps the host OS' to which snapchat caters, don't all have a devent ntp subststem available? I have vague recollections of some other software (I'm sure we all know which) implemented it's own malloc layer for every system it ran on, for less trivial reasons. ;) ________________________________________ From: NANOG [nanog-bounces@nanog.org] On Behalf Of Tim Raphael [raphael.timothy@gmail.com] Sent: Tuesday, December 20, 2016 5:34 PM To: Gary E. Miller Cc: nanog@nanog.org Subject: Re: Recent NTP pool traffic increase This was my thought actually, Apple does offer some time services as part of the OS but it’s becoming common with larger / more popular apps to provide some of these services internally. Look at the FB app for example, there are a lot of “system” things they do themselves due to the ability to control specifics. Users don’t want to have to install a second “specialised app” for this either. With regard to an ephemeral chat app requiring time sync, I can think of quite a few use cases and mechanisms in the app that might require time services. - Tim
On 21 Dec. 2016, at 9:26 am, Gary E. Miller <gem@rellim.com> wrote:
Yo Valdis.Kletnieks@vt.edu!
On Tue, 20 Dec 2016 20:20:48 -0500 Valdis.Kletnieks@vt.edu wrote:
On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
Mostly out of curiosity, what was the reason for the change in the Snapchat code, and what plans does Snap have for whatever reason the NTP change was put in place?
From other comments in the thread, it sounds like the app was simply linked against a broken version of a library....
But why is a chat app doing NTP at all? it should rely on the OS, or a specialized app, to keep local time accurate.
RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
Exactly, Also they’re across Android and iOS and getting parity of operations across those two OSs isn’t easy. Better to just embed what they need inside the app if it is specialised enough. - Tim
On 21 Dec. 2016, at 10:13 am, Emille Blanc <emille@abccommunications.com> wrote:
Perhaps the host OS' to which snapchat caters, don't all have a devent ntp subststem available? I have vague recollections of some other software (I'm sure we all know which) implemented it's own malloc layer for every system it ran on, for less trivial reasons. ;)
________________________________________ From: NANOG [nanog-bounces@nanog.org] On Behalf Of Tim Raphael [raphael.timothy@gmail.com] Sent: Tuesday, December 20, 2016 5:34 PM To: Gary E. Miller Cc: nanog@nanog.org Subject: Re: Recent NTP pool traffic increase
This was my thought actually, Apple does offer some time services as part of the OS but it’s becoming common with larger / more popular apps to provide some of these services internally. Look at the FB app for example, there are a lot of “system” things they do themselves due to the ability to control specifics. Users don’t want to have to install a second “specialised app” for this either.
With regard to an ephemeral chat app requiring time sync, I can think of quite a few use cases and mechanisms in the app that might require time services.
- Tim
On 21 Dec. 2016, at 9:26 am, Gary E. Miller <gem@rellim.com> wrote:
Yo Valdis.Kletnieks@vt.edu!
On Tue, 20 Dec 2016 20:20:48 -0500 Valdis.Kletnieks@vt.edu wrote:
On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
Mostly out of curiosity, what was the reason for the change in the Snapchat code, and what plans does Snap have for whatever reason the NTP change was put in place?
From other comments in the thread, it sounds like the app was simply linked against a broken version of a library....
But why is a chat app doing NTP at all? it should rely on the OS, or a specialized app, to keep local time accurate.
RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
Better for whom? I'm sure all mobile operating systems provide some access to time, with a least 'seconds' resolution. If an app deems this time source untrustworthy for some reason, I don't think the reasonable response is to make independent time requests from a volunteer-operated pool for public servers designed for host synchronization. Particularly on mobile, the compartmentalization of applications means that this 'better' time will only be accessible to one application, and many applications may have this 'better time' requirement. These developers should be lobbying Apple and Google for better time, if they need it, not making many millions of calls to the NTP pool. To make things worse, I'm fairly sure that Apple's 'no background tasks' policy means that an application can't *maintain* its sense of time, so it would not surprise me if it fires off NTP requests every time it is focused, further compounding the burden. Time is already available, and having every application query for its own time against a public resource doesn't seem very friendly. It certainly doesn't scale. If they are unsuccessful lobbying the OS, why not trust the time provided by the API calls they are surely doing to their own servers? Most HTTP responses include a timestamp; surely this is good enough for expiring Snaps. Or at least operate their own NTP infrastructure. I'm sure that Snap had no malicious intent and commend them for their quick and appropriate response once the issue was identified, but I don't think this behaviour is very defensible. I for one was not harmed by the ~10x increase in load and traffic on my NTP pool node, but the 100x increase if a handful of similar apps decided they 'need' more accurate time than the OS provides would be cause for concern, and I suspect a great many pool nodes would simply disappear, compounding the problem. Please make use of these and similar services as they are designed to be used, and as efficiently as possible, especially if you are responsible for millions of users / machines. In a similar vein, I've always been curious what the ratio Google sees of ICMP echo vs. DNS traffic to 8.8.8.8 is... Keenan On 2016-12-20 18:16, Tim Raphael wrote:
Exactly,
Also they’re across Android and iOS and getting parity of operations across those two OSs isn’t easy. Better to just embed what they need inside the app if it is specialised enough.
- Tim
On 21 Dec. 2016, at 10:13 am, Emille Blanc <emille@abccommunications.com> wrote:
Perhaps the host OS' to which snapchat caters, don't all have a devent ntp subststem available? I have vague recollections of some other software (I'm sure we all know which) implemented it's own malloc layer for every system it ran on, for less trivial reasons. ;)
________________________________________ From: NANOG [nanog-bounces@nanog.org] On Behalf Of Tim Raphael [raphael.timothy@gmail.com] Sent: Tuesday, December 20, 2016 5:34 PM To: Gary E. Miller Cc: nanog@nanog.org Subject: Re: Recent NTP pool traffic increase
This was my thought actually, Apple does offer some time services as part of the OS but it’s becoming common with larger / more popular apps to provide some of these services internally. Look at the FB app for example, there are a lot of “system” things they do themselves due to the ability to control specifics. Users don’t want to have to install a second “specialised app” for this either.
With regard to an ephemeral chat app requiring time sync, I can think of quite a few use cases and mechanisms in the app that might require time services.
- Tim
On 21 Dec. 2016, at 9:26 am, Gary E. Miller <gem@rellim.com> wrote:
Yo Valdis.Kletnieks@vt.edu!
On Tue, 20 Dec 2016 20:20:48 -0500 Valdis.Kletnieks@vt.edu wrote:
Mostly out of curiosity, what was the reason for the change in the Snapchat code, and what plans does Snap have for whatever reason the NTP change was put in place? From other comments in the thread, it sounds like the app was simply
On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said: linked against a broken version of a library.... But why is a chat app doing NTP at all? it should rely on the OS, or a specialized app, to keep local time accurate.
RGDS GARY --------------------------------------------------------------------------- Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
I do think that the point of the Pool network is to be used by both consumers and vendors. And as mentioned before, there is a process if you are a vendor and want to use the pool within a commercial product. I have 3 NTP servers running and I don't really care who is using it. That said, setting up your own infrastructure is also worth considering if it's a business critical feature. I assume that a Snapchat app that fails to have accurate time or correct itself could be abused. To be honest, the fact that NTP is still something managed by volunteers and not a regulated entity (a bit like DNS) is mind boggling. On 12/20/2016 09:41 PM, Keenan Tims wrote:
Better for whom? I'm sure all mobile operating systems provide some access to time, with a least 'seconds' resolution. If an app deems this time source untrustworthy for some reason, I don't think the reasonable response is to make independent time requests from a volunteer-operated pool for public servers designed for host synchronization. Particularly on mobile, the compartmentalization of applications means that this 'better' time will only be accessible to one application, and many applications may have this 'better time' requirement. These developers should be lobbying Apple and Google for better time, if they need it, not making many millions of calls to the NTP pool. To make things worse, I'm fairly sure that Apple's 'no background tasks' policy means that an application can't *maintain* its sense of time, so it would not surprise me if it fires off NTP requests every time it is focused, further compounding the burden.
Time is already available, and having every application query for its own time against a public resource doesn't seem very friendly. It certainly doesn't scale. If they are unsuccessful lobbying the OS, why not trust the time provided by the API calls they are surely doing to their own servers? Most HTTP responses include a timestamp; surely this is good enough for expiring Snaps. Or at least operate their own NTP infrastructure.
I'm sure that Snap had no malicious intent and commend them for their quick and appropriate response once the issue was identified, but I don't think this behaviour is very defensible. I for one was not harmed by the ~10x increase in load and traffic on my NTP pool node, but the 100x increase if a handful of similar apps decided they 'need' more accurate time than the OS provides would be cause for concern, and I suspect a great many pool nodes would simply disappear, compounding the problem. Please make use of these and similar services as they are designed to be used, and as efficiently as possible, especially if you are responsible for millions of users / machines.
In a similar vein, I've always been curious what the ratio Google sees of ICMP echo vs. DNS traffic to 8.8.8.8 is...
Keenan
On 2016-12-20 18:16, Tim Raphael wrote:
Exactly,
Also they’re across Android and iOS and getting parity of operations across those two OSs isn’t easy. Better to just embed what they need inside the app if it is specialised enough.
- Tim
On 21 Dec. 2016, at 10:13 am, Emille Blanc <emille@abccommunications.com> wrote:
Perhaps the host OS' to which snapchat caters, don't all have a devent ntp subststem available? I have vague recollections of some other software (I'm sure we all know which) implemented it's own malloc layer for every system it ran on, for less trivial reasons. ;)
________________________________________ From: NANOG [nanog-bounces@nanog.org] On Behalf Of Tim Raphael [raphael.timothy@gmail.com] Sent: Tuesday, December 20, 2016 5:34 PM To: Gary E. Miller Cc: nanog@nanog.org Subject: Re: Recent NTP pool traffic increase
This was my thought actually, Apple does offer some time services as part of the OS but it’s becoming common with larger / more popular apps to provide some of these services internally. Look at the FB app for example, there are a lot of “system” things they do themselves due to the ability to control specifics. Users don’t want to have to install a second “specialised app” for this either.
With regard to an ephemeral chat app requiring time sync, I can think of quite a few use cases and mechanisms in the app that might require time services.
- Tim
On 21 Dec. 2016, at 9:26 am, Gary E. Miller <gem@rellim.com> wrote:
Yo Valdis.Kletnieks@vt.edu!
On Tue, 20 Dec 2016 20:20:48 -0500 Valdis.Kletnieks@vt.edu wrote:
Mostly out of curiosity, what was the reason for the change in the Snapchat code, and what plans does Snap have for whatever reason the NTP change was put in place? From other comments in the thread, it sounds like the app was simply
On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said: linked against a broken version of a library.... But why is a chat app doing NTP at all? it should rely on the OS, or a specialized app, to keep local time accurate.
RGDS GARY ---------------------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
On 12/20/16 7:27 PM, Laurent Dumont wrote:
I do think that the point of the Pool network is to be used by both consumers and vendors. And as mentioned before, there is a process if you are a vendor and want to use the pool within a commercial product. I have 3 NTP servers running and I don't really care who is using it.
That said, setting up your own infrastructure is also worth considering if it's a business critical feature. I assume that a Snapchat app that fails to have accurate time or correct itself could be abused.
To be honest, the fact that NTP is still something managed by volunteers and not a regulated entity (a bit like DNS) is mind boggling.
Time *is* managed by regulated entities - the National Time Labs. And Network Time Foundation's NTP Project (the reference implementation for NTP) could do lots more if we had a useful budget. Folks pay money for DNS registrations. There's no revenue stream around "time". Help us get enough support to NTF, and we'll have the staff and infrastructure to do more for folks. -- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!
On Dec 20, 2016, at 8:02 PM, Harlan Stenn <stenn@nwtime.org> wrote:
On 12/20/16 7:27 PM, Laurent Dumont wrote: To be honest, the fact that NTP is still something managed by volunteers and not a regulated entity (a bit like DNS) is mind boggling.
Time *is* managed by regulated entities - the National Time Labs.
That was pretty clearly not what Laurent was talking about.
And Network Time Foundation's NTP Project (the reference implementation for NTP) could do lots more if we had a useful budget.
Folks pay money for DNS registrations. There's no revenue stream around "time".
Help us get enough support to NTF, and we'll have the staff and infrastructure to do more for folks.
What does the NTF have to do with the NTP Pool (or the “recent NTP pool traffic increase”)? The NTP Pool is run by volunteers, as you very well know. Both the management and DNS system and the thousands of people who contribute their NTP service to the system. (And we manage on a pretty scarce budget). Ask -- http://www.askask.com
On 12/22/16 4:11 PM, Ask Bjørn Hansen wrote:
On Dec 20, 2016, at 8:02 PM, Harlan Stenn <stenn@nwtime.org> wrote:
On 12/20/16 7:27 PM, Laurent Dumont wrote: To be honest, the fact that NTP is still something managed by volunteers and not a regulated entity (a bit like DNS) is mind boggling.
Time *is* managed by regulated entities - the National Time Labs.
That was pretty clearly not what Laurent was talking about.
Not all that clearly, at least to me.
And Network Time Foundation's NTP Project (the reference implementation for NTP) could do lots more if we had a useful budget.
Folks pay money for DNS registrations. There's no revenue stream around "time".
Help us get enough support to NTF, and we'll have the staff and infrastructure to do more for folks.
What does the NTF have to do with the NTP Pool (or the “recent NTP pool traffic increase”)?
Nothing. But it has to do with companies putting NTP into their products. When they do this with problematic configurations, it causes trouble. In this case, it was trouble for the Pool project. The NTP Pool project certainly isn't the place to solve this issue.
The NTP Pool is run by volunteers, as you very well know. Both the management and DNS system and the thousands of people who contribute their NTP service to the system. (And we manage on a pretty scarce budget).
What's your point? This sort of misconfiguration will happen and the NTP Pool Project clearly isn't the place to solve this problem overall. It *is* something NTF is in a position to address. And we're almost exclusively an overstretched volunteer organization, too, as you very well know. Do we have different goals around this issue? -- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!
On Thu, Dec 22, 2016 at 4:05 PM, Harlan Stenn <stenn@nwtime.org> wrote:
This sort of misconfiguration will happen and the NTP Pool Project clearly isn't the place to solve this problem overall. It *is* something NTF is in a position to address.
Harlan, could you be more specific about how NTF can address this? Would it require modification of the NTP protocol, or something else? Royce
On 12/22/16 5:25 PM, Royce Williams wrote:
On Thu, Dec 22, 2016 at 4:05 PM, Harlan Stenn <stenn@nwtime.org> wrote:
This sort of misconfiguration will happen and the NTP Pool Project clearly isn't the place to solve this problem overall. It *is* something NTF is in a position to address.
Harlan, could you be more specific about how NTF can address this? Would it require modification of the NTP protocol, or something else?
Network Time Foundation has two projects to directly help here. One is a Certification and Compliance program. The other is a project to analyze/produce BCP-compliant ntp.conf files. Due to limited resources we're making slow progress on each of these. That's the backwards way of saying we can make much more useful progress on these and other issues (like the development of NTPv5 and the General Timestamp API) if we can get useful support. With more support we can also make more pool servers available. -- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!
Sorry if I wasn't being clear. What I mostly meant is that there should be a regulated, industry-wide effort in order to provide a stable and active pool program. With the current models, a protocol that is widely used by commercial devices is being supported by the time and effort of volunteers around the world. Initiative like the NTF and the Pool program are awesome, but I believe that NTP is something that shouldn't relegated to the wayside as the internet continues to evolve. Reading a bit more, NTF could be "upgraded" into a more official role too. I'm just a bit puzzled that this entire mixup actually happened with the modern internet. Laurent On 12/22/2016 08:05 PM, Harlan Stenn wrote:
On 12/22/16 4:11 PM, Ask Bjørn Hansen wrote:
On Dec 20, 2016, at 8:02 PM, Harlan Stenn<stenn@nwtime.org> wrote:
On 12/20/16 7:27 PM, Laurent Dumont wrote: To be honest, the fact that NTP is still something managed by volunteers and not a regulated entity (a bit like DNS) is mind boggling. Time *is* managed by regulated entities - the National Time Labs. That was pretty clearly not what Laurent was talking about. Not all that clearly, at least to me.
And Network Time Foundation's NTP Project (the reference implementation for NTP) could do lots more if we had a useful budget.
Folks pay money for DNS registrations. There's no revenue stream around "time".
Help us get enough support to NTF, and we'll have the staff and infrastructure to do more for folks. What does the NTF have to do with the NTP Pool (or the “recent NTP pool traffic increase”)? Nothing. But it has to do with companies putting NTP into their products. When they do this with problematic configurations, it causes trouble. In this case, it was trouble for the Pool project.
The NTP Pool project certainly isn't the place to solve this issue.
The NTP Pool is run by volunteers, as you very well know. Both the management and DNS system and the thousands of people who contribute their NTP service to the system. (And we manage on a pretty scarce budget). What's your point?
This sort of misconfiguration will happen and the NTP Pool Project clearly isn't the place to solve this problem overall. It *is* something NTF is in a position to address.
And we're almost exclusively an overstretched volunteer organization, too, as you very well know.
Do we have different goals around this issue?
What I mostly meant is that there should be a regulated, industry-wide effort in order to provide a stable and active pool program. With the current models, a protocol that is widely used by commercial devices is being supported by the time and effort of volunteers around the world.
My employer has a budget 'for the good of the Internet' So I'd suggest that people involved in NTP (certainly pool) submit proposals. Likewise, there are country code DNS registries that are non-profits and have budgets for research, etc. Given that time is important for DNSSEC, it maybe worth contacting them.
On Thu, Dec 22, 2016 at 11:31:08PM -0500, Laurent Dumont wrote:
What I mostly meant is that there should be a regulated, industry-wide effort in order to provide a stable and active pool program. With the current models, a protocol that is widely used by commercial devices is being supported by the time and effort of volunteers around the world.
Who's authoritative for time? Even the national labs aren't -- UTC is figured well after the fact. Thing about the pool is -- you may use it, you don't have to. You're welcome to provide your own services, including to your customers. There has to be one sole DNS -- there isn't one sole source of time. --msa
What I mostly meant is that there should be a regulated, industry-wide effort in order to provide a stable and active pool program. With
On 12/30/2016 at 1:20 PM, "Majdi S. Abbas" wrote:On Thu, Dec 22, 2016 at 11:31:08PM -0500, Laurent Dumont wrote: the
current models, a protocol that is widely used by commercial devices is being supported by the time and effort of volunteers around the world.
Who's authoritative for time? Even the national labs aren't -- UTC is figured well after the fact. In the United States that would the United States Naval Observatory (USNO) Master Clock (http://tycho.usno.navy.mil/). You can read more about it here: http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock allan
What I mostly meant is that there should be a regulated, industry-wide effort in order to provide a stable and active pool program. With
Ah, but who do you trust? Trump, Putin, or Xi's clock? That said, we use a Stratum2 clock for our AS, which syncs using GPS at $dayjob. So... I guess we trust Trump's clock. Perhaps there's a market for a device that takes GPS, GLONASS, and Beidou, and references the three for sanity checks in the event of $unforseen_circumstance. Assuming such a thing were possible - admittedly I know little about GLONASS, and even less about Beidou. "Perhaps some kind of death clock?" -----Original Message----- From: NANOG [mailto:nanog-bounces+emille=abccommunications.com@nanog.org] On Behalf Of Allan Liska Sent: December-30-16 11:09 AM To: Majdi S. Abbas; Laurent Dumont Cc: nanog@nanog.org Subject: Re: Recent NTP pool traffic increase On 12/30/2016 at 1:20 PM, "Majdi S. Abbas" wrote:On Thu, Dec 22, 2016 at 11:31:08PM -0500, Laurent Dumont wrote: the
current models, a protocol that is widely used by commercial devices is being supported by the time and effort of volunteers around the world.
Who's authoritative for time? Even the national labs aren't -- UTC is figured well after the fact. In the United States that would the United States Naval Observatory (USNO) Master Clock (http://tycho.usno.navy.mil/). You can read more about it here: http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock allan
On 12/30/2016 at 2:26 PM, "Emille Blanc" wrote:Ah, but who do you trust? Trump, Putin, or Xi's clock? That said, we use a Stratum2 clock for our AS, which syncs using GPS at $dayjob. So... I guess we trust Trump's clock. Perhaps there's a market for a device that takes GPS, GLONASS, and Beidou, and references the three for sanity checks in the event of $unforseen_circumstance. Assuming such a thing were possible - admittedly I know little about GLONASS, and even less about Beidou. "Perhaps some kind of death clock?" Personally, when it goes online, I plan on trusting the "Jeff Bezos" clock :) http://www.10000yearclock.net/ allan
On Fri, December 30, 2016 14:26, Emille Blanc wrote:
Ah, but who do you trust? Trump, Putin, or Xi's clock?
That said, we use a Stratum2 clock for our AS, which syncs using GPS at $dayjob. So... I guess we trust Trump's clock.
Perhaps there's a market for a device that takes GPS, GLONASS, and Beidou, and references the three for sanity checks in the event of $unforseen_circumstance. Assuming such a thing were possible - admittedly I know little about GLONASS, and even less about Beidou.
I was casually (yes, really) looking around at some GNSS modules available the other day, and readily found quite affordable ones that can receive at least 2 at a time, of those systems. GPS and GLONASS being the most complete, so probably the best bang-for-the-buck. I'm only an armchair time-nut, so I can't really speak to their precision and such, so this is very much a FWIW post.
-----Original Message----- From: NANOG [mailto:nanog-bounces+emille=abccommunications.com@nanog.org] On Behalf Of Allan Liska Sent: December-30-16 11:09 AM To: Majdi S. Abbas; Laurent Dumont Cc: nanog@nanog.org Subject: Re: Recent NTP pool traffic increase
On 12/30/2016 at 1:20 PM, "Majdi S. Abbas" wrote:On Thu, Dec 22, 2016 at 11:31:08PM -0500, Laurent Dumont wrote:
What I mostly meant is that there should be a regulated,
effort in order to provide a stable and active pool program. With
industry-wide the
current models, a protocol that is widely used by commercial devices is being supported by the time and effort of volunteers around the world.
Who's authoritative for time? Even the national labs aren't -- UTC is figured well after the fact.
In the United States that would the United States Naval Observatory (USNO) Master Clock (http://tycho.usno.navy.mil/). You can read more about it here: http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock
allan
-- Jeff
On 12/30/16 11:26 AM, Emille Blanc wrote:
Ah, but who do you trust? Trump, Putin, or Xi's clock?
That said, we use a Stratum2 clock for our AS, which syncs using GPS at $dayjob. So... I guess we trust Trump's clock.
Perhaps there's a market for a device that takes GPS, GLONASS, and Beidou, and references the three for sanity checks in the event of $unforseen_circumstance. Assuming such a thing were possible - admittedly I know little about GLONASS, and even less about Beidou.
Why are you trying to re-invent a properly configured S2 NTP server? H --
"Perhaps some kind of death clock?"
-----Original Message----- From: NANOG [mailto:nanog-bounces+emille=abccommunications.com@nanog.org] On Behalf Of Allan Liska Sent: December-30-16 11:09 AM To: Majdi S. Abbas; Laurent Dumont Cc: nanog@nanog.org Subject: Re: Recent NTP pool traffic increase
What I mostly meant is that there should be a regulated, industry-wide effort in order to provide a stable and active pool program. With
On 12/30/2016 at 1:20 PM, "Majdi S. Abbas" wrote:On Thu, Dec 22, 2016 at 11:31:08PM -0500, Laurent Dumont wrote: the
current models, a protocol that is widely used by commercial devices is being supported by the time and effort of volunteers around the world.
Who's authoritative for time? Even the national labs aren't -- UTC is figured well after the fact.
In the United States that would the United States Naval Observatory (USNO) Master Clock (http://tycho.usno.navy.mil/). You can read more about it here: http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock
allan
-- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!
On Fri, Dec 30, 2016 at 02:08:50PM -0500, Allan Liska wrote:
In the United States that would the United States Naval Observatory (USNO) Master Clock (http://tycho.usno.navy.mil/). You can read more about it here: http://motherboard.vice.com/read/demetrios-matsakis-and-the-master-clock
One of the things I have learned as a time hobbyist is that if something involves time, and you think there is a simple answer, you are probably wrong. :) USNO is our military time keeper -- NIST keeps time for civil purposes, and while they coordinate to stay in reasonably close proximity, even they don't agree. Even better, the GPS clocks are run by (and corrections distributed) by the Air Force, not the Navy. And they have made mistakes in recent memory. From an international perspective, BIPM is responsible for UTC, but it is only figured well after the fact. We distribute "UTC" via NTP, but it's not true UTC since that is not figured in real time, it's much, much coarser, and everyone's local views differ anyway. For an idea just how many components there are, take a look at BIPM's Circular T: ftp://ftp2.bipm.org/pub/tai//Circular-T/cirthtm/cirt.347.html But back to the point...while UTC is an international time scale, individual national labs and institutions keep their own views of it, and correct periodically...then they distribute these timescales, and in some fashion we attempt to get a coarse version of it onto the Internet in real time. There is no one authority responsible for this, and you may take time from any one (or more) of them that you choose. And for this reason, there is no single authority for time distribution on the Internet -- because there is none for the world as a whole, either. We /can/ have an authoritative system for something like host naming, where it's comparatively easy to produce a single authoritative source. Timing is not nearly such a simple subject. Cheers, --msa
Google announced public NTP service some time ago: https://developers.google.com/time/ On Tue, Dec 20, 2016 at 7:29 PM Laurent Dumont <admin@coldnorthadmin.com> wrote:
I do think that the point of the Pool network is to be used by both
consumers and vendors. And as mentioned before, there is a process if
you are a vendor and want to use the pool within a commercial product. I
have 3 NTP servers running and I don't really care who is using it.
That said, setting up your own infrastructure is also worth considering
if it's a business critical feature. I assume that a Snapchat app that
fails to have accurate time or correct itself could be abused.
To be honest, the fact that NTP is still something managed by volunteers
and not a regulated entity (a bit like DNS) is mind boggling.
On 12/20/2016 09:41 PM, Keenan Tims wrote:
Better for whom? I'm sure all mobile operating systems provide some
access to time, with a least 'seconds' resolution. If an app deems
this time source untrustworthy for some reason, I don't think the
reasonable response is to make independent time requests from a
volunteer-operated pool for public servers designed for host
synchronization. Particularly on mobile, the compartmentalization of
applications means that this 'better' time will only be accessible to
one application, and many applications may have this 'better time'
requirement. These developers should be lobbying Apple and Google for
better time, if they need it, not making many millions of calls to the
NTP pool. To make things worse, I'm fairly sure that Apple's 'no
background tasks' policy means that an application can't *maintain*
its sense of time, so it would not surprise me if it fires off NTP
requests every time it is focused, further compounding the burden.
Time is already available, and having every application query for its
own time against a public resource doesn't seem very friendly. It
certainly doesn't scale. If they are unsuccessful lobbying the OS, why
not trust the time provided by the API calls they are surely doing to
their own servers? Most HTTP responses include a timestamp; surely
this is good enough for expiring Snaps. Or at least operate their own
NTP infrastructure.
I'm sure that Snap had no malicious intent and commend them for their
quick and appropriate response once the issue was identified, but I
don't think this behaviour is very defensible. I for one was not
harmed by the ~10x increase in load and traffic on my NTP pool node,
but the 100x increase if a handful of similar apps decided they 'need'
more accurate time than the OS provides would be cause for concern,
and I suspect a great many pool nodes would simply disappear,
compounding the problem. Please make use of these and similar services
as they are designed to be used, and as efficiently as possible,
especially if you are responsible for millions of users / machines.
In a similar vein, I've always been curious what the ratio Google sees
of ICMP echo vs. DNS traffic to 8.8.8.8 is...
Keenan
On 2016-12-20 18:16, Tim Raphael wrote:
Exactly,
Also they’re across Android and iOS and getting parity of operations
across those two OSs isn’t easy.
Better to just embed what they need inside the app if it is
specialised enough.
- Tim
On 21 Dec. 2016, at 10:13 am, Emille Blanc
<emille@abccommunications.com> wrote:
Perhaps the host OS' to which snapchat caters, don't all have a
devent ntp subststem available?
I have vague recollections of some other software (I'm sure we all
know which) implemented it's own malloc layer for every system it
ran on, for less trivial reasons. ;)
________________________________________
From: NANOG [nanog-bounces@nanog.org] On Behalf Of Tim Raphael
[raphael.timothy@gmail.com]
Sent: Tuesday, December 20, 2016 5:34 PM
To: Gary E. Miller
Cc: nanog@nanog.org
Subject: Re: Recent NTP pool traffic increase
This was my thought actually, Apple does offer some time services as
part of the OS but it’s becoming common with larger / more popular
apps to provide some of these services internally.
Look at the FB app for example, there are a lot of “system” things
they do themselves due to the ability to control specifics. Users
don’t want to have to install a second “specialised app” for this
either.
With regard to an ephemeral chat app requiring time sync, I can
think of quite a few use cases and mechanisms in the app that might
require time services.
- Tim
On 21 Dec. 2016, at 9:26 am, Gary E. Miller <gem@rellim.com> wrote:
Yo Valdis.Kletnieks@vt.edu!
On Tue, 20 Dec 2016 20:20:48 -0500
Valdis.Kletnieks@vt.edu wrote:
On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
> Mostly out of curiosity, what was the reason for the change in the
> Snapchat code, and what plans does Snap have for whatever reason
> the NTP change was put in place?
From other comments in the thread, it sounds like the app was simply
linked against a broken version of a library....
But why is a chat app doing NTP at all? it should rely on the OS, or
a specialized app, to keep local time accurate.
RGDS
GARY
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
gem@rellim.com Tel:+1 541 382 8588
On Tue, Dec 20, 2016 at 8:04 PM, Yury Shefer <shefys@gmail.com> wrote:
Google announced public NTP service some time ago: https://developers.google.com/time/
Leap smearing does look interesting as way to sidestep the potentially-jarring leap-second problem ... but a note of caution. I've had multiple time geeks tell me that leap-smearing is pretty different from strict-RFC NTP, and Google themselves say on that page: "We recommend that you don’t configure Google Public NTP together with non-leap-smearing NTP servers." So it looks like we shouldn't mix and match. And since most folks should probably want some heterogeneity in their NTP, it may be a little premature to jump on the leap-smear bandwagon just yet. I'm vague on the details, so I could be wrong. Does anyone know of any other (non Google) leap-smearing NTP implementations? Royce
On Tue, Dec 20, 2016 at 8:19 PM, Royce Williams <royce@techsolvency.com> wrote:
On Tue, Dec 20, 2016 at 8:04 PM, Yury Shefer <shefys@gmail.com> wrote:
Google announced public NTP service some time ago: https://developers.google.com/time/
Leap smearing does look interesting as way to sidestep the potentially-jarring leap-second problem ... but a note of caution.
I've had multiple time geeks tell me that leap-smearing is pretty different from strict-RFC NTP, and Google themselves say on that page:
"We recommend that you don’t configure Google Public NTP together with non-leap-smearing NTP servers."
So it looks like we shouldn't mix and match. And since most folks should probably want some heterogeneity in their NTP, it may be a little premature to jump on the leap-smear bandwagon just yet.
I'm vague on the details, so I could be wrong.
This is informative: https://docs.ntpsec.org/latest/leapsmear.html
Does anyone know of any other (non Google) leap-smearing NTP implementations?
Royce
On 12/20/16 9:21 PM, Royce Williams wrote:
On Tue, Dec 20, 2016 at 8:19 PM, Royce Williams <royce@techsolvency.com> wrote:
On Tue, Dec 20, 2016 at 8:04 PM, Yury Shefer <shefys@gmail.com> wrote:
Google announced public NTP service some time ago: https://developers.google.com/time/
Leap smearing does look interesting as way to sidestep the potentially-jarring leap-second problem ... but a note of caution.
I've had multiple time geeks tell me that leap-smearing is pretty different from strict-RFC NTP, and Google themselves say on that page:
"We recommend that you don’t configure Google Public NTP together with non-leap-smearing NTP servers."
So it looks like we shouldn't mix and match. And since most folks should probably want some heterogeneity in their NTP, it may be a little premature to jump on the leap-smear bandwagon just yet.
I'm vague on the details, so I could be wrong.
This is informative:
https://docs.ntpsec.org/latest/leapsmear.html
Does anyone know of any other (non Google) leap-smearing NTP implementations?
The NTP Project has had a leap-smear implementation for a while. We also have a proposal for a REFID that indicates the provided time is a leap-smear time, and Network Time Foundation is working on a new timestamp format and API that will easily allow time exchange between systems using different timescales. -- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!
On Tue, Dec 20, 2016 at 10:27:18PM -0500, Laurent Dumont wrote:
To be honest, the fact that NTP is still something managed by volunteers and not a regulated entity (a bit like DNS) is mind boggling.
I don't see why. Look back on 30-ish years of domain registration, I think that it was far more responsibly, ethically, and professionally managed when it was handled by volunteers. ---rsk
On Tue, 20 Dec 2016 18:41:37 -0800, Keenan Tims said:
Better for whom? I'm sure all mobile operating systems provide some access to time, with a least 'seconds' resolution. If an app deems this time source untrustworthy for some reason, I don't think the reasonable response is to make independent time requests from a volunteer-operated pool for public servers designed for host synchronization.
This is possibly at least partly due to "dependency hell". For a worked example from earlier this year: http://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/ So a lot of people had their stuff blow up, even though their code called left_pad exactly noplace....
On Tue, Dec 20, 2016 at 6:41 PM, Keenan Tims <ktims@stargate.ca> wrote:
In a similar vein, I've always been curious what the ratio Google sees of ICMP echo vs. DNS traffic to 8.8.8.8 is...
The more fun question is how many pagers would go off around the world if Google stopped responding to ICMP echo. Damian
The Snapchat mobile app does not currently have a need to talk with NTP servers, it was an error. Also, this afternoon we spun off NTP server instances in Australia and South America to help with the load. Thanks, Jad On Tue, Dec 20, 2016 at 5:34 PM, Tim Raphael <raphael.timothy@gmail.com> wrote:
This was my thought actually, Apple does offer some time services as part of the OS but it’s becoming common with larger / more popular apps to provide some of these services internally. Look at the FB app for example, there are a lot of “system” things they do themselves due to the ability to control specifics. Users don’t want to have to install a second “specialised app” for this either.
With regard to an ephemeral chat app requiring time sync, I can think of quite a few use cases and mechanisms in the app that might require time services.
- Tim
On 21 Dec. 2016, at 9:26 am, Gary E. Miller <gem@rellim.com> wrote:
Yo Valdis.Kletnieks@vt.edu!
On Tue, 20 Dec 2016 20:20:48 -0500 Valdis.Kletnieks@vt.edu wrote:
On Tue, 20 Dec 2016 18:11:11 -0500, Peter Beckman said:
Mostly out of curiosity, what was the reason for the change in the Snapchat code, and what plans does Snap have for whatever reason the NTP change was put in place?
From other comments in the thread, it sounds like the app was simply linked against a broken version of a library....
But why is a chat app doing NTP at all? it should rely on the OS, or a specialized app, to keep local time accurate.
RGDS GARY ------------------------------------------------------------
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703 gem@rellim.com Tel:+1 541 382 8588
On 20/12/16 15:18, Laurent Dumont wrote:
If anything comes from this, I'd love to hear about it. As a student in the field, this is the kind of stuff I live for! ;)
Pretty awesome to see the chain of events after seeing a post on the [pool] list!
https://news.ntppool.org/2016/12/load/ https://community.ntppool.org/t/recent-ntp-pool-traffic-increase/18
On 20 Dec 2016, at 12:18, Laurent Dumont wrote:
As a student in the field, this is the kind of stuff I live for! ;)
<https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#Notable_cases> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>
On Mon, Dec 19, 2016 at 12:49 PM, Dan Drown <dan-nanog@drown.org> wrote:
Quoting David <opendak@shaw.ca>:
On 2016-12-19 1:55 PM, Jan Tore Morken wrote:
On Mon, Dec 19, 2016 at 01:32:50PM -0700, David wrote:
I found devices doing lookups for all of these at the same time
{0,0.uk,0.us,asia,europe,north-america,south-america,oceania,africa}.pool.ntp.org and then it proceeds to use everything returned, which explains why everyone is seeing an increase.
Thanks, David. That perfectly matches the list of servers used by older versions of the ios-ntp library[1][2], which would point toward some iPhone app being the source of the traffic.
[1] https://github.com/jbenet/ios-ntp/blob/d5eade6a99041094f12f0c976dd4aaeed37e0... [2] https://github.com/jbenet/ios-ntp/blob/5cc3b6e437a6422dcee9dec9da5183e283eff...
That would make sense - I see a lot of iCloud related lookups from these hosts as well.
Also, app.snapchat.com generally seems to follow just after the NTP pool DNS lookups. I don't have an iPhone to test that though.
Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs.
Anyone have a contact at Snapchat?
Looks like folks got in touch with them. Thanks! https://community.ntppool.org/t/recent-ntp-pool-traffic-increase/18 Royce
On Mon, 19 Dec 2016 12:52:59 -0700, David said:
From a source network point of view we see devices come online and hit ~35 unique NTP servers within a few seconds.
Am I the only one who read that and started wondering if some engineer writing CPE code read a recommendation someplace to "query 3-5 different servers" and managed to miss the "-"?
participants (26)
-
Allan Liska -
Ask Bjørn Hansen -
Ca By -
Damian Menscher -
Dan Drown -
David -
Denys Fedoryshchenko -
Emille Blanc -
Gary E. Miller -
Harlan Stenn -
Jad Boutros -
Jan Tore Morken -
Jeff McAdams -
Justin Paine -
Keenan Tims -
Laurent Dumont -
Majdi S. Abbas -
Paul Gear -
Peter Beckman -
Philip Homburg -
Rich Kulawiec -
Roland Dobbins -
Royce Williams -
Tim Raphael -
Valdis.Kletnieks＠vt.edu -
Yury Shefer