Hello, I'm not sure i should continue to CC nanog, if someone interested to be in CC for further updates this story please let me know. TP-Link not related, it was misunderstanding or wrong customer report. Tenda routers i believe most of cheap models are affected by this problem. On ISPs i have access i see too many of them sending requests to 133.100.9.2 (if it is unreachable, repeating each 10 seconds), this particular ip seems hardcoded there. I am sure as soon as your server is down, way they coded - it will make all this routers to ddos this particular ip, repeating NTP queries very often without any backoff/protection mechanism. Particular model i tested - W308R / Wireless N300 Home Router, but i believe many models are affected. Firmware: System Version: 5.0.7.53_en hw version : v1.0 Another possible vendor is LB-Link, but i dont have yet any info from customers who own them. On 2016-12-21 11:00, FUJIMURA Sho wrote:
Hello.
I'm Sho FUJIMURA. Thank you for your information. I operate the public NTP Services as 133.100.9.2 and 133.100.11.8. I'd like to reduce the traffic because I have trouble with too much traffic recently. So, I'm interested in the root of the the problem. If possible, would you please tell me the model numbers of Tenda and TP-Link??
-- Sho FUJIMURA Information Technology Center, Fukuoka University. 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan
fujimura@fukuoka-u.ac.jp
2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko <denys@visp.net.lb>:
I'm not sure if this issue relevant to discussed topic, Tenda routers here for a while on market, and i think i noticed this issue just now, because NTP servers they are using supposedly for healthcheck went down (or NTP owners blocked ISP's i support, due such routers).
At least after checking numerous users, i believe Tenda hardcoded those NTP IPs. What worsen issue, that in Lebanon several times per day, for example at 18pm - short electricity cutoff, and majority of users routers will reboot and surely reconnect, so it will look like a countrywide spike in NTP traffic.
I checked for a 10min also this NTP ips in dns responses, none of thousands of users tried to resolve any name with them over any DNS server, so i conclude they are hardcoded somewhere in firmware.
Here is traffic of Tenda router after reconnecting (but not full powercycle, i dont have it in my hands). But as you can see, no DNS resolution attempts:
20:15:59.305739 PPPoE [ses 0x1483] CHAP, Success (0x03), id 1, Msg S=XXXXXX M=Authentication succeeded 20:15:59.306100 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length 12 20:15:59.317840 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length 24 20:15:59.317841 PPPoE [ses 0x1483] IPCP, Conf-Ack (0x02), id 1, length 12 20:15:59.317867 PPPoE [ses 0x1483] IPCP, Conf-Nack (0x03), id 1, length 18 20:15:59.325253 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 2, length 24 20:15:59.325273 PPPoE [ses 0x1483] IPCP, Conf-Ack (0x02), id 2, length 24 20:15:59.335589 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 133.100.9.2.123: NTPv3, Client, length 48 20:15:59.335588 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.41.123: NTPv3, Client, length 48 20:15:59.335588 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.40.123: NTPv3, Client, length 48
Here is example of Tenda traffic if it is unable to reach destination, it repeats request each 10 seconds endlessly, my guess they are using ntp to show status of internet connection. So, now that NTP servers getting quite significant DDoS such way.
19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:58:02.164781 IP (tos 0x0, ttl 64, id 38518, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) 19:58:02.164884 IP (tos 0x0, ttl 64, id 38519, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) 19:58:02.165061 IP (tos 0x0, ttl 64, id 38520, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53)
On 2016-12-19 21:40, Roland Dobbins wrote: On 20 Dec 2016, at 2:22, Denys Fedoryshchenko wrote:
If it is necessary i can investigate further.
Yes, please!
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Hello. I operate the public NTP Service as 133.100.9.2 and 133.100.11.8 at Fukuoka University, Japan. I have a lot of trouble with too much NTP traffic from many routers which 133.100.9.2 as default setting of NTP has been set like Tenda or LB-Link etc. So, although I'd like to contact Firmware developpers of these company and would like them to change the default settins, is there the person knowing the contact information? -- Sho FUJIMURA Information Technology Center, Fukuoka University. 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan 2016-12-21 18:36 GMT+09:00 Denys Fedoryshchenko <denys@visp.net.lb>:
Hello,
I'm not sure i should continue to CC nanog, if someone interested to be in CC for further updates this story please let me know.
TP-Link not related, it was misunderstanding or wrong customer report.
Tenda routers i believe most of cheap models are affected by this problem. On ISPs i have access i see too many of them sending requests to 133.100.9.2 (if it is unreachable, repeating each 10 seconds), this particular ip seems hardcoded there. I am sure as soon as your server is down, way they coded - it will make all this routers to ddos this particular ip, repeating NTP queries very often without any backoff/protection mechanism. Particular model i tested - W308R / Wireless N300 Home Router, but i believe many models are affected. Firmware: System Version: 5.0.7.53_en hw version : v1.0
Another possible vendor is LB-Link, but i dont have yet any info from customers who own them.
On 2016-12-21 11:00, FUJIMURA Sho wrote:
Hello.
I'm Sho FUJIMURA. Thank you for your information. I operate the public NTP Services as 133.100.9.2 and 133.100.11.8. I'd like to reduce the traffic because I have trouble with too much traffic recently. So, I'm interested in the root of the the problem. If possible, would you please tell me the model numbers of Tenda and TP-Link??
-- Sho FUJIMURA Information Technology Center, Fukuoka University. 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan
fujimura@fukuoka-u.ac.jp
2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko <denys@visp.net.lb>:
I'm not sure if this issue relevant to discussed topic, Tenda routers here for a while on market, and i think i noticed this issue just now, because NTP servers they are using supposedly for healthcheck went down (or NTP owners blocked ISP's i support, due such routers).
At least after checking numerous users, i believe Tenda hardcoded those NTP IPs. What worsen issue, that in Lebanon several times per day, for example at 18pm - short electricity cutoff, and majority of users routers will reboot and surely reconnect, so it will look like a countrywide spike in NTP traffic.
I checked for a 10min also this NTP ips in dns responses, none of thousands of users tried to resolve any name with them over any DNS server, so i conclude they are hardcoded somewhere in firmware.
Here is traffic of Tenda router after reconnecting (but not full powercycle, i dont have it in my hands). But as you can see, no DNS resolution attempts:
20:15:59.305739 PPPoE [ses 0x1483] CHAP, Success (0x03), id 1, Msg S=XXXXXX M=Authentication succeeded 20:15:59.306100 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length 12 20:15:59.317840 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length 24 20:15:59.317841 PPPoE [ses 0x1483] IPCP, Conf-Ack (0x02), id 1, length 12 20:15:59.317867 PPPoE [ses 0x1483] IPCP, Conf-Nack (0x03), id 1, length 18 20:15:59.325253 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 2, length 24 20:15:59.325273 PPPoE [ses 0x1483] IPCP, Conf-Ack (0x02), id 2, length 24 20:15:59.335589 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 133.100.9.2.123: NTPv3, Client, length 48 20:15:59.335588 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.41.123: NTPv3, Client, length 48 20:15:59.335588 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.40.123: NTPv3, Client, length 48
Here is example of Tenda traffic if it is unable to reach destination, it repeats request each 10 seconds endlessly, my guess they are using ntp to show status of internet connection. So, now that NTP servers getting quite significant DDoS such way.
19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:58:02.164781 IP (tos 0x0, ttl 64, id 38518, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) 19:58:02.164884 IP (tos 0x0, ttl 64, id 38519, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) 19:58:02.165061 IP (tos 0x0, ttl 64, id 38520, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53)
On 2016-12-19 21:40, Roland Dobbins wrote: On 20 Dec 2016, at 2:22, Denys Fedoryshchenko wrote:
If it is necessary i can investigate further.
Yes, please!
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Hello, Those servers aren’t (and have never been) part of the NTP Pool - https://www.ntppool.org/en/ If they were you could remove them from the system and over the next hours, days and months the traffic would go away. We also have features to change the relative amount of clients you get (to just get less queries instead of withdrawing from the pool altogether). Anyway, it looks like your IPs are listed on support.ntp.org as “public servers”, so removing them from there would be step 1. However there’s no working mechanism for you to tell the clients that they should go away after they’ve hard coded your IP in their configuration. (That’s the point of the NTP Pool system really, to let you offer a public service and have a avenue to stop doing it, too). support.ntp.org appears to be down, but your IPs are listed on the site according to a Google search: https://www.google.com/search?q=133.100.9.2+ntp Ask
On Dec 21, 2016, at 7:13 PM, FUJIMURA Sho <fujimura@fukuoka-u.ac.jp> wrote:
Hello.
I operate the public NTP Service as 133.100.9.2 and 133.100.11.8 at Fukuoka University, Japan. I have a lot of trouble with too much NTP traffic from many routers which 133.100.9.2 as default setting of NTP has been set like Tenda or LB-Link etc. So, although I'd like to contact Firmware developpers of these company and would like them to change the default settins, is there the person knowing the contact information?
-- Sho FUJIMURA Information Technology Center, Fukuoka University. 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan
Hello, I know 133.100.9.2 and 133.100.11.8 are listed. The Server Contact is old information. So, I sent e-mail to webmaster@ntp.org a few times. But, I have't received e-mail from them. I'd like them to change the information. Is there the person knowing the contact information to ntp.org? -- Sho FUJIMURA Information Technology Center, Fukuoka University. 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan 2016-12-23 9:04 GMT+09:00 Ask Bjørn Hansen <ask@develooper.com>:
Hello,
Those servers aren’t (and have never been) part of the NTP Pool - https://www.ntppool.org/en/
If they were you could remove them from the system and over the next hours, days and months the traffic would go away. We also have features to change the relative amount of clients you get (to just get less queries instead of withdrawing from the pool altogether).
Anyway, it looks like your IPs are listed on support.ntp.org as “public servers”, so removing them from there would be step 1. However there’s no working mechanism for you to tell the clients that they should go away after they’ve hard coded your IP in their configuration. (That’s the point of the NTP Pool system really, to let you offer a public service and have a avenue to stop doing it, too).
support.ntp.org appears to be down, but your IPs are listed on the site according to a Google search: https://www.google.com/search?q=133.100.9.2+ntp
Ask
On Dec 21, 2016, at 7:13 PM, FUJIMURA Sho <fujimura@fukuoka-u.ac.jp> wrote:
Hello.
I operate the public NTP Service as 133.100.9.2 and 133.100.11.8 at Fukuoka University, Japan. I have a lot of trouble with too much NTP traffic from many routers which 133.100.9.2 as default setting of NTP has been set like Tenda or LB-Link etc. So, although I'd like to contact Firmware developpers of these company and would like them to change the default settins, is there the person knowing the contact information?
-- Sho FUJIMURA Information Technology Center, Fukuoka University. 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan
Hi Fujimura-san, On 12/24/16 6:11 AM, FUJIMURA Sho wrote:
Hello,
I know 133.100.9.2 and 133.100.11.8 are listed. The Server Contact is old information. So, I sent e-mail to webmaster@ntp.org a few times. But, I have't received e-mail from them. I'd like them to change the information. Is there the person knowing the contact information to ntp.org?
I don't recall seeing the emails you sent to webmaster, but we do have a new group of folks watching the Servers web. We would be happy to work with you to give you access to those entries so you can update them. -- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!
participants (4)
-
Ask Bjørn Hansen -
Denys Fedoryshchenko -
FUJIMURA Sho -
Harlan Stenn