Re: Recent NTP pool traffic increase (update)
I'm not sure if this issue relevant to discussed topic, Tenda routers here for a while on market, and i think i noticed this issue just now, because NTP servers they are using supposedly for healthcheck went down (or NTP owners blocked ISP's i support, due such routers). At least after checking numerous users, i believe Tenda hardcoded those NTP IPs. What worsen issue, that in Lebanon several times per day, for example at 18pm - short electricity cutoff, and majority of users routers will reboot and surely reconnect, so it will look like a countrywide spike in NTP traffic. I checked for a 10min also this NTP ips in dns responses, none of thousands of users tried to resolve any name with them over any DNS server, so i conclude they are hardcoded somewhere in firmware. Here is traffic of Tenda router after reconnecting (but not full powercycle, i dont have it in my hands). But as you can see, no DNS resolution attempts: 20:15:59.305739 PPPoE [ses 0x1483] CHAP, Success (0x03), id 1, Msg S=XXXXXX M=Authentication succeeded 20:15:59.306100 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length 12 20:15:59.317840 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length 24 20:15:59.317841 PPPoE [ses 0x1483] IPCP, Conf-Ack (0x02), id 1, length 12 20:15:59.317867 PPPoE [ses 0x1483] IPCP, Conf-Nack (0x03), id 1, length 18 20:15:59.325253 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 2, length 24 20:15:59.325273 PPPoE [ses 0x1483] IPCP, Conf-Ack (0x02), id 2, length 24 20:15:59.335589 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 133.100.9.2.123: NTPv3, Client, length 48 20:15:59.335588 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.41.123: NTPv3, Client, length 48 20:15:59.335588 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.40.123: NTPv3, Client, length 48 Here is example of Tenda traffic if it is unable to reach destination, it repeats request each 10 seconds endlessly, my guess they are using ntp to show status of internet connection. So, now that NTP servers getting quite significant DDoS such way. 19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:58:02.164781 IP (tos 0x0, ttl 64, id 38518, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) 19:58:02.164884 IP (tos 0x0, ttl 64, id 38519, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) 19:58:02.165061 IP (tos 0x0, ttl 64, id 38520, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) On 2016-12-19 21:40, Roland Dobbins wrote:
On 20 Dec 2016, at 2:22, Denys Fedoryshchenko wrote:
If it is necessary i can investigate further.
Yes, please!
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
Hello. I'm Sho FUJIMURA. I operate the public NTP Services as 133.100.9.2 and 133.100.11.8. I'd like to reduce the traffic because I have trouble with too much traffic recently. So, I'm interested in the root of the the problem. If possible, would you please tell me the model numbers of Tenda and TP-Link?? -- Sho FUJIMURA Information Technology Center, Fukuoka University. 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan 2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko <denys@visp.net.lb>:
I'm not sure if this issue relevant to discussed topic, Tenda routers here for a while on market, and i think i noticed this issue just now, because NTP servers they are using supposedly for healthcheck went down (or NTP owners blocked ISP's i support, due such routers).
At least after checking numerous users, i believe Tenda hardcoded those NTP IPs. What worsen issue, that in Lebanon several times per day, for example at 18pm - short electricity cutoff, and majority of users routers will reboot and surely reconnect, so it will look like a countrywide spike in NTP traffic.
I checked for a 10min also this NTP ips in dns responses, none of thousands of users tried to resolve any name with them over any DNS server, so i conclude they are hardcoded somewhere in firmware.
Here is traffic of Tenda router after reconnecting (but not full powercycle, i dont have it in my hands). But as you can see, no DNS resolution attempts:
20:15:59.305739 PPPoE [ses 0x1483] CHAP, Success (0x03), id 1, Msg S=XXXXXX M=Authentication succeeded 20:15:59.306100 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length 12 20:15:59.317840 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 1, length 24 20:15:59.317841 PPPoE [ses 0x1483] IPCP, Conf-Ack (0x02), id 1, length 12 20:15:59.317867 PPPoE [ses 0x1483] IPCP, Conf-Nack (0x03), id 1, length 18 20:15:59.325253 PPPoE [ses 0x1483] IPCP, Conf-Request (0x01), id 2, length 24 20:15:59.325273 PPPoE [ses 0x1483] IPCP, Conf-Ack (0x02), id 2, length 24 20:15:59.335589 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 133.100.9.2.123: NTPv3, Client, length 48 20:15:59.335588 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.41.123: NTPv3, Client, length 48 20:15:59.335588 PPPoE [ses 0x1483] IP 172.17.49.245.123 > 192.5.41.40.123: NTPv3, Client, length 48
Here is example of Tenda traffic if it is unable to reach destination, it repeats request each 10 seconds endlessly, my guess they are using ntp to show status of internet connection. So, now that NTP servers getting quite significant DDoS such way.
19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177063.000000000 (2016/12/19 22:57:43) 19:58:02.164781 IP (tos 0x0, ttl 64, id 38518, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) 19:58:02.164884 IP (tos 0x0, ttl 64, id 38519, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) 19:58:02.165061 IP (tos 0x0, ttl 64, id 38520, offset 0, flags [none], proto UDP (17), length 76) 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48 Client, Leap indicator: (0), Stratum 0 (unspecified), poll 0 (1s), precision 0 Root Delay: 0.000000, Root dispersion: 0.000000, Reference-ID: (unspec) Reference Timestamp: 0.000000000 Originator Timestamp: 0.000000000 Receive Timestamp: 0.000000000 Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53) Originator - Receive Timestamp: 0.000000000 Originator - Transmit Timestamp: 3691177073.000000000 (2016/12/19 22:57:53)
On 2016-12-19 21:40, Roland Dobbins wrote:
On 20 Dec 2016, at 2:22, Denys Fedoryshchenko wrote:
If it is necessary i can investigate further.
Yes, please!
----------------------------------- Roland Dobbins <rdobbins@arbor.net>
participants (2)
-
Denys Fedoryshchenko -
FUJIMURA Sho