30% packet loss between cox.net and hetzner.de, possibly at tinet.net
Hello, There has been at least a 25% packet loss between hetzner.de and cox.net in the last couple of hours. Tried contacting hetzner.de, but they said it's not on their network. This has already happened a couple of days ago, too (strangely, on April 1), but then was good for the rest of the week -- no problems whatsoever. I wouldn't really care about this, if not for ssh: it just doesn't work on such huge loss. No other routes or networks seem affected. Any advice? # mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" ip68-97-XX-XXX.ok.ok.cox.net ; date HOST: xxxxxxxxxx Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- static.33.203.4.46.clients.your-server.de 60 60 0.0% 0.5 1.1 1.5 3.8 1.1 2.|-- hos-tr3.juniper2.rz13.hetzner.de 60 60 0.0% 0.2 0.4 3.4 46.2 9.8 3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 3.0 3.6 38.1 4.8 4.|-- r1nue1.core.init7.net 60 60 0.0% 2.8 4.9 6.0 13.3 3.9 5.|-- r1nue2.core.init7.net 60 60 0.0% 2.9 3.9 4.6 13.8 3.1 6.|-- r1fra2.core.init7.net 60 60 0.0% 5.7 7.6 8.2 17.3 3.8 7.|-- r1fra1.core.init7.net 60 60 0.0% 5.9 8.2 8.9 17.1 3.9 8.|-- xe-4-2-2.fra23.ip4.tinet.net 60 60 0.0% 5.9 6.4 7.1 38.2 5.2 9.|-- xe-3-0-0.dal33.ip4.tinet.net 60 39 35.0% 165.8 169.7 170.1 226.3 13.1 10.|-- cox-communications-gw.ip4.tinet.net 60 47 21.7% 159.0 162.3 162.6 207.9 10.4 11.|-- mtc3dsrj01-ae1.0.rd.ok.cox.net 60 46 23.3% 163.1 166.1 166.3 196.3 8.0 12.|-- 68.12.14.1 60 49 18.3% 161.3 161.6 161.6 161.9 0.1 13.|-- COX-68-12-10-114-static.coxinet.net 60 40 33.3% 162.5 162.8 162.8 163.1 0.1 14.|-- COX-68-12-10-114-static.coxinet.net 60 44 26.7% 162.5 162.8 162.8 163.2 0.1 15.|-- ip68-97-XX-XXX.ok.ok.cox.net 60 43 28.3% 170.9 173.5 173.5 179.5 1.5 Fri Apr 5 16:21:56 PDT 2013 % mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" static.88-198-xx-xx.clients.your-server.de ; date HOST: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- 192.168.xxxx 60 60 0.0% 1.0 1.9 2.5 14.0 2.5 2.|-- 10.0.x.x 60 60 0.0% 1.3 2.3 2.6 7.6 1.6 3.|-- 10.6.0.1 60 60 0.0% 8.9 13.7 14.3 38.6 5.1 4.|-- COX-68-12-10-113-static.coxinet.net 60 60 0.0% 9.6 14.1 15.6 97.2 11.9 5.|-- COX-68-12-10-2-static.coxinet.net 60 60 0.0% 10.1 14.9 15.4 31.2 4.3 6.|-- 68.1.5.161 60 60 0.0% 55.8 61.8 62.1 93.1 6.6 7.|-- nyk-s2-rou-1001.US.eurorings.net 60 60 0.0% 55.7 63.6 64.8 162.4 16.7 8.|-- nntr-s1-rou-1101.FR.eurorings.net 60 60 0.0% 149.5 168.7 169.7 213.1 19.3 9.|-- kehl-s2-rou-1103.DE.eurorings.net 60 47 21.7% 147.2 151.6 151.6 161.6 3.3 10.|-- ffm-s1-rou-1102.DE.eurorings.net 60 60 0.0% 144.1 148.9 149.1 174.2 6.4 11.|-- nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 147.1 156.2 157.3 290.9 22.4 12.|-- kpn-gw.hetzner.de 60 48 20.0% 173.8 178.9 179.0 198.2 5.1 13.|-- hos-bb2.juniper2.rz13.hetzner.de 60 43 28.3% 173.4 179.6 179.9 230.1 11.8 14.|-- hos-tr4.ex3k11.rz13.hetzner.de 60 48 20.0% 176.9 181.9 181.9 198.9 4.4 15.|-- static.88-198-xx-xx.clients.your-server.de 60 37 38.3% 173.2 177.0 177.0 186.6 3.4 Fri 5 Apr 2013 16:22:51 PDT The huge packet loss even extends to the regular cox.net web-site, it seems: # mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" cox.net ; date HOST: xxxxxxxxxx Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- static.33.203.4.46.clients.your-server.de 60 60 0.0% 0.5 1.2 1.7 5.6 1.4 2.|-- hos-tr1.juniper1.rz13.hetzner.de 60 60 0.0% 0.2 0.3 2.1 29.3 6.0 3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 3.5 5.8 62.5 10.1 4.|-- r1nue1.core.init7.net 60 60 0.0% 2.8 4.2 5.2 13.5 3.8 5.|-- r1nue2.core.init7.net 60 60 0.0% 2.9 4.7 6.0 28.7 5.0 6.|-- r1fra2.core.init7.net 60 60 0.0% 5.7 7.4 8.1 15.9 3.6 7.|-- r1fra1.core.init7.net 60 60 0.0% 5.9 7.6 8.1 16.8 3.2 8.|-- xe-4-2-2.fra23.ip4.tinet.net 60 60 0.0% 5.9 6.4 7.1 36.2 5.6 9.|-- xe-9-0-0.was14.ip4.tinet.net 60 43 28.3% 142.3 145.0 145.2 190.0 9.6 10.|-- cox-communications-gw.ip4.tinet.net 60 48 20.0% 124.8 129.6 130.2 176.6 13.2 11.|-- dukedsrj02-ge210.0.rd.at.cox.net 60 45 25.0% 137.7 141.2 141.6 194.7 11.6 12.|-- 68.1.15.238 60 49 18.3% 138.1 138.7 138.7 140.0 0.4 13.|-- 68.99.123.4 60 36 40.0% 140.4 140.9 140.9 141.6 0.3 14.|-- ww2.cox.com 60 44 26.7% 140.6 140.9 140.9 141.8 0.3 Fri Apr 5 18:19:21 PDT 2013 Best regards, Constantine.
On 2013-04-06 04:32, Constantine A. Murenin wrote:
Hello,
There has been at least a 25% packet loss between hetzner.de and cox.net in the last couple of hours.
Tried contacting hetzner.de, but they said it's not on their network. This has already happened a couple of days ago, too (strangely, on April 1), but then was good for the rest of the week -- no problems whatsoever.
I wouldn't really care about this, if not for ssh: it just doesn't work on such huge loss.
No other routes or networks seem affected.
Any advice?
Doesnt looks like tinet for me. I have colo in Europe and it is connected over tinet and mtr to static.33.203.4.46.clients.your-server.de is clean visp-probe ~ # mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" static.33.203.4.46.clients.your-server.de HOST: visp-probe Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- X.X.X.X 60 60 0.0% 0.1 0.9 6.1 50.7 11.7 2.|-- r1fra1.core.init7.net 60 60 0.0% 1.0 2.6 3.9 12.8 3.6 3.|-- r1fra3.core.init7.net 60 60 0.0% 1.0 2.2 3.2 12.0 3.2 4.|-- r1nue2.core.init7.net 60 60 0.0% 3.7 5.3 6.0 15.5 3.7 5.|-- r1nue1.core.init7.net 60 60 0.0% 3.9 5.8 6.5 15.3 3.6 6.|-- gw-hetzner.init7.net 60 60 0.0% 3.9 5.1 8.1 89.6 15.2 7.|-- hos-bb2.juniper2.rz13.hetzner.de 60 60 0.0% 6.1 7.7 10.4 74.9 14.8 8.|-- static.33.203.4.46.clients.your-server.de 60 60 0.0% 6.5 7.7 7.8 13.1 1.4 --- Denys Fedoryshchenko, Network Engineer, Virtual ISP S.A.L.
On 2013-W14-6 05:04 +0300, Denys Fedoryshchenko wrote:
On 2013-04-06 04:32, Constantine A. Murenin wrote:
Hello,
There has been at least a 25% packet loss between hetzner.de and cox.net in the last couple of hours.
Tried contacting hetzner.de, but they said it's not on their network. This has already happened a couple of days ago, too (strangely, on April 1), but then was good for the rest of the week -- no problems whatsoever.
I wouldn't really care about this, if not for ssh: it just doesn't work on such huge loss.
No other routes or networks seem affected.
Any advice?
Doesnt looks like tinet for me.
Might have been eurorings.net, as your Amazon EC2 to Hetzner traceroute seemed to suggest? This loss was apparent even with their own main websites: cox.net from a hetzner.de node: # mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" cox.net ; date ... 3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 3.5 5.8 62.5 10.1 4.|-- r1nue1.core.init7.net 60 60 0.0% 2.8 4.2 5.2 13.5 3.8 5.|-- r1nue2.core.init7.net 60 60 0.0% 2.9 4.7 6.0 28.7 5.0 6.|-- r1fra2.core.init7.net 60 60 0.0% 5.7 7.4 8.1 15.9 3.6 7.|-- r1fra1.core.init7.net 60 60 0.0% 5.9 7.6 8.1 16.8 3.2 8.|-- xe-4-2-2.fra23.ip4.tinet.net 60 60 0.0% 5.9 6.4 7.1 36.2 5.6 9.|-- xe-9-0-0.was14.ip4.tinet.net 60 43 28.3% 142.3 145.0 145.2 190.0 9.6 10.|-- cox-communications-gw.ip4.tinet.net 60 48 20.0% 124.8 129.6 130.2 176.6 13.2 11.|-- dukedsrj02-ge210.0.rd.at.cox.net 60 45 25.0% 137.7 141.2 141.6 194.7 11.6 12.|-- 68.1.15.238 60 49 18.3% 138.1 138.7 138.7 140.0 0.4 13.|-- 68.99.123.4 60 36 40.0% 140.4 140.9 140.9 141.6 0.3 14.|-- ww2.cox.com 60 44 26.7% 140.6 140.9 140.9 141.8 0.3 Fri Apr 5 18:19:21 PDT 2013 hetzner.de from a cox.net node: ... 5.|-- COX-68-12-8-132-static.coxinet.net 60 60 0.0% 11.6 15.3 15.8 35.1 4.9 6.|-- 68.1.5.161 60 60 0.0% 55.7 59.8 60.0 95.8 5.3 7.|-- nyk-s2-rou-1001.US.eurorings.net 60 60 0.0% 55.8 63.8 64.9 139.4 14.7 8.|-- nntr-s1-rou-1101.FR.eurorings.net 60 60 0.0% 149.8 154.1 154.1 171.9 4.3 9.|-- kehl-s2-rou-1103.DE.eurorings.net 60 60 0.0% 147.2 152.8 153.1 206.1 8.7 10.|-- ffm-s1-rou-1102.DE.eurorings.net 60 60 0.0% 143.3 147.7 147.7 177.2 5.0 11.|-- nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 147.3 153.8 154.0 211.1 10.0 12.|-- kpn-gw.hetzner.de 60 44 26.7% 173.3 177.6 177.6 184.5 2.7 13.|-- hos-bb2.juniper3.rz2.hetzner.de 60 42 30.0% 170.3 175.1 175.2 203.1 5.6 14.|-- hos-tr4.ms-ex3k2.rz1.hetzner.de 60 44 26.7% 171.9 175.9 175.9 185.4 2.8 15.|-- www.hetzner.de 60 42 30.0% 170.4 175.1 175.2 187.0 4.1 Fri 5 Apr 2013 17:38:11 PDT ... 5.|-- COX-68-12-8-132-static.coxinet.net 60 60 0.0% 11.3 14.2 14.5 33.6 3.3 6.|-- 68.1.5.161 60 60 0.0% 56.2 61.1 61.8 135.6 11.6 7.|-- nyk-s2-rou-1001.US.eurorings.net 60 60 0.0% 56.0 60.6 61.0 121.1 8.7 8.|-- nntr-s1-rou-1101.FR.eurorings.net 60 58 3.3% 150.1 153.5 153.5 166.3 2.8 9.|-- kehl-s2-rou-1103.DE.eurorings.net 60 29 51.7% 147.5 152.0 152.1 170.8 5.3 10.|-- ffm-s1-rou-1102.DE.eurorings.net 60 28 53.3% 143.7 148.4 148.6 186.7 8.0 11.|-- nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 147.2 151.8 151.9 178.2 4.7 12.|-- kpn-gw.hetzner.de 60 45 25.0% 172.7 177.0 177.0 197.4 4.0 13.|-- hos-bb2.juniper3.rz2.hetzner.de 60 41 31.7% 170.4 176.5 176.6 190.9 6.2 14.|-- hos-tr4.ms-ex3k2.rz1.hetzner.de 60 50 16.7% 171.8 175.6 175.6 183.1 2.8 15.|-- www.hetzner.de 60 47 21.7% 170.5 174.2 174.3 188.8 3.2 Fri 5 Apr 2013 19:22:36 PDT But it has since subsided: ... 3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 2.9 3.4 41.4 5.0 4.|-- r1nue1.core.init7.net 60 60 0.0% 2.8 5.2 6.3 12.4 3.8 5.|-- r1nue2.core.init7.net 60 60 0.0% 2.9 4.0 4.7 14.0 3.2 6.|-- r1fra2.core.init7.net 60 60 0.0% 5.7 7.7 8.4 17.0 3.9 7.|-- r1fra1.core.init7.net 60 60 0.0% 5.9 8.7 9.3 17.1 3.7 8.|-- xe-4-2-2.fra23.ip4.tinet.net 60 60 0.0% 5.9 6.1 6.2 15.0 1.3 9.|-- xe-9-0-0.was14.ip4.tinet.net 60 60 0.0% 95.9 99.4 99.8 134.3 8.9 10.|-- cox-communications-gw.ip4.tinet.net 60 60 0.0% 100.0 100.5 100.5 104.0 0.6 11.|-- dukedsrj02-ge210.0.rd.at.cox.net 60 60 0.0% 113.2 116.7 117.1 163.6 10.9 12.|-- 68.1.15.238 60 60 0.0% 113.4 113.8 113.8 116.0 0.4 13.|-- 68.99.123.4 60 50 16.7% 115.6 115.9 115.9 116.6 0.2 14.|-- ww2.cox.com 60 59 1.7% 115.7 116.1 116.1 117.2 0.3 Fri Apr 5 19:46:48 PDT 2013 ... 5.|-- COX-68-12-8-132-static.coxinet.net 60 60 0.0% 11.5 16.2 16.8 57.0 6.1 6.|-- 68.1.5.161 60 60 0.0% 56.3 61.5 61.8 105.1 6.5 7.|-- nyk-s2-rou-1001.US.eurorings.net 60 60 0.0% 56.6 63.9 64.6 122.2 11.0 8.|-- nntr-s1-rou-1101.FR.eurorings.net 60 60 0.0% 150.4 157.6 158.0 226.0 11.8 9.|-- kehl-s2-rou-1103.DE.eurorings.net 60 60 0.0% 147.7 152.5 152.7 206.0 7.7 10.|-- ffm-s1-rou-1102.DE.eurorings.net 60 60 0.0% 143.6 150.4 150.5 184.4 7.6 11.|-- nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 147.7 154.6 155.2 261.1 15.2 12.|-- kpn-gw.hetzner.de 60 60 0.0% 148.0 153.1 153.2 175.2 4.2 13.|-- hos-bb2.juniper3.rz2.hetzner.de 60 60 0.0% 145.0 164.8 166.2 220.6 22.2 14.|-- hos-tr4.ms-ex3k2.rz1.hetzner.de 60 59 1.7% 145.8 152.5 152.6 166.7 4.4 15.|-- www.hetzner.de 60 58 3.3% 144.5 149.7 149.7 161.3 3.9 Fri 5 Apr 2013 20:53:59 PDT With the end-to-end working, too. Cheers, Constantine.
On 2013-W14-5 21:27 -0700, Constantine A. Murenin wrote:
On 2013-W14-6 05:04 +0300, Denys Fedoryshchenko wrote:
On 2013-04-06 04:32, Constantine A. Murenin wrote:
Hello,
There has been at least a 25% packet loss between hetzner.de and cox.net in the last couple of hours.
Tried contacting hetzner.de, but they said it's not on their network. This has already happened a couple of days ago, too (strangely, on April 1), but then was good for the rest of the week -- no problems whatsoever.
I wouldn't really care about this, if not for ssh: it just doesn't work on such huge loss.
No other routes or networks seem affected.
Any advice?
Doesnt looks like tinet for me.
Might have been eurorings.net, as your Amazon EC2 to Hetzner traceroute seemed to suggest?
This loss was apparent even with their own main websites:
This is now happening again today, in a similar timeframe: % mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" hetzner.de ; date HOST: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Snt Rcv Loss% Best Gmean Avg Wrst StDev 4.|-- COX-68-12-10-121-static.coxinet.net 60 60 0.0% 8.8 12.3 12.9 43.4 5.4 5.|-- COX-68-12-8-132-static.coxinet.net 60 60 0.0% 11.4 14.5 14.6 27.0 2.5 6.|-- 68.1.5.161 60 60 0.0% 56.1 60.4 60.5 74.0 4.0 7.|-- nyk-s2-rou-1001.US.eurorings.net 60 60 0.0% 55.9 60.5 60.7 93.3 5.4 8.|-- nntr-s1-rou-1101.FR.eurorings.net 60 60 0.0% 150.1 154.0 154.1 167.0 2.9 9.|-- kehl-s2-rou-1103.DE.eurorings.net 60 60 0.0% 147.8 152.3 152.5 201.1 7.1 10.|-- ffm-s1-rou-1102.DE.eurorings.net 60 60 0.0% 143.9 148.0 148.1 174.0 4.6 11.|-- nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 147.4 160.6 163.4 336.0 36.2 12.|-- kpn-gw.hetzner.de 60 51 15.0% 167.3 172.5 172.5 184.3 3.5 13.|-- hos-bb2.juniper3.rz2.hetzner.de 60 60 0.0% 144.4 148.2 148.2 159.0 2.5 14.|-- hos-tr4.ms-ex3k2.rz1.hetzner.de 60 59 1.7% 145.5 150.1 150.1 157.8 3.0 15.|-- www.hetzner.de 60 47 21.7% 167.0 173.2 173.3 203.2 7.1 Sat 6 Apr 2013 15:14:00 PDT % traceroute -w2 -I hetzner.de; date traceroute to hetzner.de (213.133.107.227), 64 hops max, 60 byte packets 4 COX-68-12-10-121-static.coxinet.net (68.12.10.121) 22.012 ms 14.112 ms 11.230 ms 5 COX-68-12-8-132-static.coxinet.net (68.12.8.132) 23.021 ms 13.239 ms 12.400 ms 6 68.1.5.161 (68.1.5.161) 60.956 ms 59.738 ms 57.743 ms 7 nyk-s2-rou-1001.US.eurorings.net (134.222.248.13) 58.906 ms 60.571 ms 70.399 ms 8 nntr-s1-rou-1101.FR.eurorings.net (134.222.226.162) 152.597 ms 153.552 ms 150.250 ms 9 kehl-s2-rou-1103.DE.eurorings.net (134.222.227.121) 152.905 ms 149.469 ms 150.648 ms 10 ffm-s1-rou-1102.DE.eurorings.net (134.222.227.177) 148.114 ms 144.579 ms 147.114 ms 11 nbg-s1-rou-1001.DE.eurorings.net (134.222.227.118) 148.728 ms 150.950 ms 148.117 ms 12 kpn-gw.hetzner.de (134.222.107.21) 170.385 ms * 167.488 ms 13 hos-bb2.juniper3.rz2.hetzner.de (213.239.240.129) 147.476 ms 149.074 ms 144.824 ms 14 hos-tr4.ms-ex3k2.rz1.hetzner.de (213.239.193.243) 161.621 ms 148.504 ms 147.931 ms 15 * www.hetzner.de (213.133.107.227) 169.137 ms 176.884 ms Sat 6 Apr 2013 15:28:32 PDT Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" cox.net ; date HOST: XXXXXXXXXX Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- static.33.203.4.46.clients.your-server.de 60 60 0.0% 0.6 1.2 1.5 3.8 1.1 2.|-- hos-tr1.juniper1.rz13.hetzner.de 60 60 0.0% 0.2 0.3 1.8 34.6 5.8 3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 3.2 5.2 77.2 10.9 4.|-- r1nue1.core.init7.net 60 60 0.0% 2.8 4.6 5.7 13.3 3.9 5.|-- r1nue2.core.init7.net 60 60 0.0% 2.9 4.1 4.7 13.9 2.9 6.|-- r1fra2.core.init7.net 60 60 0.0% 5.7 7.9 8.6 17.8 3.8 7.|-- r1fra1.core.init7.net 60 60 0.0% 5.9 8.9 9.8 17.1 4.3 8.|-- xe-4-2-2.fra23.ip4.tinet.net 60 60 0.0% 5.9 6.2 6.9 59.5 7.0 9.|-- xe-10-2-2.was14.ip4.tinet.net 60 52 13.3% 115.3 119.8 120.1 163.4 9.1 10.|-- cox-communications-gw.ip4.tinet.net 60 51 15.0% 119.5 123.7 124.0 175.8 10.3 11.|-- dukedsrj02-ge210.0.rd.at.cox.net 60 48 20.0% 132.4 135.6 135.7 165.3 5.8 12.|-- 68.1.15.238 60 47 21.7% 132.9 135.0 135.1 146.5 2.0 13.|-- 68.99.123.4 60 33 45.0% 135.1 136.7 136.7 139.1 1.1 14.|-- ww2.cox.com 60 50 16.7% 135.1 136.8 136.8 139.2 1.1 Sat Apr 6 15:13:49 PDT 2013 Cns# traceroute -w2 -I cox.net; date traceroute to cox.net (68.99.123.161), 64 hops max, 60 byte packets 1 static.33.203.4.46.clients.your-server.de (46.4.203.33) 0.790 ms 3.658 ms 0.486 ms 2 hos-tr1.juniper1.rz13.hetzner.de (213.239.224.1) 0.244 ms 0.241 ms 0.251 ms 3 hos-bb2.juniper4.rz2.hetzner.de (213.239.240.138) 2.735 ms 2.772 ms 2.752 ms 4 r1nue1.core.init7.net (77.109.135.101) 2.811 ms 2.833 ms 2.769 ms 5 r1nue2.core.init7.net (77.109.140.154) 2.955 ms 2.928 ms 2.896 ms 6 r1fra2.core.init7.net (77.109.140.49) 14.891 ms 5.720 ms 13.291 ms 7 r1fra1.core.init7.net (77.109.128.137) 5.935 ms 7.804 ms 5.905 ms 8 xe-4-2-2.fra23.ip4.tinet.net (77.67.76.237) 5.885 ms 5.920 ms 5.912 ms 9 xe-10-2-2.was14.ip4.tinet.net (141.136.110.18) 115.127 ms 121.543 ms * 10 cox-communications-gw.ip4.tinet.net (77.67.79.234) 119.574 ms 119.245 ms 119.170 ms 11 dukedsrj02-ge210.0.rd.at.cox.net (68.1.1.123) 132.619 ms 132.431 ms 132.363 ms 12 68.1.15.238 (68.1.15.238) 132.718 ms 132.689 ms 132.599 ms 13 68.99.123.4 (68.99.123.4) 134.947 ms * 134.671 ms 14 * ww2.cox.com (68.99.123.161) 135.139 ms 135 ms Sat Apr 6 15:44:12 PDT 2013 It seems like hetzner.de <-> Amazon EC2 connectivity is indeed also affected, in addition to hetzner.de <-> cox.net: Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" www.tarsnap.com ; date 2.|-- hos-tr3.juniper2.rz13.hetzner.de 60 60 0.0% 0.2 0.3 2.8 50.1 9.6 3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 3.5 5.9 51.2 10.4 4.|-- r1nue1.core.init7.net 60 60 0.0% 2.8 4.7 5.6 14.2 3.5 5.|-- r1nue2.core.init7.net 60 60 0.0% 2.9 4.3 5.2 13.2 3.8 6.|-- r1fra2.core.init7.net 60 60 0.0% 5.7 8.8 9.7 18.4 4.6 7.|-- r1fra1.core.init7.net 60 60 0.0% 5.9 8.2 9.0 24.7 4.3 8.|-- xe-4-2-2.fra23.ip4.tinet.net 60 60 0.0% 5.9 6.4 7.0 43.7 5.4 9.|-- xe-4-1-1.was14.ip4.tinet.net 60 45 25.0% 115.1 117.3 117.6 173.2 9.6 10.|-- vadata-gw.ip4.tinet.net 60 50 16.7% 117.0 117.6 117.7 130.0 2.1 11.|-- 72.21.220.31 60 50 16.7% 118.1 120.2 120.2 140.4 4.6 12.|-- 205.251.245.55 60 49 18.3% 120.2 121.1 121.1 132.0 2.1 13.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 14.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 15.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 16.|-- 216.182.224.129 60 54 10.0% 118.2 118.9 118.9 130.3 1.7 17.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 18.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 19.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 20.|-- ec2-23-21-149-109.compute-1.amazonaws.com 59 53 10.2% 118.6 119.4 119.4 139.8 2.9 Sat Apr 6 15:21:06 PDT 2013 On the other hand, comcast.net is not affected, att.net not affected, he.net not affected, lots of other routes not affected. Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" comcast.net ; date 2.|-- hos-tr3.juniper2.rz13.hetzner.de 60 60 0.0% 0.2 0.3 3.9 56.6 11.6 3.|-- hos-bb2.juniper8.rz1.hetzner.de 60 60 0.0% 2.8 2.9 3.1 24.9 2.9 4.|-- nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 3.0 3.6 7.3 177.0 24.2 5.|-- mchn-s1-rou-1021.DE.eurorings.net 60 60 0.0% 6.9 7.6 8.0 24.1 3.8 6.|-- mchn-s1-rou-1103.DE.eurorings.net 60 60 0.0% 91.8 93.7 93.9 133.0 6.7 7.|-- kehl-s2-rou-1103.DE.eurorings.net 60 60 0.0% 93.9 95.8 96.1 134.2 7.3 8.|-- nntr-s1-rou-1101.FR.eurorings.net 60 60 0.0% 91.8 94.2 94.3 127.6 5.3 9.|-- nyk-s2-rou-1021.US.eurorings.net 60 60 0.0% 92.1 95.7 96.0 128.3 8.2 10.|-- nyk-s2-rou-1001.US.eurorings.net 60 60 0.0% 91.7 93.2 93.7 188.9 12.6 11.|-- te-1-10-0-4-pe01.111eighthave.ny.ibone.comcast.net 60 60 0.0% 91.7 92.3 92.3 93.7 0.3 12.|-- pos-1-6-0-0-cr01.newyork.ny.ibone.comcast.net 60 60 0.0% 92.2 94.0 94.0 95.8 1.1 13.|-- he-0-0-0-0-cr01.350ecermak.il.ibone.comcast.net 60 60 0.0% 111.5 117.1 117.2 123.3 3.6 14.|-- pos-0-3-0-0-ar02.northlake.il.ndcchgo.comcast.net 60 60 0.0% 112.8 113.0 113.0 113.5 0.1 15.|-- te-0-3-0-0-ur04.northlake.il.ndcchgo.comcast.net 60 60 0.0% 112.8 113.0 113.0 113.3 0.1 16.|-- ge-0-1-0-0-ur03.northlake.il.ndcchgo.comcast.net 60 60 0.0% 112.9 113.0 113.0 113.3 0.1 17.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 Sat Apr 6 15:47:41 PDT 2013 Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" att.net ; date 2.|-- hos-tr3.juniper2.rz13.hetzner.de 60 60 0.0% 0.2 0.2 1.0 45.2 5.8 3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 3.7 8.0 86.2 17.5 4.|-- ae55.edge7.Frankfurt1.Level3.net 60 60 0.0% 7.2 7.9 8.9 57.8 7.9 5.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 6.|-- ae-62-62.ebr2.Frankfurt1.Level3.net 60 60 0.0% 85.3 85.3 85.3 87.2 0.3 7.|-- ae-23-23.ebr2.London1.Level3.net 60 60 0.0% 85.3 85.3 85.3 85.4 0.0 8.|-- ae-43-43.ebr1.NewYork1.Level3.net 60 60 0.0% 85.4 85.5 85.5 88.6 0.4 9.|-- ae-71-71.csw2.NewYork1.Level3.net 60 60 0.0% 85.4 87.6 87.7 97.7 3.9 10.|-- ae-2-70.edge3.NewYork1.Level3.net 60 60 0.0% 85.5 87.5 88.0 154.7 10.9 11.|-- att-level3.newyork1.level3.net 60 60 0.0% 86.5 88.3 88.3 91.8 1.2 12.|-- cr2.n54ny.ip.att.net 60 60 0.0% 111.5 116.7 116.8 124.9 4.9 13.|-- cr2.wswdc.ip.att.net 60 60 0.0% 110.1 115.4 115.5 123.9 5.1 14.|-- cr1.attga.ip.att.net 60 60 0.0% 111.3 116.7 116.8 124.9 4.8 ... Sat Apr 6 15:36:40 PDT 2013 Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" he.net ; date 2.|-- hos-tr3.juniper2.rz13.hetzner.de 60 60 0.0% 0.2 0.3 4.2 71.3 13.9 3.|-- hos-bb2.juniper4.ffm.hetzner.de 60 60 0.0% 5.8 5.9 6.0 9.9 0.7 4.|-- 30gigabitethernet4-3.core1.fra1.he.net 60 60 0.0% 5.7 8.3 8.8 17.6 3.3 5.|-- 10gigabitethernet1-4.core1.par2.he.net 60 60 0.0% 15.1 18.1 18.5 26.0 3.7 6.|-- 10gigabitethernet7-1.core1.ash1.he.net 60 60 0.0% 93.1 95.9 95.9 105.7 3.5 7.|-- 10gigabitethernet11-1.core1.pao1.he.net 60 60 0.0% 163.4 167.5 167.6 177.0 4.1 8.|-- 10gigabitethernet1-2.core1.fmt1.he.net 60 60 0.0% 164.5 170.2 170.5 248.5 12.6 9.|-- he.net 60 59 1.7% 164.0 164.6 164.6 165.2 0.2 Sat Apr 6 15:52:46 PDT 2013 Although hetzner.de claims that this whole loss is outside of their own network, I'm inclined to deduce that the loss might actually be concentrated on their own KPN / eurorings.net router -- kpn-gw.hetzner.de (134.222.107.21), and perhaps occurs only in one direction. Although there is no traffic loss from he.net if you try to traceroute the router itself (I'm not sure what that means, though, other than a potential attack vector from exposing a router globally like that): # mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" -4 134.222.107.21 ; date HOST: XXXXXXXXX Snt Rcv Loss% Best Gmean Avg Wrst StDev 1. router1-fmt.linode.com 60 60 0.0% 0.5 1.5 4.8 70.2 10.6 2. 10gigabitethernet2-3.core1.fmt1.he.net 60 60 0.0% 0.6 1.0 3.5 72.1 11.9 3. 10gigabitethernet1-1.core1.pao1.he.net 60 60 0.0% 1.0 1.2 1.4 12.4 1.5 4. eqix-sv8.kpn.com 60 60 0.0% 1.8 2.5 5.9 77.9 15.2 5. chg-s1-rou-1041.US.eurorings.net 60 60 0.0% 161.5 167.1 167.3 197.9 7.5 6. ahbn-s1-rou-1041.US.eurorings.net 60 60 0.0% 71.5 74.0 74.3 108.9 7.5 7. nyk-s2-rou-1021.US.eurorings.net 60 60 0.0% 71.6 75.7 76.2 131.7 10.3 8. nntr-s1-rou-1101.FR.eurorings.net 60 60 0.0% 161.6 164.5 164.6 188.8 6.8 9. kehl-s2-rou-1103.DE.eurorings.net 60 60 0.0% 161.5 162.9 163.0 206.3 6.0 10. ffm-s1-rou-1102.DE.eurorings.net 60 60 0.0% 157.9 166.9 167.1 193.8 9.4 11. nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 161.6 163.6 163.9 235.2 10.3 12. kpn-gw.hetzner.de 60 59 1.7% 161.3 161.9 161.9 167.5 1.0 Sat Apr 6 15:58:04 PDT 2013 li163-159:~# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" -4 hetzner.de ; date HOST: xxxxxxxxx Snt Rcv Loss% Best Gmean Avg Wrst StDev 1. router1-fmt.linode.com 60 60 0.0% 0.5 8.6 34.8 396.6 66.0 2. 10gigabitethernet2-3.core1.fmt1.he.net 60 60 0.0% 0.6 2.7 6.7 75.1 12.2 3. 10gigabitethernet1-1.core1.pao1.he.net 60 60 0.0% 0.9 2.2 3.8 22.1 4.3 4. 10gigabitethernet9-3.core1.ash1.he.net 60 60 0.0% 71.2 74.9 75.1 88.7 4.9 5. 10gigabitethernet1-2.core1.par2.he.net 60 60 0.0% 149.0 151.9 152.0 159.8 3.5 6. 10gigabitethernet4-4.core1.fra1.he.net 60 60 0.0% 158.2 161.3 161.4 169.1 3.8 7. decix-gw.hetzner.de 60 60 0.0% 158.6 161.2 161.4 209.3 8.9 8. hos-bb1.juniper3.rz2.hetzner.de 60 60 0.0% 161.8 173.2 174.0 220.2 18.4 9. hos-tr3.ms-ex3k2.rz1.hetzner.de 60 60 0.0% 162.6 163.7 163.7 166.0 1.1 10. www.hetzner.de 60 59 1.7% 161.8 162.4 162.4 163.1 0.3 Sat Apr 6 16:14:48 PDT 2013 I've been a fan of hetzner.de, but I think it's staggering that they won't do anything about this huge and persistent packet loss. Best regards, Constantine.
On 2013-04-07 02:20, Constantine A. Murenin wrote:
Although hetzner.de claims that this whole loss is outside of their own network, I'm inclined to deduce that the loss might actually be concentrated on their own KPN / eurorings.net router -- kpn-gw.hetzner.de (134.222.107.21), and perhaps occurs only in one direction.
I think too. Btw as i said, have host on tinet, and it is 100% clear from EC2. So seems tinet is fine for sure. HOST: ip-10-203-61-X Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- ip-10-203-60-2.ec2.internal 60 60 0.0% 0.3 0.6 1.1 19.1 2.7 2.|-- ip-10-1-36-21.ec2.internal 60 60 0.0% 0.4 0.6 0.8 9.3 1.3 3.|-- ip-10-1-34-0.ec2.internal 60 60 0.0% 0.4 0.7 1.0 14.5 2.0 4.|-- 100.64.20.43 60 60 0.0% 0.4 0.6 0.6 2.0 0.2 5.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 6.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 7.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 8.|-- 100.64.16.157 60 60 0.0% 0.5 2.4 9.7 68.8 16.1 9.|-- 72.21.222.154 60 60 0.0% 1.5 1.9 2.6 36.4 4.8 10.|-- 72.21.220.46 60 60 0.0% 1.5 2.1 3.3 59.2 7.6 11.|-- xe-7-2-0.was10.ip4.tinet.net 60 60 0.0% 1.6 2.1 2.6 17.2 3.1 12.|-- xe-0-1-0.fra23.ip4.tinet.net 60 60 0.0% 92.2 92.8 92.8 104.3 1.9 13.|-- ge-1-1-0.pr1.g310.fra.de.eurotransit.net 60 60 0.0% 92.2 93.1 93.2 112.8 3.3 14.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0 last hop icmp blocked, it is my host
Although there is no traffic loss from he.net if you try to traceroute the router itself (I'm not sure what that means, though, other than a potential attack vector from exposing a router globally like that):
I don't think there is attack vector, proper control plane ACL will make them safe.
I've been a fan of hetzner.de, but I think it's staggering that they won't do anything about this huge and persistent packet loss. Indeed, i noticed that transfers from EC2 are terrible last days to Hetzner.
Maybe worth to open topic at www.webhostingtalk.com ?
Best regards, Constantine.
--- Denys Fedoryshchenko, Network Engineer, Virtual ISP S.A.L.
participants (2)
-
Constantine A. Murenin -
Denys Fedoryshchenko