For anyone's following the story: the weeks-long congestion on he.net remains, however, hetzner has switched my original route to an alternative uplink. I'm no longer experiencing the he.net evening jitter that would bring my avg rtt from the non-congested 114ms to an average of 140ms and more during the busy times. Other sites that have an uplink from he.net are still affected, with a noticeable jitter, delay and some packet loss (which seems to increase with the increase in the avg rtt). Was: Tue Dec 3 22:25:38 PST 2013 Wed Dec 4 07:25:38 CET 2013 HOST: Cns??????? Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- static.??.???.4.46.clients.your-server.de 900 900 0.0% 0.5 0.9 1.3 7.3 1.2 2.|-- hos-tr1.juniper1.rz13.hetzner.de 900 900 0.0% 0.2 0.2 2.1 101.2 9.0 3.|-- core21.hetzner.de 900 900 0.0% 0.2 0.2 0.2 19.7 1.1 4.|-- core22.hetzner.de 900 897 0.3% 0.2 0.2 0.2 19.6 1.1 5.|-- core1.hetzner.de 900 900 0.0% 4.8 4.8 4.8 16.4 1.1 6.|-- juniper1.ffm.hetzner.de 900 893 0.8% 4.8 4.8 4.8 19.1 0.7 7.|-- 30gigabitethernet1-3.core1.ams1.he.net 900 900 0.0% 11.2 14.0 14.8 123.4 6.5 8.|-- 10ge1-4.core1.lon1.he.net 900 900 0.0% 18.2 19.9 20.2 59.7 3.9 9.|-- 10ge10-4.core1.nyc4.he.net 900 896 0.4% 86.9 113.6 114.2 148.4 11.4 10.|-- 100ge7-2.core1.chi1.he.net 900 898 0.2% 106.6 133.0 133.6 184.6 12.4 11.|-- ??? 900 0 100.0 0.0 0.0 0.0 0.0 0.0 12.|-- et-11-0-0.945.rtr.ictc.indiana.gigapop.net 900 899 0.1% 113.3 136.7 137.1 162.3 10.8 13.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu 900 895 0.6% 113.3 137.2 137.7 183.3 11.1 14.|-- ae-0.0.br2.bldc.net.uits.iu.edu 900 900 0.0% 114.3 137.8 138.2 162.8 10.7 15.|-- ae-10.0.cr3.bldc.net.uits.iu.edu 900 899 0.1% 114.4 137.9 138.4 167.5 10.7 16.|-- m???c????????.indiana.edu 900 898 0.2% 114.5 138.1 138.5 159.0 10.4 Tue Dec 3 22:43:16 PST 2013 Wed Dec 4 07:43:16 CET 2013 0 154.8 155 = 94 + 61 1 127.9 128 = 67 + 61 2 125.0 125 = 64 + 61 3 131.5 132 = 71 + 61 4 131.2 132 = 71 + 61 5 117.4 118 = 57 + 61 6 132.9 133 = 72 + 61 7 143.4 144 = 83 + 61 8 138.7 139 = 78 + 61 9 131.5 131 = 70 + 61 10 142.2 142 = 81 + 61 11 152.2 152 = 91 + 61 12 125.2 125 = 64 + 61 13 125.6 125 = 64 + 61 14 140.8 141 = 80 + 61 15 150.9 151 = 89 + 62 Tue Dec 3 22:43:31 PST 2013 Wed Dec 4 07:43:31 CET 2013 Now: Tue Dec 3 23:01:20 PST 2013 Wed Dec 4 08:01:20 CET 2013 HOST: Cns??????? Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- static.??.???.4.46.clients.your-server.de 900 900 0.0% 0.5 0.9 1.3 7.0 1.1 2.|-- hos-tr1.juniper1.rz13.hetzner.de 900 900 0.0% 0.1 0.2 4.3 245.4 18.9 3.|-- core21.hetzner.de 900 900 0.0% 0.2 0.2 0.3 99.4 3.5 4.|-- core12.hetzner.de 900 900 0.0% 2.7 2.8 2.8 23.1 1.9 5.|-- juniper4.rz2.hetzner.de 900 899 0.1% 2.8 2.8 2.8 29.0 1.1 6.|-- te0-0-2-1.nr21.b040138-0.nue01.atlas.cogentco.com 900 900 0.0% 3.1 3.3 3.4 7.1 0.6 7.|-- te2-4.ccr01.nue01.atlas.cogentco.com 900 900 0.0% 3.1 6.6 25.9 244.4 51.0 | `|-- 154.25.0.13 8.|-- te0-2-0-3.ccr22.muc01.atlas.cogentco.com 900 900 0.0% 5.7 5.8 5.9 11.1 0.5 9.|-- be2229.ccr22.fra03.atlas.cogentco.com 900 900 0.0% 11.1 11.4 11.4 14.4 0.5 | `|-- 154.54.38.189 10.|-- te0-3-0-2.mpd22.ams03.atlas.cogentco.com 900 900 0.0% 17.7 17.9 17.9 49.1 1.1 | `|-- 154.54.39.193 11.|-- be2276.ccr22.lon13.atlas.cogentco.com 900 900 0.0% 25.3 27.4 27.4 33.2 1.3 | `|-- 154.54.37.106 | |-- 154.54.58.69 12.|-- te0-7-0-33.ccr22.bos01.atlas.cogentco.com 900 900 0.0% 93.1 93.9 93.9 98.8 1.2 | `|-- 154.54.44.189 | |-- 130.117.0.185 13.|-- te7-8.ccr02.alb02.atlas.cogentco.com 900 900 0.0% 96.8 119.9 128.8 364.2 56.9 | `|-- 154.54.27.153 14.|-- te7-8.ccr01.buf02.atlas.cogentco.com 900 900 0.0% 103.3 126.8 135.8 444.9 59.0 15.|-- te0-5-0-2.ccr21.cle04.atlas.cogentco.com 900 900 0.0% 108.6 109.5 109.5 113.5 1.3 16.|-- te3-2.ccr01.cmh02.atlas.cogentco.com 900 900 0.0% 111.1 134.8 143.5 467.0 59.4 17.|-- te4-7.ccr01.cvg02.atlas.cogentco.com 900 900 0.0% 113.8 122.5 126.2 467.9 39.6 18.|-- te3-3.ccr01.ind01.atlas.cogentco.com 900 899 0.1% 116.1 137.8 145.4 457.7 56.0 19.|-- 38.104.214.6 900 900 0.0% 115.8 118.3 118.5 206.0 8.1 20.|-- xe-0-1-0.11.rtr.ictc.indiana.gigapop.net 900 900 0.0% 115.9 116.7 116.7 128.4 1.5 21.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu 900 900 0.0% 115.9 117.0 117.1 168.6 3.6 22.|-- ae-0.0.br2.bldc.net.uits.iu.edu 900 900 0.0% 116.8 117.7 117.7 130.8 1.8 23.|-- ae-10.0.cr3.bldc.net.uits.iu.edu 900 899 0.1% 116.8 117.8 117.8 135.1 1.9 24.|-- m???c????????.indiana.edu 900 899 0.1% 117.1 117.9 117.9 122.0 1.2 Tue Dec 3 23:19:56 PST 2013 Wed Dec 4 08:19:56 CET 2013 0 120.1 120 = 57 + 63 1 117.3 117 = 54 + 63 2 120.3 120 = 57 + 63 3 117.3 117 = 54 + 63 4 117.4 118 = 54 + 64 5 117.3 118 = 54 + 64 6 120.0 120 = 57 + 63 7 117.3 118 = 54 + 64 8 117.3 118 = 54 + 64 9 117.6 118 = 54 + 64 10 117.3 118 = 54 + 64 11 117.4 118 = 54 + 64 12 117.4 117 = 53 + 64 13 118.6 118 = 55 + 63 14 117.5 117 = 53 + 64 15 117.3 117 = 53 + 64 Tue Dec 3 23:20:11 PST 2013 Wed Dec 4 08:20:11 CET 2013 Yet: Cns# sh -c 'while (true); do date; env TZ=Europe/Berlin date; mtr --report{,-wide,-cycles=900} --interval 1 --order "SRL BGAWV" -4 ntp1.yycix.ca; date; env TZ=Europe/Berlin date; unbuffer hping --icmp-ts --count 16 ntp1.yycix.ca | perl -ne "if (/icmp_seq=(\d+) rtt=(\d+\.\d)/) {(\$s,\$p)=(\$1,\$2);} if (/ate=(\d+) Receive=(\d+) Transmit=(\d+)/) {(\$o,\$r,\$t)=(\$1,\$2,\$3);} if (/tsrtt=(\d+)/) {print \$s,qq/\t/,\$p,qq/\t/,\$1,qq/ = /,\$r-\$o,qq/ + /,\$o+\$1-\$t,qq/\n/;}"; done' Tue Dec 3 23:32:37 PST 2013 Wed Dec 4 08:32:37 CET 2013 HOST: Cns??????? Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- static.??.???.4.46.clients.your-server.de 900 900 0.0% 0.5 0.9 1.2 9.7 1.2 2.|-- hos-tr4.juniper2.rz13.hetzner.de 900 900 0.0% 0.2 0.2 2.0 138.3 8.9 3.|-- core22.hetzner.de 900 900 0.0% 0.2 0.2 0.2 8.5 0.4 4.|-- core1.hetzner.de 900 900 0.0% 4.8 4.8 4.8 16.0 0.7 5.|-- juniper1.ffm.hetzner.de 900 900 0.0% 4.8 4.8 4.8 13.8 0.7 6.|-- 30gigabitethernet1-3.core1.ams1.he.net 900 900 0.0% 11.2 14.2 14.9 106.2 6.2 7.|-- 10ge1-4.core1.lon1.he.net 900 900 0.0% 18.0 20.5 20.8 55.6 4.1 8.|-- 10ge10-4.core1.nyc4.he.net 900 895 0.6% 86.4 119.3 120.0 158.6 13.1 9.|-- 10ge1-2.core1.tor1.he.net 900 896 0.4% 96.4 128.4 129.1 159.6 12.9 10.|-- 10ge3-1.core1.ywg1.he.net 900 895 0.6% 121.8 152.2 152.7 181.5 12.6 11.|-- 10ge1-1.core1.yyc1.he.net 900 888 1.3% 135.7 168.0 168.5 197.4 13.0 12.|-- sebo-systems-inc.gigabitethernet2-23.core1.yyc1.he.net 900 890 1.1% 138.4 168.2 168.6 190.9 12.5 13.|-- ??? 900 0 100.0 0.0 0.0 0.0 0.0 0.0 14.|-- ntp1.yycix.ca 900 893 0.8% 138.7 168.8 169.2 191.9 12.2 Tue Dec 3 23:49:07 PST 2013 Wed Dec 4 08:49:07 CET 2013 0 180.0 180 = 109 + 71 1 155.9 156 = 84 + 72 2 143.6 144 = 72 + 72 3 179.8 180 = 108 + 72 4 161.0 161 = 89 + 72 5 155.9 156 = 84 + 72 6 168.0 168 = 97 + 71 7 164.5 165 = 93 + 72 8 177.9 178 = 107 + 71 9 159.6 160 = 88 + 72 10 179.1 180 = 108 + 72 11 178.6 178 = 106 + 72 12 149.8 149 = 78 + 71 13 166.2 166 = 94 + 72 14 146.8 147 = 75 + 72 15 153.7 154 = 82 + 72 Tue Dec 3 23:49:22 PST 2013 Wed Dec 4 08:49:22 CET 2013 Cheers, Constantine. On 2013-W48-7 01:11 -0800, Constantine A. Murenin wrote:
Anyhow, is this better?
I now saw a 2% traffic loss this night at a random test time, and the 151ms avg rtt on this 114ms rtt route.
Cns# date ; mtr --report{,-wide,-cycles=600} --interval 0.5 --order "SRL BGAWV" -4 ????c????????.indiana.edu ; date Sat Nov 30 23:17:13 PST 2013 HOST: Cns??????? Snt Rcv Loss% Best Gmean Avg Wrst StDev 1.|-- static.??.???.4.46.clients.your-server.de 600 600 0.0% 0.5 1.0 1.3 4.6 1.1 2.|-- hos-tr1.juniper1.rz13.hetzner.de 600 600 0.0% 0.1 0.2 2.0 58.5 7.9 3.|-- core21.hetzner.de 600 600 0.0% 0.2 0.2 0.2 10.2 0.7 4.|-- core22.hetzner.de 600 600 0.0% 0.2 0.2 0.2 11.2 0.8 5.|-- core1.hetzner.de 600 600 0.0% 4.8 4.8 4.8 25.1 1.3 6.|-- juniper1.ffm.hetzner.de 600 600 0.0% 4.8 4.8 4.8 13.9 0.6 7.|-- 30gigabitethernet1-3.core1.ams1.he.net 600 595 0.8% 11.2 14.3 15.2 121.4 7.4 8.|-- 10gigabitethernet1-4.core1.lon1.he.net 600 600 0.0% 18.2 21.0 21.3 51.2 4.0 9.|-- 10gigabitethernet10-4.core1.nyc4.he.net 600 592 1.3% 86.9 125.9 126.4 160.7 10.6 10.|-- 100gigabitethernet7-2.core1.chi1.he.net 600 591 1.5% 106.6 145.1 145.4 190.9 10.5 11.|-- ??? 600 0 100.0 0.0 0.0 0.0 0.0 0.0 12.|-- et-11-0-0.945.rtr.ictc.indiana.gigapop.net 600 589 1.8% 114.3 148.9 149.2 167.9 9.1 13.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu 600 589 1.8% 113.4 149.2 149.5 173.4 9.3 14.|-- ae-0.0.br2.bldc.net.uits.iu.edu 600 590 1.7% 114.5 150.2 150.5 175.6 9.3 15.|-- ae-10.0.cr3.bldc.net.uits.iu.edu 600 589 1.8% 114.3 150.5 150.8 181.0 9.1 16.|-- ????c????????.indiana.edu 600 589 1.8% 114.8 150.7 151.0 170.7 9.0 Sat Nov 30 23:24:06 PST 2013
The ICMP timestamp request/reply test still indicates that only one path is affected: the one from Europe to US over he.net.
Cns# date ; unbuffer hping --icmp-ts --count 30 ????c????????.indiana.edu | \ perl -ne 'if (/icmp_seq=(\d+) rtt=(\d+\.\d)/) {($s, $p) = ($1, $2);} \ if (/ate=(\d+) Receive=(\d+) Transmit=(\d+)/) {($o, $r, $t) = ($1, $2, $3);} \ if (/tsrtt=(\d+)/) { \ print $s, "\t", $p, "\t", $1, " = ", $r - $o, " + ", $o + $1 - $t, "\n"; }' Sun Dec 1 00:55:46 PST 2013 0 151.3 151 = 91 + 60 1 154.2 154 = 93 + 61 2 127.8 127 = 67 + 60 3 123.6 123 = 63 + 60 4 136.9 137 = 76 + 61 5 149.6 149 = 89 + 60 6 147.4 147 = 87 + 60 7 133.5 133 = 73 + 60 8 152.2 152 = 92 + 60 9 137.3 137 = 77 + 60 10 143.7 144 = 84 + 60 11 124.5 124 = 64 + 60 12 141.4 141 = 81 + 60 13 118.0 118 = 58 + 60 14 153.6 154 = 94 + 60 15 137.7 138 = 78 + 60 16 119.9 120 = 60 + 60 17 130.6 131 = 71 + 60 18 144.6 145 = 85 + 60 19 138.8 139 = 79 + 60 20 155.7 156 = 96 + 60 21 128.8 129 = 69 + 60 22 153.0 153 = 93 + 60 23 146.5 147 = 87 + 60 24 137.2 138 = 77 + 61 25 153.3 154 = 94 + 60 26 146.3 147 = 87 + 60 27 150.1 151 = 91 + 60 28 150.5 150 = 90 + 60 29 143.5 143 = 83 + 60
Cheers, Constantine.