July 2013 - Test - lists.nanog.org

Diverse sub-sea carrier links London / Tokyo
by Rodrick Brown 10 Jul '13

10 Jul '13

I'm looking for a diverse carrier circuits to/from the following POP's over different sub-sea links. I'm not too comfortable getting diverse path(s) from the same carrier (Hibernia) so I'm interested in getting input from others who have dealt with other carriers in these locations who can offer connectivity over different sub-sea links than my current provider. 755 Secaucus-350 Cermak-Seattle-*PC1 Cable*-Ajigaura-TY3 (Equinix) 755 Secaucus-300 Blvd-111 8th-*AC1 Standard*-Slough-Telehouse North-London Hosting Center (Savvis) Sorry if this is off-list. --RB

1 0

google troubles?
by Blair Trosper 10 Jul '13

10 Jul '13

Seeing lots of reports of people unable to get to many Google services. Seems to be affecting Comcast users disproportionately. It's fine for me, but a lot of my staff are basically out of luck...but according to the Google Apps Status page, everything is fine. It's anecdotal, but it would seem like there's an issue based on these reports. Oh, and this: http://www.cnn.com/2013/07/10/tech/web/google-down/index.html Anyone know what's up? Fiber cut? DC outages? -- blair

6 6

www.att.net ipv6 traceroute
by David Hill 10 Jul '13

10 Jul '13

Anyone else noticing odd ipv6 traceroutes to www.att.net (2001:1890:1c01:2::40)? For example, the first traceroute originating in Chicago, and if the hostnames are anything to go by: Chicago -> Dallas -> Atlanta -> Miama -> NYC -> Amsterdam -> Frankfurt -> France -> D.C. -> Atlanta... The first 2 traceroutes are on SixXs tunnels and the third is native. From 2604:8800::/32 (Chicago, Sixxs) 2 gw-122.chi-03.us.sixxs.net 25.925 ms 14.816 ms 13.645 ms 3 uschi03.sixxs.net 18.651 ms 15.544 ms 34.06 ms 4 2620:0:6b0:a::1 27.401 ms 15.677 ms 14.83 ms 5 tge6-3.fr3.ord4.ipv6.llnw.net 14.482 ms 18.875 ms 21.112 ms 6 ve8.fr3.ord.ipv6.llnw.net 28.965 ms 24.031 ms 15.947 ms 7 tge2-1.fr3.dal.ipv6.llnw.net 39.672 ms 41.431 ms 49.888 ms 8 tge4-1.fr3.atl1.ipv6.llnw.net 59.908 ms 62.198 ms 72.244 ms 9 tge2-1.fr3.mia1.ipv6.llnw.net 78.641 ms 75.351 ms 74.425 ms 10 2600:804:11f::11 91.243 ms 90.002 ms 87.802 ms 11 2600:802::23 119.804 ms 100.727 ms 100.737 ms 12 ge-0-3-0.IL1.NYC12.ALTER.NET 147.803 ms 145.406 ms 147.878 ms 13 ae1.XT2.AMS2.ALTER.NET 148.699 ms ae1.XT1.AMS2.ALTER.NET 146.322 ms 152.503 ms 14 GigabitEthernet6-0-0.GW5.AMS6.ALTER.NET 145.967 ms 148.903 ms 165.926 ms 15 FastEthernet0-0.6B1.AMS6.ALTER.NET 183.716 ms 156.131 ms 194.141 ms 16 tu3.6B1.FFT4.ALTER.NET 184.867 ms 184.005 ms 170.47 ms 17 ffm-s1-rou-1006.DE.eurorings.net 159.565 ms * 173.012 ms 18 ffm-s1-rou-1102.DE.eurorings.net 200.82 ms 225.979 ms 207.712 ms 19 kehl-s2-rou-1103.DE.eurorings.net 228.884 ms 208.576 ms 203.722 ms 20 nntr-s1-rou-1101.FR.eurorings.net 192.901 ms 210.818 ms 210.279 ms 21 * 2001:680:0:134:222:226:160:1 189.452 ms 171.714 ms 22 2001:1890:c00:c001::1171:ecca 192.306 ms 164.33 ms 192.835 ms 23 2001:1890:c00:c001::ee71:ecca 222.488 ms 218.21 ms 194.537 ms 24 2001:1890:ff:ffff:12:122:105:90 214.309 ms 216.21 ms 216.51 ms 25 wswdc22crs.ipv6.att.net 206.215 ms 220.884 ms 202.472 ms 26 attga21crs.ipv6.att.net 213.776 ms 220.918 ms 205.205 ms 27 attga409me3.ipv6.att.net 190.66 ms 228.532 ms 200.606 ms 28 2001:1890:c00:8802::11cc:6c6d 217.65 ms 222.145 ms 210.213 ms From 2a02:2528::/32 (Switzerland, Sixxs) 2 gw-91.gva-01.ch.sixxs.net 28.932 ms 28.415 ms 29.091 ms 3 2a02:2528:ff:1::4 28.082 ms 28.873 ms 41.525 ms 4 ge0-1.gv1.ip-max.sixxs.net 29.546 ms 37.835 ms 29.366 ms 5 ge3-14.ar01.gva253.ip-max.net 30.502 ms 32.925 ms 29.370 ms 6 ae1.cr01.gva253.ip-max.net 45.019 ms 41.335 ms 47.675 ms 7 xe0-0-1.cr01.gva252.ip-max.net 28.903 ms 33.530 ms 41.954 ms 8 po1.br01.gva252.ip-max.net 39.794 ms 34.197 ms 28.843 ms 9 r1gva1.core.init7.net 28.849 ms 32.812 ms 31.988 ms 10 r1zug1.core.init7.net 47.583 ms 42.633 ms 37.785 ms 11 r1glb1.core.init7.net 36.323 ms 56.574 ms 34.909 ms 12 r1zrh1.core.init7.net 50.896 ms 56.617 ms 42.967 ms 13 ae0-12.zur11.ip6.tinet.net 42.462 ms 41.909 ms 41.557 ms 14 xe-0-1-1.nyc20.ip6.tinet.net 128.395 ms xe-0-1-2.nyc20.ip6.tinet.net 127.736 ms xe-0-1-1.nyc20.ip6.tinet.net 120.968 ms 15 nyiix.iij.net 121.285 ms 121.758 ms 120.904 ms 16 nyc002bb10.iij.net 121.436 ms 122.211 ms 120.475 ms 17 abn001bb10.iij.net 121.829 ms 123.181 ms 121.787 ms 18 2600:803:92f::1 143.860 ms 125.196 ms 179.323 ms 19 2600:802::23 139.937 ms 132.728 ms 140.416 ms 20 ge-1-2-0.il1.nyc12.alter.net 130.477 ms ge-0-3-0.il1.nyc12.alter.net 130.349 ms 140.231 ms 21 * * * 22 gigabitethernet6-0-0.gw5.ams6.alter.net 349.594 ms 129.648 ms gigabitethernet7-0-0.gw5.ams6.alter.net 130.583 ms 23 fastethernet0-0.6b1.ams6.alter.net 138.677 ms 169.896 ms 402.511 ms 24 tu3.6b1.fft4.alter.net 379.628 ms 156.939 ms 138.227 ms 25 * * * 26 ffm-s1-rou-1102.de.eurorings.net 219.035 ms 227.354 ms 236.133 ms 27 kehl-s2-rou-1103.de.eurorings.net 228.896 ms 221.190 ms 239.280 ms 28 nntr-s1-rou-1101.fr.eurorings.net 219.410 ms 226.779 ms 230.940 ms 29 2001:680::134:222:226:160:1 247.598 ms 243.938 ms 218.783 ms 30 2001:1890:c00:c001::1171:ecca 217.493 ms 242.119 ms 217.778 ms 31 2001:1890:c00:c001::ee71:ecca 238.297 ms 219.211 ms 218.524 ms 32 2001:1890:ff:ffff:12:122:105:90 257.048 ms 250.699 ms 252.160 ms 33 wswdc22crs.ipv6.att.net 248.764 ms 268.970 ms 416.443 ms 34 * attga21crs.ipv6.att.net 239.030 ms 241.492 ms 35 attga409me3.ipv6.att.net 249.902 ms 268.800 ms 253.631 ms 36 2001:1890:c00:8802::11cc:6c6d 260.203 ms 236.484 ms 245.413 ms From 2001:1b60::/32 (Germany, Native) 2 2001:1b60:0:1:0:1:0:1 0.275 ms 0.247 ms 0.237 ms 3 Pironet.FRA-12-eth012-615.v6.lambdanet.net 3.199 ms 3.198 ms 4.578 ms 4 ae5.irt1.fra44.de.v6.as13237.net 6.518 ms 6.499 ms 6.507 ms 5 xe-0-3-0-0.fra21.ip6.tinet.net 6.570 ms 6.563 ms 6.543 ms 6 xe-0-1-2.nyc20.ip6.tinet.net 86.740 ms xe-0-0-2.nyc20.ip6.tinet.net 86.833 ms xe-0-2-1.nyc20.ip6.tinet.net 86.715 ms 7 NYIIX.IIJ.Net 88.196 ms 88.046 ms 88.016 ms 8 nyc002bb10.iij.net 85.712 ms 85.844 ms 85.978 ms 9 abn001bb10.iij.net 92.920 ms 93.113 ms 92.785 ms 10 2600:803:92f::1 93.331 ms 92.965 ms 92.713 ms 11 2600:802::23 94.009 ms 94.048 ms 100.621 ms 12 ge-0-3-0.IL1.NYC12.ALTER.NET 99.927 ms 99.896 ms ge-1-2-0.IL1.NYC12.ALTER.NET 99.866 ms 13 ae1.XT2.AMS2.ALTER.NET 97.016 ms ae1.XT1.AMS2.ALTER.NET 96.994 ms 96.919 ms 14 GigabitEthernet7-0-0.GW5.AMS6.ALTER.NET 97.120 ms 98.054 ms GigabitEthernet6-0-0.GW5.AMS6.ALTER.NET 97.111 ms 15 FastEthernet0-0.6B1.AMS6.ALTER.NET 104.272 ms 106.899 ms 106.249 ms 16 tu3.6B1.FFT4.ALTER.NET 107.443 ms 100.970 ms 98.980 ms 17 * * ffm-s1-rou-1006.DE.eurorings.net 99.641 ms 18 ffm-s1-rou-1102.DE.eurorings.net 190.998 ms 183.395 ms 190.432 ms 19 kehl-s2-rou-1103.DE.eurorings.net 190.216 ms 203.128 ms 192.594 ms 20 nntr-s1-rou-1101.FR.eurorings.net 183.713 ms 193.837 ms 202.330 ms 21 2001:680:0:134:222:226:164:1 209.937 ms 182.641 ms 182.782 ms 22 2001:1890:c00:c001::1171:ecca 188.257 ms 183.682 ms 216.084 ms 23 2001:1890:c00:c001::ee71:ecca 205.741 ms 183.258 ms 204.891 ms 24 2001:1890:ff:ffff:12:122:105:90 247.476 ms 242.229 ms 251.679 ms 25 wswdc22crs.ipv6.att.net 237.167 ms 216.849 ms 228.173 ms 26 attga21crs.ipv6.att.net 208.246 ms 230.028 ms 232.396 ms 27 attga409me3.ipv6.att.net 255.300 ms 251.860 ms 249.848 ms 28 2001:1890:c00:8802::11cc:6c6d 233.211 ms 215.398 ms 206.866 ms

4 4

What to expect after a cooling failure
by Erik Levinson 10 Jul '13

10 Jul '13

As some may know, yesterday 151 Front St suffered a cooling failure after Enwave's facilities were flooded. One of the suites that we're in recovered quickly but the other took much longer and some of our gear shutdown automatically due to overheating. We shut down remotely many redundant and non-essential systems in the hotter suite, and transferred remotely some others to the cooler suite, to ensure that we had a minimum of all core systems running in the hotter suite. We waited until the temperatures returned to normal, and brought everything back online. The entire event lasted from approx 18:45 until 01:15. Apparently ambient temperature was above 43 degrees Celcius at one point on the cool side of cabinets in the hotter suite. For those who have gone through such events in the past, what can one expect in terms of long-term impact...should we expect some premature component failures? Does anyone have any stats to share? Thanks -- Erik Levinson CTO, Uberflip 416-900-3830 1183 King Street West, Suite 100 Toronto ON M6K 3C5 www.uberflip.com

11 12

Re: What to expect after a cooling failure
by Johnny Eriksson 10 Jul '13

10 Jul '13

Jake Khuon <khuon(a)NEEBU.Net> wrote: > While others have already talked about what to look out for in terms of > systems and drives, I haven't seen anyone mention things like your UPS > batteries. Were they also heat-soaked? At one place I worked at, we > lost a whole bank of batteries in the UPS room when it overheated. I > think that was somewhere around a $95,000 replacement and required > rush-delivery of a lot of SLAs from all over the place. That is one reason to have the UPS and the batteries in separate rooms. --Johnny

1 0

Re: What to expect after a cooling failure
by Tri Tran 10 Jul '13

10 Jul '13

I have seen DDR2 RAM give random errors from inadequate cooling. The cabinets were stacked to the max with severs but the doors were not meshed. DDR2 run fairly hot, especially when all the banks are filled. Tri Tran -----Original Message----- From: Jay Ashworth <jra(a)baylink.com> Date: Wed, 10 Jul 2013 00:04:23 To: NANOG<nanog(a)nanog.org> Subject: Re: What to expect after a cooling failure ----- Original Message ----- > From: "Erik Levinson" <erik.levinson(a)uberflip.com> > For those who have gone through such events in the past, what can one > expect in terms of long-term impact...should we expect some premature > component failures? Does anyone have any stats to share? If the HDDs were spinning while above rated maximum ambient intake temp, *especially* if they're not *right out front in the intake path* (is anything not built that way anymore? Yeah; the back side of 45-drive Supermicro racks, among other things), you should probably plan on doing a preemptive replacement cycle, or at the very least, pay *very* close attention to smartctld, and have a good stock of pre-trayed replacements. Remember that you may fall in the RAID Hole if you wait for failures, and hence lose data which isn't backed up anyway -- if more drives in a raid group fail *during rebuilds*, you're essentially screwed. If your raid groups were properly dispersed across drive build dates, then this will probably be *slightly* less dangerous, but still. Also watch bearing-type fans. Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

1 0

Anyone from frontiernet.net on here?
by Janet Sullivan 10 Jul '13

10 Jul '13

I've been seeing really bad packet loss between PCCW and frontier, and so far haven't been able to make any traction with anyone on either side. I'm betting that the ??? is a peering point either in London or Ashburn. uk.bgp4.net (0.0.0.0) Tue Jul 9 20:39:53 2013 Keys: Help Display mode Restart statistics Order of fields qu it Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 212.111.33.230 0.0% 43 1.7 0.7 0.5 1.8 0.3 2. 212.111.33.237 0.0% 43 2.3 1.9 1.1 22.7 3.3 3. 63.218.13.221 0.0% 43 2.3 15.6 1.1 230.3 45.9 4. ??? 5. 74.40.2.173 23.3% 43 177.9 150.3 147.8 177.9 7.0 6. 74.40.2.193 20.9% 43 149.9 149.9 149.1 161.2 2.1 7. 74.40.3.241 18.6% 43 149.6 152.4 149.1 193.2 8.8 8. 74.40.5.49 28.6% 43 148.1 150.8 147.8 192.1 9.9 9. 74.40.5.54 26.2% 43 148.3 150.5 147.9 218.7 12.6 10. 74.40.5.46 33.3% 42 149.5 154.0 149.2 212.8 14.0 11. 74.40.3.137 16.7% 42 147.4 148.7 146.9 163.0 4.1 12. 74.40.1.154 29.3% 42 148.2 153.6 147.7 206.7 14.2 13. 50.34.2.162 35.7% 42 150.2 150.4 149.8 156.3 1.2 14. 50.46.150.55 26.2% 42 150.7 151.0 150.5 152.0 0.4

3 3

Re: What to expect after a cooling failure
by Larry Sheldon 10 Jul '13

10 Jul '13

On 7/9/2013 10:28 PM, Erik Levinson wrote: > As some may know, yesterday 151 Front St suffered a cooling failure > after Enwave's facilities were flooded. > > One of the suites that we're in recovered quickly but the other took > much longer and some of our gear shutdown automatically due to > overheating. We shut down remotely many redundant and non-essential > systems in the hotter suite, and transferred remotely some others to > the cooler suite, to ensure that we had a minimum of all core systems > running in the hotter suite. We waited until the temperatures > returned to normal, and brought everything back online. The entire > event lasted from approx 18:45 until 01:15. Apparently ambient > temperature was above 43 degrees Celcius at one point on the cool > side of cabinets in the hotter suite. > > For those who have gone through such events in the past, what can one > expect in terms of long-term impact...should we expect some premature > component failures? Does anyone have any stats to share? No stats, but way back in the day of very large computers (1 each) in very large facilities, it seems like the thing we worried most about at restart was too-rapid cooling and the resulting condensation if the conditions were right. After power-up the next thing was disk crashes that occurred on the way down (this was a long time ago discs and drums are different now). Lastly was overheat failures which were relatively few and always in components with a weakness reputation. -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. (Adapted from Stephen Pinker)

1 0

Google's QUIC
by Michael Thomas 09 Jul '13

09 Jul '13

http://arstechnica.com/information-technology/2013/06/google-making-the-web… Sorry if this is a little more on the dev side, and less on the ops side but since it's Google, it will almost certainly affect the ops side eventually. My first reaction to this was why not SCTP, but apparently they think that middle boxen/firewalls make it problematic. That may be, but UDP based port filtering is probably not far behind on the flaky front. The second justification was TLS layering inefficiencies. That definitely has my sympathies as TLS (especially cert exchange) is bloated and the way that it was grafted onto TCP wasn't exactly the most elegant. Interestingly enough, their main justification wasn't a security concern so much as "helpful" middle boxen getting their filthy mitts on the traffic and screwing it up. The last thing that occurs to me reading their FAQ is that they are seemingly trying to send data with 0 round trips. That is, SYN, data, data, data... That really makes me wonder about security/dos considerations. As in, it sounds too good to be true. But maybe that's just the security cruft? But what about SYN cookies/dos? Hmmm. Other comments or clue? Mike

20 36

Comcast using private space and trying to use pool.ntp.org
by Mike McLaughlin 09 Jul '13

09 Jul '13

This is a friendly note to any Comcast net admins who might be interested: We have a Business Fiber circuit and run a public NTP server in the pool. It seems Comcast is using private IP space (mostly 10.29.0.0/16 and 10.183.0.0/16) for equipment and are trying to use the NTP pool for time services (without NAT). Clearly that isn't working out so hot - my firewall logs are chalk full of requests from those two networks, so I'd assume it's a pretty large number of misconfigured devices since I'm one of many many pool servers - and it's fairly safe to assume they're Comcast devices since they aren't getting eaten before hitting my firewall from Cocmast. Just for laughs, I suspect, Comcast blocks bogons at my gateway, so I couldn't even let them get time from me if I wanted to. Clearly not a critical (to me) issue, but I figured it was worth passing along since I count on Comcast for interwebs and it's likely Comcast could see time sync issues related to this. Sincerely, Mike McLaughlin

1 0