- Test - lists.nanog.org

v6 cdn problems
by Pete Carah 10 Nov '14

10 Nov '14

Prefix this - I'm on fios in the Baltimore area, using a HE tunnel terminating in ashburn. (*still* no native v6 on fios :-( Speedtest shows little or no congestion, and didn't change significantly when I reduced mtu by 8. (interestingly, speedtest.net usually reads faster than verizon's internal speedtest, and rarely averages less than my billed speed.) I've recently had problems (started a few weeks ago with www.att.com, 4-5 days ago with *.google.com) which unfortunately happy eyeballs doesn't help. att.com uses akamai, google uses their own cdn (per dns; I don't know if there are any connections behind the scenes.) This occurs on several google sites, all of which resolve to the same netblocks from here (maps.google.com, www.google.com, maps.gstatic.com, and at least one of the ad servers). Symptom with akamai is that it connects immediately then data transfer times out. With google, symptom involves both slow connection, and data transfer timing out. I don't know if chrome's happy eyeballs is working since if it was, and absent address caching, I shouldn't see the slow connection. v6 connections to my hosts in Los Angeles (not on HE address space, but we do peer with them on any2) work fine transferring graphics and large database files both ways, so I'm pretty sure I don't have an mtu problem nor some other fios or HE problem. Just to be sure, I dropped the 1500 to 1492 on the fios link and did the same adjustment to the mtu on my tunnel (becomes 1472). No change on my hosts. att.com appears a little better, though still very slow. Google shows no change at all. I saw some of the same problem yesterday from Frederick on comcast (only to google, didn't look at att), but couldn't take the time to do traceroutes. If desired, I'm likely to go out there tomorrow and can do that too. (we use a freebsd+pf router there). Is this a provisioning problem where v6 eyeballs are outstripping cdn provisioning (since win7 and 8 both default to v6)? Or is something else going on. Since this seems to affect more than one cdn, it seems odd. I run my own resolver locally instead of using verizon's. (and my own (vyatta) router instead of theirs. Actually I'm still using theirs as a bridge to talk to the set-top box (I don't know if Motorola still makes the coax terminal that would bridge it directly...) Resolve and traceroutes of relevant sites: [pete@port5 ~]$ host www.att.com www.att.com is an alias for prod-www.zr-att.com.akadns.net. prod-www.zr-att.com.akadns.net is an alias for www.att.com.edgekey.net. www.att.com.edgekey.net is an alias for e2318.dscb.akamaiedge.net. e2318.dscb.akamaiedge.net has address 23.76.217.145 e2318.dscb.akamaiedge.net has IPv6 address 2600:807:320:202:9200::90e e2318.dscb.akamaiedge.net has IPv6 address 2600:807:320:202:8600::90e Traceroute (v4) to this shows it is in Newark or environs: [pete@port5 ~]$ traceroute e2318.dscb.akamaiedge.net traceroute to e2318.dscb.akamaiedge.net (23.76.217.145), 30 hops max, 60 byte packets 1 rtr.east.altadena.net (192.168.170.1) 2.008 ms 2.450 ms 3.091 ms 2 L300.BLTMMD-VFTTP-64.verizon-gni.net (108.3.150.1) 9.021 ms 9.054 ms 9.045 ms 3 G0-7-4-5.BLTMMD-LCR-21.verizon-gni.net (100.41.195.94) 10.670 ms 10.683 ms 10.677 ms 4 ae4-0.RES-BB-RTR2.verizon-gni.net (130.81.209.230) 9.002 ms ae20-0.RES-BB-RTR1.verizon-gni.net (130.81.151.112) 8.995 ms so-1-1-0-0.RES-BB-RTR1.verizon-gni.net (130.81.199.2) 8.953 ms 5 * * * 6 * * * 7 0.xe-5-0-4.XL3.EWR6.ALTER.NET (140.222.225.73) 51.202 ms 41.102 ms 40.345 ms 8 0.ae1.XL4.EWR6.ALTER.NET (140.222.228.41) 33.065 ms TenGigE0-6-0-3.GW8.EWR6.ALTER.NET (152.63.19.158) 11.550 ms TenGigE0-6-0-6.GW8.EWR6.ALTER.NET (152.63.25.10) 11.659 ms 9 TenGigE0-7-0-1.GW8.EWR6.ALTER.NET (152.63.19.166) 19.854 ms akamai-gw.customer.alter.net (152.179.185.126) 1766.402 ms TenGigE0-7-0-7.GW8.EWR6.ALTER.NET (152.63.25.30) 18.227 ms 10 akamai-gw.customer.alter.net (152.179.185.126) 1747.269 ms a23-76-217-145.deploy.static.akamaitechnologies.com (23.76.217.145) 10.672 ms 11.263 ms Traceroute6 to it appears to be local (but is hard to tell. Next-to-last hop looks like either a router or load-balancer is overloaded. Same for the v4 traceroute... [pete@port5 ~]$ traceroute6 www.att.com traceroute to www.att.com (2600:807:320:202:9200::90e), 30 hops max, 80 byte packets 1 rtr.east.altadena.net (2001:470:e160:1::1) 5.182 ms 5.274 ms 5.254 ms 2 altadenamd-1.tunnel.tserv13.ash1.ipv6.he.net (2001:470:7:126::1) 11.452 ms 15.040 ms 18.592 ms 3 ge4-12.core1.ash1.he.net (2001:470:0:90::1) 20.273 ms 20.574 ms 20.567 ms 4 eqx.br6.iad8.verizonbusiness.com (2001:504:0:2::701:1) 20.522 ms 20.232 ms 20.475 ms 5 * * * 6 2600:802:44f::2 (2600:802:44f::2) 1283.113 ms 1296.115 ms 1296.181 ms 7 2600:807:320:200::1743:f397 (2600:807:320:200::1743:f397) 20.181 ms 16.169 ms 14.073 ms [pete@port5 ~]$ host www.google.com www.google.com has address 74.125.228.16 www.google.com has address 74.125.228.20 www.google.com has address 74.125.228.17 www.google.com has address 74.125.228.19 www.google.com has address 74.125.228.18 www.google.com has IPv6 address 2607:f8b0:4004:800::1012 Traceroute (v4) to this shows something odd, but I don't know where "burl" is for verizon. Also I appear to hit two nodes for the terminal. At least one google node appears to be ashburn (or environs) : [pete@port5 ~]$ traceroute www.google.com traceroute to www.google.com (74.125.228.20), 30 hops max, 60 byte packets 1 rtr.east.altadena.net (192.168.170.1) 2.646 ms 2.816 ms 3.536 ms 2 L300.BLTMMD-VFTTP-64.verizon-gni.net (108.3.150.1) 4.109 ms 4.194 ms 4.186 ms 3 G0-7-4-4.BLTMMD-LCR-22.verizon-gni.net (130.81.170.84) 7.928 ms 8.096 ms 8.088 ms 4 ae20-0.PHIL-BB-RTR1.verizon-gni.net (130.81.151.120) 10.881 ms ae20-0.PHIL-BB-RTR2.verizon-gni.net (130.81.151.124) 11.074 ms so-6-1-0-0.PHIL-BB-RTR2.verizon-gni.net (130.81.199.4) 11.047 ms 5 0.xe-7-0-2.XL2.IAD8.ALTER.NET (152.63.4.93) 14.872 ms 0.xe-3-0-1.XL3.IAD8.ALTER.NET (152.63.3.61) 37.703 ms 12.268 ms 6 0.xe-9-2-0.GW9.IAD8.ALTER.NET (152.63.36.30) 12.866 ms 0.xe-11-2-1.GW9.IAD8.ALTER.NET (152.63.42.2) 14.442 ms 0.xe-9-2-0.GW9.IAD8.ALTER.NET (152.63.36.30) 11.918 ms 7 * 0.xe-10-1-1.GW9.IAD8.ALTER.NET (152.63.35.113) 16.901 ms pool-96-236-104-66.burl.east.verizon.net (96.236.104.66) 136.110 ms 8 pool-96-236-104-66.burl.east.verizon.net (96.236.104.66) 137.977 ms 216.239.46.248 (216.239.46.248) 13.875 ms pool-96-236-104-66.burl.east.verizon.net (96.236.104.66) 134.602 ms 9 216.239.46.248 (216.239.46.248) 15.918 ms 10.708 ms 10.162 ms 10 72.14.238.173 (72.14.238.173) 11.347 ms 12.111 ms iad23s05-in-f20.1e100.net (74.125.228.20) 12.769 ms Corresponding traceroute6 (shows lack of reverse on most hits...): [pete@port5 ~]$ traceroute6 www.google.com traceroute to www.google.com (2607:f8b0:4004:800::1011), 30 hops max, 80 byte packets 1 rtr.east.altadena.net (2001:470:e160:1::1) 1.640 ms 1.811 ms 1.801 ms 2 altadenamd-1.tunnel.tserv13.ash1.ipv6.he.net (2001:470:7:126::1) 11.977 ms 15.279 ms 19.265 ms 3 ge4-12.core1.ash1.he.net (2001:470:0:90::1) 19.779 ms 20.776 ms 22.303 ms 4 2001:4860:1:1:0:1b1b:0:d (2001:4860:1:1:0:1b1b:0:d) 22.267 ms 22.514 ms 22.507 ms 5 2001:4860::1:0:9ff (2001:4860::1:0:9ff) 22.508 ms 22.471 ms 22.455 ms 6 2001:4860:0:1::551 (2001:4860:0:1::551) 22.467 ms 19.139 ms 19.116 ms 7 2607:f8b0:8000:18::c (2607:f8b0:8000:18::c) 19.054 ms 2607:f8b0:8000:18::f (2607:f8b0:8000:18::f) 7.716 ms 2607:f8b0:4004:800::1b (2607:f8b0:4004:800::1b) 8.379 ms Again shows multiple terminals for the given address. Ping works fine to all of the addresses, both v4 and v6, and the att one always connects immediately. The google one doesn't always. When I disable the HE tunnel, (and restart the browser - apparently happy-eyeballs caches), everything works just fine, so the problem does appear to relate to v6. For reference, I mostly use chrome in linux. My daughter sees the same problem with google, mostly using chrome in win 7. I see the problem with firefox (in linux) also (to both sites). -- Pete

6 10

Re: I am about to inherit 26 miles of dark fiber
by Scott Weeks 10 Nov '14

10 Nov '14

:: Ah, the famous good-will of NANOG. But you got more of the good than the other. :: I knew I would get some interesting responses. And you got more of that than non-interesting... :-) scott

1 0

Re: I am about to inherit 26 miles of dark fiber
by Lorell Hathcock 10 Nov '14

10 Nov '14

Ah, the famous good-will of NANOG. I knew I would get some interesting responses. I was part of the Field Ops group of Enron Broadband years ago. We deployed DWDM extensively. Admittedly it has been a while. This 26 miles of dark fiber is deployed by a municipality in and around their fair city. Part of a deal with this new firm is that the firm will use the aforementioned six strands. So the fiber is deployed throughout this city that has been largely under-serviced. By lack of resources, the city could not deploy services to businesses/enterprises. So as I ponder the opportunity, I seek to tap the creative juices of NANOG. Thanks, Lorell Hathcock

1 0

FW: M-Lab-Related PCAPs
by Livingood, Jason 10 Nov '14

10 Nov '14

FYI to this list since I suspect few of you are on the M-Lab Discuss list. Srikanth from ICSI has kindly taken on consolidating some PCAPs. If anyone wishes to send any to him, he is at srknth.s(a)gmail.com<mailto:srknth.s@gmail.com>. JL On 11/6/14, 7:24 PM, "Srikanth S" <srknth.s(a)gmail.com<mailto:srknth.s@gmail.com>> wrote: So it looks as though marking is not done for all MLab traffic. Also, some web traffic (to CNN) is marked at a lower priority than streaming (Netflix), which is strange as web traffic is likely more sensitive to degradation than streaming (?). Here are the traces: http://www1.icsi.berkeley.edu/~srikanth/pcaps/google.pcap http://www1.icsi.berkeley.edu/~srikanth/pcaps/youtube-image.pcap http://www1.icsi.berkeley.edu/~srikanth/pcaps/cnn.pcap http://www1.icsi.berkeley.edu/~srikanth/pcaps/netflix-streaming.pcap On Tuesday, November 4, 2014 1:29:16 PM UTC-8, Jason Livingood wrote: Another follow-up. Someone emailed me a PCAP off-list from an enterprise type of customer. Their PCAP was somewhat incomplete (so I still need more) but they noticed that some traffic at the next priority down from 0x48 at 0x28 (priority). And some other traffic was marked with the next priority down again at 0x00 (routine). So it appears there are three DSCP / ToS markings in use rather than just two (0x00, 0x28, 0x48). So safe to say more research is needed here – anyone collecting PCAPs should IMHO continue. :-) Jason

2 1

Re: Reporting DDOS reflection attacks
by Larry Sheldon 10 Nov '14

10 Nov '14

On 11/9/2014 13:40, Doug Barton wrote: > On 11/8/14 6:33 PM, Roland Dobbins wrote: >> this is incorrect and harmful, and should be removed: >> >> iii. Consider dropping any DNS reply packets which are larger >> than 512 Bytes – these are commonly found in DNS DoS Amplification >> attacks. >> >> This *breaks the Internet*. Don't do it. > > +1 > The whole thing> Really? -- The unique Characteristics of System Administrators: The fact that they are infallible; and, The fact that they learn from their mistakes. Quis custodiet ipsos custodes

2 1

Reporting DDOS reflection attacks
by srn.nanog＠prgmr.com 09 Nov '14

09 Nov '14

Like most small providers, we occasionally get hit by DoS attacks. We got hammered by an SSDP reflection attack (udp port 1900) last week. We took a 27 second log and from there extracted about 160k unique IPs. It is really difficult to find abuse emails for 160k IPs. We know about abuse.net but abuse.net requires hostnames, not IPs for lookups and not all IP addresses have valid DNS entries. The only other way we know of to report problems is to grab the abuse email addresses is whois. However, whois is not structured and is not set up to deal with this number of requests - even caching whois data based on subnets will result in many thousands of lookups. Long term it seems like structured data and some kind of authentication would be ideal for reporting attacks. But right now how should we be doing it?

12 16

DDOS, IDS, RTBH, and Rate limiting
by Eric C. Miller 09 Nov '14

09 Nov '14

Today, we experienced (3) separate DDoS attacks from Eastern Asia, all generating > 2Gbps towards a single IP address in our network. All 3 attacks targeted different IP addresses with dst UDP 19, and the attacks lasted for about 5 minutes and stopped as fast as they started. Does anyone have any suggestions for mitigating these type of attacks? A couple of things that we've done already... We set up BGP communities with our upstreams, and tested that RTBH can be set and it does work. However, by the time that we are able to trigger the black hole, the attack is almost always over. For now, we've blocked UDP 19 incoming at our edge, so that if future, similar attacks occur, it doesn't affect our internal links. What I think that I need is an IDS that can watch our edge traffic and automatically trigger a block hole advertisement for any internal IP beginning to receive > 100Mbps of traffic. A few searches are initially coming up dry... Eric Miller, CCNP Network Engineering Consultant (407) 257-5115

12 21

Re: Clueful Jive Communications Contact?
by Bryan Fields 08 Nov '14

08 Nov '14

On 11/7/14, 6:24 PM, chris wrote: > I have a very frustrated mutual customer we provide network services to who > utilizes Jive for their voice and we can see that there is intermittent > reachability problems and all attempts to go through normal support with > the information we have provided are going nowhere. Where's June Cleaver when you need her.. -- Bryan Fields 727-409-1194 - Voice 727-214-2508 - Fax http://bryanfields.net

1 0

NYC logistics issue
by Andy Brezinsky 08 Nov '14

08 Nov '14

Due to a comedy of errors, I have a pallet of gear sitting in Elizabeth NJ that needs to get to the 9th floor of 60 Hudson by EOD Monday that can't seem to get delivered due to no company having a COI on file with the building. We did have UPS freight lined up to pick it up from the original shipper tonight and deliver it on Monday but there was an "issue" and the driver left without the pallet. Does have any contacts at courier companies that could possibly break down 600lbs of gear (48"x48"x44" pallet with 14 boxes ranging from 10 to 75lbs) at the warehouse and deliver it to the building? Any other suggestions for how to deal with this problem? Replies off-list are fine. Thanks in advance. -- Andy Brezinsky

1 0

BGP Update Report
by cidr-report＠potaroo.net 07 Nov '14

07 Nov '14

BGP Update Report Interval: 30-Oct-14 -to- 06-Nov-14 (7 days) Observation Point: BGP Peering with AS131072 TOP 20 Unstable Origin AS Rank ASN Upds % Upds/Pfx AS-Name 1 - AS14287 399233 9.1% 66538.8 -- TRIAD-TELECOM - Triad Telecom, Inc.,US 2 - AS23752 198523 4.5% 1965.6 -- NPTELECOM-NP-AS Nepal Telecommunications Corporation, Internet Services,NP 3 - AS7545 172069 3.9% 80.5 -- TPG-INTERNET-AP TPG Telecom Limited,AU 4 - AS9829 161433 3.7% 145.4 -- BSNL-NIB National Internet Backbone,IN 5 - AS12897 143079 3.2% 4471.2 -- HEAGMEDIANET HSE Medianet GmbH,DE 6 - AS3816 49561 1.1% 96.6 -- COLOMBIA TELECOMUNICACIONES S.A. ESP,CO 7 - AS27738 32552 0.7% 41.7 -- Ecuadortelecom S.A.,EC 8 - AS3313 32485 0.7% 10828.3 -- INET-AS BT Italia S.p.A.,IT 9 - AS9808 30902 0.7% 26.5 -- CMNET-GD Guangdong Mobile Communication Co.Ltd.,CN 10 - AS8402 28525 0.7% 53.6 -- CORBINA-AS OJSC "Vimpelcom",RU 11 - AS6713 27862 0.6% 44.4 -- IAM-AS,MA 12 - AS48159 27319 0.6% 98.6 -- TIC-AS Telecommunication Infrastructure Company,IR 13 - AS28573 25645 0.6% 17.6 -- NET Serviços de Comunicação S.A.,BR 14 - AS45899 25081 0.6% 57.0 -- VNPT-AS-VN VNPT Corp,VN 15 - AS23342 22220 0.5% 22220.0 -- UNITEDLAYER - Unitedlayer, Inc.,US 16 - AS10620 20754 0.5% 8.4 -- Telmex Colombia S.A.,CO 17 - AS25003 19893 0.5% 1989.3 -- INTERNET_BINAT Internet Binat Ltd,IL 18 - AS4134 19833 0.5% 126.3 -- CHINANET-BACKBONE No.31,Jin-rong Street,CN 19 - AS13489 19479 0.4% 46.9 -- EPM Telecomunicaciones S.A. E.S.P.,CO 20 - AS22773 19470 0.4% 4.5 -- ASN-CXA-ALL-CCI-22773-RDC - Cox Communications Inc.,US TOP 20 Unstable Origin AS (Updates per announced prefix) Rank ASN Upds % Upds/Pfx AS-Name 1 - AS14287 399233 9.1% 66538.8 -- TRIAD-TELECOM - Triad Telecom, Inc.,US 2 - AS23342 22220 0.5% 22220.0 -- UNITEDLAYER - Unitedlayer, Inc.,US 3 - AS61039 16767 0.4% 16767.0 -- ZMZ OAO ZMZ,RU 4 - AS18135 12424 0.3% 12424.0 -- BTV BTV Cable television,JP 5 - AS3313 32485 0.7% 10828.3 -- INET-AS BT Italia S.p.A.,IT 6 - AS47680 6477 0.1% 6477.0 -- NHCS EOBO Limited,IE 7 - AS3 18160 0.4% 1306.0 -- MIT-GATEWAYS - Massachusetts Institute of Technology,US 8 - AS62174 6033 0.1% 6033.0 -- INTERPAN-AS INTERPAN LTD.,BG 9 - AS24768 4849 0.1% 4849.0 -- ALMOUROLTEC ALMOUROLTEC SERVICOS DE INFORMATICA E INTERNET LDA,PT 10 - AS12897 143079 3.2% 4471.2 -- HEAGMEDIANET HSE Medianet GmbH,DE 11 - AS51314 4321 0.1% 4321.0 -- TEVIANT-AS LLC Joint Small Enterprise "Teviant",UA 12 - AS53823 3066 0.1% 3066.0 -- SMTA - Scio Mutual Telephone Association,US 13 - AS56636 2928 0.1% 2928.0 -- ASVEDARU VEDA Ltd.,RU 14 - AS60725 16553 0.4% 2758.8 -- O3B-AS O3b Limited,JE 15 - AS35093 4473 0.1% 2236.5 -- RO-HTPASSPORT High Tech Passport Ltd SUA California San Jose SUCURSALA BUCURESTI ROMANIA,RO 16 - AS25003 19893 0.5% 1989.3 -- INTERNET_BINAT Internet Binat Ltd,IL 17 - AS23752 198523 4.5% 1965.6 -- NPTELECOM-NP-AS Nepal Telecommunications Corporation, Internet Services,NP 18 - AS61051 1834 0.0% 1834.0 -- DEKARD Dekard LLC,RU 19 - AS3 1716 0.0% 5517.0 -- MIT-GATEWAYS - Massachusetts Institute of Technology,US 20 - AS47805 3355 0.1% 1677.5 -- MCIT Ministry of Communications and Information,SA TOP 20 Unstable Prefixes Rank Prefix Upds % Origin AS -- AS Name 1 - 202.70.88.0/21 99367 2.2% AS23752 -- NPTELECOM-NP-AS Nepal Telecommunications Corporation, Internet Services,NP 2 - 202.70.64.0/21 98016 2.1% AS23752 -- NPTELECOM-NP-AS Nepal Telecommunications Corporation, Internet Services,NP 3 - 208.70.20.0/22 79857 1.7% AS14287 -- TRIAD-TELECOM - Triad Telecom, Inc.,US 4 - 208.78.116.0/22 79845 1.7% AS14287 -- TRIAD-TELECOM - Triad Telecom, Inc.,US 5 - 208.73.244.0/22 79843 1.7% AS14287 -- TRIAD-TELECOM - Triad Telecom, Inc.,US 6 - 208.88.232.0/22 79841 1.7% AS14287 -- TRIAD-TELECOM - Triad Telecom, Inc.,US 7 - 216.162.0.0/20 79841 1.7% AS14287 -- TRIAD-TELECOM - Triad Telecom, Inc.,US 8 - 64.29.130.0/24 22220 0.5% AS23342 -- UNITEDLAYER - Unitedlayer, Inc.,US 9 - 94.16.80.0/20 20461 0.4% AS12897 -- HEAGMEDIANET HSE Medianet GmbH,DE 10 - 185.9.28.0/22 20434 0.4% AS12897 -- HEAGMEDIANET HSE Medianet GmbH,DE 11 - 94.16.72.0/21 20433 0.4% AS12897 -- HEAGMEDIANET HSE Medianet GmbH,DE 12 - 94.16.64.0/21 20417 0.4% AS12897 -- HEAGMEDIANET HSE Medianet GmbH,DE 13 - 194.127.204.0/23 20402 0.4% AS12897 -- HEAGMEDIANET HSE Medianet GmbH,DE 14 - 194.99.108.0/23 20400 0.4% AS12897 -- HEAGMEDIANET HSE Medianet GmbH,DE 15 - 194.45.104.0/23 20368 0.4% AS12897 -- HEAGMEDIANET HSE Medianet GmbH,DE 16 - 192.115.44.0/22 19883 0.4% AS25003 -- INTERNET_BINAT Internet Binat Ltd,IL 17 - 130.0.192.0/21 18136 0.4% AS3 -- MIT-GATEWAYS - Massachusetts Institute of Technology,US 18 - 91.235.169.0/24 16767 0.4% AS61039 -- ZMZ OAO ZMZ,RU 19 - 71.19.134.0/23 16604 0.4% AS3313 -- INET-AS BT Italia S.p.A.,IT 20 - 194.153.99.0/24 15878 0.3% AS3313 -- INET-AS BT Italia S.p.A.,IT Details at http://bgpupdates.potaroo.net ------------------------------------ Copies of this report are mailed to: nanog(a)nanog.org eof-list(a)ripe.net apops(a)apops.net routing-wg(a)ripe.net afnog(a)afnog.org

1 0