September 2018 - Test - lists.nanog.org

QWEST you have broken DNS servers
by Mark Andrews 14 Sep '18

14 Sep '18

I know it takes some time to upgrade DNS servers to ones that are actually protocol compliant but 4+ years is ridiculous. Your servers are the only ones serving the Alexa top 1M sites or the GOV zone that still return BADVERS to EDNS queries with a EDNS option present. This was behaviour made up by your DNS vendor. The correct response to EDNS options that are not understood is to IGNORE them. This allows clients and servers to deploy support for new options independently of each other. Additionally this is breaking DNSSEC validation of the signed zones your clients have you serving. They expect you to be using EDNS compliant name servers for this role which you are not. No, we are not working around this breakage in the resolver. Mark % dig soa frc.gov. @208.44.130.121 +norec ; <<>> DiG 9.12.1 <<>> soa frc.gov. @208.44.130.121 +norec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: BADVERS, id: 59707 ;; flags: qr ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; Query time: 66 msec ;; SERVER: 208.44.130.121#53(208.44.130.121) ;; WHEN: Tue Sep 11 06:08:41 UTC 2018 ;; MSG SIZE rcvd: 23 % dig soa frc.gov. @208.44.130.121 +norec +nocookie ; <<>> DiG 9.12.1 <<>> soa frc.gov. @208.44.130.121 +norec +nocookie ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16876 ;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;frc.gov. IN SOA ;; ANSWER SECTION: frc.gov. 86400 IN SOA sauthns2.qwest.net. dns-admin.qwestip.net. 2180320527 10800 3600 604800 86400 ;; AUTHORITY SECTION: frc.gov. 86400 IN NS sauthns1.qwest.net. frc.gov. 86400 IN NS sauthns2.qwest.net. ;; Query time: 66 msec ;; SERVER: 208.44.130.121#53(208.44.130.121) ;; WHEN: Tue Sep 11 06:19:33 UTC 2018 ;; MSG SIZE rcvd: 145 % grep ednsopt=badvers reports/alexa1m.2018-08-26T00:00:06Z | grep edns=ok | awk '{print $3}' | sort -u (sauthns1.qwest.net.) (sauthns2.qwest.net.) % grep ednsopt=badvers reports-full/gov-full.2018-09-11T00:00:06Z | grep edns=ok | awk '{print $3}' | sort -u (sauthns1.qwest.net.) (sauthns2.qwest.net.) % -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka(a)isc.org

2 3

Database that netflix/hulu use to determine who is a proxy and who isnt?
by Drew Weaver 14 Sep '18

14 Sep '18

It seems like recently one of the sources for IP info that Netflix and Hulu uses was updated with erroneous information as access to both of the services was revoked pretty much at the same time. Does anyone know what source they use for that information so I can request that they fix some of their information? Thanks, -Drew

2 1

Re: Puerto Rico Internet Exchange
by Mel Beckman 14 Sep '18

14 Sep '18

Mike, But why would you want, as a content provider, to have your content hosted on the island? Backhauling it over fiber is no big deal across the short distances involved. As far as I can tell, PR has a glut of ocean floor fiber capacity, just installed a couple years ago. We're not talking stock market trades here, where milliseconds matter. We're talking Netflix movie reruns, which could be easily delivered with seconds of latency. Those who hold to the "if you build it they will come" business model forget that that model was a fantasy in a movie. A movie currently being streamed to PR without difficulty :-) -mel via cell On Sep 14, 2018, at 6:14 AM, Mike Hammett <nanog(a)ics-il.net<mailto:nanog@ics-il.net>> wrote: Agreed. Very chicken or the egg. Any recently formed IX is largely a conduit for big content to connect to local eyeballs. As some critical mass of eyeballs is achieved, local content is interested as are large networks like Hurricane Electric. In the case of PR, if there are no local content providers, an IX provides an avenue for one to form to connect to other operators on the island, avoiding underwater cables to the mainland. If I were a company in PR, I'd want my web site and other services hosted in PR, not Miami or Virginia. ----- Mike Hammett Intelligent Computing Solutions<http://www.ics-il.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL> Midwest Internet Exchange<http://www.midwest-ix.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix> The Brothers WISP<http://www.thebrotherswisp.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> ________________________________ From: "Sander Steffann" <sander(a)steffann.nl<mailto:sander@steffann.nl>> To: "Mel Beckman" <mel(a)beckman.org<mailto:mel@beckman.org>> Cc: "nanog" <nanog(a)nanog.org<mailto:nanog@nanog.org>> Sent: Friday, September 14, 2018 8:08:58 AM Subject: Re: Puerto Rico Internet Exchange Hi, > In general an IX only makes sense when there are local resources to exchange. It doesn’t seem like PR has a lot of, if any, content providers of its own, so most consumer content is coming from offshore anyway. This can also work the other way: once there is a local IXP, it can open opportunities for local content providers. Cheers, Sander

2 1

SAFNOG-4/EANOG/tzNOG - Just Under 2 Weeks To Go!
by Mark Tinka 13 Sep '18

13 Sep '18

Hi all. This is just a reminder that with a little under 2 weeks to go the upcoming SAFNOG-4/EANOG/tzNOG meeting to be held in Dar Es Salaam, 24th - 29th September, please remember to register your attendance at http://www.safnog.org/ so you don't miss the meeting. The agenda is now up on the web site at http://www.safnog.org/#agenda with just a few slots left which are filling fast. In case you feel like you have an interesting technical and operational talk that you'd like to share with the community at this meeting, please feel free to submit it for the program committee to review as soon as possible, at: https://papers.safnog.org/user/login.php?event=76 Key talks that you would be looking forward include: * Ben Roberts (Liquid Telecom), talking about the latency realities of intra-African Internet communication. * The 3 major data centre projects that are happening in South Africa, Kenya and Uganda. * The DNSSEC key rollover due on 11th October, and what to look forward to, as explained by Amreesh Phokeer. * An IPv6 Deployathon 2-day workshop headed by Mukom Akong Tamon. * ... and so many more. There will be 2 social events where delegates can network, share a drink, some good food, and stand a chance to win either an iPhone X or AppleTV 4K. We have secured a special discount rate at the hotel venue for all SAFNOG/EANOG/tzNOG participants. More details are available at http://www.safnog.org/#venue . We look forward to seeing you in Dar Es Salaam. Mark Tinka On Behalf of the SAFNOG-4/EANOG/tzNOG Organising Committee

1 0

OpenDNS CGNAT Issues
by Darin Steffl 12 Sep '18

12 Sep '18

Hello, I have a ticket open with OpenDNS about filtering happening on some of our CGNAT IP space where a customer has "claimed" the IP as theirs so other customers using that same IP and OpenDNS are being filtered and not able to access sites that fall under their chosen filter. I have a ticket open from 6 days ago but it's not going anywhere fast. Can someone from OpenDNS contact me or point me to a contact there to help get this resolved? I believe we need to claim our CGNAT IP space so residential users can't claim IP's of their own. Thank you! -- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi <http://www.facebook.com/minnesotawifi> Like us on Facebook <http://www.facebook.com/minnesotawifi>

14 21

Bell Canada IP transit - NID required?
by JASON BOTHE 11 Sep '18

11 Sep '18

All, Just curious if anyone uses Bell Canada for IP transit in a colo environment and whether or not you were able to obtain a traditional cross-connect without having Bell place a managed CPE device in your cage. I can’t seem to find anyone intelligent enough to tell my any different other than “it’s never been done before” and “we need remote testing capabilities.” J~

6 5

Microsoft your DNS servers are broken
by Mark Andrews 11 Sep '18

11 Sep '18

While we are talking about DNS server that are broken, Microsoft your servers are as well. As none of the zones you serve are DNSSEC signed there isn’t as much breakage possible but there are still interoperability problems and unnecessary additional traffic. It’s not like the EDNS specification is complicated. The microsoftonline servers will cause DNSSEC validation to fail if they ever serve a DNSSEC signed zone in this state. The FORMERR will cause EDNS servers to fallback to plain DNS and the validators won’t get the records they need. The azure servers cause problems for anyone deploying a new EDNS options as they have to cope with your servers incorrectly echoing back the option. Additionally if EDNS(1) is ever deployed there is a good chance that resolvers will assume the broken answers indicate that there is no data at the name. Mark cityofharrison-mi.gov. @207.46.15.59 (ns1.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @2a01:111:f406:1804::59 (ns1.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @191.232.83.138 (ns3.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @2a01:111:f406:b400::22 (ns3.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @157.56.81.41 (ns2.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @2a01:111:f406:3403::41 (ns2.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @13.107.24.7 (ns3-07.azure-dns.org.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @2a01:111:4000::7 (ns3-07.azure-dns.org.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @13.107.160.7 (ns4-07.azure-dns.info.): dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @2620:1ec:bda::7 (ns4-07.azure-dns.info.): dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @64.4.48.7 (ns2-07.azure-dns.net.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @2620:1ec:8ec::7 (ns2-07.azure-dns.net.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @40.90.4.7 (ns1-07.azure-dns.com.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @2603:1061::7 (ns1-07.azure-dns.com.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka(a)isc.org

2 1

Proofpoint Contact?
by Benoit Garcia 10 Sep '18

10 Sep '18

Hello, Could a Proofpoint e-mail blacklist contact me off list? I'm looking for some help to solve an undelivered email issue. Thanks in advance. -- Regards, Benoit.

2 1

Service provider story about tracking down TCP RSTs
by frnkblk＠iname.com 10 Sep '18

10 Sep '18

I want to share a little bit of our journey in tracking down the TCP RSTs that impacted some of our customers for almost ten weeks. Almost immediately after we turned up two new Arista border routers in late July we started receiving a trickle of complaints from customers regarding their inability to access certain websites (mostly B2B). All the packet captures showed the standard TCP SYN/SYN-ACK pair, then a TCP RST from the website after the client sent a TLS/SSL Client Hello. As the reports continued to come in, we built a Google Doc to keep track and it became clear that most of the sites were hosted by Incapsula/Imperva, but there were also a few by Sucuri and Fastly. Knowing that Incapsula provides DoS protection, we attempted to work with them (providing websites, source/destination IPs, traceroutes, and packet captures) to find out why their hosts were issuing our customers a TCP RST, but we made little progress. We moved some of the affected customers to different IP addresses but that didn't resolve the issue. We also asked our customer to work with the website to see if they would be willing to open a ticket with Incapsula. In the meantime, customers were getting frustrated! They couldn't visit Incapsula-hosted healthcare websites, financial firms, product dealers, etc. Over the weeks, a few of those customers purchased/borrowed different routers and some of those didn't have website issues anymore. And more than a few of them discovered that the websites worked fine from home or their mobile phone/hotspot, but not from their Internet connection with us. You can guess where they were applying pressure! That said, we didn't know why a small handful of companies, known for DoS protection, were issuing TCP RSTs to just some of our customers. Earlier this week we received four or five more websites from yet another affected customer, but most of those were with Fastly. By this time, we had been able to replicate the issue in our lab. Feeling desperate to make some tangible progress on this issue, I reached out to the Fastly NOC. In less than 12 hours they provided some helpful feedback, pointing out that a single traceroute to a Fastly site was hitting two of their POPs (they use anycast) and because they don't sync state between POPs the second POP would naturally issue a TCP RST (sidebar: fascinating blog article on Fastly's infrastructure here: https://www.fastly.com/blog/building-and-scaling-fastly-network-part-2-balan cing-requests). In subsequent email exchanges, the Fastly NOC suggested that it appeared that we were "spraying flows" (that is, packets related to single client session were egressing our network via different paths). Because Fastly is also present with us at an IX (though they weren't advertising their anycast IPs at the time), they suggested that we look at how our traffic egresses our network (IX versus transit) and our routers' outbound load-balancing/hashing schemes. The IX turned up to be a red herring, so I turned my attention to our transit. Each of our border routers has two BGP sessions over two circuits to transit provider POP A and two BGP sessions over two circuits to transit provider POP B, for a total of four BGP sessions per border router, a total of eight BGP sessions altogether. Starting with our core router, I confirmed that its ECMP hashing was consistent such that Fastly-bound traffic always went to border router 1 or border router 2. Then I looked at the ECMP hashing scheme on our border routers and noticed something unique - by default Arista also uses TTL: IPv4 hash fields: Source IPv4 Address is ON Protocol is ON Time-To-Live is ON Destination IPv4 Address is ON Since the source and destination IPs and protocol weren't changing, perhaps the TTL was not consistent? I opened the first packet trace in Wireshark and jackpot - the TTL value was 128 on the SYN but 127 on the TLS/SSL Client Hello. I adjusted the Arista's load-balancing profile not to use TTL and immediately my MTR in the background changed and all the sites on the lab machine that couldn't load before . were now loading. Fastly also pointed me to another article written by Joel Jaeggli (https://blog.apnic.net/2018/01/11/ipv6-flow-label-misuse-hashing/) that discusses IPv6 flow labels - we removed that from the border routers' IPv6 hash fields, too. I reviewed the packet traces today and noticed that TTL values remained consistent at 128 *behind* the router CPE. In packet captures on the WAN interface of the router CPE I see that the SYN remains at 128, but the TLS/Client Hello is properly decremented to 127. So, it appears that some router CPE (and there were a variety of makes and models) are doing something special to certain packets and not decrementing the TTL. This explains why: * our customers had issues with all their devices behind their router CPE * the issue remained regardless of what public IP address their router CPE obtained via DHCP or was assigned * some customers who changed their router CPE didn't have the issue anymore - they got lucky with a router that doesn't adjust/reset the TTL * why customers who used our managed Wi-Fi router did not see the issue, because that model doesn't apparently manipulate the TTL, at least not in an inconsistent way. Lesson learned: review a device's hashing mechanism before going into production. For those interested, I have links to the packet traces below my signature, showing the inconsistent TTL values. Thanks again to the fantastic group of folk at the Fastly NOC who so ably pointed us in the right direction! Frank https://www.premieronline.net/~fbulk/example1.pcapng https://www.premieronline.net/~fbulk/example2.pcapng https://www.premieronline.net/~fbulk/example3.pcapng

11 16

Vultr contact
by Mehmet Akcin 10 Sep '18

10 Sep '18

Hello, I have been looking for a contact in Vultr for awhile now. Can anyone help me contact with them? (i need help beyond their sales/support channels) mehmet

4 5