September 2018 - Test - lists.nanog.org

SAFNOG-4/EANOG/tzNOG - Just Under 2 Weeks To Go!
by Mark Tinka 13 Sep '18

13 Sep '18

Hi all. This is just a reminder that with a little under 2 weeks to go the upcoming SAFNOG-4/EANOG/tzNOG meeting to be held in Dar Es Salaam, 24th - 29th September, please remember to register your attendance at http://www.safnog.org/ so you don't miss the meeting. The agenda is now up on the web site at http://www.safnog.org/#agenda with just a few slots left which are filling fast. In case you feel like you have an interesting technical and operational talk that you'd like to share with the community at this meeting, please feel free to submit it for the program committee to review as soon as possible, at: https://papers.safnog.org/user/login.php?event=76 Key talks that you would be looking forward include: * Ben Roberts (Liquid Telecom), talking about the latency realities of intra-African Internet communication. * The 3 major data centre projects that are happening in South Africa, Kenya and Uganda. * The DNSSEC key rollover due on 11th October, and what to look forward to, as explained by Amreesh Phokeer. * An IPv6 Deployathon 2-day workshop headed by Mukom Akong Tamon. * ... and so many more. There will be 2 social events where delegates can network, share a drink, some good food, and stand a chance to win either an iPhone X or AppleTV 4K. We have secured a special discount rate at the hotel venue for all SAFNOG/EANOG/tzNOG participants. More details are available at http://www.safnog.org/#venue . We look forward to seeing you in Dar Es Salaam. Mark Tinka On Behalf of the SAFNOG-4/EANOG/tzNOG Organising Committee

1 0

OpenDNS CGNAT Issues
by Darin Steffl 12 Sep '18

12 Sep '18

Hello, I have a ticket open with OpenDNS about filtering happening on some of our CGNAT IP space where a customer has "claimed" the IP as theirs so other customers using that same IP and OpenDNS are being filtered and not able to access sites that fall under their chosen filter. I have a ticket open from 6 days ago but it's not going anywhere fast. Can someone from OpenDNS contact me or point me to a contact there to help get this resolved? I believe we need to claim our CGNAT IP space so residential users can't claim IP's of their own. Thank you! -- Darin Steffl Minnesota WiFi www.mnwifi.com 507-634-WiFi <http://www.facebook.com/minnesotawifi> Like us on Facebook <http://www.facebook.com/minnesotawifi>

14 21

Bell Canada IP transit - NID required?
by JASON BOTHE 11 Sep '18

11 Sep '18

All, Just curious if anyone uses Bell Canada for IP transit in a colo environment and whether or not you were able to obtain a traditional cross-connect without having Bell place a managed CPE device in your cage. I can’t seem to find anyone intelligent enough to tell my any different other than “it’s never been done before” and “we need remote testing capabilities.” J~

6 5

Microsoft your DNS servers are broken
by Mark Andrews 11 Sep '18

11 Sep '18

While we are talking about DNS server that are broken, Microsoft your servers are as well. As none of the zones you serve are DNSSEC signed there isn’t as much breakage possible but there are still interoperability problems and unnecessary additional traffic. It’s not like the EDNS specification is complicated. The microsoftonline servers will cause DNSSEC validation to fail if they ever serve a DNSSEC signed zone in this state. The FORMERR will cause EDNS servers to fallback to plain DNS and the validators won’t get the records they need. The azure servers cause problems for anyone deploying a new EDNS options as they have to cope with your servers incorrectly echoing back the option. Additionally if EDNS(1) is ever deployed there is a good chance that resolvers will assume the broken answers indicate that there is no data at the name. Mark cityofharrison-mi.gov. @207.46.15.59 (ns1.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @2a01:111:f406:1804::59 (ns1.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @191.232.83.138 (ns3.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @2a01:111:f406:b400::22 (ns3.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @157.56.81.41 (ns2.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok cityofharrison-mi.gov. @2a01:111:f406:3403::41 (ns2.bdm.microsoftonline.com.) dns=ok edns=ok edns1=ok edns@512=ok ednsopt=formerr,echoed edns1opt=formerr,version-not-zero,echoed do=ok ednsflags=ok optlist=formerr,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @13.107.24.7 (ns3-07.azure-dns.org.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @2a01:111:4000::7 (ns3-07.azure-dns.org.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @13.107.160.7 (ns4-07.azure-dns.info.): dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @2620:1ec:bda::7 (ns4-07.azure-dns.info.): dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @64.4.48.7 (ns2-07.azure-dns.net.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @2620:1ec:8ec::7 (ns2-07.azure-dns.net.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @40.90.4.7 (ns1-07.azure-dns.com.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok clintoncounty-ia.gov. @2603:1061::7 (ns1-07.azure-dns.com.) dns=ok edns=ok edns1=noerror,badversion edns@512=ok ednsopt=echoed edns1opt=noerror,badversion do=ok ednsflags=ok optlist=ok,subnet signed=ok ednstcp=ok -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka(a)isc.org

2 1

Proofpoint Contact?
by Benoit Garcia 10 Sep '18

10 Sep '18

Hello, Could a Proofpoint e-mail blacklist contact me off list? I'm looking for some help to solve an undelivered email issue. Thanks in advance. -- Regards, Benoit.

2 1

Service provider story about tracking down TCP RSTs
by frnkblk＠iname.com 10 Sep '18

10 Sep '18

I want to share a little bit of our journey in tracking down the TCP RSTs that impacted some of our customers for almost ten weeks. Almost immediately after we turned up two new Arista border routers in late July we started receiving a trickle of complaints from customers regarding their inability to access certain websites (mostly B2B). All the packet captures showed the standard TCP SYN/SYN-ACK pair, then a TCP RST from the website after the client sent a TLS/SSL Client Hello. As the reports continued to come in, we built a Google Doc to keep track and it became clear that most of the sites were hosted by Incapsula/Imperva, but there were also a few by Sucuri and Fastly. Knowing that Incapsula provides DoS protection, we attempted to work with them (providing websites, source/destination IPs, traceroutes, and packet captures) to find out why their hosts were issuing our customers a TCP RST, but we made little progress. We moved some of the affected customers to different IP addresses but that didn't resolve the issue. We also asked our customer to work with the website to see if they would be willing to open a ticket with Incapsula. In the meantime, customers were getting frustrated! They couldn't visit Incapsula-hosted healthcare websites, financial firms, product dealers, etc. Over the weeks, a few of those customers purchased/borrowed different routers and some of those didn't have website issues anymore. And more than a few of them discovered that the websites worked fine from home or their mobile phone/hotspot, but not from their Internet connection with us. You can guess where they were applying pressure! That said, we didn't know why a small handful of companies, known for DoS protection, were issuing TCP RSTs to just some of our customers. Earlier this week we received four or five more websites from yet another affected customer, but most of those were with Fastly. By this time, we had been able to replicate the issue in our lab. Feeling desperate to make some tangible progress on this issue, I reached out to the Fastly NOC. In less than 12 hours they provided some helpful feedback, pointing out that a single traceroute to a Fastly site was hitting two of their POPs (they use anycast) and because they don't sync state between POPs the second POP would naturally issue a TCP RST (sidebar: fascinating blog article on Fastly's infrastructure here: https://www.fastly.com/blog/building-and-scaling-fastly-network-part-2-balan cing-requests). In subsequent email exchanges, the Fastly NOC suggested that it appeared that we were "spraying flows" (that is, packets related to single client session were egressing our network via different paths). Because Fastly is also present with us at an IX (though they weren't advertising their anycast IPs at the time), they suggested that we look at how our traffic egresses our network (IX versus transit) and our routers' outbound load-balancing/hashing schemes. The IX turned up to be a red herring, so I turned my attention to our transit. Each of our border routers has two BGP sessions over two circuits to transit provider POP A and two BGP sessions over two circuits to transit provider POP B, for a total of four BGP sessions per border router, a total of eight BGP sessions altogether. Starting with our core router, I confirmed that its ECMP hashing was consistent such that Fastly-bound traffic always went to border router 1 or border router 2. Then I looked at the ECMP hashing scheme on our border routers and noticed something unique - by default Arista also uses TTL: IPv4 hash fields: Source IPv4 Address is ON Protocol is ON Time-To-Live is ON Destination IPv4 Address is ON Since the source and destination IPs and protocol weren't changing, perhaps the TTL was not consistent? I opened the first packet trace in Wireshark and jackpot - the TTL value was 128 on the SYN but 127 on the TLS/SSL Client Hello. I adjusted the Arista's load-balancing profile not to use TTL and immediately my MTR in the background changed and all the sites on the lab machine that couldn't load before . were now loading. Fastly also pointed me to another article written by Joel Jaeggli (https://blog.apnic.net/2018/01/11/ipv6-flow-label-misuse-hashing/) that discusses IPv6 flow labels - we removed that from the border routers' IPv6 hash fields, too. I reviewed the packet traces today and noticed that TTL values remained consistent at 128 *behind* the router CPE. In packet captures on the WAN interface of the router CPE I see that the SYN remains at 128, but the TLS/Client Hello is properly decremented to 127. So, it appears that some router CPE (and there were a variety of makes and models) are doing something special to certain packets and not decrementing the TTL. This explains why: * our customers had issues with all their devices behind their router CPE * the issue remained regardless of what public IP address their router CPE obtained via DHCP or was assigned * some customers who changed their router CPE didn't have the issue anymore - they got lucky with a router that doesn't adjust/reset the TTL * why customers who used our managed Wi-Fi router did not see the issue, because that model doesn't apparently manipulate the TTL, at least not in an inconsistent way. Lesson learned: review a device's hashing mechanism before going into production. For those interested, I have links to the packet traces below my signature, showing the inconsistent TTL values. Thanks again to the fantastic group of folk at the Fastly NOC who so ably pointed us in the right direction! Frank https://www.premieronline.net/~fbulk/example1.pcapng https://www.premieronline.net/~fbulk/example2.pcapng https://www.premieronline.net/~fbulk/example3.pcapng

11 16

Vultr contact
by Mehmet Akcin 09 Sep '18

09 Sep '18

Hello, I have been looking for a contact in Vultr for awhile now. Can anyone help me contact with them? (i need help beyond their sales/support channels) mehmet

4 5

Study on configuration change practices
by Aaron Gember-Jacobson 08 Sep '18

08 Sep '18

We are a team of networking researchers at the University of Wisconsin-Madison and Colgate University investigating methods for automatically synthesizing router configurations from high-level requirements (or intents). To guide our research, we seek to better understand the configuration change practices used in production networks. We would appreciate if you could take 3 minutes to complete our brief, anonymous survey: https://colgate.co1.qualtrics.com/jfe/form/SV_3ee26xayy70jP3D To learn more about our research, visit http://aaron.gember-jacobson.com/research/repair Thanks, Aaron Gember-Jacobson *Assistant Professor of Computer Science* *Colgate University*

2 2

Spoofer Report for NANOG for Aug 2018
by CAIDA Spoofer Project 08 Sep '18

08 Sep '18

In response to feedback from operational security communities, CAIDA's source address validation measurement project (https://spoofer.caida.org) is automatically generating monthly reports of ASes originating prefixes in BGP for systems from which we received packets with a spoofed source address. We are publishing these reports to network and security operations lists in order to ensure this information reaches operational contacts in these ASes. This report summarises tests conducted within usa, can. Inferred improvements during Aug 2018: ASN Name Fixed-By 33651 CMCS 2018-08-15 2152 CSUNET-NW 2018-08-20 54417 STIMULUS-TECHNOLOGIES 2018-08-22 21949 BEANFIELD 2018-08-29 Further information for the inferred remediation is available at: https://spoofer.caida.org/remedy.php Source Address Validation issues inferred during Aug 2018: ASN Name First-Spoofed Last-Spoofed 3356 LEVEL3 2016-03-06 2018-08-13 577 BACOM 2016-03-09 2018-08-27 7922 COMCAST-7922 2016-06-04 2018-08-28 20115 CHARTER-NET-HKY-NC 2016-06-09 2018-08-23 209 CENTURYLINK-US-LEGACY-QWEST 2016-08-16 2018-08-29 6128 CABLE-NET-1 2016-09-03 2018-08-11 11232 MIDCO-NET 2016-09-22 2018-08-11 20412 CLARITY-TELECOM 2016-09-30 2018-08-28 6181 FUSE-NET 2016-10-10 2018-08-31 15305 SYRINGANETWORKS 2016-10-21 2018-08-25 25787 ROWE-NETWORKS 2016-10-21 2018-08-29 174 COGENT-174 2016-10-21 2018-08-22 271 BCNET 2016-10-24 2018-08-17 2828 XO-AS15 2016-10-25 2018-08-24 30341 SCTA-ASN 2016-10-31 2018-08-26 32440 LONI 2016-11-03 2018-08-29 33182 DimeNOC 2016-11-08 2018-08-30 12083 WOW-INTERNET 2016-11-09 2018-08-31 1403 EBOX 2016-11-12 2018-08-19 22561 CENTURYLINK-LEGACY-LIGHTCORE 2016-11-27 2018-08-06 21947 BCI 2017-01-06 2018-08-27 7106 INDEPENDENTSFIBERNETWORK 2017-01-09 2018-08-17 9009 M247 2017-01-10 2018-08-24 21832 KELLINCOM-1 2017-02-03 2018-08-25 7122 MTS-ASN 2017-05-16 2018-08-27 701 UUNET 2017-06-14 2018-08-13 22402 NEXTCO 2017-06-19 2018-08-16 6461 ZAYO-6461 2017-06-21 2018-08-28 20394 MASHELL-TELECOM 2017-06-29 2018-08-07 63296 AMARILLO-WIRELESS 2017-09-01 2018-08-30 7233 YAHOO-US 2017-09-07 2018-08-09 2152 CSUNET-NW 2017-11-08 2018-08-30 546 PARSONS-PGS-1 2017-11-20 2018-08-29 54540 INCERO 2018-01-15 2018-08-27 12222 AKAMAI 2018-02-14 2018-08-31 3257 GTT-BACKBONE 2018-03-06 2018-08-31 29384 Qatar-Foundation 2018-03-08 2018-08-26 23148 TERRENAP 2018-03-15 2018-08-30 54417 STIMULUS-TECHNOLOGIES 2018-03-29 2018-08-28 20009 WADSNET 2018-04-13 2018-08-29 4201 ORST 2018-04-19 2018-08-24 11827 WSU 2018-04-19 2018-08-29 393564 SPOKANE 2018-06-05 2018-08-15 35911 BNQ-1 2018-06-06 2018-08-29 225 VIRGINIA 2018-06-18 2018-08-21 30174 UTA 2018-08-03 2018-08-10 22773 ASN-CXA-ALL-CCI-22773-RDC 2018-08-04 2018-08-28 33588 BRESNAN 2018-08-05 2018-08-05 22742 CT-EDU-NET 2018-08-08 2018-08-29 53646 HARRIS-BROADBAND 2018-08-10 2018-08-10 55053 COLOWAREHOUSEONTARIO 2018-08-14 2018-08-15 54804 CSMIII-BUNKIELA 2018-08-21 2018-08-30 20057 ATT-MOBILITY-LLC-AS20057 2018-08-23 2018-08-23 15320 PRIMELINK 2018-08-23 2018-08-23 17025 ZAYO-CUSTOMER-17025 2018-08-23 2018-08-23 33452 RW 2018-08-24 2018-08-24 11260 EASTLINK-HSI 2018-08-26 2018-08-26 11996 LOBOIS 2018-08-28 2018-08-28 394153 VELOXINET 2018-08-28 2018-08-28 20448 VPNTRANET-LLC 2018-08-29 2018-08-29 40911 L2NC 2018-08-31 2018-08-31 Further information for these tests where we received spoofed packets is available at: https://spoofer.caida.org/recent_tests.php?country_include=usa,can&no_block… Please send any feedback or suggestions to spoofer-info(a)caida.org

1 0

uCPE or SD-WAN Devices with LTE Support
by Colton Conor 08 Sep '18

08 Sep '18

We have been using cradlepoint routers, but we want a device that has the functionality of a cradlepoint with the ability to run own own applications naively on Linux. cradlepoint does not allow you to run your own apps on the device itself eventhogh the newer cradlepoint has a quad core processor. Does anyone have recommendations for a UCPE or SD-WAN device that: cost about what a Cradlepoint cost ($500 to $750) have built in LTE (or supports a M.2 LTE card) has at least one Ethernet port has remote management, and can load your own application onto? We could probably build out own with a marker board of some sort and a Sierra Wireless LTE card, but what linux operating system would be best for this?

1 0