August 2017 - Test - lists.nanog.org

Last Week's Canadian Fiber Cut
by Rod Beck 24 Aug '17

24 Aug '17

Did we ever get any resolution on why this was such a big outage? Appears there were two fiber cuts. Were the fibers damaged in the same conduit? Is this a collapsed ring scenario? http://www.cbc.ca/news/canada/newfoundland-labrador/concerns-about-backup-b… Roderick Beck Director of Global Sales United Cable Company DRG Undersea Consulting Affiliate Member www.unitedcablecompany.com<http://www.unitedcablecompany.com> 85 Király utca, 1077 Budapest rod.beck(a)unitedcablecompany.com 36-30-859-5144 [1467221477350_image005.png]

11 17

Cogent Chicago
by Mike Hammett 24 Aug '17

24 Aug '17

I'm looking for someone knowledgeable as to how some of their datacenter POPs interconnect. Trying to determine what level of diversity other than POP location there are between two datacenters. Cogent staff is fine, maybe even preferred. Unsurprisingly, the sales person I talked to wasn't incredibly useful. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP

1 0

Contact at Charter Communications?
by Chris Rhode 24 Aug '17

24 Aug '17

Hello, I apologize if this is not the appropriate place to ask, however we have been trying to get in touch with someone at Charter Communications to see if they are blocking part of our IP range and have been unsuccessful in getting in touch with anybody. I've contacted both the email address and called the phone number listed under the tech contact for brighthouse.com (which is the domain in question) listed on whois.icann.org and have not received any response. Would someone please contact me off-list with someone who I might be able to reach out to in order to check on this? Thank you! -- *Chris Rhode *Network Engineer University of South Florida – Information Technology crhode(a)usf.edu <mailto:crhode@usf.edu>

2 1

Re: AT&T NOC contact?
by Sahil Ganguly 24 Aug '17

24 Aug '17

Hello, Thank you for checking, the issue was resolved. On Thu, Aug 24, 2017 at 7:52 AM, Nimrod Levy <nimrodl(a)gmail.com> wrote: > There was a message to the outages list over the weekend on this, has this > issue not been resolved? > > On Thu, Aug 24, 2017 at 10:38 AM Sahil Ganguly via NANOG <nanog(a)nanog.org> > wrote: > >> Hello, >> >> Is there someone at AT&T on the mailing list I can talk to regarding a >> possible routing loop getting from AT&T to Box? >> >> Thanks! >> >> -- >> Sahil Ganguly >> Senior Network Operations Engineer >> >> >> M: 303.250.8893 <(303)%20250-8893> >> 900 Jefferson Ave >> Redwood City, CA 94063 >> > -- > > -- > Nimrod > -- Sahil Ganguly Senior Network Operations Engineer M: 303.250.8893 900 Jefferson Ave Redwood City, CA 94063

1 0

AT&T NOC contact?
by Sahil Ganguly 24 Aug '17

24 Aug '17

Hello, Is there someone at AT&T on the mailing list I can talk to regarding a possible routing loop getting from AT&T to Box? Thanks! -- Sahil Ganguly Senior Network Operations Engineer M: 303.250.8893 900 Jefferson Ave Redwood City, CA 94063

2 1

DevOps workflow for networking
by Kasper Adel 24 Aug '17

24 Aug '17

We are pretty new to those new-age network orchestrators and automation, I am curious to ask what everyone is the community is doing? sorry for such a long and broad question. What is your workflow? What tools are your teams using? What is working what is not? What do you really like and what do you need to improve? How mature do you think your process is? etc etc Wanted to ask and see what approaches the many different teams here are taking! We are going to start working from a GitLab based workflow. Projects are created, issues entered and developed with a gitflow branching strategy. GitLab CI pipelines run package loadings and run tests inside a lab. Tests are usually python unit tests that are run to do both functional and service creation, modification and removal tests. For unit testing we typically use python libraries to open transactions to do the service modifications (along with functional tests) against physical lab devices. For our prod deployment we leverage 'push on green' and gating to push package changes to prod devices. Thanks

11 11

Google DNS --- Figuring out which DNS Cluster you are using
by Erik Sundberg 24 Aug '17

24 Aug '17

I sent this out on the outage list, with a lots of good feedback sent to me. So I figured it would be useful to share the information on nanog as well. A couple months ago had to troubleshoot a google DNS issue with Google’s NOC. Below is some helpful information on how to determine which DNS Cluster you are going to. Let’s remember that Google runs DNS Anycast for DNS queries to 8.8.8.8 and 8.8.4.4. Anycast routes your DNS queries to the closes DNS cluster based on the best route / lowest metric to 8.8.8.8/8.8.4.4. Google has deployed multiple DNS clusters across the world and each DNS Cluster has multiple servers. So a DNS query in Chicago will go to a different DNS clusters than queries from a device in Atlanta or New York. How to get a list of google DNS Cluster’s. dig -t TXT +short locations.publicdns.goog. @8.8.8.8 How to print this list in a table format. Script from: https://developers.google.com/speed/public-dns/faq --------------- #!/bin/bash IFS="\"$IFS" for LOC in $(dig -t TXT +short locations.publicdns.goog. @8.8.8.8) do case $LOC in '') : ;; *.*|*:*) printf '%s ' ${LOC} ;; *) printf '%s\n' ${LOC} ;; esac done --------------- Which will give you a list like below. This is all of the IP network’s that google uses for their DNS Clusters and their associated locations. 74.125.18.0/26 iad 74.125.18.64/26 iad 74.125.18.128/26 syd 74.125.18.192/26 lhr 74.125.19.0/24 mrn 74.125.41.0/24 tpe 74.125.42.0/24 atl 74.125.44.0/24 mrn 74.125.45.0/24 tul 74.125.46.0/24 lpp 74.125.47.0/24 bru 74.125.72.0/24 cbf 74.125.73.0/24 bru 74.125.74.0/24 lpp 74.125.75.0/24 chs 74.125.76.0/24 cbf 74.125.77.0/24 chs 74.125.79.0/24 lpp 74.125.80.0/24 dls 74.125.81.0/24 dub 74.125.92.0/24 mrn 74.125.93.0/24 cbf 74.125.112.0/24 lpp 74.125.113.0/24 cbf 74.125.115.0/24 tul 74.125.176.0/24 mrn 74.125.177.0/24 atl 74.125.179.0/24 cbf 74.125.181.0/24 bru 74.125.182.0/24 cbf 74.125.183.0/24 cbf 74.125.184.0/24 chs 74.125.186.0/24 dls 74.125.187.0/24 dls 74.125.190.0/24 sin 74.125.191.0/24 tul 172.217.32.0/26 lhr 172.217.32.64/26 lhr 172.217.32.128/26 sin 172.217.33.0/26 syd 172.217.33.64/26 syd 172.217.33.128/26 fra 172.217.33.192/26 fra 172.217.34.0/26 fra 172.217.34.64/26 bom 172.217.34.192/26 bom 172.217.35.0/24 gru 172.217.36.0/24 atl 172.217.37.0/24 gru 173.194.90.0/24 cbf 173.194.91.0/24 scl 173.194.93.0/24 tpe 173.194.94.0/24 cbf 173.194.95.0/24 tul 173.194.97.0/24 chs 173.194.98.0/24 lpp 173.194.99.0/24 tul 173.194.100.0/24 mrn 173.194.101.0/24 tul 173.194.102.0/24 atl 173.194.103.0/24 cbf 173.194.168.0/26 nrt 173.194.168.64/26 nrt 173.194.168.128/26 nrt 173.194.168.192/26 iad 173.194.169.0/24 grq 173.194.170.0/24 grq 173.194.171.0/24 tpe 2404:6800:4000::/48 bom 2404:6800:4003::/48 sin 2404:6800:4006::/48 syd 2404:6800:4008::/48 tpe 2404:6800:400b::/48 nrt 2607:f8b0:4001::/48 cbf 2607:f8b0:4002::/48 atl 2607:f8b0:4003::/48 tul 2607:f8b0:4004::/48 iad 2607:f8b0:400c::/48 chs 2607:f8b0:400d::/48 mrn 2607:f8b0:400e::/48 dls 2800:3f0:4001::/48 gru 2800:3f0:4003::/48 scl 2a00:1450:4001::/48 fra 2a00:1450:4009::/48 lhr 2a00:1450:400b::/48 dub 2a00:1450:400c::/48 bru 2a00:1450:4010::/48 lpp 2a00:1450:4013::/48 grq There are IPv4 Networks: 68 IPv6 Networks: 20 DNS Cluster’s Identified by POP Code’s: 20 DNS Clusters identified by POP Code to City, State, or Country. Not all of these are Google’s Core Datacenters, some of them are Edge Points of Presences (POPs). https://peering.google.com/#/infrastructure and https://www.google.com/about/datacenters/inside/locations/ Most of these are airport codes, it did my best to get the location correct. iad Washington, DC syd Sydney, Australia lhr London, UK mrn Lenoir, NC tpe Taiwan atl Altanta, GA tul Tulsa, OK lpp Findland bru Brussels, Belgium cbf Council Bluffs, IA chs Charleston, SC dls The Dalles, Oregon dub Dublin, Ireland sin Singapore fra Frankfort, Germany bom Mumbai, India gru Sao Paulo, Brazil scl Santiago, Chile nrt Tokyo, Japan grq Groningen, Netherlans Which Google DNS Server Cluster am I using. I am testing this from Chicago, IL # dig o-o.myaddr.l.google.com -t txt +short @8.8.8.8 "173.194.94.135" <<<<<<DNS Server IP, reference the list above to get the cluster, Council Bluffs, IA "edns0-client-subnet 207.xxx.xxx.0/24" <<<< Your Source IP Block Side note, the google dns servers will not respond to DNS queries to the Cluster’s Member’s IP, they will only respond to dns queries to 8.8.8.8 and 8.8.4.4. So the following will not work. dig google.com @173.194.94.135 Now to see the DNS Cluster load balancing in action. I am doing a dig query from our Telx\Digital Realty POP in Atlanta, GA. We do peer with google at this location. I dig a dig query about 10 times and received the following unique dns cluster member ip’s as responses. dig o-o.myaddr.l.google.com -t txt +short @8.8.8.8 "74.125.42.138" "173.194.102.132" "74.125.177.5" "74.125.177.74" "74.125.177.71" "74.125.177.4" Which all are Google DNS Networks in Atlanta. 74.125.42.0/24 atl 74.125.177.0/24 atl 172.217.36.0/24 atl 173.194.102.0/24 atl 2607:f8b0:4002::/48 atl Just thought it would be helpful when troubleshooting google DNS issues. ________________________________ CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

9 11

AWS internal networking team contact?
by Robbie Trencheny 24 Aug '17

24 Aug '17

Hey all, We are seeing major packet loss and high latency at a Level3 node just before the hop into AWS us-west-2. We had a go live planned for today which has now been scrapped because T-Mobile customers (a significant chunk of our customer base) nationwide are unable to login to our app. AWS Support is dragging their feet waiting to hear back from their internal networking team and didn't even believe they peered with Level3 at us-west-2 and we just don't have the time. CEO is talking about pulling the plug entirely on us-west-2 if not all of AWS and re-deploying to us-east-X or GCP. If someone from AWS could reach out off list to help expedite my ticket and get in contact with Level3 to fix their Seattle nodes i'd really appreciate it. It's already 740 on the west coast and I don't think i'm going home anytime soon. :( For those interested, failing nodes i'm seeing are ae-1-51.ear2.Seattle1.Level3.net <http://ae-1-51.ear2.seattle1.level3.net/> and ae-2-52.ear2.Seattle1.Level3.net <http://ae-2-52.ear2.seattle1.level3.net/>. Confirmed that T-Mobile is routing through those nodes from both SF and NYC. Apple also rejected our app during review this morning because of network failures while on wifi which I have to assume are the same ones we've been seeing for the last 96 hours. Thanks

1 0

Frontier SoCal FIOS contact? - gateway proxy arp issue
by Paul B. Henson 24 Aug '17

24 Aug '17

So yesterday I started seeing some arp warnings in my server logs: Aug 23 16:09:29 lisa /bsd: arp info overwritten for 96.251.22.154 by f0:1c:2d:8d:0e:cf on em2 Aug 23 16:12:24 lisa /bsd: arp info overwritten for 96.251.22.154 by f0:1c:2d:8d:0e:cf on em2 Aug 23 16:21:28 lisa /bsd: arp info overwritten for 96.251.22.154 by 00:25:90:da:ea:f9 on em2 f0:1c:2d:8d:0e:cf is the MAC address of 96.251.22.1, L100.LSANCA-VFTTP-55.gni.frontiernet.net, my FIOS gateway. It seems that for some reason proxy arp has been enabled on the router providing my gateway, and it is arp'ing for my static IP addresses as shown below: 0c:c4:7a:b3:ca:54 - MAC of lisa.pbhware.com (96.251.22.156) 00:25:90:da:ea:f9 - MAC of bart.pbhware.com (96.251.22.154) f0:1c:2d:8d:0e:cf - MAC of L100.LSANCA-VFTTP-55.gni.frontiernet.net (96.251.22.1) On 96.251.22.156: $ ping 96.251.22.154 16:12:24.416146 0c:c4:7a:b3:ca:54 Broadcast arp 42: arp who-has bart.pbhware.com tell lisa.pbhware.com 16:12:24.416405 00:25:90:da:ea:f9 0c:c4:7a:b3:ca:54 arp 60: arp reply bart.pbhware.com is-at 00:25:90:da:ea:f9 16:12:24.419522 f0:1c:2d:8d:0e:cf 0c:c4:7a:b3:ca:54 arp 60: arp reply bart.pbhware.com is-at f0:1c:2d:8d:0e:cf Another example for an IP address (96.251.22.158/static-5.pbhware.com) not currently in use: $ ping 96.251.22.158 16:26:15.784494 0c:c4:7a:b3:ca:54 Broadcast arp 42: arp who-has static-5.pbhware.com tell lisa.pbhware.com 16:26:15.787624 f0:1c:2d:8d:0e:cf 0c:c4:7a:b3:ca:54 arp 60: arp reply static-5.pbhware.com is-at f0:1c:2d:8d:0e:cf 16:26:15.787677 0c:c4:7a:b3:ca:54 f0:1c:2d:8d:0e:cf ip 98: lisa.pbhware.com > static-5.pbhware.com: icmp: echo request The gateway starts arp'ing looking for the real owner of that IP so it can proxy the traffic: 16:28:07.074869 f0:1c:2d:8d:0e:cf Broadcast arp 60: arp who-has static-5.pbhware.com tell L100.LSANCA-VFTTP-55.gni.frontiernet.net When I googled it, I found a number of complaints about this dating back years regarding Verizon FIOS, where proxy arp deployments seemed to be a standard practice in some areas, but for the decade I have had business FIOS this has never happened before. So I contacted frontier technical support to ask about it; unfortunately, the interaction was less than successful :(. While the level 1 support person was very helpful and didn't take very long to convince to escalate the issue, the first level 2 support person didn't quite seem to understand the concept, as demonstrated by the following excerpts from the chat: "turning off the ARP is done from the device that is receiving the request" "that is something he would need to turn off on his router, because we do not support 3rd party equipment" "sorry but we cannot make the change that he is requesting, it needs to be done on his device" "I understand what you are saying but the ARP request needs to be turned off through his equipment it is what is allowing signal in, just like a Ping can be turned off in the router even if something else is requesting the OPING" Technically, that last comment is true - if my system did not make an arp request, the frontier router would indeed not respond with a proxy arp reply. Of course, my systems would have a really hard time talking to each other if they weren't running arp 8-/. The last thing he had to say was "its not, on a router if you turn off Ping requests it will not get pinged, same thing with ARP requests" and I decided to stop wasting my time on him. Sadly, the level 1 tech informed me that the level 2 tech was actually working with his lead while responding to my issue. As I mentioned, the level 1 tech was very helpful, he offered to try to reach a different level 2 tech. The second level 2 tech refused to have anything to do with me unless I hooked up the original actiontech router that came with the Fios service 10 years ago, so I wrote off the official tech-support channel for now. So, long story short, are there any fios employees hanging out here that could possibly get me in contact with someone who understands the concept of proxy arp and would be able to determine why it suddenly was enabled on the gateway for my service yesterday and hopefully get it turned back off? That would be much appreciated. In the worst case I suppose I can work around this mess with arp inspection on the switch or static arp entries on the servers s, but I'd rather avoid being kludgy. Thanks much.

1 0

Call for Participation -- ICANN DNSSEC Workshop at ICANN60 in Abu Dhabi, UAE
by Jacques Latour 23 Aug '17

23 Aug '17

Call for Participation -- ICANN DNSSEC Workshop at ICANN60 in Abu Dhabi, UAE The DNSSEC Deployment Initiative and the Internet Society Deploy360 Programme, in cooperation with the ICANN Security and Stability Advisory Committee (SSAC), are planning a DNSSEC Workshop during the ICANN60 meeting held from 01 November 2017 in Abu Dhabi, UAE tentatively from 0900-1500 local time. The DNSSEC Workshop has been a part of ICANN meetings for several years and has provided a forum for both experienced and new people to meet, present and discuss current and future DNSSEC deployments. For reference, the most recent session was held at the ICANN Policy Forum in Johannesburg, South Africa. The presentations and transcripts are available at: https://schedule.icann.org/event/B3zi/dnssec-workshop[schedule.icann.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__schedule.icann.org_eve…>. At ICANN60 we are particularly interested in live demonstrations of uses of DNSSEC or DANE. Examples might include: * Innovative uses of APIs to do something new and different using DNSSEC/DANE. * Email clients and servers using DNSSEC, OPENPGPKEY, or S/MIME for secure email. * DNSSEC automation and deployment using CDS, CDNSKey, and CSYNC. * DNSSEC signing solutions and innovation. * Tools for automating the generation of DNSSEC/DANE records. * Services for monitoring or managing DNSSEC signing or validation. * Tools or services for using DNSSEC/DANE along with other existing protocols and services such as SSH, XMPP, SMTP, S/MIME or PGP/GPG. Our interest is to provide current examples of the state of development and to show real-world examples of how DNSSEC and DANE related innovation can be used to increase the overall security of the Internet. We are open to presentations and demonstrations related to any topic associated with DNSSEC and DANE. Examples of the types of topics we are seeking include: 1. DNSSEC activities in the Middle East Region For this panel we are seeking participation from those who have been involved in DNSSEC deployment in the region and also from those who have not deployed DNSSEC but who have a keen interest in the challenges and benefits of deployment. In particular, we will consider the following questions: Are you interested in reporting on DNSSEC validation of your ISPs? What can DNSSEC do for you? What doesn't it do? What are the internal tradeoffs to implementing DNSSEC? What did you learn in your deployment of DNSSEC? We are interested in presentations from both people involved with the signing of domains and people involved with the deployment of DNSSEC-validating DNS resolvers. 2. Impact and Results of Root Key Rollover Following the Root Key Rollover, we would like to bring together a panel of people who can talk about the impacts to ISPs, equipment providers and end users, and also what was done to mitigate those issues. In particular, we are seeking participation from vendors, ISPs, and the community that may have been affected by distribution of new root keys. you have a specific concern about the Root Key Rollover we would like to hear from you. 3. Implementing next generation DNSSEC signing at Registries and DNS Operators Now that DNSSEC technology has matured many Registries and DNS Operators have upgraded their legacy DNSSEC signing services with innovative solutions. * Real world use cases of HSMs and key management. * Signing at the edge We would be interested in seeing presentations or demonstrations on those topics. 4. The operational realities of running DNSSEC Now that DNSSEC has become an operational norm for many registries, registrars, and ISPs, what have we learned about how we manage DNSSEC? What is the best practice around your local key rollovers? How often do you review your disaster recovery procedures? Is there operational familiarity within your customer support teams? What operational statistics have we gathered about DNSSEC? Are there experiences being documented in the form of best practices, or something similar, for transfer of signed zones? 5. DANE and DNSSEC application automation For DNSSEC to reach massive deployment levels it is clear that a higher level of automation is required than is currently available. There also is strong interest for DANE usage within web transactions as well as for securing email and Voice-over-IP (VoIP). We are seeking presentations on topics such as: * How can the industry use DANE and other DNSSEC applications as a mechanism for creating a more secure Internet? * What tools, systems and services are available to help automate DNSSEC key management? * Can you provide an analysis of current tools/services and identify gaps? * What are some of the new and innovative uses of DANE and other DNSSEC applications in new areas or industries? * What tools and services are now available that can support DANE usage? We would be particularly interested in any live demonstrations of DNSSEC / DANE application automation and services. Demonstrations of new tools that make the setup of DNSSEC or DANE more automated would also be welcome. 6. DNSSEC and DANE in the enterprise and in the enterprise tool set Enterprises and enterprise software can play a critical role in both providing DNSSEC validation to their internal networks and also through signing of the domains owned by the enterprise. We are seeking presentations from enterprises and enterprise software providers that have implemented DNSSEC on validation and/or signing processes and can address questions such as: * What enterprise software support or plan do you have to support DNSSEC? * What are the benefits to enterprises of rolling out DNSSEC validation? And how do they do so? * What are the challenges to deployment for these organizations and how could DANE and other DNSSEC applications address those challenges? * How should an enterprise best prepare its IT staff and network to implement DNSSEC? * What enterprise tools and systems are available to assist enterprises in the deployment of DNSSEC? * How can the DANE protocol be used within an enterprise to bring a higher level of security to transactions using SSL/TLS certificates? 7. Implementing DNSSEC validation at Internet Service Providers (ISPs) Internet Service Providers (ISPs) play a critical role by enabling DNSSEC validation for the caching DNS resolvers used by their customers. We have now seen massive rollouts of DNSSEC validation within large North American ISPs and at ISPs around the world. We are interested in presentations on topics such as: * Can you describe your experiences with negative Trust Anchors and operational realities? * What does an ISP need to do to prepare its network for implementing DNSSEC validation? * How does an ISP need to prepare its support staff and technical staff for the rollout of DNSSEC validation? * Can you provide results and/or impacts of the impact of root key rollover? * What rollover technique do you use, i.e., RFC 5011 or other? In addition, we welcome suggestions for additional topics. If you are interested in participating, please send a brief (1-2 sentence) description of your proposed presentation to dnssec-abudhabi(a)isoc.org<mailto:dnssec-abudhabi@isoc.org> by **08 September 2017** We hope that you can join us. Thank you, Julie Hedlund On behalf of the DNSSEC Workshop Program Committee: Jean Robert Hountomey, AfricaCERT Jacques Latour, .CA Xiaodong Lee, CNNIC Russ Mundy, Parsons Ondřej Filip, CZ.NIC Yoshiro Yoneya, JPRS Dan York, Internet Society Mark Elkins, DNS/ZACR

1 0