On Fri, Oct 20, 2017 at 1:10 AM, David Sotnick <sotnickd-nanog@ddv.com> wrote:
Well well, it looks like a Direct Connect circuit to Google was leaking the route to this DMZ 153.7.233.0/24 back to Google via BGP.
Return traffic from Google (for only some fraction of DNS queries) was passing back across this leaked route, and being dropped on this Direct Connect peering point at Disney.
Gotta love it when a problem is solved, by the OP, within an hour of resorting to mailing the NANOG community.
This shows some issues as well, I think? http://dnsviz.net/d/studio.disney.com/servers/ $ dig NS disney.com ;; ANSWER SECTION: disney.com. 4676 IN NS huey11.disney.com. disney.com. 4676 IN NS huey.disney.com. disney.com. 4676 IN NS Orns02.dig.com. disney.com. 4676 IN NS Orns01.dig.com. disney.com. 4676 IN NS Sens02.dig.com. disney.com. 4676 IN NS Sens01.dig.com. $ dig NS studio.disney.com @huey11.disney.com. ;; AUTHORITY SECTION: studio.disney.com. 600 IN NS wallyb.pixar.com. studio.disney.com. 600 IN NS andre.pixar.com. studio.disney.com. 600 IN NS cliff.studio.disney.com. studio.disney.com. 600 IN NS norm.studio.disney.com. $ for d in $(dig +short NS disney.com); do dig +short SOA disney.com @$d; done huey.disney.com. root.huey.disney.com. 2017102000 3600 900 3600000 3600 huey.disney.com. root.huey.disney.com. 2017102000 3600 900 3600000 3600 huey.disney.com. root.huey.disney.com. 2017102000 3600 900 3600000 3600 huey.disney.com. root.huey.disney.com. 2017102000 3600 900 3600000 3600 huey.disney.com. root.huey.disney.com. 2017102000 3600 900 3600000 3600 huey.disney.com. root.huey.disney.com. 2017102000 3600 900 3600000 3600 $ for d in $(dig +short NS studio.disney.com); do dig +short SOA studio.disney.com @$d; done cliff.studio.disney.com. admin.studio.disney.com. 2017101904 10800 3600 604800 86400 cliff.studio.disney.com. admin.studio.disney.com. 2017101904 10800 3600 604800 86400 cliff.studio.disney.com. admin.studio.disney.com. 2017101904 10800 3600 604800 86400 cliff.studio.disney.com. admin.studio.disney.com. 2017101904 10800 3600 604800 86400 cliff.studio.disney.com. admin.studio.disney.com. 2017101904 10800 3600 604800 86400 it looks like the second-level and third-level don't agree with each other on whom should be the NS for the third-level? that shouldn't be fatal, but is something to cleanup. Thanks all, nothing to see here!
-David
On Thu, Oct 19, 2017 at 8:41 PM, David Sotnick <sotnickd-nanog@ddv.com> wrote:
Hi Nanog,
I am principal network engineer for sister-studio to Disney Studios. They have been struggling with DNS issues since Thursday 12th October.
By all accounts it appears as though *some* of the Google DNS resolvers cannot reach the authoritative nameservers for "studio.disney.com".
This is causing ~20-30% of all DNS requests against Google Public DNS 8.8.8.8 / 8.8.4.4 to fail for requests in this subdomain.
The name servers reside in 153.7.233.0/24.
Might someone be able to *connect me* with someone at Google to assist my poor colleagues who are banging their heads against a brick wall here.
Thank you, David