On Wed, Jan 16, 2013 at 2:00 PM, Erik Levinson <erik.levinson@uberflip.com> wrote:
Hi everyone,
I'm having an unusual DNS problem and would appreciate feedback.
For the zones in question, primary DNS is provided by GoDaddy and secondary DNS by DNS Made Easy. Over a week ago we made changes to several A records (including wildcards on two different zones), all already having a TTL no greater than one hour.
The new IPs on those A records have taken many millions of requests since the changes. Occasionally, a small amount of traffic appears at the old IPs that those A records had. This is HTTP traffic. Packet captures of this traffic show various Host headers.
Attempting to resolve those various Host headers from various networks in Canada against various random private and public resolvers and against the authoritative NSs all yield correct results (i.e. new IPs).
However, both GoDaddy and DNS Made Easy use anycast, which makes it less likely that I can see the entire picture of what's happening.
I suspect that somewhere, one of their servers has the wrong data, or some resolver is misbehaving, but based on the pattern/traffic/volume/randomization of hostnames, the resolver theory is less likely. I haven't analyzed the source IPs yet to see if they're in a particular set of countries.
I've opened a ticket with DNS Made Easy and they replied very quickly suggesting the problem is not with them. I've opened a ticket with GoDaddy and...well, it's GoDaddy, so I don't expect much (no response yet).
Any ideas? Can folks try resolving eriktest.uberflip.com and post here with details only if it resolves to an IP starting with 76.9 (old IPs)?
Thanks
Erik
The other likely cause of this is local cacheing nameservers somewhere at some ISP or major site, that do not respect TTL values for some reason. This is sadly a common problem - not statistically, most nameservers do the right thing, but if you run big sites and flip things, there's always a long tail of people whose nameservers just didn't get it. -- -george william herbert george.herbert@gmail.com