Is there perhaps an about.com/nytimes.com admin around? I was wondering if they perhaps knew that their loadbalancer for www.nytimes.com is fairly broken wrt answering AAAA queries: (who's NS for nytimes.com) dig NS nytimes.com +short ns1t.nytimes.com. nydns2.about.com. nydns1.about.com. (who do they think is the NS for www.nytimes.com) dig www.nytimes.com @ns1t.nytimes.com. NS ;; QUESTION SECTION: ;www.nytimes.com. IN NS ;; AUTHORITY SECTION: www.nytimes.com. 60 IN NS nss1.sea1.nytimes.com. www.nytimes.com. 60 IN NS nss1.lga2.nytimes.com. (what is the AAAA for www.nytimes.com ?? ) dig www.nytimes.com @nss1.sea1.nytimes.com. AAAA ;www.nytimes.com. IN AAAA ;; AUTHORITY SECTION: . 3600000 IN NS k.root-servers.net. . 3600000 IN NS l.root-servers.net. . 3600000 IN NS m.root-servers.net. . 3600000 IN NS a.root-servers.net. . 3600000 IN NS b.root-servers.net. . 3600000 IN NS c.root-servers.net. . 3600000 IN NS d.root-servers.net. . 3600000 IN NS e.root-servers.net. . 3600000 IN NS f.root-servers.net. . 3600000 IN NS g.root-servers.net. . 3600000 IN NS h.root-servers.net. . 3600000 IN NS i.root-servers.net. . 3600000 IN NS j.root-servers.net. ;; ADDITIONAL SECTION: k.root-servers.net. 3600000 IN A 193.0.14.129 l.root-servers.net. 3600000 IN A 198.32.64.12 m.root-servers.net. 3600000 IN A 202.12.27.33 ;; Query time: 89 msec ;; SERVER: 170.149.172.35#53(170.149.172.35) wha??? <ricky voice>Lucy, your loadbalancer is foobar'd</ricky voice> In an effort to make v6 things work a tad better in this hostile world, could the NYTimes folks let us know what sort of LB that is? and why it wants to not be a good Intenet Citizen?? -Chris
I hate to reply to myself, but... (and I'm sure this isn't the only other example) what the heck is ETrade's LB doing here? (who is NS for etrade.com) ;etrade.com. IN NS ;; ANSWER SECTION: etrade.com. 3212 IN NS dnsauth2.sys.gtei.net. etrade.com. 3212 IN NS dnsauth1.sys.gtei.net. etrade.com. 3212 IN NS ns1m7.etrade.com. etrade.com. 3212 IN NS ns2m7.etrade.com. etrade.com. 3212 IN NS auth40.ns.uu.net. etrade.com. 3212 IN NS ns1m4.etrade.com. etrade.com. 3212 IN NS ns2m3.etrade.com. (what's A for www.etrade.com @ns1m4.etrade.com) ;; QUESTION SECTION: ;www.etrade.com. IN A ;; AUTHORITY SECTION: www.etrade.com. 3600 IN NS gsched8.etrade.com. www.etrade.com. 3600 IN NS gsched4.etrade.com. www.etrade.com. 3600 IN NS gsched5.etrade.com. www.etrade.com. 3600 IN NS gsched7.etrade.com. sweet, now who is AAAA for www.etrade.com? ; <<>> DiG 9.4.0 <<>> AAAA @gsched5.etrade.com. www.etrade.com ; (1 server found) ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29630 ;; flags: qr aa rd; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; WARNING: Messages has 20 extra bytes at end ;; Query time: 28 msec ;; SERVER: 198.93.34.30#53(198.93.34.30) ;; WHEN: Sat Sep 27 02:42:27 2008 (or without recursion in the request: ; <<>> DiG 9.4.0 <<>> AAAA @gsched5.etrade.com. www.etrade.com +norecurse ; (1 server found) ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3362 ;; flags: qr aa; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: Messages has 20 extra bytes at end ;; Query time: 26 msec ;; SERVER: 198.93.34.30#53(198.93.34.30) ;; WHEN: Sat Sep 27 02:58:35 2008 ) what?? maybe the packet trace would help? Frame 1 (74 bytes on wire, 74 bytes captured) Arrival Time: Sep 27, 2008 03:02:52.198866000 [Time delta from previous captured frame: 0.000000000 seconds] [Time delta from previous displayed frame: 0.000000000 seconds] [Time since reference or first frame: 0.000000000 seconds] Frame Number: 1 Frame Length: 74 bytes Capture Length: 74 bytes [Frame is marked: False] [Protocols in frame: eth:ip:udp:dns] Ethernet II, Src: Intel_5c:b0:00 (00:0e:0c:5c:b0:00), Dst: Unispher_a0:3d:a5 (00:90:1a:a0:3d:a5) Destination: Unispher_a0:3d:a5 (00:90:1a:a0:3d:a5) Address: Unispher_a0:3d:a5 (00:90:1a:a0:3d:a5) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Source: Intel_5c:b0:00 (00:0e:0c:5c:b0:00) Address: Intel_5c:b0:00 (00:0e:0c:5c:b0:00) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Type: IP (0x0800) Internet Protocol, Src: 1.1.1.1 (1.1.1.1), Dst: 198.93.34.30 (198.93.34.30) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length: 60 Identification: 0x0000 (0) Flags: 0x04 (Don't Fragment) 0... = Reserved bit: Not set .1.. = Don't fragment: Set ..0. = More fragments: Not set Fragment offset: 0 Time to live: 64 Protocol: UDP (0x11) Header checksum: 0x23c3 [correct] [Good: True] [Bad : False] Source: 1.1.1.1 (1.1.1.1) Destination: 198.93.34.30 (198.93.34.30) User Datagram Protocol, Src Port: 22479 (22479), Dst Port: domain (53) Source port: 22479 (22479) Destination port: domain (53) Length: 40 Checksum: 0x1728 [incorrect, should be 0x06ba (maybe caused by "UDP checksum offload"?)] [Good Checksum: False] [Bad Checksum: True] Domain Name System (query) Transaction ID: 0xfd35 Flags: 0x0000 (Standard query) 0... .... .... .... = Response: Message is a query .000 0... .... .... = Opcode: Standard query (0) .... ..0. .... .... = Truncated: Message is not truncated .... ...0 .... .... = Recursion desired: Don't do query recursively .... .... .0.. .... = Z: reserved (0) .... .... ...0 .... = Non-authenticated data OK: Non-authenticated data is unacceptable Questions: 1 Answer RRs: 0 Authority RRs: 0 Additional RRs: 0 Queries www.etrade.com: type AAAA, class IN Name: www.etrade.com Type: AAAA (IPv6 address) Class: IN (0x0001) Frame 2 (74 bytes on wire, 74 bytes captured) Arrival Time: Sep 27, 2008 03:02:52.226523000 [Time delta from previous captured frame: 0.027657000 seconds] [Time delta from previous displayed frame: 0.027657000 seconds] [Time since reference or first frame: 0.027657000 seconds] Frame Number: 2 Frame Length: 74 bytes Capture Length: 74 bytes [Frame is marked: False] [Protocols in frame: eth:ip:udp:dns] Ethernet II, Src: Unispher_a0:3d:a5 (00:90:1a:a0:3d:a5), Dst: Intel_5c:b0:00 (00:0e:0c:5c:b0:00) Destination: Intel_5c:b0:00 (00:0e:0c:5c:b0:00) Address: Intel_5c:b0:00 (00:0e:0c:5c:b0:00) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Source: Unispher_a0:3d:a5 (00:90:1a:a0:3d:a5) Address: Unispher_a0:3d:a5 (00:90:1a:a0:3d:a5) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Type: IP (0x0800) Internet Protocol, Src: 198.93.34.30 (198.93.34.30), Dst:1.1.1.1 (1.1.1.1) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length: 60 Identification: 0x9fb6 (40886) Flags: 0x04 (Don't Fragment) 0... = Reserved bit: Not set .1.. = Don't fragment: Set ..0. = More fragments: Not set Fragment offset: 0 Time to live: 253 Protocol: UDP (0x11) Header checksum: 0xc70b [correct] [Good: True] [Bad : False] Source: 198.93.34.30 (198.93.34.30) Destination: 1.1.1.1 (1.1.1.1) User Datagram Protocol, Src Port: domain (53), Dst Port: 22479 (22479) Source port: domain (53) Destination port: 22479 (22479) Length: 40 Checksum: 0x82ba [correct] [Good Checksum: True] [Bad Checksum: False] Domain Name System (response) [Request In: 1] [Time: 0.027657000 seconds] Transaction ID: 0xfd35 Flags: 0x8400 (Standard query response, No error) 1... .... .... .... = Response: Message is a response .000 0... .... .... = Opcode: Standard query (0) .... .1.. .... .... = Authoritative: Server is an authority for domain .... ..0. .... .... = Truncated: Message is not truncated .... ...0 .... .... = Recursion desired: Don't do query recursively .... .... 0... .... = Recursion available: Server can't do recursive queries .... .... .0.. .... = Z: reserved (0) .... .... ..0. .... = Answer authenticated: Answer/authority portion was not authenticated by the server .... .... .... 0000 = Reply code: No error (0) Questions: 0 Answer RRs: 0 Authority RRs: 0 Additional RRs: 0 2 packets captured It's interesting as an aside that the LB here pushes out a TTL255 packet... Maybe the ETrade folks are also listening and could comment public/private or just fix this? :) It'd be good to see what kind of LB this is, and what version of software it is running. -Chris On Fri, Sep 26, 2008 at 10:13 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
Is there perhaps an about.com/nytimes.com admin around? I was wondering if they perhaps knew that their loadbalancer for www.nytimes.com is fairly broken wrt answering AAAA queries:
(who's NS for nytimes.com) dig NS nytimes.com +short ns1t.nytimes.com. nydns2.about.com. nydns1.about.com.
(who do they think is the NS for www.nytimes.com) dig www.nytimes.com @ns1t.nytimes.com. NS ;; QUESTION SECTION: ;www.nytimes.com. IN NS
;; AUTHORITY SECTION: www.nytimes.com. 60 IN NS nss1.sea1.nytimes.com. www.nytimes.com. 60 IN NS nss1.lga2.nytimes.com.
(what is the AAAA for www.nytimes.com ?? ) dig www.nytimes.com @nss1.sea1.nytimes.com. AAAA ;www.nytimes.com. IN AAAA
;; AUTHORITY SECTION: . 3600000 IN NS k.root-servers.net. . 3600000 IN NS l.root-servers.net. . 3600000 IN NS m.root-servers.net. . 3600000 IN NS a.root-servers.net. . 3600000 IN NS b.root-servers.net. . 3600000 IN NS c.root-servers.net. . 3600000 IN NS d.root-servers.net. . 3600000 IN NS e.root-servers.net. . 3600000 IN NS f.root-servers.net. . 3600000 IN NS g.root-servers.net. . 3600000 IN NS h.root-servers.net. . 3600000 IN NS i.root-servers.net. . 3600000 IN NS j.root-servers.net.
;; ADDITIONAL SECTION: k.root-servers.net. 3600000 IN A 193.0.14.129 l.root-servers.net. 3600000 IN A 198.32.64.12 m.root-servers.net. 3600000 IN A 202.12.27.33
;; Query time: 89 msec ;; SERVER: 170.149.172.35#53(170.149.172.35)
wha??? <ricky voice>Lucy, your loadbalancer is foobar'd</ricky voice>
In an effort to make v6 things work a tad better in this hostile world, could the NYTimes folks let us know what sort of LB that is? and why it wants to not be a good Intenet Citizen??
-Chris
Hey Chris, I'll reply to you off list. Thanks for the heads up. -rjb On 9/26/08 10:13 PM, "Christopher Morrow" <morrowc.lists@gmail.com> wrote:
Is there perhaps an about.com/nytimes.com admin around? I was wondering if they perhaps knew that their loadbalancer for www.nytimes.com is fairly broken wrt answering AAAA queries:
(who's NS for nytimes.com) dig NS nytimes.com +short ns1t.nytimes.com. nydns2.about.com. nydns1.about.com.
(who do they think is the NS for www.nytimes.com) dig www.nytimes.com @ns1t.nytimes.com. NS ;; QUESTION SECTION: ;www.nytimes.com. IN NS
;; AUTHORITY SECTION: www.nytimes.com. 60 IN NS nss1.sea1.nytimes.com. www.nytimes.com. 60 IN NS nss1.lga2.nytimes.com.
(what is the AAAA for www.nytimes.com ?? ) dig www.nytimes.com @nss1.sea1.nytimes.com. AAAA ;www.nytimes.com. IN AAAA
;; AUTHORITY SECTION: . 3600000 IN NS k.root-servers.net. . 3600000 IN NS l.root-servers.net. . 3600000 IN NS m.root-servers.net. . 3600000 IN NS a.root-servers.net. . 3600000 IN NS b.root-servers.net. . 3600000 IN NS c.root-servers.net. . 3600000 IN NS d.root-servers.net. . 3600000 IN NS e.root-servers.net. . 3600000 IN NS f.root-servers.net. . 3600000 IN NS g.root-servers.net. . 3600000 IN NS h.root-servers.net. . 3600000 IN NS i.root-servers.net. . 3600000 IN NS j.root-servers.net.
;; ADDITIONAL SECTION: k.root-servers.net. 3600000 IN A 193.0.14.129 l.root-servers.net. 3600000 IN A 198.32.64.12 m.root-servers.net. 3600000 IN A 202.12.27.33
;; Query time: 89 msec ;; SERVER: 170.149.172.35#53(170.149.172.35)
wha??? <ricky voice>Lucy, your loadbalancer is foobar'd</ricky voice>
In an effort to make v6 things work a tad better in this hostile world, could the NYTimes folks let us know what sort of LB that is? and why it wants to not be a good Intenet Citizen??
-Chris
On Sat, Sep 27, 2008 at 3:12 AM, Robert Manning <riches@about.com> wrote:
Hey Chris, I'll reply to you off list.
awesome, thanks!
Thanks for the heads up.
-rjb
On 9/26/08 10:13 PM, "Christopher Morrow" <morrowc.lists@gmail.com> wrote:
Is there perhaps an about.com/nytimes.com admin around? I was wondering if they perhaps knew that their loadbalancer for www.nytimes.com is fairly broken wrt answering AAAA queries:
(who's NS for nytimes.com) dig NS nytimes.com +short ns1t.nytimes.com. nydns2.about.com. nydns1.about.com.
(who do they think is the NS for www.nytimes.com) dig www.nytimes.com @ns1t.nytimes.com. NS ;; QUESTION SECTION: ;www.nytimes.com. IN NS
;; AUTHORITY SECTION: www.nytimes.com. 60 IN NS nss1.sea1.nytimes.com. www.nytimes.com. 60 IN NS nss1.lga2.nytimes.com.
(what is the AAAA for www.nytimes.com ?? ) dig www.nytimes.com @nss1.sea1.nytimes.com. AAAA ;www.nytimes.com. IN AAAA
;; AUTHORITY SECTION: . 3600000 IN NS k.root-servers.net. . 3600000 IN NS l.root-servers.net. . 3600000 IN NS m.root-servers.net. . 3600000 IN NS a.root-servers.net. . 3600000 IN NS b.root-servers.net. . 3600000 IN NS c.root-servers.net. . 3600000 IN NS d.root-servers.net. . 3600000 IN NS e.root-servers.net. . 3600000 IN NS f.root-servers.net. . 3600000 IN NS g.root-servers.net. . 3600000 IN NS h.root-servers.net. . 3600000 IN NS i.root-servers.net. . 3600000 IN NS j.root-servers.net.
;; ADDITIONAL SECTION: k.root-servers.net. 3600000 IN A 193.0.14.129 l.root-servers.net. 3600000 IN A 198.32.64.12 m.root-servers.net. 3600000 IN A 202.12.27.33
;; Query time: 89 msec ;; SERVER: 170.149.172.35#53(170.149.172.35)
wha??? <ricky voice>Lucy, your loadbalancer is foobar'd</ricky voice>
In an effort to make v6 things work a tad better in this hostile world, could the NYTimes folks let us know what sort of LB that is? and why it wants to not be a good Intenet Citizen??
-Chris
I worked with Chris on this outside of the list. Replying here just to close the loop in case anyone else was interested. This situation is explained in this Case Study: http://support.citrix.com/article/CTX117947 The key sentence being: "In NetScaler software release 7.0, when the DNS server looks up AAAA records, the response was “0” and errors “0”. However, in NetScaler software release 8.0, with standard response “0”, the NetScaler appliance sends the delegation records to root. " To summarize, if you don't have your NS records in place on the Netscalers, you will see a loop for AAAA queries (root>auth>netscaler>root....), eventually resulting in a SERVFAIL. Christopher Morrow wrote:
On Sat, Sep 27, 2008 at 3:12 AM, Robert Manning <riches@about.com> wrote:
Hey Chris, I'll reply to you off list.
awesome, thanks!
Thanks for the heads up.
-rjb
On 9/26/08 10:13 PM, "Christopher Morrow" <morrowc.lists@gmail.com> wrote:
Is there perhaps an about.com/nytimes.com admin around? I was wondering if they perhaps knew that their loadbalancer for www.nytimes.com is fairly broken wrt answering AAAA queries:
(who's NS for nytimes.com) dig NS nytimes.com +short ns1t.nytimes.com. nydns2.about.com. nydns1.about.com.
(who do they think is the NS for www.nytimes.com) dig www.nytimes.com @ns1t.nytimes.com. NS ;; QUESTION SECTION: ;www.nytimes.com. IN NS
;; AUTHORITY SECTION: www.nytimes.com. 60 IN NS nss1.sea1.nytimes.com. www.nytimes.com. 60 IN NS nss1.lga2.nytimes.com.
(what is the AAAA for www.nytimes.com ?? ) dig www.nytimes.com @nss1.sea1.nytimes.com. AAAA ;www.nytimes.com. IN AAAA
;; AUTHORITY SECTION: . 3600000 IN NS k.root-servers.net. . 3600000 IN NS l.root-servers.net. . 3600000 IN NS m.root-servers.net. . 3600000 IN NS a.root-servers.net. . 3600000 IN NS b.root-servers.net. . 3600000 IN NS c.root-servers.net. . 3600000 IN NS d.root-servers.net. . 3600000 IN NS e.root-servers.net. . 3600000 IN NS f.root-servers.net. . 3600000 IN NS g.root-servers.net. . 3600000 IN NS h.root-servers.net. . 3600000 IN NS i.root-servers.net. . 3600000 IN NS j.root-servers.net.
;; ADDITIONAL SECTION: k.root-servers.net. 3600000 IN A 193.0.14.129 l.root-servers.net. 3600000 IN A 198.32.64.12 m.root-servers.net. 3600000 IN A 202.12.27.33
;; Query time: 89 msec ;; SERVER: 170.149.172.35#53(170.149.172.35)
wha??? <ricky voice>Lucy, your loadbalancer is foobar'd</ricky voice>
In an effort to make v6 things work a tad better in this hostile world, could the NYTimes folks let us know what sort of LB that is? and why it wants to not be a good Intenet Citizen??
-Chris
-- -Brendan Cleary Senior Network Engineer NYTIMES.COM 212.556.8041
On Tue, Sep 30, 2008 at 5:04 PM, Brendan Cleary <cleary@nytimes.com> wrote:
I worked with Chris on this outside of the list. Replying here just to close the loop in case anyone else was interested.
This situation is explained in this Case Study: http://support.citrix.com/article/CTX117947
The key sentence being: "In NetScaler software release 7.0, when the DNS server looks up AAAA records, the response was "0" and errors "0". However, in NetScaler software release 8.0, with standard response "0", the NetScaler appliance sends the delegation records to root. "
To summarize, if you don't have your NS records in place on the Netscalers, you will see a loop for AAAA queries (root>auth>netscaler>root....), eventually resulting in a SERVFAIL.
Thanks Brendan! Hopefully Citrix can improve their standard config for this sort of deployment to make this a little simpler? I can't believe NYTimes is the only user of Netscalers for this function. -Chris
participants (4)
-
Brendan Cleary
-
Christopher Morrow
-
Florian Weimer
-
Robert Manning