Yes, we have our own Bind9 caching servers in different geographic locations, direct on fiber using SSD drives... Seem very quick when using GRC's DNS Benchmark tests. Eric Rogers PDS Connect (317) 831-3000 x200 -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mike Hammett Sent: Saturday, April 2, 2016 2:01 PM Cc: nanog@nanog.org Subject: Re: Someone Please Help Me Understand Are you using locally resolving DNS servers? I don't know how FB determines where your content comes from, but some CDNs test the performance to your resolving DNS server. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Eric Rogers" <ecrogers@precisionds.com> To: nanog@nanog.org Sent: Saturday, April 2, 2016 12:54:40 PM Subject: Someone Please Help Me Understand Ok, I'm trying to learn, so bear with me. We are an ISP in Indianapolis that has full routes from 3 different providers HE.Net in Columbus OH being one. We also are peered with 2 peering exchanges, including EquinixIX in Chicago. The problem is Instagram and Facebook (same company, I know) for our customers seems very slow. This is where I need a way to troubleshoot/understand more. I did a traceroute to the IP that is serving the pictures, and it resolves to the FBCDN servers in Dallas, and is showing packet loss and pings once it hits Dallas, and are in the 1xxs of ms. Tracing route to instagram-p3-shv-01-dfw1.fbcdn.net [31.13.66.52] over a maximum of 30 hops: 1 4 ms 3 ms 4 ms 10.7.0.1 2 20 ms 43 ms 42 ms inmtvlobs-rtr-01.dynamic.pdsconnect.me [192.69.57.1] 3 25 ms 47 ms 29 ms inmtvlmwt-rtr-01.infrastructure.pdsconnect.me [192.69.48.162] 4 46 ms 32 ms 58 ms inindyhen-core1.infrastructure.pdsconnect.me [192.69.48.193] 5 36 ms 53 ms 51 ms ge2-4.core1.cmh1.he.net [184.105.32.1] 6 47 ms 41 ms 75 ms 10ge1-2.core1.chi1.he.net [184.105.222.165] 7 57 ms 57 ms 53 ms 100ge14-1.core2.chi1.he.net [184.105.81.97] 8 57 ms 73 ms 84 ms 100ge12-1.core1.mci3.he.net [184.105.81.209] 9 75 ms 73 ms 102 ms 10ge15-6.core1.dal1.he.net [184.105.222.10] 10 93 ms 103 ms 92 ms eqix-da1.facebook.com [206.223.118.176] 11 102 ms 101 ms * psw01c.dfw1.tfbnw.net [173.252.65.196] 12 92 ms 97 ms 105 ms msw1aq.01.dfw1.tfbnw.net [204.15.21.89] 13 110 ms * 98 ms instagram-p3-shv-01-dfw1.fbcdn.net [31.13.66.52] Since I am peered with the route servers in EquinixIX Chicago, shouldn't the data be coming from there, or at least hit their routers? In my trace, it shows HE to Chicago, then to Dallas. How does FB decide what IP the content gets displayed from, and is there anything I can do as a provider? If it is DNS, I can obviously clear the cache to see if it gets new IPs. If I'm not getting FB peering IPs in Chicago, do I need to peer directly? Should I get FaceBook involved? Eric Rogers PDS Connect (317) 831-3000 x200