Facebook more specific via Level3 ?
Hi, is anyone else receiving Facebooks /24 more specific from Level3 (AS3356)? 31.13.70.0/24 *[BGP/170] 1w5d 17:21:28, MED 0, localpref 100, from a.b.c.d AS path: 3356 32934 I, validation-state: unverified This more specific is only visible via AS3356 Facebook isnt even announcing it via direct peering. Any Facebook admin on or off list available to get this debugged and tracked down? Thanks & best regards Jürgen
I see that specific route both of my upstreams and not going through level 3. Network Next Hop MED LocPrf Weight Path *>x 31.13.70.0/24 x.x.x.x 0 80 0 6461 32934 i *i 31.13.70.0/24 x.x.x.x 0 80 0 209 32934 i * 31.13.70.0/24 x.x.x.x 10 80 0 209 32934 i On Tue, Mar 21, 2017 at 12:56 PM, Jürgen Jaritsch <juergen@jaritsch.at> wrote:
Hi,
is anyone else receiving Facebook’s /24 more specific from Level3 (AS3356)?
31.13.70.0/24 *[BGP/170] 1w5d 17:21:28, MED 0, localpref 100, from a.b.c.d
AS path: 3356 32934 I, validation-state: unverified
This more specific is only visible via AS3356 … Facebook isn’t even announcing it via direct peering.
Any Facebook admin on or off list available to get this debugged and tracked down?
Thanks & best regards
Jürgen
31.13.70.0/24
This more specific is only visible via AS3356 ... Facebook isn’t even announcing it via direct peering.
This specific, and many others, are only announced to peers in the metro they originate in. To receive this prefix directly you'll need to peer with us in Los Angeles. It appears you're in Austria though, which means there's likely no use in you peering with us in LA. We globally load balance people to the best point of presence. You shouldn't see much, if any, traffic to or from the above prefix.
Any Facebook admin on or off list available to get this debugged and tracked down?
Please reach out to noc@fb.com in the future. Regards, -- dsp
Hi Doug, looks like this is also affecting other prefixes: 157.240.3.0/24 *[BGP/170] 18w5d 16:50:37, MED 0, localpref 150 AS path: 3356 32934 I, validation-state: unverified > to 80.239.128.178 via ae9.0 I understand that FB is using some type of DNS geo-loadbalancing and other mechanism to redirect users to (possibly) nearer mirrors. The used DNS is directly requesting the root DNS and not any other public DNS (e.g. not 8.8.8.8). Running some requests within 3 minutes gives me the below results: www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36 www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68 Trace to 31.13.77.36: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev [...] 6. win-b4-link.telia.net 0.0% 3 21.5 27.5 21.5 35.7 7.3 7. level-ic-1573273-wien-b4.c.telia.net 0.0% 3 21.7 20.8 20.0 21.7 0.8 8. ae-1-9.edge2.sanjose3.level3.net 0.0% 3 177.1 179.8 177.1 182.6 2.8 9. 4.53.210.78 0.0% 3 178.3 178.8 178.3 179.2 0.5 10. po131.asw04.sjc1.tfbnw.net 0.0% 3 184.6 180.0 177.6 184.6 4.0 11. po241.psw02.sjc2.tfbnw.net 0.0% 3 177.8 177.6 177.3 177.8 0.3 12. 173.252.67.79 0.0% 3 176.3 177.4 176.3 177.9 0.9 13. edge-star-mini-shv-01-sjc2.facebook.com 0.0% 3 177.6 177.4 177.1 177.6 0.3 Trace to 157.240.2.35: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev [...] 6. win-b4-link.telia.net 0.0% 3 20.9 25.8 20.8 35.8 8.6 7. level-ic-1573273-wien-b4.c.telia.net 0.0% 3 20.5 21.4 20.2 23.4 1.8 8. ??? 9. 4.15.85.102 0.0% 3 142.9 138.5 135.8 142.9 3.8 10. po121.asw01.ord3.tfbnw.net 0.0% 3 135.5 135.7 135.5 136.0 0.4 11. po211.psw01c.ort2.tfbnw.net 0.0% 2 136.1 135.7 135.3 136.1 0.6 12. 173.252.67.127 0.0% 2 151.6 147.6 143.6 151.6 5.7 13. edge-star-mini-shv-01-ort2.facebook.com 0.0% 2 136.0 135.9 135.8 136.0 0.2 Trace to 31.13.93.36 => this one is going via the FB DE-CIX peering and latency of 34ms is perfect: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. router 0.0% 5 0.5 0.5 0.5 0.6 0.0 2. 192.168.8.3 0.0% 4 0.7 0.7 0.7 0.8 0.1 3. 37.252.236.88 0.0% 4 37.8 38.0 31.8 46.1 6.0 4. er-03.00-09-23.anx04.vie.at.anexia-it.com 0.0% 4 25.3 25.2 23.1 29.2 2.9 5. cr-02.0v-08-72.anx03.vie.at.anexia-it.com 0.0% 4 21.3 30.4 20.6 40.1 10.9 6. cr-01.0v-08-73.anx25.fra.de.anexia-it.com 0.0% 4 45.7 39.1 33.1 45.7 5.3 7. ae1.br02.fra1.tfbnw.net 0.0% 4 33.0 34.6 32.1 40.3 3.8 8. po114.asw01.fra2.tfbnw.net 0.0% 4 33.6 35.6 33.3 41.7 4.0 9. po211.psw01d.fra3.tfbnw.net 0.0% 4 40.8 36.6 32.9 40.8 3.4 10. 173.252.67.171 0.0% 4 41.2 35.6 33.3 41.2 3.7 11. edge-star-mini-shv-01-fra3.facebook.com 0.0% 4 34.6 33.9 33.2 34.6 0.6 Trace to 31.13.76.68: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev [...] 6. win-b4-link.telia.net 0.0% 14 28.9 24.7 20.3 29.3 3.5 7. level-ic-1573273-wien-b4.c.telia.net 0.0% 14 22.8 22.5 20.0 30.9 3.7 8. ae-2-3613.edge1.seattle3.level3.net 0.0% 13 195.2 189.2 186.0 195.2 3.5 9. 4.59.232.46 0.0% 13 187.9 188.2 186.7 196.1 2.5 10. po101.psw03.sea1.tfbnw.net 0.0% 13 194.7 188.3 186.5 194.7 2.0 11. 173.252.67.125 0.0% 13 188.0 188.8 186.9 195.5 2.2 12. edge-star-mini-shv-01-sea1.facebook.com 0.0% 13 186.6 189.0 186.2 196.0 3.1 Best regards Jürgen -----Ursprüngliche Nachricht----- Von: Doug Porter [mailto:dsp@fb.com] Gesendet: Dienstag, 21. März 2017 18:41 An: Jürgen Jaritsch <juergen@jaritsch.at>; nanog@nanog.org Betreff: Re: Facebook more specific via Level3 ?
31.13.70.0/24
This more specific is only visible via AS3356 ... Facebook isnt even announcing it via direct peering.
This specific, and many others, are only announced to peers in the metro they originate in. To receive this prefix directly you'll need to peer with us in Los Angeles. It appears you're in Austria though, which means there's likely no use in you peering with us in LA. We globally load balance people to the best point of presence. You shouldn't see much, if any, traffic to or from the above prefix.
Any Facebook admin on or off list available to get this debugged and tracked down?
Please reach out to noc@fb.com in the future. Regards, -- dsp
looks like this is also affecting other prefixes:
Many of our prefixes are only announced to peers in the metro they originate in. Please stop obsessing about this detail; it's not the problem.
I understand that FB is using some type of DNS geo-loadbalancing and other mechanism to redirect users to (possibly) nearer mirrors.
We target traffic two ways. One is relatively traditional dns global load balancing, using the resolver ip. The other method---which steers the vast majority of our traffic---vends urls that send people to a specific PoP based on their client ip. It appears you're having a targeting problem. Please reach out to our NOC to get support. -- dsp
On Tue, Mar 21, 2017, at 20:38, Jürgen Jaritsch wrote:
I understand that FB is using some type of DNS geo-loadbalancing and other mechanism to redirect users to (possibly) nearer mirrors. The used DNS is directly requesting the root DNS and not any other public DNS (e.g. not 8.8.8.8). Running some requests within 3 minutes gives me the below results:
www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36 www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68
Hi, the load-balancing definitely doesn't choose the *nearest* mirror. We are in France and unless we do dirty tricks, we *always* get directed to US sites (as far as LA), with horrible performance. Everything since end of December. As a consequence we let the dirty tricks in place (query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive resolving) and we get directed to Frankfurt or Amsterdam (never London or Paris).
Hi,
(query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive resolving) and we get directed to Frankfurt or Amsterdam (never London or Paris).
This is exactly what I've implemented yesterday on my end :). Best regards Jürgen -----Ursprüngliche Nachricht----- Von: Radu-Adrian Feurdean [mailto:nanog@radu-adrian.feurdean.net] Gesendet: Mittwoch, 22. März 2017 11:02 An: Jürgen Jaritsch <juergen@jaritsch.at>; Doug Porter <dsp@fb.com>; nanog@nanog.org Betreff: Re: Facebook more specific via Level3 ? On Tue, Mar 21, 2017, at 20:38, Jürgen Jaritsch wrote:
I understand that FB is using some type of DNS geo-loadbalancing and other mechanism to redirect users to (possibly) nearer mirrors. The used DNS is directly requesting the root DNS and not any other public DNS (e.g. not 8.8.8.8). Running some requests within 3 minutes gives me the below results:
www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36 www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68
Hi, the load-balancing definitely doesn't choose the *nearest* mirror. We are in France and unless we do dirty tricks, we *always* get directed to US sites (as far as LA), with horrible performance. Everything since end of December. As a consequence we let the dirty tricks in place (query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive resolving) and we get directed to Frankfurt or Amsterdam (never London or Paris).
Are your DNS resolvers on your network? No DNS forwarding? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Radu-Adrian Feurdean" <nanog@radu-adrian.feurdean.net> To: "Jürgen Jaritsch" <juergen@jaritsch.at>, "Doug Porter" <dsp@fb.com>, nanog@nanog.org Sent: Wednesday, March 22, 2017 5:02:12 AM Subject: Re: Facebook more specific via Level3 ? On Tue, Mar 21, 2017, at 20:38, Jürgen Jaritsch wrote:
I understand that FB is using some type of DNS geo-loadbalancing and other mechanism to redirect users to (possibly) nearer mirrors. The used DNS is directly requesting the root DNS and not any other public DNS (e.g. not 8.8.8.8). Running some requests within 3 minutes gives me the below results:
www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36 www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36 www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68
Hi, the load-balancing definitely doesn't choose the *nearest* mirror. We are in France and unless we do dirty tricks, we *always* get directed to US sites (as far as LA), with horrible performance. Everything since end of December. As a consequence we let the dirty tricks in place (query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive resolving) and we get directed to Frankfurt or Amsterdam (never London or Paris).
Hi, got the same from Kiev, Ukraine: dig fbcdn.com fbcdn.com. 300 IN A 31.13.74.1 which is slow and routed through USA and dig fbcdn.com @8.8.8.8 fbcdn.com. 299 IN A 31.13.93.3 which is fast and routed through Germany Same is for IPv6. Is there any solutions without dirty hacks? On 22.03.17 12:02, Radu-Adrian Feurdean wrote:
Hi, the load-balancing definitely doesn't choose the *nearest* mirror. We are in France and unless we do dirty tricks, we *always* get directed to US sites (as far as LA), with horrible performance. Everything since end of December. As a consequence we let the dirty tricks in place (query facebook.com and fbcdn.com on 8.8.8.8 instead of regular recursive resolving) and we get directed to Frankfurt or Amsterdam (never London or Paris).
On Sun, Apr 16, 2017 at 04:20:20PM +0300, Max Tulyev wrote:
got the same from Kiev, Ukraine:
dig fbcdn.com fbcdn.com. 300 IN A 31.13.74.1 which is slow and routed through USA
and dig fbcdn.com @8.8.8.8 fbcdn.com. 299 IN A 31.13.93.3 which is fast and routed through Germany
Same is for IPv6.
Is there any solutions without dirty hacks?
As mentioned in this thread, talk to the Facebook NOC: On Tue, Mar 21, 2017 at 05:41:23PM +0000, Doug Porter wrote: > > Please reach out to noc@fb.com in the future." > Kind regards, Job
If John is on the list, please have him contact me. Thanks. Regards, Roderick. Sales Cross Lake SeaX-1 and SeaX-2
Hi.... john@vanoppen.com if you need me or anything from 11404 in general. -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rod Beck Sent: Sunday, April 16, 2017 9:40 AM To: nanog@nanog.org Subject: John Van Oppen - Wave Broadband If John is on the list, please have him contact me. Thanks. Regards, Roderick. Sales Cross Lake SeaX-1 and SeaX-2
participants (9)
-
Doug Porter -
Jay Nakamura -
Job Snijders -
John van Oppen -
Jürgen Jaritsch -
Max Tulyev -
Mike Hammett -
Radu-Adrian Feurdean -
Rod Beck