Google's GSLB gone wrong: Southern Europe goes Middle East (UAE / lcmcta)
Hello, here's an interesting case of a global load balancing SNAFU: Google DNS-load-balances traffic based on the customers /24 between the available PoPs for the specific geolocation area. *SOME* those /24s at *ALL* ISPs in Southern Europe are going to the Middle East, more specifically the UAE (PTR prefix is "lcmcta"). So the the customers in Southern Europe hit Google Frontends on another continent, obviously with a latency well above 100ms. It appears all of Southern Europe is impacted, I have found this to be the case for at least: - Italy - Croatia - Hungary - Serbia - Romania - Bulgaria - Albania - Greece - Malta This affects websites like google.com, mail.google.com and youtube.com (although most likely not the video streaming itself though). The impact due to the latency and traffic is probably not extreme; AFAIK it does not affect Youtube stream itself (just the main website). It has been going on for probably a week or more, Google NOC Ticket 2588682096 is open since March 10th, 2176711238 since March 11th, but no luck so far in those tickets. The balancing occurs via the authoritative DNS servers at Google, looking either at the EDNS client subnet (also used with 8.8.8.8 but not 1.1.1.1), or without ENDS at the source IP of the DNS request. Thanks to EDNS client subnet this can be easily simulated; in the following example we see that for clients in 2.112.4.0/24 (Telecom Italia), the Google Frontend IP is 142.250.186.238 which is a unicast Google node in the United Arab Emirates (PTR prefix is "lcmcta"): $ dig +short +subnet=2.112.4.0/24 -tA @ns1.google.com. google.com. 142.250.186.238 $ dig +short -x 142.250.186.238 lcmcta-af-in-f14.1e100.net. $ More data from 6 large networks in Italy is available here, but again it affects every single ISP in the multiple countries: https://github.com/lukastribus/ggc-italy-middle-east-routing The most likely reason for this I believe is a misconfigured GGC cluster or Edge PoP in the United Arab Emirates - the error being that it is erroneously set to serve content to clients in Southern Europe as well a Middle East. If I were to speculate even further, this may be have been an emergency response related to the ongoing attacks in the region. If someone would happen to have a contact that can escalate the aforementioned Google Tickets 2588682096 and 2176711238 to the correct teams, that would be very much appreciated. I hope this was an interesting read :) cheers, lukas
On 12/03/2026 22:12, Lukas Tribus via NANOG wrote: I do not see a site in UAE: https://gcping.com/ Doha and Dammam but no UAE. Perhaps gcping hasn't been updated with the UAE site. Don't know. -Hank
Hello,
here's an interesting case of a global load balancing SNAFU:
Google DNS-load-balances traffic based on the customers /24 between the available PoPs for the specific geolocation area. *SOME* those /24s at *ALL* ISPs in Southern Europe are going to the Middle East, more specifically the UAE (PTR prefix is "lcmcta"). So the the customers in Southern Europe hit Google Frontends on another continent, obviously with a latency well above 100ms.
It appears all of Southern Europe is impacted, I have found this to be the case for at least:
- Italy - Croatia - Hungary - Serbia - Romania - Bulgaria - Albania - Greece - Malta
This affects websites like google.com, mail.google.com and youtube.com (although most likely not the video streaming itself though). The impact due to the latency and traffic is probably not extreme; AFAIK it does not affect Youtube stream itself (just the main website).
It has been going on for probably a week or more, Google NOC Ticket 2588682096 is open since March 10th, 2176711238 since March 11th, but no luck so far in those tickets.
The balancing occurs via the authoritative DNS servers at Google, looking either at the EDNS client subnet (also used with 8.8.8.8 but not 1.1.1.1), or without ENDS at the source IP of the DNS request.
Thanks to EDNS client subnet this can be easily simulated; in the following example we see that for clients in 2.112.4.0/24 (Telecom Italia), the Google Frontend IP is 142.250.186.238 which is a unicast Google node in the United Arab Emirates (PTR prefix is "lcmcta"):
$ dig +short +subnet=2.112.4.0/24 -tA @ns1.google.com. google.com. 142.250.186.238 $ dig +short -x 142.250.186.238 lcmcta-af-in-f14.1e100.net. $
More data from 6 large networks in Italy is available here, but again it affects every single ISP in the multiple countries:
https://github.com/lukastribus/ggc-italy-middle-east-routing
The most likely reason for this I believe is a misconfigured GGC cluster or Edge PoP in the United Arab Emirates - the error being that it is erroneously set to serve content to clients in Southern Europe as well a Middle East. If I were to speculate even further, this may be have been an emergency response related to the ongoing attacks in the region.
If someone would happen to have a contact that can escalate the aforementioned Google Tickets 2588682096 and 2176711238 to the correct teams, that would be very much appreciated.
I hope this was an interesting read :)
cheers, lukas _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/3EJUOZJK...
Hi Hank, On Fri, 13 Mar 2026 at 07:11, Hank Nussbacher via NANOG <nanog@lists.nanog.org> wrote:
On 12/03/2026 22:12, Lukas Tribus via NANOG wrote:
I do not see a site in UAE: https://gcping.com/ Doha and Dammam but no UAE. Perhaps gcping hasn't been updated with the UAE site. Don't know.
It may also be Doha, the only way I arrived at the UAE conclusion is RIPE Atlas measurements, and there is a Atlas probe 3 ms away within the UAE: https://atlas.ripe.net/measurements/158843055/ But gcping is about Google Cloud regions, which is not necessarily the same as Google Edge Locations and certainly not Google Global Cache deployments. Lukas
Hi Lukas, all, Thanks for reporting this, and sorry for your trouble in getting it to the appropriate team inside Google. I think we have everything we need to diagnose this, and I've done as you requested: the new "magic number" is in the bug system: 492415305 (those two NOC cases are linked to it). Best Regards, Phil Sykes (day job: Google Networking SRE) On Fri, Mar 13, 2026 at 8:36 AM Lukas Tribus via NANOG < nanog@lists.nanog.org> wrote:
Hi Hank,
On Fri, 13 Mar 2026 at 07:11, Hank Nussbacher via NANOG <nanog@lists.nanog.org> wrote:
On 12/03/2026 22:12, Lukas Tribus via NANOG wrote:
I do not see a site in UAE: https://gcping.com/ Doha and Dammam but no UAE. Perhaps gcping hasn't been updated with the UAE site. Don't know.
It may also be Doha, the only way I arrived at the UAE conclusion is RIPE Atlas measurements, and there is a Atlas probe 3 ms away within the UAE: https://atlas.ripe.net/measurements/158843055/
But gcping is about Google Cloud regions, which is not necessarily the same as Google Edge Locations and certainly not Google Global Cache deployments.
Lukas _______________________________________________ NANOG mailing list
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/TS463KW6...
Hi Phil, On Friday, 13 March 2026, Phil Sykes via NANOG <nanog@lists.nanog.org> wrote:
Hi Lukas, all,
Thanks for reporting this, and sorry for your trouble in getting it to the appropriate team inside Google.
I think we have everything we need to diagnose this, and I've done as you requested: the new "magic number" is in the bug system: 492415305 (those two NOC cases are linked to it).
Thank you, thats very much appreciated. Best regards, Lukas
participants (3)
-
Hank Nussbacher -
Lukas Tribus -
Phil Sykes