Here is what I think would happen if you were to try this setup. Let's assume you deployed in eu-west-2 (London) and eu-central-1 (Frankfurt). You would find that you could successfully connect to a number of networks but also that some of them would work from the "wrong" site. Eg. you would have clients in London that require you to use the Frankfurt instance to connect to. You would also find a lot of networks that you could not connect to from either site. And you would have some instability where some networks at random switch between these states. For example you could have a client that one day works from London, the next day it is Frankfurt and then someday it won't be working at all.

As you add more sites, you also add more ways for the traffic to end up in the wrong places. You could have something that works with just the two sites above but then when you add eu-west-1 (Ireland) you could suddenly not connect to them from any of the three sites.

There IS a possible solution which is to tunnel the traffic back to the correct site. This still requires you to use unique IP addresses for each site, but all addresses could be in the same subnet. You would have IP a.b.c.1 to be London, a.b.c.2 Frankfurt, a.b.c.3 Ireland. Then if London receives traffic to a.b.c.2 you would have a tunnel to send the traffic to Frankfurt.

Regards,

Baldur


On Wed, Jul 28, 2021 at 11:07 AM Baldur Norddahl <baldur.norddahl@gmail.com> wrote:


> On Jul 27, 2021, at 17:20, Vimal <j.vimal@gmail.com> wrote:
> Yes, this makes sense as the destination can be anywhere around the world, and that routing is asymmetric as others mentioned.  However, if the destination service is "close" (in the routing metric sense) to the initiating host, anycast return IP ought to work well, right?  I understand this is a very important caveat and impractical to implement correctly in the real world.

No, there is no such thing as "close". You could have a direct peering with some ISP and have them still deliver the responses on the other side of earth. You do not control the routing of other networks and can not be sure what they will do.

For larger networks you may also have multiple peering points. Say you have a peering with them in city A and city B. How do you know which of their IP ranges are closer to A or B? You don't. And the same goes for them, they have no idea if you prefer A or B. Therefore you could select A and they may reply to B. They may even load balance between A and B if you are really unlucky.

Routing is asymmetric. That means you have absolutely no idea where the replies end up going. Often it will not be what you think is "close".

I do not run anycast, but I understand that the usual way of dealing with these issues is to do as little as possible with anycast before redirecting to a unicast address. For example you could have just your DNS on anycast and each site would reply with unique unicast addresses. Since DNS is just a single pair of UDP request/response, with the first packet originating from a unicast client, this works well.

Regards,

Baldur