Hi, I'm working on some DR design and we want to not only have this site as a DR but also performing some active/active for some of the services we hosts and I was wondering if someone had some experience with using anycast for TFTP or DHCP services? What are some of the pains/challenges you experienced and things we should lookout for? Any input is greatly appreciated. Kind regards, Javier Gutierrez
On Thu, Feb 22, 2024 at 10:47 AM Javier Gutierrez <GutierrezJ@westmancom.com> wrote:
I was wondering if someone had some experience with using anycast for TFTP or DHCP services?
Hi Javier, Anycast for TFTP is more or less the same as anycast for TCP-based protocols: it has corner cases which fail and fail hard, but otherwise it works. Outside the corner cases, the failure mode for tftp clients should be the same as the failure mode when the tftp server goes down in the middle of a transfer and comes back up some time later. The corner cases are variations on the theme that your routing causes packets from a particular source to oscillate between the tftp servers. In the corner cases, the tftp client can't communicate with the -same- tftp server long enough to complete a transfer. Experiments with anycast TCP suggest that the corner cases happen for less than 1% of client sources, but when they do happen tend to be persistent, affecting all communication between that client and the anycast IP address for an extended duration, sometimes weeks or months. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
I do NTP, DHCP, TFTP, DNS, HTTP anycast. NTP, DNS and HTTP with ECMP, TFTP and DHCP as active/active on a per Datacenter Basis. These are small Datacenters with less than 50k Servers each. In every datacenter an anycast node is active and the router just chooses the shortest path. It becomes tricky for DHCP if a location has the same cost to more than one anycast Node. For this case we have setup a DHCP nodes in two datacenters using different local-preferences to simulate a failover active/passive setup. Cheers Thomas Am 22.02.24 um 19:47 schrieb Javier Gutierrez:
Hi, I'm working on some DR design and we want to not only have this site as a DR but also performing some active/active for some of the services we hosts and I was wondering if someone had some experience with using anycast for TFTP or DHCP services? What are some of the pains/challenges you experienced and things we should lookout for?
Any input is greatly appreciated.
Kind regards,
*Javier Gutierrez*
On Feb 22, 2024, at 12:52, Thomas Mieslinger <miesi@mail.com> wrote:
It becomes tricky for DHCP if a location has the same cost to more than one anycast Node. For this case we have setup a DHCP nodes in two datacenters using different local-preferences to simulate a failover active/passive setup.
The relay server `dhcplb` could, maybe, help in that scenario (dhcplb runs on the anycast IP, the “real” DHCP servers on unicast IPs behind dhcplb). https://github.com/facebookincubator/dhcplb Ask
On Fri, Feb 23, 2024 at 6:34 PM Ask Bjørn Hansen <ask@develooper.com> wrote:
The relay server `dhcplb` could, maybe, help in that scenario (dhcplb runs on the anycast IP, the “real” DHCP servers on unicast IPs behind dhcplb).
Although they used the word "anycast", they're just load balancing. Devices behind a load balancer are not "anycast," since the load balancer explicitly decides which machine gets which transaction. Even with clever load balancers like Linux Virtual Server in "routing" mode where the back-end servers all share the virtual IP address, that's load balancing, not anycast routing. An IP is not "anycast" unless it moves via anycast routing. Anycast routing means it's announced into the _routing protocol_ from multiple sources on behalf of multiple distinct machines. In their readme, they comment that their load balancer replaced attempts to use anycast routing with equal cost multipath. That makes good sense. Relying on ECMP for anycasted DHCP would be a disaster during any sort of failure. Add or remove a single route from an ECMP set and the hashed path selection changes for most of the connections. All the DHCP renewals would very suddenly be going to the wrong DHCP server. Where anycast works, it works because ECMP only rarely comes into play. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
On Feb 23, 2024, at 20:32, William Herrin <bill@herrin.us> wrote:
The relay server `dhcplb` could, maybe, help in that scenario (dhcplb runs on the anycast IP, the “real” DHCP servers on unicast IPs behind dhcplb).
Although they used the word "anycast", they're just load balancing.
The idea is to run the relays on an anycasted IP (so the load balancer / relay IP is anycasted).
[….] Relying on ECMP for anycasted DHCP would be a disaster during any sort of failure. Add or remove a single route from an ECMP set and the hashed path selection changes for most of the connections.
Consistent hashing (which I thought was widely supported now in ECMP implementations) and a bit of automation in how announcements are added can greatly mitigate this. Ask
The system Ask is describing is the traditional method of using anycast to geographically load-balance long-lived flows. The first time I did that was with FTP servers in Berkeley and Santa Cruz, in 1989. I did a bigger system, also load balancing FTP servers for Oracle, their public-facing documentation stores, with servers in San Jose and Washington DC, a couple of years later. A couple of years further on and the World Wide Web was a thing, and everybody was doing it. -Bill
On Feb 24, 2024, at 7:38 AM, Ask Bjørn Hansen <ask@develooper.com> wrote:
On Feb 23, 2024, at 20:32, William Herrin <bill@herrin.us> wrote:
The relay server `dhcplb` could, maybe, help in that scenario (dhcplb runs on the anycast IP, the “real” DHCP servers on unicast IPs behind dhcplb).
Although they used the word "anycast", they're just load balancing.
The idea is to run the relays on an anycasted IP (so the load balancer / relay IP is anycasted).
[….] Relying on ECMP for anycasted DHCP would be a disaster during any sort of failure. Add or remove a single route from an ECMP set and the hashed path selection changes for most of the connections.
Consistent hashing (which I thought was widely supported now in ECMP implementations) and a bit of automation in how announcements are added can greatly mitigate this.
Ask
Thanks to you all for your answers, it has helped me a lot already. My design is very simplistic, I have 2 sets of firewalls that I will have advertising a /32 unicast to the network at each location and it will have a TFTP server behind each firewall. I have no intention to have this be part of the internet as it will be used to serve internal customers devices that require TFTP For the setup where you are running Anycast on a datacenter, are you running it inside the datacenter only or across multiple datacenters? other than having to replicate IPs and file services between datacenters have you seen any other issues? Kind regards, Javier Gutierrez, Network Architect – AS19016 https://www.peeringdb.com/net/4073 Westman Communications Group [cid:2db642a4-fcf9-40b4-a719-2afd8097f2e9]1906 Park Ave. • Brandon, MB • R7B 0R9 [cid:8862c057-cdef-45f6-a0e3-497508d0d64a]204.720.1158 [cid:6a35147d-b3b0-44cf-bc96-6822377f5231] gutierrezj@westmancom.com<mailto:gutierrezj@westmancom.com> [A close up of a sign Description automatically generated]<https://westmancom.com/personal> [cid:486e0290-5d40-48dd-80eb-3be9a705b1e6]<https://www.facebook.com/WestmanCom>[cid:425d7b57-d7e3-491d-9d22-910d4072b88a]<https://twitter.com/WestmanCom> [cid:ee77dd48-8761-498b-b45b-82b00e5bf553] <https://www.youtube.com/user/WestmanCom> [cid:547ce68d-d61c-40e3-b150-39bff72b8d6b] <https://www.instagram.com/westmancom> [cid:ba4751b3-edc0-484e-bb40-731ca94e8c84] <https://www.linkedin.com/company/westmancom> This e-mail and any attachments contain confidential and privileged information. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this e-mail and destroy any copies. Any dissemination or use of this information by a person other than intended recipient is unauthorized and may be illegal. ________________________________ From: NANOG <nanog-bounces+gutierrezj=westmancom.com@nanog.org> on behalf of Bill Woodcock <woody@pch.net> Sent: Saturday, February 24, 2024 1:09 AM To: Ask Bjørn Hansen <ask@develooper.com> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: TFTP over anycast CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and know the content is safe. The system Ask is describing is the traditional method of using anycast to geographically load-balance long-lived flows. The first time I did that was with FTP servers in Berkeley and Santa Cruz, in 1989. I did a bigger system, also load balancing FTP servers for Oracle, their public-facing documentation stores, with servers in San Jose and Washington DC, a couple of years later. A couple of years further on and the World Wide Web was a thing, and everybody was doing it. -Bill On Feb 24, 2024, at 7:38 AM, Ask Bjørn Hansen <ask@develooper.com> wrote: On Feb 23, 2024, at 20:32, William Herrin <bill@herrin.us> wrote: The relay server `dhcplb` could, maybe, help in that scenario (dhcplb runs on the anycast IP, the “real” DHCP servers on unicast IPs behind dhcplb). Although they used the word "anycast", they're just load balancing. The idea is to run the relays on an anycasted IP (so the load balancer / relay IP is anycasted). [….] Relying on ECMP for anycasted DHCP would be a disaster during any sort of failure. Add or remove a single route from an ECMP set and the hashed path selection changes for most of the connections. Consistent hashing (which I thought was widely supported now in ECMP implementations) and a bit of automation in how announcements are added can greatly mitigate this. Ask
On Tue, Feb 27, 2024 at 10:02 AM Javier Gutierrez <GutierrezJ@westmancom.com> wrote:
My design is very simplistic, I have 2 sets of firewalls that I will have advertising a /32 unicast to the network at each location and it will have a TFTP server behind each firewall.
Hi Javier, That sounds straightforward to me with no major failure modes. I would make the firewall part of my OSPF network and then add the tftp servers to OSPF using FRR. Then I'd write a script to monitor the local tftp server and stop frr if it detects any problems with the tftp server. The local tftp server will always be closer than the remote one via OSPF link costs, unless it goes offline. I assume you also have an encrypted channel between the firewalls to handle traffic that stays "inside" your security boundary, as tftp generally should. Where you could get into trouble is if you add a third or additional sites. If there's ever an equal routing cost from any one site to two others, there's a non-zero risk of the failover process failing... and you won't know it until you need it. Regards, Bill Herrin -- William Herrin bill@herrin.us https://bill.herrin.us/
On 27/02/2024 18:47, William Herrin wrote:
Then I'd write a script to monitor the local tftp server and stop frr if it detects any problems with the tftp server. There are other ways to achieve this without actually stopping the routing daemon.
We have DNS servers where the anycast service address is added to a loopback interface (lo1) and only advertised when present. The monitoring script drops and adds the service address to the loopback, without otherwise touching the routing daemon. We use Bird rather than FRR, though. Ray
On Apr 6, 2024, at 10:30, Ray Bellis <ray@bellis.me.uk> wrote: On 27/02/2024 18:47, William Herrin wrote:
Then I'd write a script to monitor the local tftp server and stop frr if it detects any problems with the tftp server. There are other ways to achieve this without actually stopping the routing daemon. We have DNS servers where the anycast service address is added to a loopback interface (lo1) and only advertised when present.
That’s been the normal way of doing it for some 35 years now. iBGP advertise, or don’t advertise, the service address, which is attached to the loopback, depending whether you’re ready to service traffic. -Bill
On Sat, 6 Apr 2024 at 12:00, Bill Woodcock <woody@pch.net> wrote:
That’s been the normal way of doing it for some 35 years now. iBGP advertise, or don’t advertise, the service address, which is attached to the loopback, depending whether you’re ready to service traffic.
If we are talking about eBGP, then pulling routes makes sense. If we are talking about iBGP and controlled environment, you should never pull anycast routes, because eventually you will have failure mode, where the check mechanism itself is broken, and you'll pull all routes. If instead of pulling the routes, you make them inferior, you are covered for the failure mode of check itself being broken. -- ++ytti
Others have addressed some of the issues, but one easy win for DHCP (which is otherwise a PITA to make redundany in *any* way) is to (a) not block ICMP anywhere, including on the client devices, and (b) have the DHCP ping before assignment. That’s not always on by default, and it’ll eliminate ~90% of the conflicts you would otherwise encounter if the anycast node isn’t extremely stable. If you become aware of a distributed DHCP server that actually works well in this environment, that’s worth a post to the list all by itself. -Adam Adam Thompson Consultant, Infrastructure Services [cid:image001.png@01DA663E.AD5957D0] 100 - 135 Innovation Drive Winnipeg, MB R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) https://www.merlin.mb.ca<https://www.merlin.mb.ca/> [cid:image002.png@01DA663E.AD5957D0]Chat with me on Teams<https://teams.microsoft.com/l/chat/0/0?users=athompson@merlin.mb.ca> [https://res-h3.public.cdn.office.net/assets/bookwithme/misc/CalendarPerson20px.png]<https://outlook.office.com/bookwithme/user/a3e87142ccc14503bf5ec7ea59afd6e3@merlin.mb.ca?anonymous&ep=bwmEmailSignature> Book time to meet with me<https://outlook.office.com/bookwithme/user/a3e87142ccc14503bf5ec7ea59afd6e3@merlin.mb.ca?anonymous&ep=bwmEmailSignature> From: NANOG <nanog-bounces+athompson=merlin.mb.ca@nanog.org> On Behalf Of Javier Gutierrez Sent: Thursday, February 22, 2024 12:47 PM To: nanog@nanog.org Subject: TFTP over anycast Hi, I'm working on some DR design and we want to not only have this site as a DR but also performing some active/active for some of the services we hosts and I was wondering if someone had some experience with using anycast for TFTP or DHCP services? What are some of the pains/challenges you experienced and things we should lookout for? Any input is greatly appreciated. Kind regards, Javier Gutierrez
participants (9)
-
Adam Thompson -
Ask Bjørn Hansen -
Bill Woodcock -
Dan Sneddon -
Javier Gutierrez -
Ray Bellis -
Saku Ytti -
Thomas Mieslinger -
William Herrin