IPv6 Server Load Balancing - DSR
Dear Colleagues, I've been scratching my head over this for the past couple of months and have come up with blanks, and several weeks of scouring various resources on the net have not yielded anything more fruitful. I'm looking at server load balancing for IPv6 and specifically need DSR (direct server return). Additionally, I need to support both TCP and UDP. I have evaluated a number of different load balancing solutions purporting to support IPv6 with varying results (and costs)... a few examples: F5 : according to marketing blurb supposedly supports IPv6 in NAT and DSR mode, both UDP and TCP. Their documentation, however, has no mention of IPv6 capability. Other disadvantage = cost... Brocade/Foundry: Similar situation to F5 Zeus: IPv6 in NAT only, and even more expensive than F5. Exceliance Aloha: IPv6 in NAT only, and ONLY in TCP (no UDP) A few others also tested... including LVM/HAProxy (same situation as Exceliance Aloha), and others... Finally in the end, only OpenSolaris ILB seems to put all the checks in the right boxes for my requirements. But there is still a problem. 1. IPv4 TCP and UDP work fine in NAT, Half-NAT, and DSR 2. IPv6 I've managed to get working, complete with healthchecks, in TCP and UDP in NAT only although the documentation stipulates that DSR is also possible (but not HalfNAT for the moment). The problem with #2: Using the same server farm behind, but in dual-stack, and configuring ILB for TCP and UDP services using NAT, everything is fine. If I configure it for DSR, immediately it fails (both with and without healthchecks). Although from the ILB host itself, I can certainly do a manual heathcheck.. (e.g. telnet <server_real_ipv6_addr> 80 and do GET / or HEAD / with no problems. Using ARP poisoning from the shell I can also perform the healthcheck on the real server via telnet using the virtual ip. The servers are configured normally for DSR.. with the virtual IP attached to a local dummy or loopback interface, and with IPv4 DSR works fine. Nevertheless, I've been unable to get DSR working with ILB -- and have found absolutely nothing around the net with working examples of IPv6 SLB with DSR. NAT mode works fine, but the real server loses visibility of the end user's IP as the requests come from the internal IP of the ILB host, and with a system that uses client IP address as part of the various criteria for session tracking, it creates a few problems... I am suspecting that the issue may be related to ND, as the behaviour is similar to the old story with doing DSR on real-servers using older linux distributions that do not by default disable proxy-ARP replies by the server for IP addresses on dummy or loopback interfaces, and of course the proxy ARP causes confusion to the load balancer and breaks the whole thing. But the real servers are recent Debian distributions, and both ipv4 ARP and ipv6 ND is disabled on the dummy interfaces, as is proxy ARP. Would anyone happen to have any useful pointers, tips, or other on how to resolve the issue? Many thanks in advance. Leland
On 2010-08-12 08:32, Leland Vandervort wrote:
I'm looking at server load balancing for IPv6 and specifically need DSR (direct server return). Additionally, I need to support both TCP and UDP.
This is easily done with OpenBSD. See here for starters: http://www.undeadly.org/cgi?action=article&sid=20080617010016 Simon -- NAT64/DNS64 open-source --> http://ecdysis.viagenie.ca STUN/TURN server --> http://numb.viagenie.ca vCard 4.0 --> http://www.vcarddav.org
On Thu, 12 Aug 2010, Simon Perreault wrote:
On 2010-08-12 08:32, Leland Vandervort wrote:
I'm looking at server load balancing for IPv6 and specifically need DSR (direct server return). Additionally, I need to support both TCP and UDP.
This is easily done with OpenBSD. See here for starters:
http://www.undeadly.org/cgi?action=article&sid=20080617010016
And FreeBSD: http://www.freshports.org/net/relayd/
Simon -- NAT64/DNS64 open-source --> http://ecdysis.viagenie.ca STUN/TURN server --> http://numb.viagenie.ca vCard 4.0 --> http://www.vcarddav.org
Hi Leland, Seems that hardware vendors doesn't like IPv6... for load balancing. I had a look to relayd from OpenBSD, and it seems this can be used a LoadBalancing with DSR... Even if they don't recommand this ... Maybe the is is the time to move from hardware / closed solutions to open ones.. ? Xavier
OpenSolaris ILB is open solution ;) but yea, that's what we've started looking at -- hence LVM / HAProxy as well.. (though LVM is IPv4 only, and HAProxy is NAT only for IPv6) does relayd support UDP as well as TCP or is it layer7 only like HAProxy ? In the case of ILB, I'm not convinced that it's a problem with the LB itself, but rather the idiosyncrasies of ND in IPv6 that is causing the problem.. but I may be wrong... at any rate, something's amiss ... cheers, Leland On 12 Aug 2010, at 15:05, Xavier Beaudouin wrote:
Hi Leland,
Seems that hardware vendors doesn't like IPv6... for load balancing.
I had a look to relayd from OpenBSD, and it seems this can be used a LoadBalancing with DSR... Even if they don't recommand this ...
Maybe the is is the time to move from hardware / closed solutions to open ones.. ?
Xavier
Hi Leland, Le 12 août 2010 à 15:11, Leland Vandervort a écrit :
OpenSolaris ILB is open solution ;)
but yea, that's what we've started looking at -- hence LVM / HAProxy as well.. (though LVM is IPv4 only, and HAProxy is NAT only for IPv6)
does relayd support UDP as well as TCP or is it layer7 only like HAProxy ?
It does everything... :) L2 -> L7...
In the case of ILB, I'm not convinced that it's a problem with the LB itself, but rather the idiosyncrasies of ND in IPv6 that is causing the problem.. but I may be wrong... at any rate, something's amiss ...
Maybe on some setup you should desactivate ND... Xavier
On 12 Aug 2010, at 15:19, Xavier Beaudouin wrote:
In the case of ILB, I'm not convinced that it's a problem with the LB itself, but rather the idiosyncrasies of ND in IPv6 that is causing the problem.. but I may be wrong... at any rate, something's amiss ...
Maybe on some setup you should desactivate ND...
Yea.. well. .that's the point... can't deactivate ND on the real interface of the server as that's required for the server itself.. but it, according to the kernel, deactivated on the dummy interface carrying the virtual IP of the server farm... exactly as is done for IPv4 and ARP manipulation. Hmmmmm... L.
On Aug 12, 2010, at 6:19 AM, Xavier Beaudouin wrote:
Hi Leland,
Le 12 août 2010 à 15:11, Leland Vandervort a écrit :
OpenSolaris ILB is open solution ;)
but yea, that's what we've started looking at -- hence LVM / HAProxy as well.. (though LVM is IPv4 only, and HAProxy is NAT only for IPv6)
does relayd support UDP as well as TCP or is it layer7 only like HAProxy ?
It does everything... :) L2 -> L7...
In the case of ILB, I'm not convinced that it's a problem with the LB itself, but rather the idiosyncrasies of ND in IPv6 that is causing the problem.. but I may be wrong... at any rate, something's amiss ...
Maybe on some setup you should desactivate ND...
Xavier
If you're putting the DSR address on an interface other than loopback, you probably need to turn of DAD on the interface with the DSR address otherwise DAD will shut down that address on the interface when it sees other servers with the same address. Sometimes it will shut down all but one, sometimes it will shut down all. Owen
Hi Owen, The DSR address is indeed on a loopback in our case. lo Link encap:Local Loopback inet6 addr: ::1/128 Scope:Host inet6 addr: xxxx:xxxx:x:xxxx::xx/128 Scope:Global The mystery continues... Leland On 12 Aug 2010, at 18:28, Owen DeLong wrote:
On Aug 12, 2010, at 6:19 AM, Xavier Beaudouin wrote:
Hi Leland,
Le 12 août 2010 à 15:11, Leland Vandervort a écrit :
OpenSolaris ILB is open solution ;)
but yea, that's what we've started looking at -- hence LVM / HAProxy as well.. (though LVM is IPv4 only, and HAProxy is NAT only for IPv6)
does relayd support UDP as well as TCP or is it layer7 only like HAProxy ?
It does everything... :) L2 -> L7...
In the case of ILB, I'm not convinced that it's a problem with the LB itself, but rather the idiosyncrasies of ND in IPv6 that is causing the problem.. but I may be wrong... at any rate, something's amiss ...
Maybe on some setup you should desactivate ND...
Xavier
If you're putting the DSR address on an interface other than loopback, you probably need to turn of DAD on the interface with the DSR address otherwise DAD will shut down that address on the interface when it sees other servers with the same address. Sometimes it will shut down all but one, sometimes it will shut down all.
Owen
On Thu, 12 Aug 2010 14:32:25 +0200 Leland Vandervort <leland@taranta.discpro.org> wrote:
I'm looking at server load balancing for IPv6 and specifically need DSR (direct server return). Additionally, I need to support both TCP and UDP.
IPVS has had IPv6 support for a while: http://www.mindbasket.com/ipvs/ We're using it on our mirror site, http://ftp.heanet.ie, with DSR for http, ftp and rsync load balancing. rg -- Rob Gallagher | Public Key: 0x1DD13A78 HEAnet Limited, Ireland's Education and Research Network 1st Floor, 5 George's Dock, IFSC, Dublin 1. Registered in Ireland, no 275301 T: (+353-1) 6609040 F: (+353-1) 6603666 WWW: http://www.heanet.ie/ HEAnet National Networking Conference, 10-12 November 2010 - Registration is now open at: http://www.heanet.ie/conferences/2010/
Brocade basically sucks when it comes to loadbalancing IPv6, the old serveriron platform is EOL and a complete mess which offers some IPv6 support, but not much. The new ADX platform seems to be in a pre-alfa stage at the moment. So normally I would say stand clear, however we do run a (larger) usenet platform on v6 which uses DSR and that part works on the serveriron, running a pre-relase of the 11.0.0f software. Must admit we don't do anything fancy, it's all unprotected and statically routed, ACLs are all done on the reals and on the Juniper in front of the serveriron etc. But it seems to hold, haven't heard any complains yet. But be warned this is a really specifc subset of features. For regular operations like web we still have loads and loads of issues. Basically the other choice is F5. We are busy setting up a PoC with A10, who claim IPv6 support. Hopefully in a few weeks time they can be added to the list of potential suppliers. Other then these two I haven't come across any dedicated stuff and what's left is Linux/BSD based solutions. MarcoH
participants (7)
-
Leland Vandervort -
Marco Hogewoning -
Mohacsi Janos -
Owen DeLong -
Rob Gallagher -
Simon Perreault -
Xavier Beaudouin