In my last blog entry on mod_backhand, I mentioned that you could implement redundancy in a web cluster by having multiple load balancers and using round robin DNS pointing to the load balancers. This technique is mentioned in the mod_backhand presentation notes by Theo Schlossnagle, one of the author's of mod_backhand.

But if you think about it carefully, in my opinion, this solution doesn't really work well:

• Round robin DNS systems simply return a list of IP addresses (sorted randomly) that resolve that domain name. Most DNS servers do not perform checks to detect if any of the IP addresses are down. So clients will get the IP addresses of load balancers that are down too. The DNS client normally just picks the first IP address in the list to use - and it could be the IP address of a load balancer that is down.
• Secondly, when a client resolves the domain name, this IP address is cached on the client. So a failure in one of the load balancers would still mean that all web browsers that previously connected to that load balancer successfully because of DNS caching would be unable to access the system for quite a long time (e,g. 10-15 minutes).

I can think of several solutions, but the best one as far as I can see (if you still want to use mod_backhand of course) is to run mod_backhand on a high availability hardware solution (or if a few minutes downtime is acceptable, keep a spare box configured with mod_backhand around to swap with any balancer that goes down), and not push the hard problem of high availability and redundancy to DNS.

Ahmad Amran Kapi from Melaka, Malaysia, points out that you can use Wackamole:

Wackamole is an application that helps with making a cluster highly available. It manages a bunch of virtual IPs, that should be available to the outside world at all times. Wackamole ensures that a single machine within a cluster is listening on each virtual IP address that Wackamole manages. If it discovers that particular machines within the cluster are not alive, it will almost immediately ensure that other machines acquire these public IPs. At no time will more than one machine listen on any virtual IP. Wackamole also works toward achieving a balanced distribution of number IPs on the machine within the cluster it manages.