I've been investigating the new improved mod_proxy in Apache 2.2.x for use in our new production environment, and in particular the built-in load balancing support. It was always possible to build a load-balanced proxy server with Apache before, using some mod_rewrite voodoo, but having a whole set of directives that do all the hard work for you is a great feature.
There is however, a catch. It won't work out of the box with PHP sessions, or many other applications. I've since worked out a way around this which enables you to continue using all the great features mod_proxy_balancer offers and still bind requests to an originating server. All you need is a little mod_rewrite magic : Read on for more details...
If you're going to use mod_proxy to build a reverse proxy (caching
or otherwise) in front of your backend application servers, you're
probably going to want to use sticky sessions so that a user of your
site always gets "bound" to the server that the session originated on.
Otherwise, you'll get people randomly getting logged out and all manner
of other nastiness. According to the docs, you can use the "stickysession" key in your configuration :
The value is usually set to something
like JSESSIONID or PHPSESSIONID,
and it depends on the backend application server that support sessions.
Sounds great, but it doesn't actually work that way with PHP. If you set your configuration to something like this :
ProxyPass / balancer://cluster/ lbmethod=byrequests \
stickysession=PHPSESSID failover=Off
You'll find that regardless of whether a session exists or not, you'll still end up randomly pinging between the backend servers, which is definitely NOT what you want to happen. So why is this happening ? Well, it turns out that the Apache documentation is actually a little misleading. If you look at the relevant code in modules/proxy/mod_proxy_balancer.c, lines 195 to 210, you'll see it's actually looking for a session ID, then a period (.) character and a route. Tomcat can be configured to do this, but PHP can't; PHP sessions can only be alphanumeric and don't specify an originating server or "route".
So, as a workaround, you can create your own cookie and use that instead when binding requests to a backend server. It does however mean that any request gets bound to a server,not just when sessions start but this shouldn't be too much of an issue, particularly if you are caching static content as well. An easy way to do this on recent versions of Apache is to use mod_rewrite to set the cookie for you (this feature was only added in later versions of 2.0.x and up - I recommend running 2.2.x to be certain all this will work).
Say you have 2 backend servers, www1.example.com and www2.example.com. You'd add the following to your backend vhost configuation :
RewriteEngine On
RewriteRule .* - [CO=BALANCEID:balancer.www1:.example.com]
And then do the same for www2, but obviously changing the cookie value to reflect this. You then need to tell your frontend proxy that it should look for this cookie, and which server each "route" refers to :
ProxyPass / balancer://cluster/ lbmethod=byrequests stickysession=BALANCEID
ProxyPassReverse / balancer://cluster/
<Proxy balancer://cluster>
BalancerMember http://www1.example.com route=www1
BalancerMember http://www2.example.com route=www2
</Proxy>
And then give it a kick. What you then should see (if you switch LogLevel to debug) is the following the first time a request comes in :
proxy: BALANCER: Found value (null) for stickysession BALANCEID
proxy: Entering byrequests for BALANCER (balancer://cluster)
So it doesn't have a preferred route, and it switches to the default
load balancing algorithm (byrequests) to get a random server. Next
time, the cookie will have been set, and you'll see :
proxy: BALANCER: Found value balancer.www1 for stickysession BALANCEID
proxy: BALANCER: Found route www1
proxy: BALANCER (balancer://cluster) worker (http://www1.example.com)
rewritten to http://www1.example.com/
And you should then be good to go. Each new request that comes in will be directed to a backend server according to your load-balancing method, and any subsequent requests from that user (assuming they have cookies enabled) will then go back to the same backend server. When they close their browser and the cookie expires, the "binding" is reset and they'll get a new random server next time they connect.
Friday, March 23. 2007 at 08:38 (Reply)
Thanks for putting me on to the right track. I ended up setting a cookie using a randomised RewriteMap (mod_rewrite) on my reverse proxy.
-ascs
Tuesday, March 27. 2007 at 01:40 (Reply)
NameVirtualHost 10.1.1.115:8080
NameVirtualHost 10.1.1.116:8080
VirtualHost 10.1.1.115:8080
RewriteEngine On
RewriteRule . - [CO=BALANCEID:balancer.app1:.example.com:120:/app]
/VirtualHost
VirtualHost 10.1.1.116:8080
RewriteEngine On
RewriteRule . - [CO=BALANCEID:balancer.app2:.example.com:120:/app]
/VirtualHost
ProxyPass /app balancer://app_cluster/app stickysession=BALANCEID
ProxyPassReverse /app http://10.1.1.115:8080/app
ProxyPassReverse /app http://10.1.1.116:8080/app
Proxy balancer://app_cluster
BalancerMember http://10.1.1.115:8080 route=app1
BalancerMember http://10.1.1.116:8080 route=app2
/Proxy
Thursday, March 29. 2007 at 21:49 (Link) (Reply)
Anyway, I will have a look and check...
Friday, March 30. 2007 at 15:46 (Reply)
Friday, March 30. 2007 at 11:31 (Reply)
Sunday, July 1. 2007 at 03:20 (Link) (Reply)
Mark, I have a similar problem here and I´ll appreciatte a lot if you help me.
I have two instances of ColdFusion running on the same server. As you know, CF runs on the JRun and therefore, is an application java that have session controlled by JSessionID.
Well, my problem consists as following. When I make login, I can´t logon in the application, because instance CF that validates login is not the same one that it answers request of the initial page of the application. As the session does not exist for this second instance, I am redirected again for the logon screen.
I use Apache 2.2x with mod_proxy_balancer. I think that this solve my problem. What is your opinion?
Regards,
Paulo Roberto
Wednesday, August 15. 2007 at 22:41 (Link) (Reply)
I'm not sure where ColdFusion configures this but in PHP you can specify the session storage directory in the php.ini file. I suspect ColdFusion has an equivalent in it's config files.
Alternatively, you could switch to storing sessions in a database. That will come with it's own set of problems such as placing extra load on your database server and potentially being slower than storing them on local disks but it will solve the problem of accessing a session from multiple web servers.
Thursday, January 31. 2008 at 19:17 (Reply)
Thanks for the post.
Paulo,
JRun's JSessionID will not help for session stickiness here. So I would propose writing an extra filter in your web application(in your case CF) and add a custom cookie in both the instances such as cookie.node1, cookie.node2 Then add this custom cookie name in the stickysession=${Custom_cookie}
This should help to maintain session stickiness in the balancer, though it will break in case of fail over.
-JR
Wednesday, September 5. 2007 at 16:20 (Reply)
My experience with sticky balancing is "best to avoid it if possible"!
Friday, August 1. 2008 at 16:00 (Reply)
I checked this and it work so far.
But what are you doing, when a server fails? I mean completely.
So far I've got a so called healthcheck script which checks the BalancerMember and reconfigures the apache.
Now, the cookie is still set and the proxy sends the request to the failed server. Do you have any idea how to get into that request and send to another BalancerMember?
Best Regards,
Uwe
Sunday, August 3. 2008 at 19:12 (Reply)
I have 5 prescription servers in the back-end that support HTTP tunneled requests. They have an internal sessionid called CustSessID that is coming from cookie. I want stickyness so that the requests from same browser go to the same Prescription Server. I also have URL rewriting that connects to Generic server or BrandName Server.
Thursday, September 25. 2008 at 16:10 (Reply)
BalancerMember http://www1.example.com
BalancerMember http://www2.example.com
in my case i have two apache as BalanceMember and they are listing at secure port 443.
so my links are
BalancerMember https://www1.example.com
BalancerMember https://www2.example.com
this is not working????? why??
more over in my real servers i am re writing the urls too.
and while i request my loadbalancer apache it just redirect my request to real server and in my browser i see the address of my real server. which i dont like.
can anybdy help me.
Thursday, October 2. 2008 at 13:59 (Reply)
proxy: BALANCER: Found value balancer%2Ewww1 for stickysession BALANCEID
So the route is never getting
discovered.
I'm setting the cookie with an ASP page. Anyone know whats going on?
Wednesday, October 15. 2008 at 09:54 (Reply)
I have Problem in getting sticky session.
i used phpBB application to test sticky session.i have installed phpbb in two diffrent machine connected to diffrent database.
here is my config
_____________________________________
NameVirtualHost *
ServerName test
ServerAlias test
ProxyRequests Off
Order deny,allow
Allow from all
ProxyPass /balancer-manager !
ProxyPass / balancer://mycluster/ stickysession=BALANCEID nofailover=On
ProxyPassReverse / http://192.168.11.23/phpBB2
ProxyPassReverse / http://192.168.11.43/phpBB2
BalancerMember http://192.168.11.23/phpBB2 route=http1
BalancerMember http://192.168.11.43/phpBB2 route=http2
ProxySet lbmethod=byrequests
SetHandler balancer-manager
Order deny,allow
Allow from all
Every time i refresh the page it takes me to diffrent server and does not allow me to login, because it's checking session on bothe the server
Please help....
Wednesday, October 15. 2008 at 11:01 (Reply)
Monday, October 20. 2008 at 11:23 (Reply)
If you have Apache running as a load balancer with two web nodes, and the user gets a cookie associating them with www1, there is an issue when www1 goes offline. It seems that Apache is smart enough to know that www1 is offline and requests should now go to www2, but that only happens with fresh requests. The previous user with the cookie pointing to www1 is now just getting sent to the www1 server, which throws them a Server Unavailable message (via the load balancer). Any way around this? I guess I would be looking for a way to check that a server is up before Apache uses that balance member. Thoughts?
-Travis
Monday, October 20. 2008 at 13:20 (Reply)
Alternatively, you could write a script that uses the balancer-manager page to dynamically add/remove backend servers.
Thursday, October 30. 2008 at 05:18 (Link) (Reply)
I see that maxattempts and nofailover parameters to the ProxyPass directive are described on this page of the HTTPD configuration site:
http://httpd.apache.org/docs/2.2/mod/mod_proxy.html
Does that look like what you need?
Thursday, November 20. 2008 at 20:09 (Reply)
Since I wrote this post, I was also messing around with other load balancing options, so forgive me if my explanation is a bit incorrect, since I'm trying to put myself back into this configuration to fully understand it...
Tuesday, May 26. 2009 at 23:09 (Reply)
One common reason for using a reverse proxy is to enable caching. By default, mod_cache records and replays the Set-Cookie header!
This means that you can wind up with a cached route, so that anyone who requests the cached file will get assigned to the same backend host.
CacheIgnoreHeaders Set-Cookie clears this up.
Friday, February 5. 2010 at 10:16 (Reply)
Any idea why I might be getting:
proxy: BALANCER: Found value balancer.www8888 for stickysession balancerID
proxy: BALANCER: Found route www8888
followed immediately by:
proxy: Entering byrequests for BALANCER (balancer://cluster)
? It appears to be ignoring the route=www888 in my BalancerMember entry.
Thanks!
Friday, February 5. 2010 at 10:49 (Reply)