I've been investigating the new improved mod_proxy in Apache 2.2.x for use in our new production environment, and in particular the built-in load balancing support. It was always possible to build a load-balanced proxy server with Apache before, using some mod_rewrite voodoo, but having a whole set of directives that do all the hard work for you is a great feature.
There is however, a catch. It won't work out of the box with PHP sessions, or many other applications. I've since worked out a way around this which enables you to continue using all the great features mod_proxy_balancer offers and still bind requests to an originating server. All you need is a little mod_rewrite magic : Read on for more details...
If you're going to use mod_proxy to build a reverse proxy (caching
or otherwise) in front of your backend application servers, you're
probably going to want to use sticky sessions so that a user of your
site always gets "bound" to the server that the session originated on.
Otherwise, you'll get people randomly getting logged out and all manner
of other nastiness. According to the docs, you can use the "stickysession" key in your configuration :
The value is usually set to something
like JSESSIONID or PHPSESSIONID,
and it depends on the backend application server that support sessions.
Sounds great, but it doesn't actually work that way with PHP. If you set your configuration to something like this :
ProxyPass / balancer://cluster/ lbmethod=byrequests \
stickysession=PHPSESSID failover=Off
You'll find that regardless of whether a session exists or not, you'll still end up randomly pinging between the backend servers, which is definitely NOT what you want to happen. So why is this happening ? Well, it turns out that the Apache documentation is actually a little misleading. If you look at the relevant code in modules/proxy/mod_proxy_balancer.c, lines 195 to 210, you'll see it's actually looking for a session ID, then a period (.) character and a route. Tomcat can be configured to do this, but PHP can't; PHP sessions can only be alphanumeric and don't specify an originating server or "route".
So, as a workaround, you can create your own cookie and use that instead when binding requests to a backend server. It does however mean that any request gets bound to a server,not just when sessions start but this shouldn't be too much of an issue, particularly if you are caching static content as well. An easy way to do this on recent versions of Apache is to use mod_rewrite to set the cookie for you (this feature was only added in later versions of 2.0.x and up - I recommend running 2.2.x to be certain all this will work).
Say you have 2 backend servers, www1.example.com and www2.example.com. You'd add the following to your backend vhost configuation :
RewriteEngine On
RewriteRule .* - [CO=BALANCEID:balancer.www1:.example.com]
And then do the same for www2, but obviously changing the cookie value to reflect this. You then need to tell your frontend proxy that it should look for this cookie, and which server each "route" refers to :
ProxyPass / balancer://cluster/ lbmethod=byrequests stickysession=BALANCEID
ProxyPassReverse / balancer://cluster/
<Proxy balancer://cluster>
BalancerMember http://www1.example.com route=www1
BalancerMember http://www2.example.com route=www2
</Proxy>
And then give it a kick. What you then should see (if you switch LogLevel to debug) is the following the first time a request comes in :
proxy: BALANCER: Found value (null) for stickysession BALANCEID
proxy: Entering byrequests for BALANCER (balancer://cluster)
So it doesn't have a preferred route, and it switches to the default
load balancing algorithm (byrequests) to get a random server. Next
time, the cookie will have been set, and you'll see :
proxy: BALANCER: Found value balancer.www1 for stickysession BALANCEID
proxy: BALANCER: Found route www1
proxy: BALANCER (balancer://cluster) worker (http://www1.example.com)
rewritten to http://www1.example.com/
And you should then be good to go. Each new request that comes in will be directed to a backend server according to your load-balancing method, and any subsequent requests from that user (assuming they have cookies enabled) will then go back to the same backend server. When they close their browser and the cookie expires, the "binding" is reset and they'll get a new random server next time they connect.
Friday, March 23. 2007 at 09:38 (Reply)
Thanks for putting me on to the right track. I ended up setting a cookie using a randomised RewriteMap (mod_rewrite) on my reverse proxy.
-ascs
Tuesday, March 27. 2007 at 02:40 (Reply)
NameVirtualHost 10.1.1.115:8080
NameVirtualHost 10.1.1.116:8080
VirtualHost 10.1.1.115:8080
RewriteEngine On
RewriteRule . - [CO=BALANCEID:balancer.app1:.example.com:120:/app]
/VirtualHost
VirtualHost 10.1.1.116:8080
RewriteEngine On
RewriteRule . - [CO=BALANCEID:balancer.app2:.example.com:120:/app]
/VirtualHost
ProxyPass /app balancer://app_cluster/app stickysession=BALANCEID
ProxyPassReverse /app http://10.1.1.115:8080/app
ProxyPassReverse /app http://10.1.1.116:8080/app
Proxy balancer://app_cluster
BalancerMember http://10.1.1.115:8080 route=app1
BalancerMember http://10.1.1.116:8080 route=app2
/Proxy
Thursday, March 29. 2007 at 22:49 (Link) (Reply)
Anyway, I will have a look and check...
Friday, March 30. 2007 at 16:46 (Reply)
Friday, March 30. 2007 at 12:31 (Reply)
Sunday, July 1. 2007 at 04:20 (Link) (Reply)
Mark, I have a similar problem here and I´ll appreciatte a lot if you help me.
I have two instances of ColdFusion running on the same server. As you know, CF runs on the JRun and therefore, is an application java that have session controlled by JSessionID.
Well, my problem consists as following. When I make login, I can´t logon in the application, because instance CF that validates login is not the same one that it answers request of the initial page of the application. As the session does not exist for this second instance, I am redirected again for the logon screen.
I use Apache 2.2x with mod_proxy_balancer. I think that this solve my problem. What is your opinion?
Regards,
Paulo Roberto
Wednesday, August 15. 2007 at 23:41 (Link) (Reply)
I'm not sure where ColdFusion configures this but in PHP you can specify the session storage directory in the php.ini file. I suspect ColdFusion has an equivalent in it's config files.
Alternatively, you could switch to storing sessions in a database. That will come with it's own set of problems such as placing extra load on your database server and potentially being slower than storing them on local disks but it will solve the problem of accessing a session from multiple web servers.
Thursday, January 31. 2008 at 20:17 (Reply)
Thanks for the post.
Paulo,
JRun's JSessionID will not help for session stickiness here. So I would propose writing an extra filter in your web application(in your case CF) and add a custom cookie in both the instances such as cookie.node1, cookie.node2 Then add this custom cookie name in the stickysession=${Custom_cookie}
This should help to maintain session stickiness in the balancer, though it will break in case of fail over.
-JR
Wednesday, September 5. 2007 at 17:20 (Reply)
My experience with sticky balancing is "best to avoid it if possible"!
Friday, August 1. 2008 at 17:00 (Reply)
I checked this and it work so far.
But what are you doing, when a server fails? I mean completely.
So far I've got a so called healthcheck script which checks the BalancerMember and reconfigures the apache.
Now, the cookie is still set and the proxy sends the request to the failed server. Do you have any idea how to get into that request and send to another BalancerMember?
Best Regards,
Uwe
Sunday, August 3. 2008 at 20:12 (Reply)
I have 5 prescription servers in the back-end that support HTTP tunneled requests. They have an internal sessionid called CustSessID that is coming from cookie. I want stickyness so that the requests from same browser go to the same Prescription Server. I also have URL rewriting that connects to Generic server or BrandName Server.
Thursday, September 25. 2008 at 17:10 (Reply)
BalancerMember http://www1.example.com
BalancerMember http://www2.example.com
in my case i have two apache as BalanceMember and they are listing at secure port 443.
so my links are
BalancerMember https://www1.example.com
BalancerMember https://www2.example.com
this is not working????? why??
more over in my real servers i am re writing the urls too.
and while i request my loadbalancer apache it just redirect my request to real server and in my browser i see the address of my real server. which i dont like.
can anybdy help me.
Thursday, October 2. 2008 at 14:59 (Reply)
proxy: BALANCER: Found value balancer%2Ewww1 for stickysession BALANCEID
So the route is never getting
discovered.
I'm setting the cookie with an ASP page. Anyone know whats going on?
Wednesday, October 15. 2008 at 10:54 (Reply)
I have Problem in getting sticky session.
i used phpBB application to test sticky session.i have installed phpbb in two diffrent machine connected to diffrent database.
here is my config
_____________________________________
NameVirtualHost *
ServerName test
ServerAlias test
ProxyRequests Off
Order deny,allow
Allow from all
ProxyPass /balancer-manager !
ProxyPass / balancer://mycluster/ stickysession=BALANCEID nofailover=On
ProxyPassReverse / http://192.168.11.23/phpBB2
ProxyPassReverse / http://192.168.11.43/phpBB2
BalancerMember http://192.168.11.23/phpBB2 route=http1
BalancerMember http://192.168.11.43/phpBB2 route=http2
ProxySet lbmethod=byrequests
SetHandler balancer-manager
Order deny,allow
Allow from all
Every time i refresh the page it takes me to diffrent server and does not allow me to login, because it's checking session on bothe the server
Please help....
Wednesday, October 15. 2008 at 12:01 (Reply)
Monday, October 20. 2008 at 12:23 (Reply)
If you have Apache running as a load balancer with two web nodes, and the user gets a cookie associating them with www1, there is an issue when www1 goes offline. It seems that Apache is smart enough to know that www1 is offline and requests should now go to www2, but that only happens with fresh requests. The previous user with the cookie pointing to www1 is now just getting sent to the www1 server, which throws them a Server Unavailable message (via the load balancer). Any way around this? I guess I would be looking for a way to check that a server is up before Apache uses that balance member. Thoughts?
-Travis
Monday, October 20. 2008 at 14:20 (Reply)
Alternatively, you could write a script that uses the balancer-manager page to dynamically add/remove backend servers.
Thursday, October 30. 2008 at 06:18 (Link) (Reply)
I see that maxattempts and nofailover parameters to the ProxyPass directive are described on this page of the HTTPD configuration site:
http://httpd.apache.org/docs/2.2/mod/mod_proxy.html
Does that look like what you need?
Thursday, November 20. 2008 at 21:09 (Reply)
Since I wrote this post, I was also messing around with other load balancing options, so forgive me if my explanation is a bit incorrect, since I'm trying to put myself back into this configuration to fully understand it...
Wednesday, January 19. 2011 at 22:05 (Reply)
Did you find solution to, "I guess I would be looking for a way to check that a server is up before Apache uses that balance member."
With the configuration I have, if a member is "OK", but the backend is down, the balancer still uses the member connected to the failed backend - only once - then will no longer use it. (I use failonstatus).
I have tried it use ping=1, but I guess that is only used for ajp13..
Hope someone can help.
Hope you can help.
Sunday, April 12. 2009 at 11:53 (Link) (Reply)
I did a solution where you only have to change the loadbalancer server. Loadbalancer is creating the balance cookie based on elected backend server based on environment properties set by mod_proxy_balancer.
http://www.jakeri.net/2009/03/sticky-load-balance-with-apache-httpd-22/
Wednesday, May 27. 2009 at 00:09 (Reply)
One common reason for using a reverse proxy is to enable caching. By default, mod_cache records and replays the Set-Cookie header!
This means that you can wind up with a cached route, so that anyone who requests the cached file will get assigned to the same backend host.
CacheIgnoreHeaders Set-Cookie clears this up.
Friday, February 5. 2010 at 11:16 (Reply)
Any idea why I might be getting:
proxy: BALANCER: Found value balancer.www8888 for stickysession balancerID
proxy: BALANCER: Found route www8888
followed immediately by:
proxy: Entering byrequests for BALANCER (balancer://cluster)
? It appears to be ignoring the route=www888 in my BalancerMember entry.
Thanks!
Friday, February 5. 2010 at 11:49 (Reply)
Thursday, March 25. 2010 at 02:28 (Reply)
"When they close their browser and the cookie expires, the "binding" is reset and they'll get a new random server next time they connect."
I manage an infrastructure that was built off of this page. The cookie never expires. A user is bound to a specific server forever. If that server dies, then the user is going to go to that dead server.
This is bad. I'm searching for a solution to this now.
Thursday, March 25. 2010 at 10:00 (Reply)
Using Firefox (purely because it's got good developer tools like the Web Developer toolbar and Firebug that lat you inspect this sort of behaviour), when I visit a site that sets the cookie, I see the following in my cookie store :
Name BALANCEID
Value balancer.www1
Host .example.com
Path /
Secure No
Expires At End Of Session
And it's set by the following header from the webserver :
Set-Cookie BALANCEID=balancer.www1; path=/; domain=.example.com
When I close my browser down completely and then restart it, I can check my cookie store and the cookie isn't there anymore. When I revisit the site, my request headers do not send any cookies.
-Mark
Tuesday, April 6. 2010 at 15:07 (Link) (Reply)
Thank you.
Tuesday, June 22. 2010 at 11:01 (Reply)
After four years, your article still valuable. In Apache 2.2 documents and many other (Tomcat, Apache) cluster configuration tutorials, they didn't specific the relation between "route" in Apache's httpd.conf
BalancerMember ajp://localhost:8009/passport route=node00
BalancerMember ajp://localhost:8019/passport route=node01
BalancerMember ajp://localhost:8029/passport route=node02
and "jvmRoute" in Tomcat's server.xml
Thanks for your article!
Thursday, August 5. 2010 at 14:15 (Reply)
ProxyPass /BLAH/ balancer://test/BLAH stickysession=ASPSESSIONID that will not works because there was many sessioncookies with different names at the end (ASPSESSIONIDABCDE; ASPSESSIONIDEFGHI...) and the stickysessionparameter in ProxyPass seems not to allow wildcards like ProxyPass /BLAH/ balancer://test/BLAH stickysession=ASPSESSIONID*.
So your solution works PERFECT.
Thanks!
Tuesday, August 31. 2010 at 01:32 (Reply)
Your article is excellent. I was wondering if it fails with rewrite instead of directly proxypassing. I have apache on frontend and another couple of httpd.s on backend serving pure content. my setup is like
______________________
BalancerMember http://10.10.10.1 route=cmshttp1
BalancerMember http://10.10.10.2 route=cmshttp2
ProxySet lbmethod=byrequests nofailover=Off
RewriteRule ^/publish/(.*)$ balancer://cmshttp/publish/$1 [P,QSA]
______________________
The cookie on backend is properly set and I can check by directly going to http://10.10.10.1 (firebug). But I see in my http://10.6.32.154/balancer-manager that they are elected almost equally .
Any thoughts ? Thanks a lot.
Thursday, September 23. 2010 at 01:56 (Reply)
Monday, November 8. 2010 at 17:29 (Reply)
I have a problem with my load balancer, I have an application in jsp.
Enter the login but even there, is reloaded to the point that keeps me writing, it appears that the balancer makes this leaping route map
But when he entered the balancer-manager and disable all instances and leave a normalmentey if you load the application works fine.
I need urgent help
this is my configuration
isten 9095
RewriteEngine On
ProxyRequests Off
Order deny,allow
Allow from all
SetHandler balancer-manager
Order Deny,Allow
Allow from all
SetHandler server-status
Order Deny,Allow
Allow from all
RewriteRule . - [CO=BALANCEID:balancer.d2:.192.168.0.177:38084]
RewriteRule . - [CO=BALANCEID:balancer.d3:.192.168.0.177:38085]
RewriteRule . - [CO=BALANCEID:balancer.d4:.192.168.0.177:38086]
RewriteRule . - [CO=BALANCEID:balancer.d5:.192.168.0.177:38087]
RewriteRule . - [CO=BALANCEID:balancer.d6:.192.168.0.177:38090]
RewriteRule . - [CO=BALANCEID:balancer.d7:.192.168.0.177:38091]
ProxyPass /Draco balancer://mycluster/Draco/ stickysession=BALANCEID
ProxyPassReverse /app balancer://mycluster/app/
BalancerMember http://192.168.0.177:38086 route=d2
BalancerMember http://192.168.0.177:38084 route=d3
BalancerMember http://192.168.0.177:38085 route=d4
BalancerMember http://192.168.0.177:38087 route=d7
BalancerMember http://192.168.0.177:38090 route=d7
BalancerMember http://192.168.0.177:38091 route=d7
Order allow,deny
Allow from all
#
#Order allow,deny
#Allow from all
#
Order allow,deny
Allow from all
ProxyPass /balancer-manager !
ProxyPass / balancer://mycluster/
ProxyPass /server-status !
ErrorLog logs/error4
CustomLog logs/error5 common
Sunday, June 26. 2011 at 22:29 (Reply)
ServerAdmin webmaster@localhost
ProxyRequests Off
Order deny,allow
Allow from all
ProxyPass /error1 http://192.168.1.41/error/
ProxyPass /error2 http://192.168.1.42/error/
ProxyPass /balancer-manager !
SetHandler balancer-manager
Order deny,allow
Allow from 192.168.1.0/24
Allow from 10.1.0.0/24
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
BalancerMember http://192.168.1.41:80 route=1
BalancerMember http://192.168.1.42:80 route=2
#ProxySet lbmethod=byrequests
ProxySet stickysession=ROUTEID
ProxyPass / balancer://cluster/ nofailover=On
#stickysession=BALANCEID
ProxyPassReverse / balancer://cluster/
Options FollowSymLinks
AllowOverride None
ErrorLog /var/log/apache2/error.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
CustomLog /var/log/apache2/access.log combined
CookieTracking on
CookieStyle Cookie
CookieName USERINFO
# LogSQLTransferLogFormat huSUsbTvRAc
LogSQLWhichCookie USERINFO
LogSQLTransferLogTable access_log
LogSQLCookieLogTable cookies
LogSQLCreateTables on
Does anynyone know how to redirect to working route if one route fails?
Wednesday, May 30. 2012 at 05:48 (Reply)
But When my website use ajax( xajax). it can not load data.
How fix when website load by ajax?