一篇关于Tomcat的文章

Connect Tomcat servers to Apache and to each other to keep your site running

Summary

If you run only one instance of Tomcat, you lose requests/sessions whenever you upgrade or restart your site. In this article, author Graham King presents simple steps for connecting a pair (or more) of Tomcats to Apache using the JK2/AJP (Apache JServ Protocol) connector and to each other using Tomcat 5's clustering capabilities. Any of the Tomcat servers can be stopped or started without affecting users. With an Apache/Tomcat cluster in place, you can easily adjust your configuration for a range of load-balancing and failover scenarios. --By Graham King

ou've written a Web application that runs happily in production on Tomcat, probably on port 80. Some early problems might include complaints from your security team because your server runs as a root facing the outside world and excessive traffic—traffic beyond what a single server can deal with.

A far more likely problem, however, is that you will need to issue new releases of your application. You may even need to upgrade your version of Tomcat or the operating system. These tasks will require you to restart Tomcat. And whenever you do that, your site will become unavailable.

To prevent such an outcome, you move your main server to port 8081, set up a secondary server identical to the primary one on 8082, and some form of dispatcher on port 80. The dispatcher directs traffic to the secondary server while you upgrade the primary one.

Options for the dispatcher are:

iptables: Part of the Linux kernel, primarily used for enforcing firewall rules. They allow you to map all traffic from one port to another port. For example, this command maps all traffic from port 80 to the primary server on 8081:

iptables -t nat -A PREROUTING -p tcp -d lo --dport 80 -j DNAT --to 127.0.0.1:8081

This option is only possible on a Linux system with iptables compiled into the kernel.

Apache with mod_proxy: Set up Apache as a reverse proxy. To switch servers, update the configuration file (httpd.conf or apache2.conf) and ask Apache to reload it. This is a good solution if you only need to switch servers. A potential downside is the difficulty inherent in automating the switching, since only the root user can ask Apache to reload its configuration file.

Apache with mod_jk2: This option allows you to cluster Tomcat servers together, implement load-balancing and failover strategies, and dynamically add and remove members from the cluster. It makes for a great setup, even if you simply want to switch traffic from a primary to a secondary server. Throughout the rest of this article, I cover how to set up this option.

Note: For more details on iptables and mod_proxy, see Resources.

Step-by-step setup

If you don't already have Tomcat 5.0.x, download a recent copy and unpack it twice. I put mine in /usr/local/tomat1 and /usr/local/tomcat2.

Edit the server.xml file (/usr/local/tomcat1/conf/server.xml or /usr/local/tomcat2/conf/server.xml) on each Tomcat by changing the non-SSL (Secure Socket Layer) Coyote HTTP/1.1 Connector element for the first server to:

and the second server to:

Also on the second server, change the port in the server entry at the top. For example:

Start both servers and check if you can get the default homepage. Download and install Apache 2 and make sure it runs.

Install mod_jk2

A binary build or packaged build of mod_jk2 is available for many platforms. A binary or packaged build is the easiest way to install mod_jk2—if you have one of these, skip to the next section ("Apache mod_jk2 Configuration"). Otherwise, you need to build mod_jk2 from source, which is still quite easy.

To build from source, you need the apxs tool, which is usually found in the apache2-devel package. On Linux, the apache2-devel package will probably come with your distribution in RPM format. Unpack the source archive, change into jakarta-tomcat-connectors-jk2-2.0.4-src/jk/native2 and follow the instructions in BUILD.txt.

Once built, you need to tell Apache to load the mod_jk2 module. Edit your httpd2.conf or apache2.conf by adding the line:

LoadModule jk2_module modules/mod_jk2.so

The Gentoo e-build creates file modules.d/89_mod_jk.conf for you so you don't have to. On Gentoo, to switch that module on, you must edit /etc/config.d/apache2 by adding the line:

APACHE2_OPTS="-D JK2"

Restart Apache and check that no errors occur during startup.

Apache mod_jk2 configuration

Having installed mod_jk2, we now need to configure it. The configuration file tells the mod_jk2 connector where the Tomcat workers are and how to manage their connections.

The cluster setup on the Apache side is completed in conf/workers2.properties, in the same directory as your httpd2.conf or apache2.conf file.

For our setup, here is a workers2.properties file:

[shm:]

info=Shared memory

file=anonymous

[channel.socket:server1]

host=127.0.0.1

port=8009

tomcatId=server1

[channel.socket:server2]

host=127.0.0.1

port=8010

tomcatId=server2

ver=0

[lb:lb]

info=Load balancer

sticky=1

ver=0

[uri:/servlets-examples/*]

info=Examples Web application

group=lb:lb

ver=1

[status:status]

info=Status worker, displays runtime informations

[uri:/jkstatus/*]

info=Display status information and checks the config file for changes.

group=status:status

Each section starts with a [type:name] entry, and the body of the sections are the various properties available for that type.

The [shm:] section defines shared memory in Apache, which can be either a disk file or in memory. anonymous means it's in memory. You need a shared memory section for the Apache Scoreboard, which is how Apache knows your configuration file has changed. I return to this subject later in the section "Change the Cluster Dynamically: The jkstatus Page."

The [channel.socket:] sections define the location of your Tomcat servers. The tomcatId is a name you give that instance of Tomcat—it must match the engine's jvmRoute entry in Tomcat's server.xml file, which we will set up shortly.

The [lb:] section is the load balancer. By default, this includes all the servers defined in the workers2.properties file (server1 and server2 in our case). The sticky property makes sessions sticky—once Apache has dispatched a request to a certain server, this property ensures requests in that session always go to the same server as long as it is available. The tomcatId entries here and the jvmRoute entries in server.xml are needed to make the sticky property work.

The [uri:] sections define which URL patterns to forward to Tomcat. You can forward your whole Web application (as in the example) or just the dynamic parts, and let Apache serve the static sections. Unless you have specific performance problems, let Tomcat deal with serving all your content, the static and the dynamic parts. The group property declares which load balancer to use.

The [status:] section and its [uri:/jkstatus/*] section are covered later—ignore them for now.

Full details on the available types and their properties can be found in Resources.

workers2.properties: An example

If a request arrives on port 80 for URL /servlets-examples/index.html, the operating system will hand the request to Apache. Apache will see a [uri:] entry that matches that URL and look up the load balancer.

As discussed previously, the [lb:] section defines sticky sessions. If the session cookie has a name tagged onto it, Apache will look up the [channel.socket:] with that tomcatId and delegate the request to the Tomcat on that host and port. If no name is on the cookie or sticky=0, Apache uses its load-balancing rules to pick a server and dispatch to it. We have not specified any load-balancing rules, so Apache will alternate requests between servers.

Tomcat setup: server.xml

Having configured Apache as our dispatcher, we now need to configure the Tomcat workers.

Edit the first Tomcat server's conf/server.xml. Uncomment the entry labeled "Coyote/JK2 AJP 1.3 Connector." It should read something like this:

<Connector port="8009"

enableLookups="false" redirectPort="8443" debug="0"

protocol="AJP/1.3" />

Edit the Engine entry to have a jvmRoute matching the tomcatId entry in workers2.properties:

Do the same for server2, but on the AJP (Apache JServ Protocol) connector, set port="8010" and set the engine entry to jvmRoute="server2".

If you skipped the basic setup section earlier, you will also need to change the port on the secondary server's server element.

Know which server is being used

Tomcat comes with many preinstalled Web applications. We will use the servlets-examples application for our testing. Specifically, we will use the SessionExample servlet. Before doing so, we must edit the servlet to reflect which server we are using. Edit webapps/servlets-examples/WEB-INF/classes/SessionExample.java on your first server by looking for:

out.println("<h3>" + title + "</h3>");

and change it to:

out.println("<h3>" + title + " - SERVER 1</h3>");

Do the same for server2.

Recompile each of those servlets using the command line:

javac -classpath "/usr/local/tomcat1/common/lib/servlet-api.jar:." SessionExample.java

Remember to replace /usr/local/tomcat1 with the path to your server.

Now we can tell what server we are using.

Test

Start both Tomcats and Apache. Go to http://localhost/servlets-examples/ and choose the SessionExample. You should see the page we have just compiled.

Now clear your session (close all your browser windows or open a different browser) and go to the example URL again. You should get a different server. If not, try a couple more times.

Once assigned a server, the load balancer's sticky property makes us use the same one, but each new session is assigned a server according to the load-balancer rules.

Add some session values. Now stop the Tomcat you are on. The Apache error log, usually found in /var/log/http/error_log, reports the connection loss. Our dispatcher will no longer send requests to that worker. It checks regularly to see if the worker is back up, and will start using it again as soon as it is.

When you stop the worker you are using, you don't receive any errors. The dispatcher will automatically send any further requests to another worker. Your session, however, is lost. Any values you just added will no longer be there.

To prevent this session loss, we must get the two Tomcats to talk to each other.

Cluster the Tomcats

We now have Apache acting as a dispatcher for our two Tomcat workers. The next step is to get the two Tomcats to swap session data. Once they share session data, we can stop any one of the Tomcat workers without the user losing his work or state.

Edit server.xml and remove the comment tags from around the Cluster element on both servers. On server2, change the Receiver element's tcpListenPort to 4002. As both our Tomcat servers are on the same machine, the port needs to differ.

Edit the webapps/servlets-examples/WEB-INF/web.xml on both servers. Below the <display-name>..</display-name> element, include the line:

Tomcat will only replicate those sessions for Web applications marked as distributable.

Start both servers back up. In catalina.out, you should notice the two servers discovering each other and communicating.

You can now stop either Tomcat and your session will not be lost. Add some values to your session and stop Tomcat, just as we did previously. This time, your values should remain. If we had not changed the servlet to print which server it runs on, we would not know anything had happened. Tomcat workers can come and go without the site users ever knowing. Neat!

Note: You cannot cluster different versions of Tomcat. For example 5.0.25 and 5.0.28 will not work. Data is sent between them in a serialized form, and Java will only deserialize an object to the same version of the classfile that serialized it.

Using the session

Objects placed on the session must implement java.io.Serializable for Tomcat's clustering to replicate them. Every session change is replicated to every other server in the cluster, so it is wise to use the session as little as possible.

Unless you perceive URL parameters to be a security risk, use URL parameters instead of the session. Work with the idea that users will copy the URL and email it to others—i.e., the application must be able to rebuild the state from the URL parameters.

When using HTTP basic authentication (over SSL or in a trusted environment), the user does not need to be on the session since the authentication details are sent in every request. You could write a servlet filter that retrieves the user details from the request, checks them, loads your User object, and puts it where the rest of the application can find it.

By reducing session usage to a minimum, the user's browsing experience becomes more natural—the browser's Forward and Back buttons will work as intended, bookmarking pages will work, as will sending URLs to other users.

Change the cluster dynamically: The jkstatus page

In the workers2.properties file, we defined a [status:status] element mapped to URI /jkstatus. Let's look at that in more depth.

Go to http://localhost/jkstatus. You should see something like this:

jkstatus worker

This is your workers2.properties file after Apache has digested it. Click the Scoreboard info link—if that page says no Scoreboard is available, check that you have the [shm:] element. That area of shared memory is the Scoreboard.

The most interesting aspect of the jkstatus worker is that it reloads the workers2.properties file when you hit that page. This means we can change the cluster configuration without restarting any servers.

For example, edit workers2.properties and disable server2 by adding the disabled property and increasing the ver number:

[channel.socket:server2]

host=127.0.0.1

port=8010

tomcatId=server2

ver=1

disabled=1

Go to /jkstatus, and you should notice a message in the top left corner saying something like:

Updated config version to 4 Status information for child 6

You should also see that the disabled field for server2 is set. Server2 will no longer accept any new sessions. Similarly, we could add a new Tomcat to the cluster by adding a new channel.socket entry.

By changing the owner of workers2.properties, we could edit it and reload it into the server without ever needing to reboot. We could add or remove cluster workers from an Ant or Maven task, from a shell script, from a CGI script, or even from a scheduled task such as a Unix cron job. Heck, we could even write a script that parsed Apache's log, calculated the current average load, and added or removed Tomcat workers as needed!

Simple scenarios

In the current setup, the two servers equally share the work. If server2 were on a less powerful machine, we might want to give it a lower percentage of the traffic. What percentage of the traffic a worker receives is set via the channel.socket element's lb_factor property. The smaller the number, the more often Tomcat will receive requests. The lb_factor property defaults to 1. To reduce server2's load, edit its entry to:

[channel.socket:server2]

host=127.0.0.1

port=8010

tomcatId=server2

ver=1

lb_factor=2

The other scenaro we might need is failover. For server2 to receive requests only if server1 fails, use the level property:

[channel.socket:server1]

host=127.0.0.1

port=8009

tomcatId=server1

level=0

ver=2

[channel.socket:server2]

host=127.0.0.1

port=8010

tomcatId=server2

ver=2

level=1

The level can be 0-3. All the lower-numbered levels are checked before any higher levels receive a request. Server2 will receive requests only if server1 no longer responds.

Note that we change the ver number for each edit so that jkstatus will reload the file.

Cleaning up

Having tested our Apache/Tomcat cluster, we can tidy up the server.xml. You can comment out the HTTP connector in the server.xml file, as it should not be accessible in production use. Another option is to use firewall rules to block those ports from the outside world. Then we can keep the connector for use in testing or for accessing administration applications not mapped by a [uri:] element in the workers2.properties file.

Troubleshooting resources

The most helpful page on the mod_jk2/AJP 1.3 setup (though terse, it contains much information):

http://jakarta.apache.org/tomcat/connectors-doc-archive/jk2/index.html

The jkstatus page on your local server shows if your workers2.properties file has been read correctly:

http://localhost/jkstatus

The Tomcat log file should contain clues as to what went wrong: /usr/local/tomcat1/logs/catalina.out

The Apache error log file: /var/log/http/error_log

Conclusion

We started with one instance of Tomcat and lost traffic whenever we needed to work on it. Now we have Apache with any number of Tomcat workers behind it, and we can stop and start these at will. Requests and sessions are safe. We can change the cluster settings without restarting Apache.

A few years ago, you could only get this functionality with an expensive, proprietary, and cumbersome J2EE server. Today we can do it with Apache and Tomcat. Thanks to the Tomcat developers and the open source model!

Resources

iptables:

http://www.netfilter.org/index.html

mod_proxy:

http://jakarta.apache.org/tomcat/tomcat-5.0-doc/proxy-howto.html

mod_jk2 binaries (scroll down to JK 2 Binary Releases):

http://jakarta.apache.org/site/binindex.cgi

mod_jk2 Gentoo e-build (use the attachment from Comment #38 or later):

http://bugs.gentoo.org/show_bug.cgi?id=18353

mod_jk2 Debian package:

http://packages.debian.org/testing/web/libapache2-mod-jk2

mod_jk2 source (scroll down to JK2 2.0.4 Source Release tar.gz):