How to Tune the JVM for High Availability Coldfusion Servers: Part 2 Memory

In Part 1 we covered why you needed to tune your JVM and installing the updated JVM.  Now we are going to tune it for performance.  We are going to rewrite the java.args line to accomplish this.

First we are going to set the size in memory of your JVM.  On 32bit OS'es your going to be limited by the 32bit addressing more than the physical server memory.  We found that even though you should be able to address upto 2.1Gb of memory the maximum realistic stable JVM size is going to be around 1Gb.  Depending on the size of your application and the amount of data you store in memory you may never need a large amount of memory space.  So lets start building the “jvm.args” line.  Xmx and Xms are the max and min size of the JVM memory space.  These need to be the same number (did you back up your original file?).  In the example below we are setting the JVM memory space to be 1Gb.  The default setting as shipped from Adobe is 512m. Also note that much of the jvm.args line is path information.  We'll be saving that to put back on when we are done tuning.  Also I am assuming your server has plenty of physical memory is a multi-CPU/core beast.  If your server only has 1Gb of physical RAM then don't set your JVM heap to 1Gb. Better yet, upgrade your server. Memory is cheap.

original line: (as it was from one of my old installs)


java.args=-server  -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS -Xmx512m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=128m -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib -Djava.compiler=NONE -Xnoagent -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000 -XX:+UseParallelGC -Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/cfform/jars

new line (as we rebuild it):
java.args= -server -Xmx1024m -Xms1024m

(Note that -server needs to be the first option according to Sun. This sets whether to use the client or server Hotspot VM)

Recently I was asked why should I set the min and max heap to the same size since the JVM will auto expand from your min (Xmm) to your max (Xmx) as needed.  The reason is that the JVM needs a contigous block of memory and the auto expanding takes precious time to do.  If there isn't enough contigous memory for the JVM to expand then it won't.  Part of the tuning process is to determine how much memory your CF application(s) need to run and configure your heap size accordingly.  (at least this is the way I learned it)

So now we need to add PermSize and MaxPermSize settings. These relate to the internal memory sizes.  The heap is further divided into Young and Old generation: (This is also related to Garbage Collecting).

java.args= -server -Xmx1024m -Xms1024m -XX:MaxPermSize=128m -XX:PermSize=128m

Sometimes you may need to increase the Min and Max perm sizes.  It is possible that 128m will not be enough in some cases and it can be increased.

java.args= -server -Xmx1024m -Xms1024m -XX:MaxPermSize=192m -XX:PermSize=192m

There is no set “perfect” value that works for everyone.  It comes down to your server and application's performance under load.  As I said before we found these settings for our servers based on trial and error. Also there is no set perfect value for your JVM heap size.  Trial and error will determine what will work best for your situation.  And after upgrading to CF 8 we retweaked the tuning and increased our heap and perm sizes.

A peek at our CF8 settings on 64Bit Linux with JVM 1.6_10 shows we're going even bigger.
java.args= -server -Xmx1536m -Xms1536m -XX:MaxPermSize=256m -XX:PermSize=256m

UPDATE: Since I wrote this we have upgraded the memory in our servers at work to have 8Gbs of RAM. Then our Sys Admin increased the JVM heap to 4Gb on each server. The settings now look like this.

java.args= -server -Xmx4096m -Xms4096m -XX:MaxPermSize=256m -XX:PermSize=256m

This goes to show that even though you tuned your JVM and all is running well you may have to re-tune again in the future. Our taffic increased enough to cause us to rework a few things.

This concludes the memory settings.

The next settings are related to the JVM Garbage Collection.  (For a complete reference of garbage collections see this [http://java.sun.com/docs/hotspot/gc1.4.2/].)

We are concerned with the type and timing of the Garbage Collections we're going to be doing.  All types are not equal.  And some are suited to specific server environments.

Since we are concerned with high availability we are going to discuss what works for multiple CPU/core servers. (your running multi-cpu/core servers right?)  This is the “Thoughput Collector”.  The setting is -XX:+UseParallelGC.  If you really want to know why then read this http://java.sun.com/docs/hotspot/gc1.4.2/#4. Types of Collectors.  I can assure you it will help you sleep.  

Essentially it allows parallel garbage collections across multiple threads at the same time.

So now our java.args looks like this.

java.args= -server -Xmx1024m -Xms1024m -XX:MaxPermSize=192m -XX:PermSize=192m  -XX:+UseParallelGC

Now that you've chosen the type of Garbage Collection we need to determine the frequency of collections.  At your house the garbage truck probably comes around once a week.  If it came less often you would have lots of trash building up outback and if they came more often you may not have any trash to pick up on certain days.  The same is true for the JVM garbage collection.  If you do it too often your calling a complete stop to JVM processing more often than you need and if you don't do it often enough your collection will take long enough to cause your users to experience lags.

What's this complete stop I'm talking about?  In order to do Garbage Collection the JVM needs to stop all processing. Yikes! This can be fatal if you don't get it right.  Oddly enough this critical setting has no default value when you install Coldfusion which means the JVM default of once per minute is used.  If your application has an unexplained pause about once per minute then this is most likely the culprit.  I've seen this happen a few times with some of my friends servers.  We also had the similar thing happen with our servers.  Our sysadmin found this little gem and shared it with me.  

We've been pretty lucky with the following settings.  This sets the collection interval to 10 minutes.

-Dsun.rmi.dgc.client.gcInterval=600000
-Dsun.rmi.dgc.server.gcInterval=600000


Will these values work for you? Sure, they'll work.  But they may not be optimal.  Garbage collection can become the bottle neck of your application.  Since all applications have different memory usage characteristics and needs you will most likely have to experiment with these settings as well.  That sleeper of a web page that describes the types of collections also discusses the collection interval.  

Now your java.args looks like this.

java.args= -server -Xmx1024m -Xms1024m -XX:MaxPermSize=192m -XX:PermSize=192m  -XX:+UseParallelGC -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000
 
If you are really having problems with your application pausing and making adjustments to these settings don't help much then read [http://java.sun.com/docs/hotspot/gc1.4.2/example.html] this page from Sun.  It's another sleeper, but it is pack full of information on detailed diagnosis of Garbage Collection problems.

Now you need to finish off you java.args line by putting the remaining original stuff back in place.

java.args= -server -DJINTEGRA_NATIVE_MODE -DJINTEGRA_PREFETCH_ENUMS -Xmx1024m -Xms1024m -XX:MaxPermSize=192m -XX:PermSize=192m  -XX:+UseParallelGC -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib -Djava.compiler=NONE -Xnoagent -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000    -Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/cfform/jars

Make sure there are no line breaks and save your settings.  Restart ColdFusion to make them take effect.  You did remember to backup your original jvm.config file right?  Many times I've made changes to test a setting and the JVM will not restart.  The fast fix is to replace the modified jvm.config file with the original one and start Coldfusion.  While this may not be so critical on a dev server, when your making changes to live production servers this can save your backside.

Cheers and have fun tuning your JVM.  

* Settings values are not set in stone - each server environment and application needs to be custom tuned.  I simply provide some guide posts along the way.