ColdFusion JVM total memory spiking and crashing server

[Guest]

Greetings; I'm currently using ColdFusion 8 32-bit Enterprise edition on win 2003 platform and witnessing an anormally. On the ColdFusion monitor, while viewing JVM memory (used/total), I have noticed that occasionally, the 'total' memory spikes to the heap size limit (1024MB), while 'used' memory remains constant or even drops. It's like the JVM is trying to allocate more memory eventhough it is not requested/required. Is it normal for total memory to occasionally spike and crash the server, while used memory consistently remains low (about 20% of limit set in jvm.config)? This happens periodically about once or twice a week unexpectedly and has been difficult to reproduce! Any thoughts?

[Guest]

Here is a screenshot from server monitor - JVM Memory
If you notice, at about 11:58am, there was a sudden jump in JVM total memory from about 384MB up to about 1022.44MB which crashed the server, eventhough 'used' memory did not increase. Is this unexpected behavior normal?

[Melvin T.]

Please provide these info:
1. Java Args in JVM config
2. Version Info (eg. 8,0,1,xxxxxx)

Also:
1. Are you running any scheduled task?
2. Does your application uses any COM objects?
3. Could you isolate what running pages at the time of the crash?

There maybe times where it is even difficult (or not totally) to launch the CF Server monitor when a crash occurs. Getting thread dump is the best way to go. I think the technote here is for MX 6.1 - 7, but it could still be applied for CF8. Here's the link:
http://www.adobe.com/go/tn_18339
It's a pretty informative technote, one that's I've used toe most when I once worked with MM support.

Hope this helps.

[Guest]

Thanks for the reply Melvin and for the link as well. Will follow-up... In response to the questions you raised:
1. Are you running any scheduled task?
No
2. Does your application uses any COM objects?
No dll
3. Could you isolate what running pages at the time of the crash?
None that I have noticed so far.

Have you had a similar case when total memory spikes above threshhold but 'Used' memory remained low? It's like the system is suddenly asking for more memory when it doesn't need it! My environment is below:

CF Version: 8,0,1,195765 Enterprise
OS: Windows 2003
Java Version: Sun JVM 1.6.0_04

# Arguments to VM
java.args=-server -Xmx1024m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=256m -XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/

[speedpedal]

CF8 uses more memory than its CFMX7, and that mainly because of the JVM it uses. IMO, occasional memory spikes could be a hard one to decipher, but I would be more concern if it crashes the server. If it happens, then a thread dump would tell us what's going on during the crash.

I also observed that your maxPermSize is set to 256. Try to increase that to 512 and see what happens. If your site is heavy on CFC or you are running an MVC (Struts, Model-Glue) onto your site, then this is the region it is being allocated to.

HTH. If you can get the thread dump, I can help you read it.

Thanks.

[Guest]

I'll look into increasing maxPermSize. The only framework we use is fusebox. What's strange is, when the memory spikes, JRun's Virtual Memory allocation (reading from Task Manager) also spikes to about 1.3Gb before ColdFusion server crashes. However, the 'Used' memory remains low when that happens following the screenshot above. What's strange too is, the problem occurs periodically, even though the applications are accessed everyday.

I'll see if I can send you a thread dump when it happens. This is what I'm getting in my instance log file:

java.lang.OutOfMemoryError: PermGen space
Exception in thread "jrpp-106" java.lang.OutOfMemoryError: PermGen space
01/30 15:59:38 Error [jrpp-107] - PermGen space The specific sequence of files included or processed is: <path to file>
java.lang.OutOfMemoryError: PermGen space
java.lang.OutOfMemoryError: PermGen space

This is from the exception.log file:
"Error","jrpp-107","01/30/09","15:59:38","<Application Name>","PermGen space The specific sequence of files included or processed is: <path to file>'' "
java.lang.OutOfMemoryError: PermGen space
"Error","jrpp-110","01/30/09","16:09:01",,"PermGen space The specific sequence of files included or processed is: <path to file>'' "
java.lang.OutOfMemoryError: PermGen space

[speedpedal]

We might have a winner!

"java.lang.OutOfMemoryError: PermGen space"

I have a feeling that the <path to file> points to a CFC related to fusebox. if it is, then you can start investigating there.The error though is pretty self explanatory.

Here's my other suggestion:

- Upgrade your OS to 64 bit. This will enable you to allocate more memory to your jvm. On a 32-bit system, you can only allocate up to 1.5gb to the max heap. Here's a good reading on this issue:

http://www.talkingtree.com/blog/index.cfm/2007/8/29/CF8-Performance-64-bit

- Switch to a multi-server install, so you can take advantage of Jrun Load-balancing. You can google some resources on how to set this up.

HTH.

[Guest]

Thanks again. We're already exploring 64bit upgrade. Meanwhile, it's strange that we did not have PermGen errors with CF7 and before, up until we upgraded to CF8. I checked the file alongside the error, and there is nothing peculiar - it's a file that processes form submission. It is also possible that the file is processed during or right after JRun hangs, so it shows up with the PermGen error. It is also strange that the total memory would spike and crash server while the 'Used' memory remains low, meaning it could be an issue (instability) with the Java virtual machine as well.