Garbage Collection and Constant CPU Usage

[neochad]

Hello,

We have a load balanced environment running ColdFusion 10 Enterprise with 5 instances currently running. Currently we only have one instance being actively used but have been encountering a problem within a time frame of 12 to 36 hours (calculated on the times we have had to restart CF to come back online) the garbage collection is running almost constantly and CPU holds at 50% until the server just becomes unresponsive until the CF instance is restarted.

I have been monitoring the server via VisualVM to watch the CPU and Heap usage since I have found that running the server monitoring systems build into CF 10 Enterprise accelerate the problem to cause a crash within 4 to 5 hours.

This issue has been occuring regardless if there is heavy load or not on the server. I have been seeing some odd behaviour when running CF Stat where I am seeing 4 Requests Queued and Running Requests never shows anything other than 0. I increased the Minimum Heap Size to 512 MB and Maximum Heap Size to 1024 MB and below is our JVM arguments

-server -XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcom.sun.management.jmxremote.port=8701 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

We are running on a Windows 2008 64 Bit and IIS 7.5 on both of the load balanced servers and have updated the server to ColdFusion 10 Update 2. I am aware of the issues that are being fixed in Update 3 that was removed last week by Adobe due to bugs introduced.

If anyone has any insight in how to fix this issue, I would be greatly appreciative.  This is affecting a production server that did not show these problems when we put the servers under load when we did pre launch settings.

[carl type3]

Windows + CF10 both 64 bit I think:
- max 1024 can be small
- MaxPermSize 192m can be small
= also CF10 cfstat does not seem to work properly (think I have posted something on that previously)

Perhaps a screen cut of the visualvm CPU plus Heap and PermGen will be interesting to look at?

For GC stats you maybe better using another JDK tool Jconsole. Since you have the JMX settings in place use that as well. Although I note you say monitoring makes situation worse.

HTH, Carl.

[neochad]

Hello Carl,

Thanks for your response! Below is the screen shot of the CPU along with Heap and the PermGen. This is one of our servers that is showing the constant 50% CPU usage when I know we have practically nothing coming through the server right now. Since the restart of ColdFusion yesterday the Heap has been performing well. I have to admit I am not too knowledgeable as to the purpose of PermGen, if you could give a high level explanation as to what that is used for I would be in your debt.

Screen Shots: http://imgur.com/a/SHGvu

[neochad]

Hello Carl,

One other thing, I have only noticed an acceleration of the problem when using the built in monitoring in CF10 not when using Visual VM.

Regards,

[BKBK]

Have you identified ColdFusion as the definite cause of the increased CPU? If so, did you enable the use of client variables, and are you storing them in the registry? That seems to be a known cause of high CPU usage.

[neochad]

When viewing task manager on the servers the coldfusion.exe process is definately taking the CPU  but these are essentially virgin servers at this point so we really have nothing else running on them except IIS 7.5 and Coldfusion 10. Our client variables are set to Cookie storage and our web applications are designed to work with Sessions only. I have checked the registry size on each of the servers and it they are hovering around ~150 MB.

[neochad]

One very important thing I forgot to mention is that these are virtualized servers running in vSphere.

[carl type3]

if you search java permanent generation you will find plenty of information. I did a talk about CF JVM matters and in part discuss PermGen here http://experts.adobeconnect.com/p55663036/

The Visualvm shows the PermGen almost always full (blue area close to orange). Like heap PermGen offers a setting for minimum and maximum. Seeing that diagram perhaps alter JVM args to:
-server -XX:PermSize=192m -XX:MaxPermSize=324m -XX:+UseParallelGC etc

CF Monitor increases fall over?  Think that has been discussed before, you might want to search Charlie Arehart blog / discussions on CF Monitor.

I still think Jconsole would be good to see particularly where it shows Garbage Collection summary.

I am not virtual server savvy tho have CF8/9 on vmare and hyper-v.

HTH, Carl.

[carl type3]

Further from your images - while it shows CF CPU at 50% the GC activity is 0%.

So could be your not garbage collecting more than necessary and something else CF wise is taking CPU time.

What would be interesting is the Jconsole Memory tab Details section. It shows the time spent in minor and major GC's.  eg:

HTH, Carl.

[BKBK]

carl type3 wrote:

Further from your images - while it shows CF CPU at 50% the GC activity is 0%.

So could be your not garbage collecting more than necessary and something else CF wise is taking CPU time.

My thoughts exactly. High permgen very likely means the JVM is loading too many classes. GC of 0%, and constantly running, together with 50% CPU confirm this. They suggest a lot of activity going on in memory.

You're suffering from a memory leak, of sorts. One thing is for sure. Your CF instance is engaged in a process that is generating a large number of classes.

Do you use custom classloaders, or do you run applications that use them, for example, frameworks? Does CF have anything to do with the loadbalancing software? Have you ruled out infinite loops in your CF code, particularly in Application files?

[BKBK]

neochad wrote:

... the garbage collection is running almost constantly...

That jumped out at me. There are sources and there are sinks. The garbage collector is a sink. If it is running constantly, it implies there is a constant source of objects somewhere. For example, study the Application files to see where there could be a rampant generation of objects.