Skip navigation
neochad
Currently Being Moderated

Garbage Collection and Constant CPU Usage

Oct 25, 2012 10:16 AM

Tags: #64bit #garbage #administrator #cf10 #garbage_collection

Hello,

 

We have a load balanced environment running ColdFusion 10 Enterprise with 5 instances currently running. Currently we only have one instance being actively used but have been encountering a problem within a time frame of 12 to 36 hours (calculated on the times we have had to restart CF to come back online) the garbage collection is running almost constantly and CPU holds at 50% until the server just becomes unresponsive until the CF instance is restarted.

 

I have been monitoring the server via VisualVM to watch the CPU and Heap usage since I have found that running the server monitoring systems build into CF 10 Enterprise accelerate the problem to cause a crash within 4 to 5 hours.

 

This issue has been occuring regardless if there is heavy load or not on the server. I have been seeing some odd behaviour when running CF Stat where I am seeing 4 Requests Queued and Running Requests never shows anything other than 0. I increased the Minimum Heap Size to 512 MB and Maximum Heap Size to 1024 MB and below is our JVM arguments

 

-server -XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcom.sun.management.jmxremote.port=8701 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

 

We are running on a Windows 2008 64 Bit and IIS 7.5 on both of the load balanced servers and have updated the server to ColdFusion 10 Update 2. I am aware of the issues that are being fixed in Update 3 that was removed last week by Adobe due to bugs introduced.

 

If anyone has any insight in how to fix this issue, I would be greatly appreciative.  This is affecting a production server that did not show these problems when we put the servers under load when we did pre launch settings.

 
Replies
  • Currently Being Moderated
    Oct 25, 2012 8:11 PM   in reply to neochad

    Windows + CF10 both 64 bit I think:
    - max 1024 can be small
    - MaxPermSize 192m can be small
    = also CF10 cfstat does not seem to work properly (think I have posted something on that previously)

     

    Perhaps a screen cut of the visualvm CPU plus Heap and PermGen will be interesting to look at?

     

    For GC stats you maybe better using another JDK tool Jconsole. Since you have the JMX settings in place use that as well. Although I note you say monitoring makes situation worse.

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 26, 2012 3:54 AM   in reply to neochad

    neochad wrote:

     

    ... the garbage collection is running almost constantly...

    That jumped out at me. There are sources and there are sinks. The garbage collector is a sink. If it is running constantly, it implies there is a constant source of objects somewhere. For example, study the Application files to see where there could be a rampant generation of objects.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 26, 2012 7:58 AM   in reply to neochad

    Have you identified ColdFusion as the definite cause of the increased CPU? If so, did you enable the use of client variables, and are you storing them in the registry? That seems to be a known cause of high CPU usage.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 26, 2012 2:28 PM   in reply to neochad

    if you search java permanent generation you will find plenty of information. I did a talk about CF JVM matters and in part discuss PermGen here http://experts.adobeconnect.com/p55663036/

     

    The Visualvm shows the PermGen almost always full (blue area close to orange). Like heap PermGen offers a setting for minimum and maximum. Seeing that diagram perhaps alter JVM args to:
    -server -XX:PermSize=192m -XX:MaxPermSize=324m -XX:+UseParallelGC etc

     

    CF Monitor increases fall over?  Think that has been discussed before, you might want to search Charlie Arehart blog / discussions on CF Monitor.

     

    I still think Jconsole would be good to see particularly where it shows Garbage Collection summary.

     

    I am not virtual server savvy tho have CF8/9 on vmare and hyper-v.

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 28, 2012 5:02 PM   in reply to carl type3

    Further from your images - while it shows CF CPU at 50% the GC activity is 0%.

     

    So could be your not garbage collecting more than necessary and something else CF wise is taking CPU time.

     

    What would be interesting is the Jconsole Memory tab Details section. It shows the time spent in minor and major GC's.  eg:

     

    Capture1.JPG

     

     

    Capture2.JPG

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 29, 2012 1:03 AM   in reply to carl type3

    carl type3 wrote:

     

    Further from your images - while it shows CF CPU at 50% the GC activity is 0%.

     

    So could be your not garbage collecting more than necessary and something else CF wise is taking CPU time.

    My thoughts exactly. High permgen very likely means the JVM is loading too many classes. GC of 0%, and constantly running, together with 50% CPU confirm this. They suggest a lot of activity going on in memory.

     

    You're suffering from a memory leak, of sorts. One thing is for sure. Your CF instance is engaged in a process that is generating a large number of classes.

     

    Do you use custom classloaders, or do you run applications that use them, for example, frameworks? Does CF have anything to do with the loadbalancing software? Have you ruled out infinite loops in your CF code, particularly in Application files?

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points