Skip navigation
Currently Being Moderated

CF 8 JVM memory is not being garbage collected.

Sep 29, 2008 10:03 AM

I am baffled by something I am seeing on my QA server. I have an app that we load tested but when the test completed the JVM memory used was not released. I used CF Server Monitor to watch the memory usage and sometimes it spiked to the max and either the app failed or I got timeout exceptions.

This is the only app running on this server and the testing completed over an hour ago but the memory has not been released yet.

CF Admin settings:
Maximum JVM Heap Size (MB) 512

The CF Server JVM Setting arguments include: -server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC

I found a script that uses java.lang.Runtime and java.lang.management.managementFactory that dumps a JVM memory usage profile The latest dump follows:

JVM Monitor - ColdFusion Server - Enterprise v8,0,1,195765
JVM Memory Monitor - struct
Heap Memory Usage - Committed 481 MB
Heap Memory Usage - Initial 0.00 MB
Heap Memory Usage - Max 493 MB
Heap Memory Usage - Used 437 MB
JVM - Free Memory 44.0 MB
JVM - Max Memory 493 MB
JVM - Total Memory 481 MB
JVM - Used Memory 449 MB
Memory Pool - Code Cache - Used 8.80 MB
Memory Pool - PS Eden Space - Used 6.37 MB
Memory Pool - PS Old Gen - Used 428 MB
Memory Pool - PS Perm Gen - Used 52.4 MB
Memory Pool - PS Survivor Space - Used 3.50 MB
Non-Heap Memory Usage - Committed 62.8 MB
Non-Heap Memory Usage - Initial 18.3 MB
Non-Heap Memory Usage - Max 240 MB
Non-Heap Memory Usage - Used 61.2 MB

According to the CF Server Monitor JVM memory usage builds up to 477 MB then the app fails or timesout.
Session Scope memory usage: 0.27 KB
Application Scope memory usage: 1.370 KB
Server Scope memory usage: 3.12 KB

Since the test ended JVM Memory Usage has dropped back to 438 MB?

Besides CFAdmin nothing else is running on this CF Server. I've read several other memory related topics but none of them have helped.

Can someone tell me why the memory isn't being release? How can I further troubleshoot the problem?

Thx

pwp
 
Replies
  • Currently Being Moderated
    Sep 29, 2008 11:37 PM   in reply to pwp69
    What is your machine's RAM?

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 30, 2008 9:42 AM   in reply to pwp69
    Use the ColdFusion server monitor memory page or review http://www.schierberl.com/cfblog/index.cfm/2006/10/12/ColdFusion_memor yLeak_profiler. I believe Sun offers similar capabilites to jrockit now, too.
     
    |
    Mark as:
  • Currently Being Moderated
    Oct 1, 2008 3:01 AM   in reply to pwp69
    >I know I can increase the memory available to the JVM
    >using the startup params...

    OK. Then I suppose you would know the rule of thumb:
    maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the JVM)

    >...when I do [increase the memory] it may take a bit longer but
    >the system eventually crashes as well.

    There may well be something wrong with Coldfusion. But then again there might just be an application that is generating an infinite loop or rogue threads. You have to rule this out.

    The Administrator interface in Coldfusion 8 has the tools you need to find out what's causing the high memory usage. Just follow these steps:
    1) Open the Coldfusion Administrator and go to the Server Monitor page.
    2) Press the button Launch Server Monitor. The server monitoring interface opens.
    3) Click on the Statistics tab and then on Memory Usage.

    Study the various kinds of memory usage for clues. You might also want to see what happens when you click on the Run GC button in the Memory Usage Summary.


     
    |
    Mark as:
  • Currently Being Moderated
    Oct 1, 2008 4:53 AM   in reply to pwp69
    > OK. Then I suppose you would know the rule of thumb:
    > maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the JVM)

    The maximum stable heap size I've managed to get is around 1.0-1.2GB, on a
    win32 system. On Solaris (running a 32-bit JVM), about 1.4GB. It *seems*
    like GC doesn't actually clear out RAM properly if more than that much RAM
    is being addressed.

    I'ev no experience with 64-bit JVMs.

    What version JVM is in use here? And OS?

    I think your PERM size seems quite big in relation to your heap size. But
    that's not based on any specific knowledge, just the ratio I've seen in
    various different configurations.

    I suppose there was a reason to change from the defaults? Can you detail
    why the settings are what they are?

    --
    Adam
     
    |
    Mark as:
  • Currently Being Moderated
    Oct 1, 2008 12:16 PM   in reply to pwp69
    Adam Cameron wrote:
    The maximum stable heap size I've managed to get is around
    1.0-1.2GB, on a win32 system. On Solaris (running a 32-bit JVM),
    about 1.4GB. It *seems* like GC doesn't actually clear out RAM
    properly if more than that much RAM is being addressed.


    Yes, there is a well-known 1.8 GB heap-size limit for 32-bit processors

    I think your PERM size seems quite big in relation to your
    heap size. But that's not based on any specific knowledge,
    just the ratio I've seen in various different configurations.


    Reminds me of another rule of thumb: set the initial heap size(Xms) and the maximum heap size(Xmx) to the same value.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 1, 2008 12:25 PM   in reply to pwp69
    PWP69 wrote:
    I have been monitoring JVM Memory Usage with CF Server Monitor.
    Session, Server and Application scope memory usage all seem within reason. They increase and decrease as expected.
    JVM memory usage climbs and may decline a little but never enough for the app to run for long.


    What database server does the application use? How many queries are there in the application?

    I ask because there could also be the problem of query objects that the garbage collector fails to collect. This is a known problem for SQL Server. One of the solutions was to set Max Pooled Statements to zero in the application's datasource settings in the Coldfusion Administrator.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 1, 2008 12:55 PM   in reply to pwp69
    > Adam Cameron wrote:
    > The maximum stable heap size I've managed to get is around
    > 1.0-1.2GB, on a win32 system. On Solaris (running a 32-bit JVM),
    > about 1.4GB. It *seems* like GC doesn't actually clear out RAM
    > properly if more than that much RAM is being addressed.

    >
    > Yes, there is a well-known
    > http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_19359&sli ceId=1

    Not really what I was talking about. One might be able to get the CF
    instance to *start* with 1.8GB allocated to the heap, but it won't actually
    work. I've managed to get a server to idle for a reasonable length of time
    on 1.5GB, but as soon as the thing started to ramp up, it face-planted,
    once it started actually trying to *use* the higher end of the RAM
    allocated to it. At 1.2GB, it'll seem to run OK for a reasonable length of
    time, but eventually it starts leaking memory; at around 1GB, it was pretty
    stable.

    Hence my comment about it being *stable* at that allocation. Not that "it
    simply won't start if more than 1.8GB is allocated to it".

    My point was that your rule of thumb:

    maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the
    JVM)

    Is not a very good one. Plug 4GB RAM (so a small server) and one CF
    instance into that equation. Your rule suggests I should be allocating 2GB
    to the heap. Which - as you yourself pointed out - won't work.

    --
    Adam
     
    |
    Mark as:
  • Currently Being Moderated
    Oct 1, 2008 2:30 PM   in reply to pwp69
    Adam cameron wrote:
    My point was that your rule of thumb:

    maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the JVM)
    Is not a very good one. Plug 4GB RAM (so a small server) and
    one CF instance into that equation. Your rule suggests I should
    be allocating 2GB to the heap. Which - as you yourself pointed
    out - won't work.


    There are always limits, of course. We still use Newton everyday, even though relativity has shown him to be incorrect at high speeds

    A rule of thumb is just that: a rule of thumb. In these things, one is usually looking for an order of magnitude estimate. It is meant to give you an indication of the kind of value to expect or of how far you can go before the cookie begins to crumble. It is more art than exact science.

    The elaborate, one-gig-is-stable example you give is just one such rule of thumb. It will crumble at 1 GB RAM. Also, though your machine fell flat on 1.5 GB, there are 32-bit machines that successfully run on 1.5 GB heap size with, say, 4 GB RAM. However, that does not discount your experience.

    There could be other factors involved. For example, the type of application can have an influence on the workable maximum heap size. Hence my suggestion for PWP69 to look into queries.




     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points