• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
0

CF 8 JVM memory is not being garbage collected.

Participant ,
Sep 29, 2008 Sep 29, 2008

Copy link to clipboard

Copied

I am baffled by something I am seeing on my QA server. I have an app that we load tested but when the test completed the JVM memory used was not released. I used CF Server Monitor to watch the memory usage and sometimes it spiked to the max and either the app failed or I got timeout exceptions.

This is the only app running on this server and the testing completed over an hour ago but the memory has not been released yet.

CF Admin settings:
Maximum JVM Heap Size (MB) 512

The CF Server JVM Setting arguments include: -server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC

I found a script that uses java.lang.Runtime and java.lang.management.managementFactory that dumps a JVM memory usage profile The latest dump follows:

JVM Monitor - ColdFusion Server - Enterprise v8,0,1,195765
JVM Memory Monitor - struct
Heap Memory Usage - Committed 481 MB
Heap Memory Usage - Initial 0.00 MB
Heap Memory Usage - Max 493 MB
Heap Memory Usage - Used 437 MB
JVM - Free Memory 44.0 MB
JVM - Max Memory 493 MB
JVM - Total Memory 481 MB
JVM - Used Memory 449 MB
Memory Pool - Code Cache - Used 8.80 MB
Memory Pool - PS Eden Space - Used 6.37 MB
Memory Pool - PS Old Gen - Used 428 MB
Memory Pool - PS Perm Gen - Used 52.4 MB
Memory Pool - PS Survivor Space - Used 3.50 MB
Non-Heap Memory Usage - Committed 62.8 MB
Non-Heap Memory Usage - Initial 18.3 MB
Non-Heap Memory Usage - Max 240 MB
Non-Heap Memory Usage - Used 61.2 MB

According to the CF Server Monitor JVM memory usage builds up to 477 MB then the app fails or timesout.
Session Scope memory usage: 0.27 KB
Application Scope memory usage: 1.370 KB
Server Scope memory usage: 3.12 KB

Since the test ended JVM Memory Usage has dropped back to 438 MB?

Besides CFAdmin nothing else is running on this CF Server. I've read several other memory related topics but none of them have helped.

Can someone tell me why the memory isn't being release? How can I further troubleshoot the problem?

Thx

pwp
TOPICS
Advanced techniques

Views

3.2K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Sep 29, 2008 Sep 29, 2008

Copy link to clipboard

Copied

What is your machine's RAM?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Sep 30, 2008 Sep 30, 2008

Copy link to clipboard

Copied

4GB.

I know I can increase the memory available to the JVM using the startup params but when I do it may take a bit longer but the system eventually crashes as well.

thx

pwp

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Sep 30, 2008 Sep 30, 2008

Copy link to clipboard

Copied

Use the ColdFusion server monitor memory page or review http://www.schierberl.com/cfblog/index.cfm/2006/10/12/ColdFusion_memoryLeak_profiler. I believe Sun offers similar capabilites to jrockit now, too.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Oct 01, 2008 Oct 01, 2008

Copy link to clipboard

Copied

>I know I can increase the memory available to the JVM
>using the startup params...

OK. Then I suppose you would know the rule of thumb:
maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the JVM)

>...when I do [increase the memory] it may take a bit longer but
>the system eventually crashes as well.

There may well be something wrong with Coldfusion. But then again there might just be an application that is generating an infinite loop or rogue threads. You have to rule this out.

The Administrator interface in Coldfusion 8 has the tools you need to find out what's causing the high memory usage. Just follow these steps:
1) Open the Coldfusion Administrator and go to the Server Monitor page.
2) Press the button Launch Server Monitor. The server monitoring interface opens.
3) Click on the Statistics tab and then on Memory Usage.

Study the various kinds of memory usage for clues. You might also want to see what happens when you click on the Run GC button in the Memory Usage Summary.


Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Oct 01, 2008 Oct 01, 2008

Copy link to clipboard

Copied

> OK. Then I suppose you would know the rule of thumb:
> maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the JVM)

The maximum stable heap size I've managed to get is around 1.0-1.2GB, on a
win32 system. On Solaris (running a 32-bit JVM), about 1.4GB. It *seems*
like GC doesn't actually clear out RAM properly if more than that much RAM
is being addressed.

I'ev no experience with 64-bit JVMs.

What version JVM is in use here? And OS?

I think your PERM size seems quite big in relation to your heap size. But
that's not based on any specific knowledge, just the ratio I've seen in
various different configurations.

I suppose there was a reason to change from the defaults? Can you detail
why the settings are what they are?

--
Adam

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Oct 01, 2008 Oct 01, 2008

Copy link to clipboard

Copied

Thx, BKBK.

No. I didn't know the rule of thumb for setting maximum heap size but I did increase to 1024 MB and it still eventually ran out.

I have been monitoring JVM Memory Usage with CF Server Monitor.

Session, Server and Application scope memory usage all seem within reason. They increase and decrease as expected.

JVM memory usage climbs and may decline a little but never enough for the app to run for long.

Clicking the Run GC button has no affect.

I'm following up on ksmith's link but can't find the jrockit JVM or the mission control tool set to download. Apparently since Oracle and BEA merged things have changed some.

thx

pwp

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Oct 01, 2008 Oct 01, 2008

Copy link to clipboard

Copied

Adam Cameron wrote:
The maximum stable heap size I've managed to get is around
1.0-1.2GB, on a win32 system. On Solaris (running a 32-bit JVM),
about 1.4GB. It *seems* like GC doesn't actually clear out RAM
properly if more than that much RAM is being addressed.


Yes, there is a well-known 1.8 GB heap-size limit for 32-bit processors

I think your PERM size seems quite big in relation to your
heap size. But that's not based on any specific knowledge,
just the ratio I've seen in various different configurations.


Reminds me of another rule of thumb: set the initial heap size(Xms) and the maximum heap size(Xmx) to the same value.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Oct 01, 2008 Oct 01, 2008

Copy link to clipboard

Copied

PWP69 wrote:
I have been monitoring JVM Memory Usage with CF Server Monitor.
Session, Server and Application scope memory usage all seem within reason. They increase and decrease as expected.
JVM memory usage climbs and may decline a little but never enough for the app to run for long.


What database server does the application use? How many queries are there in the application?

I ask because there could also be the problem of query objects that the garbage collector fails to collect. This is a known problem for SQL Server. One of the solutions was to set Max Pooled Statements to zero in the application's datasource settings in the Coldfusion Administrator.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Oct 01, 2008 Oct 01, 2008

Copy link to clipboard

Copied

> Adam Cameron wrote:
> The maximum stable heap size I've managed to get is around
> 1.0-1.2GB, on a win32 system. On Solaris (running a 32-bit JVM),
> about 1.4GB. It *seems* like GC doesn't actually clear out RAM
> properly if more than that much RAM is being addressed.

>
> Yes, there is a well-known
> http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_19359&sliceId=1

Not really what I was talking about. One might be able to get the CF
instance to *start* with 1.8GB allocated to the heap, but it won't actually
work. I've managed to get a server to idle for a reasonable length of time
on 1.5GB, but as soon as the thing started to ramp up, it face-planted,
once it started actually trying to *use* the higher end of the RAM
allocated to it. At 1.2GB, it'll seem to run OK for a reasonable length of
time, but eventually it starts leaking memory; at around 1GB, it was pretty
stable.

Hence my comment about it being *stable* at that allocation. Not that "it
simply won't start if more than 1.8GB is allocated to it".

My point was that your rule of thumb:

maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the
JVM)

Is not a very good one. Plug 4GB RAM (so a small server) and one CF
instance into that equation. Your rule suggests I should be allocating 2GB
to the heap. Which - as you yourself pointed out - won't work.

--
Adam

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Oct 01, 2008 Oct 01, 2008

Copy link to clipboard

Copied

LATEST
Adam cameron wrote:
My point was that your rule of thumb:

maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the JVM)
Is not a very good one. Plug 4GB RAM (so a small server) and
one CF instance into that equation. Your rule suggests I should
be allocating 2GB to the heap. Which - as you yourself pointed
out - won't work.


There are always limits, of course. We still use Newton everyday, even though relativity has shown him to be incorrect at high speeds

A rule of thumb is just that: a rule of thumb. In these things, one is usually looking for an order of magnitude estimate. It is meant to give you an indication of the kind of value to expect or of how far you can go before the cookie begins to crumble. It is more art than exact science.

The elaborate, one-gig-is-stable example you give is just one such rule of thumb. It will crumble at 1 GB RAM. Also, though your machine fell flat on 1.5 GB, there are 32-bit machines that successfully run on 1.5 GB heap size with, say, 4 GB RAM. However, that does not discount your experience.

There could be other factors involved. For example, the type of application can have an influence on the workable maximum heap size. Hence my suggestion for PWP69 to look into queries.




Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Resources
Documentation