10 Replies Latest reply: Oct 1, 2008 2:30 PM by BKBK RSS

    CF 8 JVM memory is not being garbage collected.

    pwp69 Community Member
      I am baffled by something I am seeing on my QA server. I have an app that we load tested but when the test completed the JVM memory used was not released. I used CF Server Monitor to watch the memory usage and sometimes it spiked to the max and either the app failed or I got timeout exceptions.

      This is the only app running on this server and the testing completed over an hour ago but the memory has not been released yet.

      CF Admin settings:
      Maximum JVM Heap Size (MB) 512

      The CF Server JVM Setting arguments include: -server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC

      I found a script that uses java.lang.Runtime and java.lang.management.managementFactory that dumps a JVM memory usage profile The latest dump follows:

      JVM Monitor - ColdFusion Server - Enterprise v8,0,1,195765
      JVM Memory Monitor - struct
      Heap Memory Usage - Committed 481 MB
      Heap Memory Usage - Initial 0.00 MB
      Heap Memory Usage - Max 493 MB
      Heap Memory Usage - Used 437 MB
      JVM - Free Memory 44.0 MB
      JVM - Max Memory 493 MB
      JVM - Total Memory 481 MB
      JVM - Used Memory 449 MB
      Memory Pool - Code Cache - Used 8.80 MB
      Memory Pool - PS Eden Space - Used 6.37 MB
      Memory Pool - PS Old Gen - Used 428 MB
      Memory Pool - PS Perm Gen - Used 52.4 MB
      Memory Pool - PS Survivor Space - Used 3.50 MB
      Non-Heap Memory Usage - Committed 62.8 MB
      Non-Heap Memory Usage - Initial 18.3 MB
      Non-Heap Memory Usage - Max 240 MB
      Non-Heap Memory Usage - Used 61.2 MB

      According to the CF Server Monitor JVM memory usage builds up to 477 MB then the app fails or timesout.
      Session Scope memory usage: 0.27 KB
      Application Scope memory usage: 1.370 KB
      Server Scope memory usage: 3.12 KB

      Since the test ended JVM Memory Usage has dropped back to 438 MB?

      Besides CFAdmin nothing else is running on this CF Server. I've read several other memory related topics but none of them have helped.

      Can someone tell me why the memory isn't being release? How can I further troubleshoot the problem?

      Thx

      pwp
        • 1. Re: CF 8 JVM memory is not being garbage collected.
          BKBK MVP
          What is your machine's RAM?

          • 2. Re: CF 8 JVM memory is not being garbage collected.
            pwp69 Community Member
            4GB.

            I know I can increase the memory available to the JVM using the startup params but when I do it may take a bit longer but the system eventually crashes as well.

            thx

            pwp
            • 3. Re: CF 8 JVM memory is not being garbage collected.
              ksmith Community Member
              Use the ColdFusion server monitor memory page or review http://www.schierberl.com/cfblog/index.cfm/2006/10/12/ColdFusion_memoryLeak_profiler. I believe Sun offers similar capabilites to jrockit now, too.
              • 4. Re: CF 8 JVM memory is not being garbage collected.
                BKBK MVP
                >I know I can increase the memory available to the JVM
                >using the startup params...

                OK. Then I suppose you would know the rule of thumb:
                maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the JVM)

                >...when I do [increase the memory] it may take a bit longer but
                >the system eventually crashes as well.

                There may well be something wrong with Coldfusion. But then again there might just be an application that is generating an infinite loop or rogue threads. You have to rule this out.

                The Administrator interface in Coldfusion 8 has the tools you need to find out what's causing the high memory usage. Just follow these steps:
                1) Open the Coldfusion Administrator and go to the Server Monitor page.
                2) Press the button Launch Server Monitor. The server monitoring interface opens.
                3) Click on the Statistics tab and then on Memory Usage.

                Study the various kinds of memory usage for clues. You might also want to see what happens when you click on the Run GC button in the Memory Usage Summary.


                • 5. Re: CF 8 JVM memory is not being garbage collected.
                  Community Member
                  > OK. Then I suppose you would know the rule of thumb:
                  > maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the JVM)

                  The maximum stable heap size I've managed to get is around 1.0-1.2GB, on a
                  win32 system. On Solaris (running a 32-bit JVM), about 1.4GB. It *seems*
                  like GC doesn't actually clear out RAM properly if more than that much RAM
                  is being addressed.

                  I'ev no experience with 64-bit JVMs.

                  What version JVM is in use here? And OS?

                  I think your PERM size seems quite big in relation to your heap size. But
                  that's not based on any specific knowledge, just the ratio I've seen in
                  various different configurations.

                  I suppose there was a reason to change from the defaults? Can you detail
                  why the settings are what they are?

                  --
                  Adam
                  • 6. Re: CF 8 JVM memory is not being garbage collected.
                    pwp69 Community Member
                    Thx, BKBK.

                    No. I didn't know the rule of thumb for setting maximum heap size but I did increase to 1024 MB and it still eventually ran out.

                    I have been monitoring JVM Memory Usage with CF Server Monitor.

                    Session, Server and Application scope memory usage all seem within reason. They increase and decrease as expected.

                    JVM memory usage climbs and may decline a little but never enough for the app to run for long.

                    Clicking the Run GC button has no affect.

                    I'm following up on ksmith's link but can't find the jrockit JVM or the mission control tool set to download. Apparently since Oracle and BEA merged things have changed some.

                    thx

                    pwp
                    • 7. Re: CF 8 JVM memory is not being garbage collected.
                      BKBK MVP
                      Adam Cameron wrote:
                      The maximum stable heap size I've managed to get is around
                      1.0-1.2GB, on a win32 system. On Solaris (running a 32-bit JVM),
                      about 1.4GB. It *seems* like GC doesn't actually clear out RAM
                      properly if more than that much RAM is being addressed.


                      Yes, there is a well-known 1.8 GB heap-size limit for 32-bit processors

                      I think your PERM size seems quite big in relation to your
                      heap size. But that's not based on any specific knowledge,
                      just the ratio I've seen in various different configurations.


                      Reminds me of another rule of thumb: set the initial heap size(Xms) and the maximum heap size(Xmx) to the same value.

                      • 8. Re: CF 8 JVM memory is not being garbage collected.
                        BKBK MVP
                        PWP69 wrote:
                        I have been monitoring JVM Memory Usage with CF Server Monitor.
                        Session, Server and Application scope memory usage all seem within reason. They increase and decrease as expected.
                        JVM memory usage climbs and may decline a little but never enough for the app to run for long.


                        What database server does the application use? How many queries are there in the application?

                        I ask because there could also be the problem of query objects that the garbage collector fails to collect. This is a known problem for SQL Server. One of the solutions was to set Max Pooled Statements to zero in the application's datasource settings in the Coldfusion Administrator.

                        • 9. Re: CF 8 JVM memory is not being garbage collected.
                          Community Member
                          > Adam Cameron wrote:
                          > The maximum stable heap size I've managed to get is around
                          > 1.0-1.2GB, on a win32 system. On Solaris (running a 32-bit JVM),
                          > about 1.4GB. It *seems* like GC doesn't actually clear out RAM
                          > properly if more than that much RAM is being addressed.

                          >
                          > Yes, there is a well-known
                          > http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_19359&sliceId=1

                          Not really what I was talking about. One might be able to get the CF
                          instance to *start* with 1.8GB allocated to the heap, but it won't actually
                          work. I've managed to get a server to idle for a reasonable length of time
                          on 1.5GB, but as soon as the thing started to ramp up, it face-planted,
                          once it started actually trying to *use* the higher end of the RAM
                          allocated to it. At 1.2GB, it'll seem to run OK for a reasonable length of
                          time, but eventually it starts leaking memory; at around 1GB, it was pretty
                          stable.

                          Hence my comment about it being *stable* at that allocation. Not that "it
                          simply won't start if more than 1.8GB is allocated to it".

                          My point was that your rule of thumb:

                          maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the
                          JVM)

                          Is not a very good one. Plug 4GB RAM (so a small server) and one CF
                          instance into that equation. Your rule suggests I should be allocating 2GB
                          to the heap. Which - as you yourself pointed out - won't work.

                          --
                          Adam
                          • 10. Re: CF 8 JVM memory is not being garbage collected.
                            BKBK MVP
                            Adam cameron wrote:
                            My point was that your rule of thumb:

                            maximum heap size(Xmx) = RAM(in MB) / (2 * number of servers using the JVM)
                            Is not a very good one. Plug 4GB RAM (so a small server) and
                            one CF instance into that equation. Your rule suggests I should
                            be allocating 2GB to the heap. Which - as you yourself pointed
                            out - won't work.


                            There are always limits, of course. We still use Newton everyday, even though relativity has shown him to be incorrect at high speeds

                            A rule of thumb is just that: a rule of thumb. In these things, one is usually looking for an order of magnitude estimate. It is meant to give you an indication of the kind of value to expect or of how far you can go before the cookie begins to crumble. It is more art than exact science.

                            The elaborate, one-gig-is-stable example you give is just one such rule of thumb. It will crumble at 1 GB RAM. Also, though your machine fell flat on 1.5 GB, there are 32-bit machines that successfully run on 1.5 GB heap size with, say, 4 GB RAM. However, that does not discount your experience.

                            There could be other factors involved. For example, the type of application can have an influence on the workable maximum heap size. Hence my suggestion for PWP69 to look into queries.