CF10 on CentOS 6.5 memory/CPU issues

[Feb]

I have been playing with CF10 performance on the CentOS 6.5 environment and so far I have lots of issues with it. I really appreciate any help on resolving the issues. It is upgraded version of CF from 8 to 10, both are 64-bit.

It seems to be memory leaks issue with CF so I have tried to adjust jvm.config as well as when ahead and upgraded CF's java version to the 1.7.0.55 version (tried this solution, thread (Re: SEVERE: The web application [/] created a ThreadLocal). Some of the issues were resolved but not all, the CF still eats memory then gets stuck. I have tried to use the following jvm settings:

Old one:

----------------------------------------------------------------------

java.args=-server  -Djava.awt.headless=true -Xms384m -Xmx1152m -Xmn128m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=384m -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:ParallelGCThreads=2 -javaagent:/opt/coldfusion10/newrelic/newrelic.jar -Xbatch -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars

----------------------------------------------------------------------

Here is the new one:

----------------------------------------------------------------------

java.args=-server  -Djava.awt.headless=true -Xms1280m -Xmx1280m -Xmn128m -Dsun.io.useCanonCaches=false -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:NewSize=48m -XX:SurvivorRatio=4 -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:ParallelGCThreads=2 -Xbatch -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars

----------------------------------------------------------------------

still no luck, at this time I am getting the following errors:

----------------------------------------------------------------------

Exception in thread "DefaultQuartzScheduler_QuartzSchedulerThread" java.lang.OutOfMemoryError: Java heap space

Exception in thread "ajp-bio-8012-exec-41" java.lang.OutOfMemoryError: Java heap space

Exception in thread "Timer-2" Exception in thread "ajp-bio-8012-exec-26" java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

Exception in thread "ajp-bio-8012-exec-43" java.lang.OutOfMemoryError: Java heap space

Exception in thread "ajp-bio-8012-exec-42" java.lang.OutOfMemoryError: Java heap space

Exception in thread "ajp-bio-8012-exec-45" java.lang.OutOfMemoryError: Java heap space

May 22, 2014 3:14:46 AM org.apache.catalina.core.StandardWrapperValve invoke

SEVERE: Servlet.service() for servlet [CfmServlet] in context with path [/] threw exception [ROOT CAUSE:

java.lang.OutOfMemoryError: Java heap space

] with root cause

javax.servlet.ServletException: ROOT CAUSE:

java.lang.OutOfMemoryError: Java heap space

        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70)

        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

----------------------------------------------------------------------

and the CF should be restarted.

CPU: 2.20GHz (2 cores)

RAM: 16Gb

The CF handles approximately 90 CF-based sites. Any help would be appreciated.

Thanks!

[Anit_Kumar]

Hi Feb,

There are two concerns over here:

From the error below, Sever Monitoring seems to be turned on and causing OutOfMemory error. Turn it off ( Server Monitoring > Settings, disable all). On a production box, we recommend to turn monitoring off, as it consumes lot of memory resources.

Feb wrote:

javax.servlet.ServletException: ROOT CAUSE:

java.lang.OutOfMemoryError: Java heap space

        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70 )

        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:243)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:539)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

----------------------------------------------------------------------

Secondly, I noticed that you are running 90 CF based websites. You may have to, tune your ColdFusion connectors as well. You may refer to ColdFusion 11 IIS Connector Tuning — Adobe ColdFusion Blog

Feb wrote:

The CF handles approximately 90 CF-based sites. Any help would be appreciated.

Thanks!

The jvm.config also needs tuning as well.

[Feb]

Hello Anit,

Thank you for the reply. I really appreciate it.

It is definitely disabled. I already encountered with those monitoring issues when there was enabled this setting "Enable Memory Tracking".

I am sorry I forgot to mention regarding the CF connectors. Here are the settings:

/opt/coldfusion10/config/wsconfig/1/workers.properties:

worker.cfusion.max_reuse_connections=250

worker.cfusion.connection_pool_size=1000

worker.cfusion.connection_pool_timeout=60

/opt/coldfusion10/cfusion/runtime/conf/server.xml:

<Connector executor="tomcatThreadPool" maxThreads="1000" port="8500" protocol="org.apache.coyote.http11.Http11Protocol" connectionTimeout="60000" redirectPort="8445"/>

Best regards,

Feb

[carl type3]

Hi Feb,

CF on CentOS is not my particular environment but it is Java so the environment difference should not matter.

Some comments for you to consider:

Max heap setting -Xmx1280m could you be more generous with 16Gb total RAM. Maybe -Xmx6144m

Sometimes setting initial and maximum the same is good sometimes bad. One would need to perform some logging to know if the heap is starting to have contiguous memory space problems.

On both examples you have used a mix of garbage collectors that would be fair to say tend to occupy memory more. Sometimes keeping object in memory is desirable however while you are having issues it might be suitable to change
garbage collection to something that is fair to say more prone to evacuating objects from the heap. CF by default installs with -XX:+UseParallelGC which I often change myself tho given your issue you might be better using for now.

You seem to have two parameters specifying “new” size of heap space in second example.

So perhaps you could try this suggestion for JVM args. If that does not suffer from java.lang.OutOfMemoryError: Java heap space problems then enable some logging to see what other adjustments to memory or garbage collection would be more suitable. As always take a backup copy of JVM.CONFIG before applying any changes.

java.args=-server -Djava.awt.headless=true -Xms2048m -Xmx6144m -Xmn128m -Dsun.io.useCanonCaches=false -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Xbatch -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application .home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/j ars,{application.home}/wwwroot/WEB-INF/cfform/jars

HTH, Carl.

[Feb]

Hello Carl,

My thought was the same as yours, but I have done as of yet because this value is really high. I will try it and let you know.

I know these JVM settings do not look an accurate or production settings. I though, I resolved the issue, but it arose again so I have gone astray.

My testing rules was as follow:

java.args=-server -Xms2048m -Xmx2048m -XX:MaxPermSize=256m -XX:NewRatio=4 -XX:SurvivorRatio=8 -XX:+UseCompressedOops -Xss256k -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:CMSInitiatingOccupancyFraction=68 -XX:TargetSurvivorRatio=90 -Xmn1024m -Dsun.io.useCanonCaches=false -XX:+CMSParallelRemarkEnabled -XX:ParallelGCThreads=2 -XX:PermSize=192m -Djava.awt.headless=true -Xbatch -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false

Ok, thank you for the suggestions, I will try it now and let you know.

Best regards,

Feb

[carl type3]

Hello Feb,

>My testing rules was as follow
Sure testing can be one thing production another. I have used similar JVM settings to those before in production and found those are good for keeping used objects allocated, most of the heap is collected with minor garbage collections (GC), pause effect is not noticeable and while you may get a full GC, a java stop all process, full GC would not be frequent and short duration since objects not referred to are well taken care of by the minor GC.

Having said that I don't have that type of GC in production  preferring to use garbage first (G1GC) since Java 1.7.0_40, G1GC being suitable to high object allocation,
low pause effect  for full GC are not done often and are short in duration. Plus the  JVM args are easier to read with about 4 G1GC parameters versa say 12 for the “testing” combination.

As for easy reading of JVM args Java 8 makes that even simpler for 8 does not use PermGen, tho CF does not list support for 8 yet so perhaps the less said about that the better.

Lets see if the Java memory errors cease with the sanity UseParallel setting then perhaps with some Java logging work forward from there to increase the performance of JVM.

Kind Regards, Carl.

[Feb]

Hello Carl,

Thank you for clarification. So, the G1GC is more suitable to high object allocation, right? I hoped we resolved the issue, but got another one:

----------------------------------------------------------------------------------------------

SEVERE: Servlet.service() for servlet [CfmServlet] in context with path [/] threw exception [ROOT CAUSE:

java.lang.OutOfMemoryError: GC overhead limit exceeded

] with root cause

javax.servlet.ServletException: ROOT CAUSE:

java.lang.OutOfMemoryError: GC overhead limit exceeded

        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70)

        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

----------------------------------------------------------------------------------------------

So, should I just add/enable G1GC to the previous configuration settings? Please advise.

java.args=-server -Djava.awt.headless=true -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -Xms2048m -Xmx6144m -Xmn128m -Dsun.io.useCanonCaches=false -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Xbatch -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars

Also, should I disable the error check altogether, via "-XX:-UseGCOverheadLimit".

Thank you!

Best regards,

Fedor

[carl type3]

Hi Fedor,

About enabling G1GC:
-was an experimental option in some Java 6
-released in Java 7
-some faults that were not fully rectified until 1.7.0_40
-I would not apply unless you altered CF to use Java 1.7.0_40 or greater
-Java 7 no need to specify enabling experimental option
-is suitable for high object allocation
-do some development testing to know if it suits your environment
-like other GC's comes with numerous options that might be practical for workload

I am not sure changing to G1GC will be helpful yet. For some reason Java spent a lot of time trying to collect the heap but could not regain free memory. Typically I find ParallelGC is good for keeping the heap free. Perhaps for now you may be better to stick with that and apply other settings to collect the heap at regular intervals, like one of your earlier examples with "-Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000".

Disable the error check via "-XX:-UseGCOverheadLimit" may well stop the message "java.lang.OutOfMemoryError: GC overhead limit exceeded" in logs but is not likely to stop the heap from being in a failing condition.

CF Java logging can be helpful to know how the heap is performing however it does mean work since someone has to read, understand and interpret logs, so perhaps enable logging if no other idea's come up. I think for now stick with ParallelGC and time forced GC's with a bit more for NewGen.

java.args=-server -Djava.awt.headless=true -Xms2048m -Xmx6144m -Xmn256m -Dsun.io.useCanonCaches=false -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Xbatch -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 -Dcoldfusion.home={application.home} -Detc ...

Kind Regards, Carl.

[Feb]

Hello Carl,

Thank you so much for the provided info.

I am still using this 1.7.0_55 java version. I understand that java spent a lot of time trying to regain free memory, but to no avail. I just do not understand why it is happening randomly.

I always check the CF logs, but am not able to identify the issue even with CF server monitor. The logging is already enabled. I can identify some java issues with configuration, but I am not a java person either programmer.

Ok, I will use the following args:

java.args=-server -Djava.awt.headless=true -Xms2048m -Xmx6144m -Xmn256m -Dsun.io.useCanonCaches=false -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Xbatch -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars

I will let you know how it goes.

Once again, thank you!

Best regards,

Fedor

[carl type3]

Hi Fedor,

I would be interested in outcome of those last decribed JVM settings. Since your CF is using 1.7.0_55 that is a good candidate for G1GC. I have some jvm args in mind that may well suit. Let's see what happens with "ParallelGC + server.gcInterval" first.

Kind Regards, Carl.

[Feb]

Hello Carl,

I apologize for the lack of updates on my end. Yes, the memory leak has not been resolved yet. The websites became inaccessible and I had to restart the CF10 two day ago. I did it twice. The CF10 worked for almost a week and it had to be restarted due to inaccessible of the sites.

The state of memory of CF was as follows:

VIRT was 8430m

RES was 1.5G

I do not see anything unusual in the CF logs that might cause an issue. When the CF was being stopped it threw the following errors:

-----------------------------------------------------------------------------------------------

Jun 10, 2014 7:21:01 AM org.apache.catalina.core.StandardServer await

WARNING: StandardServer.await: Invalid command '' received

Jun 10, 2014 7:21:01 AM org.apache.catalina.core.StandardServer await

INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.

Jun 10, 2014 7:21:01 AM org.apache.coyote.AbstractProtocol pause

INFO: Pausing ProtocolHandler ["http-bio-8500"]

Jun 10, 2014 7:21:01 AM org.apache.coyote.AbstractProtocol pause

INFO: Pausing ProtocolHandler ["ajp-bio-8012"]

Jun 10, 2014 7:21:01 AM org.apache.catalina.core.StandardService stopInternal

INFO: Stopping service Catalina

Jun 10, 2014 7:21:01 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/manager] appears to have started a thread named [SeedGenerator Thread] but has failed to stop it. This is very likely to create a memory leak.

Jun 10, 2014 7:21:01 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/manager] appears to have started a thread named [SeedGenerator Thread] but has failed to stop it. This is very likely to create a memory leak.

Jun 10, 2014 7:21:01 AM org.apache.catalina.core.StandardWrapper unload

INFO: Waiting for 9 instance(s) to be deallocated

Jun 10, 2014 7:21:02 AM org.apache.catalina.core.StandardWrapper unload

INFO: Waiting for 9 instance(s) to be deallocated

Jun 10, 2014 7:21:03 AM org.apache.catalina.core.StandardWrapper unload

INFO: Waiting for 9 instance(s) to be deallocated

Jun 10, 2014 7:21:09 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@3db0d7d9]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1402347354667,areFieldsSet=true,areAllFieldsSet=true,lenient=false,zone=sun.util.calendar.ZoneInfo[id="America/New_York",offset=-18000000,dstSavings=3600000,useDaylight=true,transitions=235,lastRule=java.util.SimpleTimeZone[id=America/New_York,offset=-18000000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2014,MONTH=5,WEEK_OF_YEAR=24,WEEK_OF_MONTH=2,DAY_OF_MONTH=9,DAY_OF_YEAR=160,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=2,AM_PM=1,HOUR=4,HOUR_OF_DAY=16,MINUTE=55,SECOND=54,MILLISECOND=667,ZONE_OFFSET=-18000000,DST_OFFSET=3600000]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

...........................................

Jun 10, 2014 7:21:09 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4e779769]) and a value of type [coldfusion.runtime.RequestInfo] (value [coldfusion.runtime.RequestInfo@7646c1bc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

...........................................

-----------------------------------------------------------------------------------------------

There was approx 300Mb of free RAM on the server. I do not know what I should enable to get more info in the logs, but I have created another server with the same configuration and am trying to debug it deeply as well as try to simulate high traffic load on a web application. I try to identify the bottlenecks by FusionReactor. The testing environment is working much better for some reason. I configured it exactly the same as prod.

Any help to expedite identifying the bottlenecks would be appreciated.

Thank you!

Best regards,

Fedor

[carl type3]

Hi Fedor,

Not used FR very much myself. I gather FR has an active community forum, perhaps you can post there as well to have the problem considered from other angles.

I like Java log and or monitoring (Java Management Extensions - JMX) to know how the environment is behaving, but it does take some work on your part to apply changes and inform log and or JMX information. There are two types of Java logging
I tend to deploy depending if technician likes log file or visual graphic type thinking. Since you have started down FR route perhaps JMX might suit you tho there are no reasons why one could not simultaneously run traditional Java logging to a file as well. Suggest have a look at both these presentations I did via cfmeetup then let me know if you want to try traditional Java logging and or JMX monitoring.

This talk demonstrates traditional Java logging:

http://experts.adobeconnect.com/p55663036/

This talk demonstrated JMX use of JDK Jconsole, more to look at CF tomcat connector than memory but I think it shows concept:

http://experts.adobeconnect.com/p8l51p4s9m4/

Regards, Carl.

[Feb]

Hello Carl,

I have run CF Server Monitor and it seems to be much better and simpler tool with the same functionality to review/examine the site load to identify bottleneck.

I will check those presentations. Thank you!

Best regards,

Fedor

[Feb]

Hello Carl/Everyone,

Just wanted to let you know, that the issue is not resolved yet and the CF10 was down recently. I noticed the following error in the coldfusion-error.log

----------------------------------------------------------------------------------------------------------------

Jun 18, 2014 5:01:31 AM org.apache.catalina.core.StandardServer await

WARNING: StandardServer.await: Invalid command '' received

Jun 18, 2014 5:01:31 AM org.apache.catalina.core.StandardServer await

INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.

Jun 18, 2014 5:01:31 AM org.apache.coyote.AbstractProtocol pause

INFO: Pausing ProtocolHandler ["http-bio-8500"]

Jun 18, 2014 5:02:01 AM org.apache.coyote.AbstractProtocol pause

INFO: Pausing ProtocolHandler ["ajp-bio-8012"]

Jun 18, 2014 5:02:37 AM org.apache.catalina.core.StandardService stopInternal

INFO: Stopping service Catalina

Exception in thread "Timer-5" Jun 18, 2014 5:31:35 AM org.apache.catalina.core.ContainerBase stopInternal

SEVERE: A child container failed during stop

java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded

        at java.util.concurrent.FutureTask.report(FutureTask.java:122)

        at java.util.concurrent.FutureTask.get(FutureTask.java:188)

        at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:1156)

        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)

        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1540)

        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1530)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

        at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1057)

        at sun.misc.URLClassPath.getResource(URLClassPath.java:199)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:358)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:412)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:270)

        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1675)

        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1533)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:270)

        at java.sql.DriverManager.isDriverAllowed(DriverManager.java:463)

        at java.sql.DriverManager.isDriverAllowed(DriverManager.java:455)

        at java.sql.DriverManager.getDrivers(DriverManager.java:356)

        at org.apache.catalina.loader.JdbcLeakPrevention.clearJdbcDriverRegistrations(JdbcLeakPrevention.java:59)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.catalina.loader.WebappClassLoader.clearReferencesJdbc(WebappClassLoader.java:2044)

        at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:1964)

        at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1876)

        at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:659)

        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)

        at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5499)

        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)

        at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1540)

----------------------------------------------------------------------------------------------------------------

It seems that error occurred, when the CF was restarted by a script.

I wrote that script which monitor the memory usage. It sends me a message if there is an issue. I received the following last message this morning:

----------------------------------------------------------------------------------------------------------------

Apache ran out of memory at 06-18-2014 08:15:01 and restarted. There were 203 httpd processes at the time, with an average size of 42170 KB, for a total Apache usage of 8560 MB.

----------------------------------------------------------------------------------------------------------------

Additional info from apache logs and mod_jk

mod_jk.log:

----------------------------------------------------------------------------------------------------------------

[Wed Jun 18 05:18:29 2014] [60628:140485010884576] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1293): (cfusion) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8012) is down (errno=104)

[Wed Jun 18 05:18:29 2014] [60628:140485010884576] [error] ajp_get_reply::jk_ajp_common.c (2140): (cfusion) Tomcat is down or refused connection. No response has been sent to the client (yet)

[Wed Jun 18 05:18:29 2014] [60628:140485010884576] [info] ajp_service::jk_ajp_common.c (2629): (cfusion) sending request to tomcat failed (recoverable),  (attempt=1)

[Wed Jun 18 05:18:30 2014] [60747:140485010884576] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1293): (cfusion) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8012) is down (errno=104)

[Wed Jun 18 05:18:30 2014] [60747:140485010884576] [error] ajp_get_reply::jk_ajp_common.c (2140): (cfusion) Tomcat is down or refused connection. No response has been sent to the client (yet)

----------------------------------------------------------------------------------------------------------------

error_log:

----------------------------------------------------------------------------------------------------------------

[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_lock(jk_log_lock) failed

[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_unlock(jk_log_lock) failed

[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_lock(jk_log_lock) failed

[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_unlock(jk_log_lock) failed

[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_lock(jk_log_lock) failed

----------------------------------------------------------------------------------------------------------------

I am still investigating the issue and will let you know if I find something helpful.

Thanks!

Feb