• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
0

CF10 standard crashing every 2-3 hours

Community Beginner ,
Jun 18, 2014 Jun 18, 2014

Copy link to clipboard

Copied

Any ideas about what could be crashing CF 10 standard repeatedly?  My box is a virtualized Windows Server 64bit 2008 R2, Apache 2.2, 8GB RAM.

I've tried increasing the JVM memory settings, using the latest JVM externally, cleaning up code and slower queries, and rebuilding the web connector.  Restarting the CF service fixes things until the next crash. When CF is unresponsive, Apache is serving static pages just fine; FusionReactor, shows normal JVM heap memory and CPU isn't maxed and has no active requests running; MySQL doesn't show a bunch of slow queries.  The CF exception logs don't show anything amiss until a restart when I get a warning about a possible memory leak. 

This has been very frustrating because it just stops working at random intervals.

Two months ago, I posted this with no replies:

Still having this problem too and it's driving me nuts.  I don't know what changed, because CF10 was working fine for a while.  Now Coldfusion stops responding one or more times a day with a number of threads in the wait chain.  The only way to fix it is to restart the service.  When I do, I get the same error messages as others in this thread in the Coldfusion-error.log on startup:

"SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1].....Threads are going to be renewed over time to try and avoid a probable memory leak."

Some of these are followed by this error message somehow related to Solr collections:

"SEVERE: The web application [/] created a ThreadLocal with key of type [org.apache.solr.common.util.DateUtil.ThreadLocalDateFormat] (value [org.apache.solr.common.util.DateUtil$ThreadLocalDateFormat@a95aa91]) and a value of type [java.text.SimpleDateFormat] (value [java.text.SimpleDateFormat@5af7aed5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak."

May 01, 2014 9:11:28 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

It seems like the number of errors is about the same as the number of scheduled tasks.  It doesn't look like any useful error, exception, or access messages get logged before Coldfusion stops responding.

My setup is a virtualized 2008 Win Server-64bit, Apache 2.2 (stock 32bit), CF10-64bit, update 13

I've tried: 

  1. Various security improvements
  2. Searching for cfml errors or recursive code
  3. Adding RAM (8GB now)
  4. Pausing scheduled tasks
  5. Removing & reconnecting the server connector as Administrator as suggested
  6. Increasing the JVM heap size
  7. Using the latest 1.7.0_55  JRE externally

Next I guess I'll have to try reinstalling Coldfusion or rebuilding the server from scratch.  I have to say that moving to Railo is looking more appealing than upgrading to CF11.  This should just work without so much grief...

Views

727

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct answer

Enthusiast , Jun 18, 2014 Jun 18, 2014

Adam Read wrote:

"SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1].....Threads are going to be renewed over time to try and avoid a probable memory leak."

Some of these are followed by this error message somehow related to Solr collections:

"SEVERE: The web application [/] created a ThreadLocal with key of type [org.apache.solr.common.util.DateUtil.ThreadLocalDateFormat] (value [org.apache.solr.common.util.DateUtil$ThreadLocalDateFormat@a95

...

Votes

Translate

Translate
Enthusiast ,
Jun 18, 2014 Jun 18, 2014

Copy link to clipboard

Copied

Adam Read wrote:

"SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1].....Threads are going to be renewed over time to try and avoid a probable memory leak."

Some of these are followed by this error message somehow related to Solr collections:

"SEVERE: The web application [/] created a ThreadLocal with key of type [org.apache.solr.common.util.DateUtil.ThreadLocalDateFormat] (value [org.apache.solr.common.util.DateUtil$ThreadLocalDateFormat@a95aa91]) and a value of type [java.text.SimpleDateFormat] (value [java.text.SimpleDateFormat@5af7aed5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak."

May 01, 2014 9:11:28 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

You get this error only during the restart or shutdown of the server. You  need not to worry about this and you can totally ignore it. This would not cause any memory leak.

The only case when memory leak can occur would be – if ColdFusion server is deployed on a JEE server and the ColdFusion server is deployed/undeployed multiple times without restarting the JEE server.



Adam Read wrote:

Any ideas about what could be crashing CF 10 standard repeatedly?

You need to capture thread dump when you face the server crash issue.

Try to tune ColdFusion 10 Apache connector tuning : http://blogs.coldfusion.com/post.cfm/coldfusion-11-iis-connector-tuning

NOTE : The above article is for CF 11 and IIS however you can follow the same on CF 10 with Apache.

HTH

Thanks
VJ

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jun 18, 2014 Jun 18, 2014

Copy link to clipboard

Copied

The thread dump doesn't appear to have any useful information, at least to me anyway.  In FusionReactor there are no running web requests to create a dump for.  If view all running threads, and get the stack trace of those, all I see is JVM/tomcat internal stuff and as well as waiting threads for each scheduled task and each DSN.  I've tried pausing/removing all scheduled tasks with no joy. 

I've modified my Apache connector settings as suggested after reading this: Tuning ColdFusion 10 IIS Connector configuration — Adobe ColdFusion Blog using these settings: Resolve Stability Problems and SPEED UP ColdFusion 10 » Web Trenches.  I tried this once before with no luck, but have since reinstalled CF10. It's been working fine for 3 hours now which is better than it has all day, so we'll see.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jul 09, 2014 Jul 09, 2014

Copy link to clipboard

Copied

Resetting the Apache connector settings did the trick.  I don't know why this didn't work before, but I had reinstalled CF10 since then and had mucked with it quite a bit after that before trying to change the connector settings again.  Coldfusion had been crashing (or at least stopped responding) several times a day, but now hasn't failed in a couple of weeks.

It sure seems to me that manually tuning a connector is OK, but needing to do this sort of thing just to get it to work at all isn't why I purchased Coldfusion instead of an open-source solution.

Thanks for the tip.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Enthusiast ,
Jul 14, 2014 Jul 14, 2014

Copy link to clipboard

Copied

Sounds great Adam

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jul 15, 2014 Jul 15, 2014

Copy link to clipboard

Copied

Good detective work, Adam.

Please mark your post as the correct answer. It will surely help someone else in future.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jul 20, 2014 Jul 20, 2014

Copy link to clipboard

Copied

I am very hopeful this is my situation as well.  We have 2 servers on different networks that started exhibiting this exact same issue and has been waking me up with alerts 3-4 times a night in the middle of the night mostly.  This started June 20-22 2014.  A month later, also using fusion reactor, I have nothing.

I am going to try the same thing you did and report back. Both servers have become unresponsive 4 times each since midnight.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jul 22, 2014 Jul 22, 2014

Copy link to clipboard

Copied

LATEST

We have applied these changes to both servers and neither has hung or crashed in almost 48 hours.  This sure looks like it was our problem as well.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Resources
Documentation