Adam Read wrote:
"SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1].....Threads are going to be renewed over time to try and avoid a probable memory leak."
Some of these are followed by this error message somehow related to Solr collections:
"SEVERE: The web application [/] created a ThreadLocal with key of type [org.apache.solr.common.util.DateUtil.ThreadLocalDateFormat] (value [org.apache.solr.common.util.DateUtil$ThreadLocalDateFormat@a95aa91]) and a value of type [java.text.SimpleDateFormat] (value [java.text.SimpleDateFormat@5af7aed5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak."
May 01, 2014 9:11:28 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
You get this error only during the restart or shutdown of the server. You need not to worry about this and you can totally ignore it. This would not cause any memory leak.
The only case when memory leak can occur would be – if ColdFusion server is deployed on a JEE server and the ColdFusion server is deployed/undeployed multiple times without restarting the JEE server.
Adam Read wrote:
Any ideas about what could be crashing CF 10 standard repeatedly?
You need to capture thread dump when you face the server crash issue.
Try to tune ColdFusion 10 Apache connector tuning : http://blogs.coldfusion.com/post.cfm/coldfusion-11-iis-connector-tuning
NOTE : The above article is for CF 11 and IIS however you can follow the same on CF 10 with Apache.
The thread dump doesn't appear to have any useful information, at least to me anyway. In FusionReactor there are no running web requests to create a dump for. If view all running threads, and get the stack trace of those, all I see is JVM/tomcat internal stuff and as well as waiting threads for each scheduled task and each DSN. I've tried pausing/removing all scheduled tasks with no joy.
I've modified my Apache connector settings as suggested after reading this: Tuning ColdFusion 10 IIS Connector configuration — Adobe ColdFusion Blog using these settings: Resolve Stability Problems and SPEED UP ColdFusion 10 » Web Trenches. I tried this once before with no luck, but have since reinstalled CF10. It's been working fine for 3 hours now which is better than it has all day, so we'll see.
Resetting the Apache connector settings did the trick. I don't know why this didn't work before, but I had reinstalled CF10 since then and had mucked with it quite a bit after that before trying to change the connector settings again. Coldfusion had been crashing (or at least stopped responding) several times a day, but now hasn't failed in a couple of weeks.
It sure seems to me that manually tuning a connector is OK, but needing to do this sort of thing just to get it to work at all isn't why I purchased Coldfusion instead of an open-source solution.
Thanks for the tip.
Sounds great Adam
Good detective work, Adam.
Please mark your post as the correct answer. It will surely help someone else in future.
I am very hopeful this is my situation as well. We have 2 servers on different networks that started exhibiting this exact same issue and has been waking me up with alerts 3-4 times a night in the middle of the night mostly. This started June 20-22 2014. A month later, also using fusion reactor, I have nothing.
I am going to try the same thing you did and report back. Both servers have become unresponsive 4 times each since midnight.
We have applied these changes to both servers and neither has hung or crashed in almost 48 hours. This sure looks like it was our problem as well.