7 Replies Latest reply: Jul 22, 2014 9:42 AM by tmfprogrammer RSS

    CF10 standard crashing every 2-3 hours

    Adam Read Community Member

      Any ideas about what could be crashing CF 10 standard repeatedly?  My box is a virtualized Windows Server 64bit 2008 R2, Apache 2.2, 8GB RAM.

       

      I've tried increasing the JVM memory settings, using the latest JVM externally, cleaning up code and slower queries, and rebuilding the web connector.  Restarting the CF service fixes things until the next crash. When CF is unresponsive, Apache is serving static pages just fine; FusionReactor, shows normal JVM heap memory and CPU isn't maxed and has no active requests running; MySQL doesn't show a bunch of slow queries.  The CF exception logs don't show anything amiss until a restart when I get a warning about a possible memory leak. 

       

      This has been very frustrating because it just stops working at random intervals.

       

      Two months ago, I posted this with no replies:

       

      Still having this problem too and it's driving me nuts.  I don't know what changed, because CF10 was working fine for a while.  Now Coldfusion stops responding one or more times a day with a number of threads in the wait chain.  The only way to fix it is to restart the service.  When I do, I get the same error messages as others in this thread in the Coldfusion-error.log on startup:

       

      "SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1].....Threads are going to be renewed over time to try and avoid a probable memory leak."

       

      Some of these are followed by this error message somehow related to Solr collections:

       

      "SEVERE: The web application [/] created a ThreadLocal with key of type [org.apache.solr.common.util.DateUtil.ThreadLocalDateFormat] (value [org.apache.solr.common.util.DateUtil$ThreadLocalDateFormat@a95aa91]) and a value of type [java.text.SimpleDateFormat] (value [java.text.SimpleDateFormat@5af7aed5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak."

      May 01, 2014 9:11:28 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

       

      It seems like the number of errors is about the same as the number of scheduled tasks.  It doesn't look like any useful error, exception, or access messages get logged before Coldfusion stops responding.

       

      My setup is a virtualized 2008 Win Server-64bit, Apache 2.2 (stock 32bit), CF10-64bit, update 13

       

      I've tried: 

      1. Various security improvements
      2. Searching for cfml errors or recursive code
      3. Adding RAM (8GB now)
      4. Pausing scheduled tasks
      5. Removing & reconnecting the server connector as Administrator as suggested
      6. Increasing the JVM heap size
      7. Using the latest 1.7.0_55  JRE externally

       

      Next I guess I'll have to try reinstalling Coldfusion or rebuilding the server from scratch.  I have to say that moving to Railo is looking more appealing than upgrading to CF11.  This should just work without so much grief...

        • 1. Re: CF10 standard crashing every 2-3 hours
          vishu#13 Community Member

          Adam Read wrote:

           

          "SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1].....Threads are going to be renewed over time to try and avoid a probable memory leak."

           

          Some of these are followed by this error message somehow related to Solr collections:

           

          "SEVERE: The web application [/] created a ThreadLocal with key of type [org.apache.solr.common.util.DateUtil.ThreadLocalDateFormat] (value [org.apache.solr.common.util.DateUtil$ThreadLocalDateFormat@a95aa91]) and a value of type [java.text.SimpleDateFormat] (value [java.text.SimpleDateFormat@5af7aed5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak."

          May 01, 2014 9:11:28 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

           

           

          You get this error only during the restart or shutdown of the server. You  need not to worry about this and you can totally ignore it. This would not cause any memory leak.

          The only case when memory leak can occur would be – if ColdFusion server is deployed on a JEE server and the ColdFusion server is deployed/undeployed multiple times without restarting the JEE server.



          Adam Read wrote:

           

          Any ideas about what could be crashing CF 10 standard repeatedly?

           

          You need to capture thread dump when you face the server crash issue.

           

          Try to tune ColdFusion 10 Apache connector tuning : http://blogs.coldfusion.com/post.cfm/coldfusion-11-iis-connector-tuning

           

          NOTE : The above article is for CF 11 and IIS however you can follow the same on CF 10 with Apache.

           

          HTH

           

          Thanks
          VJ

          • 2. Re: CF10 standard crashing every 2-3 hours
            Adam Read Community Member

            The thread dump doesn't appear to have any useful information, at least to me anyway.  In FusionReactor there are no running web requests to create a dump for.  If view all running threads, and get the stack trace of those, all I see is JVM/tomcat internal stuff and as well as waiting threads for each scheduled task and each DSN.  I've tried pausing/removing all scheduled tasks with no joy. 

             

            I've modified my Apache connector settings as suggested after reading this: Tuning ColdFusion 10 IIS Connector configuration — Adobe ColdFusion Blog using these settings: Resolve Stability Problems and SPEED UP ColdFusion 10 » Web Trenches.  I tried this once before with no luck, but have since reinstalled CF10. It's been working fine for 3 hours now which is better than it has all day, so we'll see.

            • 3. Re: CF10 standard crashing every 2-3 hours
              Adam Read Community Member

              Resetting the Apache connector settings did the trick.  I don't know why this didn't work before, but I had reinstalled CF10 since then and had mucked with it quite a bit after that before trying to change the connector settings again.  Coldfusion had been crashing (or at least stopped responding) several times a day, but now hasn't failed in a couple of weeks.

               

              It sure seems to me that manually tuning a connector is OK, but needing to do this sort of thing just to get it to work at all isn't why I purchased Coldfusion instead of an open-source solution.

               

              Thanks for the tip.

              • 4. Re: CF10 standard crashing every 2-3 hours
                vishu#13 Community Member

                Sounds great Adam

                • 5. Re: CF10 standard crashing every 2-3 hours
                  BKBK MVP

                  Good detective work, Adam.

                  Please mark your post as the correct answer. It will surely help someone else in future.

                  • 6. Re: CF10 standard crashing every 2-3 hours
                    tmfprogrammer

                    I am very hopeful this is my situation as well.  We have 2 servers on different networks that started exhibiting this exact same issue and has been waking me up with alerts 3-4 times a night in the middle of the night mostly.  This started June 20-22 2014.  A month later, also using fusion reactor, I have nothing.

                     

                    I am going to try the same thing you did and report back. Both servers have become unresponsive 4 times each since midnight.

                    • 7. Re: CF10 standard crashing every 2-3 hours
                      tmfprogrammer Community Member

                      We have applied these changes to both servers and neither has hung or crashed in almost 48 hours.  This sure looks like it was our problem as well.