7 Replies Latest reply on Sep 19, 2016 7:16 PM by deanb69509320

    ColdFusion 9 sites going down regularly once per day

    deanb69509320

      Hello,

      I'm new to the forum, so please bare with me if I have not provided the proper details or in the desired format.

       

      Problem:  Daily, usually in the evening or early morning hours, USA time, our Cold Fusion sites are found down (4 sites, plus the default admin, cf instances) all connected through IIS and the Cold Fusion sites are configured to run as a Windows Service. 

       

      Resolution:  Start the sites via jrun.exe or start them via Windows Services (services.msc) and it works fine, no further issues until the next problem occurs again later.

       

      System: 
      Windows 2008 Enterprise 64bit
      VMware client running on VMHost - HOST info:  VMware ESXi 5.1.0 64bit, 1 CPU, 2 cores assigned.  6GB of RAM.  VMware host reports less than 50% CPU usage and memory across multiple VM Clients (Guests), no spikes of note.
      Cold Fusion 9 64bit

       

      Cold Fusion: 
      Server Product  ColdFusion 
      Version  9,0,1,274733   
      Edition  Enterprise   
      Operating System  Windows Server 2008   
      OS Version  6.1   
      Adobe Driver Version  4.0 (Build 0005)  

       

      Server has one a week backups, once monthly bounces, patches applied one a month on the server.  We have a production service, much higher usage, no problems.

       

      Environment:  Dev/test server, small team of users

       

      Users starting the service/sites via JRUN, have admin rights on the server.

       

      Storage is configured as SAN, as the F:\ drive on the server (17GB of 300GB free).  No known processes are running at this time.  Windows Security, Application, System, and Setup logs show only one error:

      "The description for Event ID 0 from source ColdFusion 9 ODBC Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

      If the event originated on another computer, the display information had to be saved with the event.

      The following information was included with the event:

      ColdFusion 9 ODBC Server@LOCALHOST,ErrorCode=2300,ErrorMessage=Network problem, connection unexpectedly closed by peer"

       

      Anyone have any thoughts as to what I can try or how can I track to see what is happening when this occurs (other than physically watching for it to happen) or if there is a known issue I haven't found?

       

      Thanks so much,

       

      DB

        • 1. Re: ColdFusion 9 sites going down regularly once per day
          carl type3 Level 4

          Hi Dean,

           

          Perhaps there will be some useful information in some CF logs.

           

          [CF]\runtime\logs server or [Jrun]\logs multiserver coldfusion-event.log & coldfusion-out.log

          check for -

          •java.lang.OutOfMemoryError: GC overhead limit exceeded

          •java.lang.OutOfMemoryError: Java heap space

          •java.lang.OutOfMemoryError: PermGen

          •java.lang.StackOverflowError:

          •java.lang.OutOfMemoryError: Out of swap space

           

           

          CF9.0.1 so perhaps Java 6 in use? CF9 some update levels (9.0.1 being one as best I recall) officially

          supports Java 7. It might not be much of an upgrade to change from EOL Java 6 to EOL Java 7 but could help.

          As I recall CF9 does not support current Java 8.

           

           

          HTH, Carl.

          • 2. Re: ColdFusion 9 sites going down regularly once per day
            deanb69509320 Level 1

            Hi Carl, thank you for taking the time.  Will engage developers to consider the same suggestion to upgrade.  Production running on like configuration/hardware has zero issues.

             

            The only recurring error I see in the event log (cfusion and all configured sites) is:

            error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@7b0acf26]) JRunPRoxyServer.invokeRunnable:

            java.lang.IllegalStateException

            at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)

            at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)

            at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)

            at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)

            at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

            at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

            at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

            ...this is a recurring issue in the logs.  I regret that I did overlook it as a possible cause, but is it the error I should be focusing on?  This is the only error that recurs in the logs

             

            and of course in the outlog:

            Information [scheduler-0] - Run Client Storage Purge recurs about every 1 to 1.5 hours

             

            In the production system, the output is pretty much the same on the outlog, but the event log does not have this above issue.

             

            And again, I was focusing on general cold fusion since all sites cashed, but when I started looking into the extra sites, I found this:

            javax.servlet.ServletException: ROOT CAUSE:

            java.lang.OutOfMemoryError: GC overhead limit exceeded

            at coldfusion.compiler.IntegerStack.<init>(JJTcfml40State.java:203)

            at coldfusion.compiler.JJTcfml40State.<init>(JJTcfml40State.java:34)

            at coldfusion.compiler.cfml40.<init>(cfml40.java:9)

            at coldfusion.compiler.EvaluateEngine.parse(EvaluateEngine.java:35)

            at coldfusion.compiler.ExprClassLoader$StatementCache.fetch(ExprClassLoader.java:73)

            at coldfusion.util.LruCache.get(LruCache.java:180)

            at coldfusion.compiler.ExprClassLoader$ExprCache.fetchSerial(ExprClassLoader.java:362)

            at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)

            at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)

            at coldfusion.util.SoftCache.get(SoftCache.java:81)

            at coldfusion.compiler.ExprClassLoader.compileStatement(ExprClassLoader.java:222)

            at coldfusion.compiler.ExprClassLoader.evaluate(ExprClassLoader.java:143)

            at coldfusion.runtime.CFPage.Evaluate(CFPage.java:1759)

            at cfact_scrolling2ecfm421729402.runPage(F:\inetpub\wwwroot\_Common\act_scrolling.cfm:8

             

             

            I'm checking the GC settings now and will adjust and test.

             

            Anything else you might suggest?  And thanks again.

             

            Dean

            • 3. Re: ColdFusion 9 sites going down regularly once per day
              carl type3 Level 4

              Hi Dean,

               

              Following up this:

              JRunPRoxyServer.invokeRunnable java.lang.IllegalStateException

               

              I searched and found this old CF9 thread where that turned out to be something in CFC code.

              Here is the post. See details refering to  - changed the following in my application.cfc

              <cfset This.Clientmanagement="false">

               

              https://forums.adobe.com/thread/776671

               

              I see you have this:

              java.lang.OutOfMemoryError: GC overhead limit exceeded

               

              JVM configuration can sometimes help with that.

               

              Regards, Carl.

              • 4. Re: ColdFusion 9 sites going down regularly once per day
                deanb69509320 Level 1

                Hi Carl,

                 

                Thanks to you putting me on possibly the right path, I have changed the Xms/Xmx settings to 256/384M.  On this server, there are 6 sites running and only 6GB of RAM.  At this value, with other processes, we gobble up 80% of physical memory.  If I bump it up to 512M, it hits 99%.  I think 384M, if they want to run all 6 sites, will be the limit.

                 

                I will also add this other value you suggested and report back (hopefully tomorrow) with an answer.

                 

                Thank you again sir.

                 

                Dean

                • 5. Re: ColdFusion 9 sites going down regularly once per day
                  carl type3 Level 4

                  Hi Dean,

                   

                  Well I am not sure what I have got to say will be helpful. I note CF9 64 bit so the Java JRE in use will be 64 bit as well. You can appreciate 384Mb on 32 bit is fair to say moderate on 64 bit I tend to think it is not near enough. As you point out with 6Gb total there is little to no wiggle room. Maybe some JVM tweaks can be applied to make the small heap evacuate frequently.

                   

                  Regards, Carl.

                  • 6. Re: ColdFusion 9 sites going down regularly once per day
                    carl type3 Level 4

                    Hi Dean,

                     

                    Perhaps enable some Java logging to see what the heap and non heap is doing then knowing that some kind if JVM tune can be figured out.

                     

                    -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfjvmGC.log

                     

                    Creates a log file in ColdFusion\runtime\bin\cfjvmgc.log Jrun4\bin\ in case multiserver

                     

                    HTH again, Carl.

                    • 7. Re: ColdFusion 9 sites going down regularly once per day
                      deanb69509320 Level 1

                      Thanks Carl - I will try that too.

                      Dean