17 Replies Latest reply: Sep 10, 2012 3:46 PM by carl type3 RSS

    CF crashes 1 to 2 times a weeks

    smiralles

      hey there guys, it does not fail! CF keeps crashing on us 1 to 2 times a week sometimes even 3! i have applied all the latest patches to the system and CF but it just keep getting worse and worse! I'm not a CF guy and im tired of google'n stuff. please let me know what kind of info you guys would need to help me out with these problems! i have researched and read eveything on this site. thanks in advanced.

       

      exception.log

      "    "Error","jrpp-799","09/06/12","03:17:45","XXX","Connection Failure: Status code unavailable The specific sequence of files included or processed is: C:\wwwroot\scripts\processEmail.cfm, line: 145 "

      coldfusion.tagext.net.HttpTag$HttpConnectionFailureException: Connection Failure: Status code unavailable

      "    "Error","jrpp-925","09/07/12","00:32:55","XXX","Java heap space The specific sequence of files included or processed is: C:\wwwroot\XXX-micro\index.cfm, line: 9 "

      java.lang.OutOfMemoryError: Java heap space

      "    "Error","jrpp-938","09/07/12","00:32:55","XXX","The request has exceeded the allowable time limit Tag: cfoutput The specific sequence of files included or processed is: C:\wwwroot\locator\search.cfm, line: 338 "

      coldfusion.runtime.RequestTimedOutException: The request has exceeded the allowable time limit Tag: cfoutput

      "    "Error","jrpp-929","09/07/12","00:34:16","XXX","The request has exceeded the allowable time limit Tag: CFLOOP The specific sequence of files included or processed is: C:\wwwroot\scripts\processEmail.cfm, line: 107 "

      coldfusion.runtime.RequestTimedOutException: The request has exceeded the allowable time limit Tag: CFLOOP

       

      server.log

      "    "Warning","jrpp-935","09/07/12","00:26:25",,"Thread: jrpp-935, processing template: C:\wwwroot\XXX-micro\index.cfm, completed in 55 seconds, exceeding the 45 second warning limit"

      "    "Warning","jrpp-913","09/07/12","00:26:27",,"Thread: jrpp-913, processing template: C:\wwwroot\XXX-micro\candidate.cfm, completed in 47 seconds, exceeding the 45 second warning limit"

      "    "Warning","jrpp-938","09/07/12","00:32:12",,"Thread: jrpp-938, processing template: C:\wwwroot\locator\search.cfm, completed in 60 seconds, exceeding the 45 second warning limit"

      "    "Warning","jrpp-925","09/07/12","00:32:53",,"Thread: jrpp-925, processing template: C:\wwwroot\XXX-micro\index.cfm, completed in 150 seconds, exceeding the 45 second warning limit"

      "    "Warning","jrpp-928","09/07/12","00:32:58",,"Thread: jrpp-928, processing template: C:\wwwroot\locator\search.cfm, completed in 163 seconds, exceeding the 45 second warning limit"

      "    "Warning","jrpp-932","09/07/12","00:33:01",,"Thread: jrpp-932, processing template: C:\wwwroot\locator\search.cfm, completed in 168 seconds, exceeding the 45 second warning

      application.log

      "    "Error","jrpp-925","09/07/12","00:32:55","XXX","Java heap space The specific sequence of files included or processed is: C:\wwwroot\XXX-micro\index.cfm, line: 9 "

      "    "Error","jrpp-938","09/07/12","00:32:55","XXX","The request has exceeded the allowable time limit Tag: cfoutput The specific sequence of files included or processed is: C:\wwwroot\locator\search.cfm, line: 338 "

      "    "Error","jrpp-929","09/07/12","00:34:16","XXX","The request has exceeded the allowable time limit Tag: CFLOOP The specific sequence of files included or processed is: C:\wwwroot\scripts\processEmail.cfm, line: 107 "

        • 1. Re: CF crashes 1 to 2 times a weeks
          Miguel-F Community Member

          Well probably a good place to start would be letting us know platforms and version numbers.  What OS?  What CF?  Stand-alone or j2ee?  What Web Server? etc.

           

          Also the entries from your app server's standard output log file when this happens.  For CF running under JRun j2ee this logfile is here: {drive}:\JRun4\logs\cfusion-out.log

          • 2. Re: CF crashes 1 to 2 times a weeks
            Dan Bracuk Community Member

            You might have some inefficient code causing the cpu to go to 100% utilization.

            • 3. Re: CF crashes 1 to 2 times a weeks
              smiralles Community Member

              We are running a a 2003 server STD with CF 9.01 i think. not sure about the standalone or if its the j2ee

              • 4. Re: CF crashes 1 to 2 times a weeks
                smiralles Community Member

                i do not have a {drive}:\JRun4\logs\cfusion-out.log location. not sure if thats bad or good

                • 5. Re: CF crashes 1 to 2 times a weeks
                  smiralles Community Member

                  how can i trace this?

                  • 6. Re: CF crashes 1 to 2 times a weeks
                    Miguel-F Community Member

                    JRun4 is the default location for ColdFusion install.  It can be changed however.  Do you have a JRun4 directory on your application server (server that is running ColdFusion)?  If so, it should have a 'logs' directory under it with the log files in that folder.

                    • 7. Re: CF crashes 1 to 2 times a weeks
                      smiralles Community Member

                      C:\ColdFusion9\logs is where i seem to have all my logs for CF i have looked almost everywhere to find this "folder" but no luck.

                      • 8. Re: CF crashes 1 to 2 times a weeks
                        Miguel-F Community Member

                        That's it.  I believe that directory structure also shows that you are running a standalone installation.  It is just a difference in how CF is installed and runs.  All-in-one instead of being deployed to a j2ee server.  JRun is still there though behind the scenes.

                         

                        Are you seeing any "out of memory" errors in those logs?

                        • 9. Re: CF crashes 1 to 2 times a weeks
                          Ernesto RA

                          I'm with Dan. The "java.lang.OutOfMemoryError: Java heap space" points to some process consuming all the memory JVM has available. Either you have lots of simultaneus requests, an incorrect CF configuration for your needs, or some inefficient code. The 47 to 168 seconds are huge for a script to complete, so I'd take a look to the code on those files.

                          • 10. Re: CF crashes 1 to 2 times a weeks
                            smiralles Community Member

                            i have in the past, not sure what else to do about them. the server has plenty of RAM and hdd space

                            • 11. Re: CF crashes 1 to 2 times a weeks
                              smiralles Community Member

                              the error code line is all blank. this only happens at night. its starting to drive me insane!! any pointers? i really have tried everything i can think off and read.

                              • 12. Re: CF crashes 1 to 2 times a weeks
                                carl type3 Community Member

                                Suggest check \coldfusion9\runtime\logs\ “out” and “event” logs for warnings errors prior to crash and when those applications slow down. Report back findings.

                                HTH, Carl.

                                • 13. Re: CF crashes 1 to 2 times a weeks
                                  smiralles Community Member

                                  this is what i got so far.

                                   

                                   

                                   

                                  09/07 00:32:58 error ROOT CAUSE:

                                  java.lang.OutOfMemoryError: Java heap space

                                      at java.util.Arrays.copyOf(Arrays.java:2786)

                                      at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)

                                      at java.io.DataOutputStream.write(DataOutputStream.java:90)

                                      at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)

                                      at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)

                                      at org.apache.bcel.classfile.ConstantUtf8.dump(ConstantUtf8.java:125)

                                      at org.apache.bcel.classfile.ConstantPool.dump(ConstantPool.java:230)

                                      at org.apache.bcel.classfile.JavaClass.dump(JavaClass.java:322)

                                      at org.apache.bcel.classfile.JavaClass.getBytes(JavaClass.java:290)

                                      at coldfusion.bytecode.JavaAssembler.getBytes(JavaAssembler.java:428)

                                      at coldfusion.compiler.TemplateAssembler.assemble(TemplateAssembler.java:111)

                                      at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:377)

                                      at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:147)

                                      at coldfusion.runtime.TemplateClassLoader$TemplateCache$1.fetch(TemplateClassLoader.java:442 )

                                      at coldfusion.util.LruCache.get(LruCache.java:180)

                                      at coldfusion.runtime.TemplateClassLoader$TemplateCache.fetchSerial(TemplateClassLoader.java :362)

                                      at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)

                                      at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)

                                      at coldfusion.util.SoftCache.get(SoftCache.java:81)

                                      at coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:609)

                                      at coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:785)

                                      at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:766)

                                      at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:337)

                                      at cfindex2ecfm1647231854.runPage(C:\wwwroot\XXX-micro\index.cfm:9)

                                      at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

                                      at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)

                                      at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)

                                      at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:381)

                                      at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)

                                      at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

                                      at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)

                                      at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)

                                   

                                  javax.servlet.ServletException: ROOT CAUSE:

                                  java.lang.OutOfMemoryError: Java heap space

                                      at java.util.Arrays.copyOf(Arrays.java:2786)

                                      at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)

                                      at java.io.DataOutputStream.write(DataOutputStream.java:90)

                                      at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)

                                      at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)

                                      at org.apache.bcel.classfile.ConstantUtf8.dump(ConstantUtf8.java:125)

                                      at org.apache.bcel.classfile.ConstantPool.dump(ConstantPool.java:230)

                                      at org.apache.bcel.classfile.JavaClass.dump(JavaClass.java:322)

                                      at org.apache.bcel.classfile.JavaClass.getBytes(JavaClass.java:290)

                                      at coldfusion.bytecode.JavaAssembler.getBytes(JavaAssembler.java:428)

                                      at coldfusion.compiler.TemplateAssembler.assemble(TemplateAssembler.java:111)

                                      at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:377)

                                      at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:147)

                                      at coldfusion.runtime.TemplateClassLoader$TemplateCache$1.fetch(TemplateClassLoader.java:442 )

                                      at coldfusion.util.LruCache.get(LruCache.java:180)

                                      at coldfusion.runtime.TemplateClassLoader$TemplateCache.fetchSerial(TemplateClassLoader.java :362)

                                      at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)

                                      at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)

                                      at coldfusion.util.SoftCache.get(SoftCache.java:81)

                                      at coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:609)

                                      at coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:785)

                                      at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:766)

                                      at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:337)

                                      at cfindex2ecfm1647231854.runPage(C:\wwwroot\XXX-micro\index.cfm:9)

                                      at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

                                      at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)

                                      at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)

                                      at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:381)

                                      at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)

                                      at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

                                      at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)

                                      at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)

                                   

                                      at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70 )

                                      at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

                                      at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)

                                      at jrun.servlet.FilterChain.service(FilterChain.java:101)

                                      at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)

                                      at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)

                                      at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)

                                      at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)

                                      at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

                                      at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)

                                      at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

                                      at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)

                                      at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

                                   

                                  09/07 00:33:03 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@51fffa02]) JRunPRoxyServer.invokeRunnable:

                                  java.lang.IllegalStateException

                                      at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)

                                      at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)

                                      at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)

                                      at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)

                                      at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

                                      at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)

                                      at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

                                      at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)

                                      at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

                                   

                                  java.lang.IllegalStateException

                                      at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)

                                      at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)

                                      at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)

                                      at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)

                                      at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

                                      at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)

                                      at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

                                      at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)

                                      at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

                                  • 14. Re: CF crashes 1 to 2 times a weeks
                                    Steve Sommers Community Member

                                    I suggest something like FusionReactor or some other CF/Java memory monitor (there is a free one I used as well but cannot think of the name). For me, it pinpointed a couple pages that were not releasing memory because of some differences between CF5 and CF9 when we ported the code. Your dump above may or may not reflect the actual culprit as memory issues are very hard to find. The prior request may have eaten all your memory and this request just happened to be the final straw.

                                    • 15. Re: CF crashes 1 to 2 times a weeks
                                      DanWilson_ MeganK

                                      You are going to want to get some monitoring on that pretty quickly.

                                       

                                      In almost every case, Heap Memory errors come from bad code.

                                       

                                      Lots of times, this comes from a bunch of queries in a loop. Do you have

                                      any entries in the log files around the time of the crash that might point

                                      to a scheduled task running or some other process?

                                       

                                      DW

                                      • 16. Re: CF crashes 1 to 2 times a weeks
                                        carl type3 Community Member

                                        Monitoring is a good idea that way you know what is going on. There are many free, donation and applications you pay for by way of tools to use.
                                        Free:
                                        -Logs (your already starting to get some useful errors)
                                        -CF Monitor (if you have an Enterprise licence ie not Standard)
                                        -JVM logging
                                        -Jconsole (part of Java JDK)
                                        -Jvisualvm (also part of Java JDK)
                                        -CF Server Manager (parts like details view might assist)
                                        -CF Metrics logging
                                        -CFSTAT (not to be disregarded)
                                        -others that do not come straight to mind
                                        Donation:
                                        -CFTracker (have not used that overly yet myself however looks good)
                                        Pay (tho may have trial period for free):
                                        -FusionReactor
                                        -SeeFusion
                                        -perhaps others

                                         

                                        Apologies to those that are missed in posting a quick forum reply.

                                         

                                        Given you know how one or more of those tools looks when CF is working well versa when CF is nearing crash then likely you can have a better idea as to what is the problem. Making an adjustment to a CF setting or CFM code and monitoring will show if adjustment has had any effect.

                                         

                                        Regards, Carl.

                                        • 17. Re: CF crashes 1 to 2 times a weeks
                                          carl type3 Community Member

                                          What is your CF9 JVM settings for minimum and maximum memory?

                                          Eg:

                                           

                                          Capture.JPG

                                           

                                          You think CF version is 9.0.1. Confirm version with -

                                           

                                          CFadmin > System Information

                                          Eg:
                                           
                                          About ColdFusion »

                                          Server Details 
                                          Server Product  ColdFusion 
                                          Version  9,0,1,274733   
                                          Edition  Enterprise   
                                          Update Level  /D:/ColdFusion9/lib/updates/chf9010001.jar   
                                          Java Version  1.6.0_17 

                                           

                                          Regards, Carl.