Skip navigation
smiralles
smiralles Calculating status...
Currently Being Moderated

CF crashes 1 to 2 times a weeks

Sep 7, 2012 5:44 AM

hey there guys, it does not fail! CF keeps crashing on us 1 to 2 times a week sometimes even 3! i have applied all the latest patches to the system and CF but it just keep getting worse and worse! I'm not a CF guy and im tired of google'n stuff. please let me know what kind of info you guys would need to help me out with these problems! i have researched and read eveything on this site. thanks in advanced.

 

exception.log

"    "Error","jrpp-799","09/06/12","03:17:45","XXX","Connection Failure: Status code unavailable The specific sequence of files included or processed is: C:\wwwroot\scripts\processEmail.cfm, line: 145 "

coldfusion.tagext.net.HttpTag$HttpConnectionFailureException: Connection Failure: Status code unavailable

"    "Error","jrpp-925","09/07/12","00:32:55","XXX","Java heap space The specific sequence of files included or processed is: C:\wwwroot\XXX-micro\index.cfm, line: 9 "

java.lang.OutOfMemoryError: Java heap space

"    "Error","jrpp-938","09/07/12","00:32:55","XXX","The request has exceeded the allowable time limit Tag: cfoutput The specific sequence of files included or processed is: C:\wwwroot\locator\search.cfm, line: 338 "

coldfusion.runtime.RequestTimedOutException: The request has exceeded the allowable time limit Tag: cfoutput

"    "Error","jrpp-929","09/07/12","00:34:16","XXX","The request has exceeded the allowable time limit Tag: CFLOOP The specific sequence of files included or processed is: C:\wwwroot\scripts\processEmail.cfm, line: 107 "

coldfusion.runtime.RequestTimedOutException: The request has exceeded the allowable time limit Tag: CFLOOP

 

server.log

"    "Warning","jrpp-935","09/07/12","00:26:25",,"Thread: jrpp-935, processing template: C:\wwwroot\XXX-micro\index.cfm, completed in 55 seconds, exceeding the 45 second warning limit"

"    "Warning","jrpp-913","09/07/12","00:26:27",,"Thread: jrpp-913, processing template: C:\wwwroot\XXX-micro\candidate.cfm, completed in 47 seconds, exceeding the 45 second warning limit"

"    "Warning","jrpp-938","09/07/12","00:32:12",,"Thread: jrpp-938, processing template: C:\wwwroot\locator\search.cfm, completed in 60 seconds, exceeding the 45 second warning limit"

"    "Warning","jrpp-925","09/07/12","00:32:53",,"Thread: jrpp-925, processing template: C:\wwwroot\XXX-micro\index.cfm, completed in 150 seconds, exceeding the 45 second warning limit"

"    "Warning","jrpp-928","09/07/12","00:32:58",,"Thread: jrpp-928, processing template: C:\wwwroot\locator\search.cfm, completed in 163 seconds, exceeding the 45 second warning limit"

"    "Warning","jrpp-932","09/07/12","00:33:01",,"Thread: jrpp-932, processing template: C:\wwwroot\locator\search.cfm, completed in 168 seconds, exceeding the 45 second warning

application.log

"    "Error","jrpp-925","09/07/12","00:32:55","XXX","Java heap space The specific sequence of files included or processed is: C:\wwwroot\XXX-micro\index.cfm, line: 9 "

"    "Error","jrpp-938","09/07/12","00:32:55","XXX","The request has exceeded the allowable time limit Tag: cfoutput The specific sequence of files included or processed is: C:\wwwroot\locator\search.cfm, line: 338 "

"    "Error","jrpp-929","09/07/12","00:34:16","XXX","The request has exceeded the allowable time limit Tag: CFLOOP The specific sequence of files included or processed is: C:\wwwroot\scripts\processEmail.cfm, line: 107 "

1846 Views   17 Replies   Latest reply: carl type3, Sep 10, 2012 3:46 PM
 
Replies
  • Miguel-F
    Currently Being Moderated
    1. Miguel-F,
    Sep 7, 2012 6:22 AM   in reply to smiralles
    Report

    Well probably a good place to start would be letting us know platforms and version numbers.  What OS?  What CF?  Stand-alone or j2ee?  What Web Server? etc.

     

    Also the entries from your app server's standard output log file when this happens.  For CF running under JRun j2ee this logfile is here: {drive}:\JRun4\logs\cfusion-out.log

     
    |
    Mark as:
  • Dan Bracuk
    Currently Being Moderated
    2. Dan Bracuk,
    Sep 7, 2012 7:44 AM   in reply to smiralles
    Report

    You might have some inefficient code causing the cpu to go to 100% utilization.

     
    |
    Mark as:
  • smiralles
    Currently Being Moderated
    3. smiralles,
    Sep 7, 2012 7:52 AM   in reply to Miguel-F
    Report

    We are running a a 2003 server STD with CF 9.01 i think. not sure about the standalone or if its the j2ee

     
    |
    Mark as:
  • smiralles
    Currently Being Moderated
    4. smiralles,
    Sep 7, 2012 7:56 AM   in reply to Miguel-F
    Report

    i do not have a {drive}:\JRun4\logs\cfusion-out.log location. not sure if thats bad or good

     
    |
    Mark as:
  • smiralles
    Currently Being Moderated
    5. smiralles,
    Sep 7, 2012 7:57 AM   in reply to Dan Bracuk
    Report

    how can i trace this?

     
    |
    Mark as:
  • Miguel-F
    Currently Being Moderated
    6. Miguel-F,
    Sep 7, 2012 8:03 AM   in reply to smiralles
    Report

    JRun4 is the default location for ColdFusion install.  It can be changed however.  Do you have a JRun4 directory on your application server (server that is running ColdFusion)?  If so, it should have a 'logs' directory under it with the log files in that folder.

     
    |
    Mark as:
  • smiralles
    Currently Being Moderated
    7. smiralles,
    Sep 7, 2012 8:41 AM   in reply to Miguel-F
    Report

    C:\ColdFusion9\logs is where i seem to have all my logs for CF i have looked almost everywhere to find this "folder" but no luck.

     
    |
    Mark as:
  • Miguel-F
    Currently Being Moderated
    8. Miguel-F,
    Sep 7, 2012 9:25 AM   in reply to smiralles
    Report

    That's it.  I believe that directory structure also shows that you are running a standalone installation.  It is just a difference in how CF is installed and runs.  All-in-one instead of being deployed to a j2ee server.  JRun is still there though behind the scenes.

     

    Are you seeing any "out of memory" errors in those logs?

     
    |
    Mark as:
  • Ernesto RA
    Currently Being Moderated
    9. Ernesto RA,
    Sep 7, 2012 12:36 PM   in reply to smiralles
    Report

    I'm with Dan. The "java.lang.OutOfMemoryError: Java heap space" points to some process consuming all the memory JVM has available. Either you have lots of simultaneus requests, an incorrect CF configuration for your needs, or some inefficient code. The 47 to 168 seconds are huge for a script to complete, so I'd take a look to the code on those files.

     
    |
    Mark as:
  • smiralles
    Currently Being Moderated
    10. smiralles,
    Sep 7, 2012 1:07 PM   in reply to Miguel-F
    Report

    i have in the past, not sure what else to do about them. the server has plenty of RAM and hdd space

     
    |
    Mark as:
  • smiralles
    Currently Being Moderated
    11. smiralles,
    Sep 7, 2012 1:07 PM   in reply to Ernesto RA
    Report

    the error code line is all blank. this only happens at night. its starting to drive me insane!! any pointers? i really have tried everything i can think off and read.

     
    |
    Mark as:
  • carl type3
    Currently Being Moderated
    12. carl type3,
    Sep 7, 2012 7:58 PM   in reply to smiralles
    Report

    Suggest check \coldfusion9\runtime\logs\ “out” and “event” logs for warnings errors prior to crash and when those applications slow down. Report back findings.

    HTH, Carl.

     
    |
    Mark as:
  • smiralles
    Currently Being Moderated
    13. smiralles,
    Sep 10, 2012 7:11 AM   in reply to carl type3
    Report

    this is what i got so far.

     

     

     

    09/07 00:32:58 error ROOT CAUSE:

    java.lang.OutOfMemoryError: Java heap space

        at java.util.Arrays.copyOf(Arrays.java:2786)

        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)

        at java.io.DataOutputStream.write(DataOutputStream.java:90)

        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)

        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)

        at org.apache.bcel.classfile.ConstantUtf8.dump(ConstantUtf8.java:125)

        at org.apache.bcel.classfile.ConstantPool.dump(ConstantPool.java:230)

        at org.apache.bcel.classfile.JavaClass.dump(JavaClass.java:322)

        at org.apache.bcel.classfile.JavaClass.getBytes(JavaClass.java:290)

        at coldfusion.bytecode.JavaAssembler.getBytes(JavaAssembler.java:428)

        at coldfusion.compiler.TemplateAssembler.assemble(TemplateAssembler.java :111)

        at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:37 7)

        at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:14 7)

        at coldfusion.runtime.TemplateClassLoader$TemplateCache$1.fetch(Template ClassLoader.java:442)

        at coldfusion.util.LruCache.get(LruCache.java:180)

        at coldfusion.runtime.TemplateClassLoader$TemplateCache.fetchSerial(Temp lateClassLoader.java:362)

        at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)

        at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)

        at coldfusion.util.SoftCache.get(SoftCache.java:81)

        at coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader. java:609)

        at coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java :785)

        at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServ iceImpl.java:766)

        at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:337)

        at cfindex2ecfm1647231854.runPage(C:\wwwroot\XXX-micro\index.cfm:9)

        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)

        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)

        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:381 )

        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.ja va:48)

        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

        at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)

        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)

     

    javax.servlet.ServletException: ROOT CAUSE:

    java.lang.OutOfMemoryError: Java heap space

        at java.util.Arrays.copyOf(Arrays.java:2786)

        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)

        at java.io.DataOutputStream.write(DataOutputStream.java:90)

        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)

        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)

        at org.apache.bcel.classfile.ConstantUtf8.dump(ConstantUtf8.java:125)

        at org.apache.bcel.classfile.ConstantPool.dump(ConstantPool.java:230)

        at org.apache.bcel.classfile.JavaClass.dump(JavaClass.java:322)

        at org.apache.bcel.classfile.JavaClass.getBytes(JavaClass.java:290)

        at coldfusion.bytecode.JavaAssembler.getBytes(JavaAssembler.java:428)

        at coldfusion.compiler.TemplateAssembler.assemble(TemplateAssembler.java :111)

        at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:37 7)

        at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:14 7)

        at coldfusion.runtime.TemplateClassLoader$TemplateCache$1.fetch(Template ClassLoader.java:442)

        at coldfusion.util.LruCache.get(LruCache.java:180)

        at coldfusion.runtime.TemplateClassLoader$TemplateCache.fetchSerial(Temp lateClassLoader.java:362)

        at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)

        at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)

        at coldfusion.util.SoftCache.get(SoftCache.java:81)

        at coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader. java:609)

        at coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java :785)

        at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServ iceImpl.java:766)

        at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:337)

        at cfindex2ecfm1647231854.runPage(C:\wwwroot\XXX-micro\index.cfm:9)

        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)

        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)

        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:381 )

        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.ja va:48)

        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

        at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)

        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)

     

        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringS ervletFilter.java:70)

        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46 )

        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)

        at jrun.servlet.FilterChain.service(FilterChain.java:101)

        at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)

        at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)

        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java: 286)

        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java: 543)

        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.ja va:203)

        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPoo l.java:320)

        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.j ava:428)

        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool. java:266)

        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

     

    09/07 00:33:03 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@51fffa02]) JRunPRoxyServer.invokeRunnable:

    java.lang.IllegalStateException

        at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)

        at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)

        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java: 328)

        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java: 543)

        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.ja va:203)

        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPoo l.java:320)

        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.j ava:428)

        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool. java:266)

        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

     

    java.lang.IllegalStateException

        at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)

        at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)

        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java: 328)

        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java: 543)

        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.ja va:203)

        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPoo l.java:320)

        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.j ava:428)

        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool. java:266)

        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

     
    |
    Mark as:
  • Steve Sommers
    Currently Being Moderated
    14. Steve Sommers,
    Sep 10, 2012 8:09 AM   in reply to smiralles
    Report

    I suggest something like FusionReactor or some other CF/Java memory monitor (there is a free one I used as well but cannot think of the name). For me, it pinpointed a couple pages that were not releasing memory because of some differences between CF5 and CF9 when we ported the code. Your dump above may or may not reflect the actual culprit as memory issues are very hard to find. The prior request may have eaten all your memory and this request just happened to be the final straw.

     
    |
    Mark as:
  • DanWilson_
    Currently Being Moderated
    15. DanWilson_,
    Sep 10, 2012 11:29 AM   in reply to Steve Sommers
    Report

    You are going to want to get some monitoring on that pretty quickly.

     

    In almost every case, Heap Memory errors come from bad code.

     

    Lots of times, this comes from a bunch of queries in a loop. Do you have

    any entries in the log files around the time of the crash that might point

    to a scheduled task running or some other process?

     

    DW

     
    |
    Mark as:
  • carl type3
    Currently Being Moderated
    16. carl type3,
    Sep 10, 2012 3:41 PM   in reply to DanWilson_
    Report

    Monitoring is a good idea that way you know what is going on. There are many free, donation and applications you pay for by way of tools to use.
    Free:
    -Logs (your already starting to get some useful errors)
    -CF Monitor (if you have an Enterprise licence ie not Standard)
    -JVM logging
    -Jconsole (part of Java JDK)
    -Jvisualvm (also part of Java JDK)
    -CF Server Manager (parts like details view might assist)
    -CF Metrics logging
    -CFSTAT (not to be disregarded)
    -others that do not come straight to mind
    Donation:
    -CFTracker (have not used that overly yet myself however looks good)
    Pay (tho may have trial period for free):
    -FusionReactor
    -SeeFusion
    -perhaps others

     

    Apologies to those that are missed in posting a quick forum reply.

     

    Given you know how one or more of those tools looks when CF is working well versa when CF is nearing crash then likely you can have a better idea as to what is the problem. Making an adjustment to a CF setting or CFM code and monitoring will show if adjustment has had any effect.

     

    Regards, Carl.

     
    |
    Mark as:
  • carl type3
    Currently Being Moderated
    17. carl type3,
    Sep 10, 2012 3:46 PM   in reply to smiralles
    Report

    What is your CF9 JVM settings for minimum and maximum memory?

    Eg:

     

    Capture.JPG

     

    You think CF version is 9.0.1. Confirm version with -

     

    CFadmin > System Information

    Eg:
     
    About ColdFusion »

    Server Details 
    Server Product  ColdFusion 
    Version  9,0,1,274733   
    Edition  Enterprise   
    Update Level  /D:/ColdFusion9/lib/updates/chf9010001.jar   
    Java Version  1.6.0_17 

     

    Regards, Carl.

     
    |
    Mark as:
Actions

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points