Copy link to clipboard
Copied
hey there guys, it does not fail! CF keeps crashing on us 1 to 2 times a week sometimes even 3! i have applied all the latest patches to the system and CF but it just keep getting worse and worse! I'm not a CF guy and im tired of google'n stuff. please let me know what kind of info you guys would need to help me out with these problems! i have researched and read eveything on this site. thanks in advanced.
exception.log
" "Error","jrpp-799","09/06/12","03:17:45","XXX","Connection Failure: Status code unavailable The specific sequence of files included or processed is: C:\wwwroot\scripts\processEmail.cfm, line: 145 "
coldfusion.tagext.net.HttpTag$HttpConnectionFailureException: Connection Failure: Status code unavailable
" "Error","jrpp-925","09/07/12","00:32:55","XXX","Java heap space The specific sequence of files included or processed is: C:\wwwroot\XXX-micro\index.cfm, line: 9 "
java.lang.OutOfMemoryError: Java heap space
" "Error","jrpp-938","09/07/12","00:32:55","XXX","The request has exceeded the allowable time limit Tag: cfoutput The specific sequence of files included or processed is: C:\wwwroot\locator\search.cfm, line: 338 "
coldfusion.runtime.RequestTimedOutException: The request has exceeded the allowable time limit Tag: cfoutput
" "Error","jrpp-929","09/07/12","00:34:16","XXX","The request has exceeded the allowable time limit Tag: CFLOOP The specific sequence of files included or processed is: C:\wwwroot\scripts\processEmail.cfm, line: 107 "
coldfusion.runtime.RequestTimedOutException: The request has exceeded the allowable time limit Tag: CFLOOP
server.log
" "Warning","jrpp-935","09/07/12","00:26:25",,"Thread: jrpp-935, processing template: C:\wwwroot\XXX-micro\index.cfm, completed in 55 seconds, exceeding the 45 second warning limit"
" "Warning","jrpp-913","09/07/12","00:26:27",,"Thread: jrpp-913, processing template: C:\wwwroot\XXX-micro\candidate.cfm, completed in 47 seconds, exceeding the 45 second warning limit"
" "Warning","jrpp-938","09/07/12","00:32:12",,"Thread: jrpp-938, processing template: C:\wwwroot\locator\search.cfm, completed in 60 seconds, exceeding the 45 second warning limit"
" "Warning","jrpp-925","09/07/12","00:32:53",,"Thread: jrpp-925, processing template: C:\wwwroot\XXX-micro\index.cfm, completed in 150 seconds, exceeding the 45 second warning limit"
" "Warning","jrpp-928","09/07/12","00:32:58",,"Thread: jrpp-928, processing template: C:\wwwroot\locator\search.cfm, completed in 163 seconds, exceeding the 45 second warning limit"
" "Warning","jrpp-932","09/07/12","00:33:01",,"Thread: jrpp-932, processing template: C:\wwwroot\locator\search.cfm, completed in 168 seconds, exceeding the 45 second warning
application.log
" "Error","jrpp-925","09/07/12","00:32:55","XXX","Java heap space The specific sequence of files included or processed is: C:\wwwroot\XXX-micro\index.cfm, line: 9 "
" "Error","jrpp-938","09/07/12","00:32:55","XXX","The request has exceeded the allowable time limit Tag: cfoutput The specific sequence of files included or processed is: C:\wwwroot\locator\search.cfm, line: 338 "
" "Error","jrpp-929","09/07/12","00:34:16","XXX","The request has exceeded the allowable time limit Tag: CFLOOP The specific sequence of files included or processed is: C:\wwwroot\scripts\processEmail.cfm, line: 107 "
Copy link to clipboard
Copied
Well probably a good place to start would be letting us know platforms and version numbers. What OS? What CF? Stand-alone or j2ee? What Web Server? etc.
Also the entries from your app server's standard output log file when this happens. For CF running under JRun j2ee this logfile is here: {drive}:\JRun4\logs\cfusion-out.log
Copy link to clipboard
Copied
We are running a a 2003 server STD with CF 9.01 i think. not sure about the standalone or if its the j2ee
Copy link to clipboard
Copied
i do not have a {drive}:\JRun4\logs\cfusion-out.log location. not sure if thats bad or good
Copy link to clipboard
Copied
You might have some inefficient code causing the cpu to go to 100% utilization.
Copy link to clipboard
Copied
how can i trace this?
Copy link to clipboard
Copied
JRun4 is the default location for ColdFusion install. It can be changed however. Do you have a JRun4 directory on your application server (server that is running ColdFusion)? If so, it should have a 'logs' directory under it with the log files in that folder.
Copy link to clipboard
Copied
C:\ColdFusion9\logs is where i seem to have all my logs for CF i have looked almost everywhere to find this "folder" but no luck.
Copy link to clipboard
Copied
That's it. I believe that directory structure also shows that you are running a standalone installation. It is just a difference in how CF is installed and runs. All-in-one instead of being deployed to a j2ee server. JRun is still there though behind the scenes.
Are you seeing any "out of memory" errors in those logs?
Copy link to clipboard
Copied
i have in the past, not sure what else to do about them. the server has plenty of RAM and hdd space
Copy link to clipboard
Copied
I'm with Dan. The "java.lang.OutOfMemoryError: Java heap space" points to some process consuming all the memory JVM has available. Either you have lots of simultaneus requests, an incorrect CF configuration for your needs, or some inefficient code. The 47 to 168 seconds are huge for a script to complete, so I'd take a look to the code on those files.
Copy link to clipboard
Copied
the error code line is all blank. this only happens at night. its starting to drive me insane!! any pointers? i really have tried everything i can think off and read.
Copy link to clipboard
Copied
Suggest check \coldfusion9\runtime\logs\ “out” and “event” logs for warnings errors prior to crash and when those applications slow down. Report back findings.
HTH, Carl.
Copy link to clipboard
Copied
this is what i got so far.
09/07 00:32:58 error ROOT CAUSE:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)
at org.apache.bcel.classfile.ConstantUtf8.dump(ConstantUtf8.java:125)
at org.apache.bcel.classfile.ConstantPool.dump(ConstantPool.java:230)
at org.apache.bcel.classfile.JavaClass.dump(JavaClass.java:322)
at org.apache.bcel.classfile.JavaClass.getBytes(JavaClass.java:290)
at coldfusion.bytecode.JavaAssembler.getBytes(JavaAssembler.java:428)
at coldfusion.compiler.TemplateAssembler.assemble(TemplateAssembler.java:111)
at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:377)
at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:147)
at coldfusion.runtime.TemplateClassLoader$TemplateCache$1.fetch(TemplateClassLoader.java:442)
at coldfusion.util.LruCache.get(LruCache.java:180)
at coldfusion.runtime.TemplateClassLoader$TemplateCache.fetchSerial(TemplateClassLoader.java:362)
at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)
at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)
at coldfusion.util.SoftCache.get(SoftCache.java:81)
at coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:609)
at coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:785)
at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:766)
at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:337)
at cfindex2ecfm1647231854.runPage(C:\wwwroot\XXX-micro\index.cfm:9)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:381)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
javax.servlet.ServletException: ROOT CAUSE:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
at java.io.DataOutputStream.writeUTF(DataOutputStream.java:306)
at org.apache.bcel.classfile.ConstantUtf8.dump(ConstantUtf8.java:125)
at org.apache.bcel.classfile.ConstantPool.dump(ConstantPool.java:230)
at org.apache.bcel.classfile.JavaClass.dump(JavaClass.java:322)
at org.apache.bcel.classfile.JavaClass.getBytes(JavaClass.java:290)
at coldfusion.bytecode.JavaAssembler.getBytes(JavaAssembler.java:428)
at coldfusion.compiler.TemplateAssembler.assemble(TemplateAssembler.java:111)
at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:377)
at coldfusion.compiler.NeoTranslator.translateJava(NeoTranslator.java:147)
at coldfusion.runtime.TemplateClassLoader$TemplateCache$1.fetch(TemplateClassLoader.java:442)
at coldfusion.util.LruCache.get(LruCache.java:180)
at coldfusion.runtime.TemplateClassLoader$TemplateCache.fetchSerial(TemplateClassLoader.java:362)
at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)
at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)
at coldfusion.util.SoftCache.get(SoftCache.java:81)
at coldfusion.runtime.TemplateClassLoader.findClass(TemplateClassLoader.java:609)
at coldfusion.runtime.RuntimeServiceImpl.getFile(RuntimeServiceImpl.java:785)
at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:766)
at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:337)
at cfindex2ecfm1647231854.runPage(C:\wwwroot\XXX-micro\index.cfm:9)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:381)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:94)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
09/07 00:33:03 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@51fffa02]) JRunPRoxyServer.invokeRunnable:
java.lang.IllegalStateException
at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)
at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
java.lang.IllegalStateException
at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)
at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
Copy link to clipboard
Copied
I suggest something like FusionReactor or some other CF/Java memory monitor (there is a free one I used as well but cannot think of the name). For me, it pinpointed a couple pages that were not releasing memory because of some differences between CF5 and CF9 when we ported the code. Your dump above may or may not reflect the actual culprit as memory issues are very hard to find. The prior request may have eaten all your memory and this request just happened to be the final straw.
Copy link to clipboard
Copied
You are going to want to get some monitoring on that pretty quickly.
In almost every case, Heap Memory errors come from bad code.
Lots of times, this comes from a bunch of queries in a loop. Do you have
any entries in the log files around the time of the crash that might point
to a scheduled task running or some other process?
DW
Copy link to clipboard
Copied
Monitoring is a good idea that way you know what is going on. There are many free, donation and applications you pay for by way of tools to use.
Free:
-Logs (your already starting to get some useful errors)
-CF Monitor (if you have an Enterprise licence ie not Standard)
-JVM logging
-Jconsole (part of Java JDK)
-Jvisualvm (also part of Java JDK)
-CF Server Manager (parts like details view might assist)
-CF Metrics logging
-CFSTAT (not to be disregarded)
-others that do not come straight to mind
Donation:
-CFTracker (have not used that overly yet myself however looks good)
Pay (tho may have trial period for free):
-FusionReactor
-SeeFusion
-perhaps others
Apologies to those that are missed in posting a quick forum reply.
Given you know how one or more of those tools looks when CF is working well versa when CF is nearing crash then likely you can have a better idea as to what is the problem. Making an adjustment to a CF setting or CFM code and monitoring will show if adjustment has had any effect.
Regards, Carl.
Copy link to clipboard
Copied
What is your CF9 JVM settings for minimum and maximum memory?
Eg:
You think CF version is 9.0.1. Confirm version with -
CFadmin > System Information
Eg:
About ColdFusion »
Server Details
Server Product ColdFusion
Version 9,0,1,274733
Edition Enterprise
Update Level /D:/ColdFusion9/lib/updates/chf9010001.jar
Java Version 1.6.0_17
Regards, Carl.