Skip navigation
Currently Being Moderated

Please Help - Randomly HTTP 503 Internal server error

Aug 9, 2012 12:04 AM

Tags: #server #iis #cf10 #tomcat_connector

Hi all,

 

 

I have a Coldfusion 10 standard Edition installation with MSSQL 2005 on Windows Server 2008 R2 server about 2 week old.

It is one of my live server and it is dealing with medium to high volume of traffic.

 

 

Since deploying this server, it has been getting random HTTP 503 internal server errors,

which require the entire server restart (as coldfusion application server, IIS restart will not able

to resolve the problem). Within this two weeks, sometimes it can last for couple of days without problem,

and sometimes it require 4-5 times server restart within one day.

 

 

I can't replicate the error on demand, but I could paste one of the windows server event log here:

 

 

          Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

          Faulting module name: isapi_redirect.dll, version: 1.2.32.0, time stamp: 0x4f745a78

          Exception code: 0xc0000005

          Fault offset: 0x000000000000cd68

          Faulting process id: 0xfd4

          Faulting application start time: 0x01cd745ad58e3393

          Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

          Faulting module path: C:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll

          Report Id: 144b3231-e04e-11e1-8e6d-00505687001e

 

 

And I started suspecting the problem is because the connector comes with coldfusion 10, based on one of the bug reports from apache

https://issues.apache.org/bugzilla/show_bug.cgi?id=48826.

Based on the information in log file, they are pretty similar.

 

 

Any suggestions on how to fix this random server error would be greatly appreciated!

 

 

Many thanks

 

 

Ryan

 
Replies
  • Currently Being Moderated
    Aug 12, 2012 4:05 PM   in reply to Ryan Luo

    Is there anything in the CF or Tomcat logs?

     

    --

    Adam

     
    |
    Mark as:
  • Currently Being Moderated
    Calculating status...
    Aug 17, 2012 5:32 PM   in reply to Ryan Luo

    https://issues.apache.org/bugzilla/show_bug.cgi?id=45063

     

    Use "LocalSystem" for DefaultAppPool identity.

     

    Van

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 18, 2012 7:03 AM   in reply to Ryan Luo

    Luo,

     

    You're not the only one having this problem. I removed CF9 after migrating to CF10 and all systems failed. I get the same results as you. I'm going back to 9.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 21, 2012 2:05 PM   in reply to Ryan Luo

    I'm having the same issue, IIS 7.5 on Windows Server 2008 64bit.  The issue will cause the application pool to crash specifically when doing a cffile action="upload".  My logs are showing the exact same errors as yours both in ColdFusion and in the Windows Event viewer. 

     

    Is there a bug submission somewhere where I can vote this issue up? 

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 21, 2012 7:47 PM   in reply to Ryan Luo

    It is in a brand new server that is being tested for production.   I can replicate the crash every time, the server will work fine until I do a cffile action=upload.  The application pool will then crash causing the server to stop responding.   This is on a Windows 2008 Server 64bit, IIS 7.5, and ColdFusion 10.  

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 24, 2012 12:06 PM   in reply to Ryan Luo

    I think I reduced my problem as far as I can go.  Its not the CFFILE operation itself, its with a http request with a large amount of content, such as doing an XHR file upload.   This causes the IIS connector to tomcat to throw numerous errors.  If your thread pool is set to die after x number of errors the pool will crash leaving the server unresponsive.  My solution was to do traditional form based file uploads instead of using XHR.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 28, 2012 8:45 AM   in reply to Ryan Luo

    Ryan, I am having the same issue.  I tried the change that Van suggested above, so we'll see that it works.  I used the advice in the article in his link to change the account that the IIS 7.5 application pool runs under.  You can learn more about changing it here:  http://technet.microsoft.com/en-us/library/cc771170%28v=ws.10%29.aspx .  I would never have found it otherwise--it is tucked in application pool > advanced settings > process model - identity.  Change to LocalService and hope for the best.

     

    If you solve it though some other method, please post as I have 5 servers ready to go into production once this issue is solved.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 10:53 AM   in reply to Ryan Luo

    We're getting the same issue on our CF 10 server too. It'll run fine for an hour or two, then fill up the coldfusion-error.log with about 4Mb of this error - over and over. Often the system will choke at this point.

    I can not say whether or not this is related to our ongoing issue with support: trying to get CF10 and ASP4 to run at the same time:

     

    Aug 29, 2012 10:43:28 AM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer

    SEVERE: Error in getRealPathFromConn

    java.net.SocketException: Connection reset by peer: socket write error

        at java.net.SocketOutputStream.socketWrite0(Native Method)

        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

        at java.net.SocketOutputStream.write(SocketOutputStream.java:124)

        at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor .java:391)

        at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpPro cessor.java:1112)

        at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirConte xt.java:198)

        at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext .java:412)

        at org.apache.catalina.core.StandardContext.getRealPath(StandardContext. java:4475)

        at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationCo ntext.java:414)

        at org.apache.catalina.core.ApplicationContextFacade.getRealPath(Applica tionContextFacade.java:335)

        at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContext Wrapper.java:184)

        at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContex tWrapper.java:116)

        at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWr apper.java:98)

        at flex.util.ServletPathResolver.resolveRealPath(ServletPathResolver.jav a:55)

        at flex.util.ServletPathResolver.deduceFile(ServletPathResolver.java:30)

        at flex.util.PathResolver.resolveFile(PathResolver.java:111)

        at flex.compiler.CachedComponentLoader.createSourceFileAppRoot(CachedCom ponentLoader.java:634)

        at flex.compiler.AppRootASClassLoader.createSourceFile(AppRootASClassLoa der.java:33)

        at flex.compiler.CachedComponentLoader.hasClassName(CachedComponentLoade r.java:832)

        at flex.compiler.linker.Loader$1.fetch(Loader.java:406)

        at macromedia.util.SoftCache.get(SoftCache.java:81)

        at flex.compiler.linker.Loader.getClassName(Loader.java:395)

        at flex.compiler.CompositeLoader.hasClassName(CompositeLoader.java:198)

        at flex.compiler.linker.Loader$1.fetch(Loader.java:406)

        at macromedia.util.SoftCache.get(SoftCache.java:81)

        at flex.compiler.linker.Loader.getClassName(Loader.java:395)

        at flex.compiler.linker.Loader.findClassName(Loader.java:383)

        at macromedia.asc.AsPackageManager.getAsPackage(AsPackageManager.java:19 0)

        at macromedia.asc.AsCompiler.loadSymbolTable(AsCompiler.java:1208)

        at macromedia.asc.AsParse.loadSymbolTable(AsParse.java:7714)

        at macromedia.asc.AsParse.loadSymbolTable(AsParse.java:7653)

        at macromedia.asc.AsParse.parseMemberExpr(AsParse.java:992)

        at macromedia.asc.AsParse.parseLeftHandSideExpr(AsParse.java:794)

        at macromedia.asc.AsParse.parsePostfixExpr(AsParse.java:757)

        at macromedia.asc.AsParse.parseUnaryExpr(AsParse.java:1472)

        at macromedia.asc.AsParse.parseCastExpr(AsParse.java:1303)

        at macromedia.asc.AsParse.parseMultiplicativeExpr(AsParse.java:676)

        at macromedia.asc.AsParse.parseAdditiveExpr(AsParse.java:637)

        at macromedia.asc.AsParse.parseShiftExpr(AsParse.java:1478)

        at macromedia.asc.AsParse.parseRelationalExpr(AsParse.java:1515)

        at macromedia.asc.AsParse.parseEqualityExpr(AsParse.java:1581)

        at macromedia.asc.AsParse.parseBitAndExpr(AsParse.java:1679)

        at macromedia.asc.AsParse.parseBitXorExpr(AsParse.java:1655)

        at macromedia.asc.AsParse.parseBitOrExpr(AsParse.java:1632)

        at macromedia.asc.AsParse.parseAndExpr(AsParse.java:1703)

        at macromedia.asc.AsParse.parseOrExpr(AsParse.java:1738)

        at macromedia.asc.AsParse.parseConditionalExpr(AsParse.java:1773)

        at macromedia.asc.AsParse.parseAssignmentExpr(AsParse.java:1801)

        at macromedia.asc.AsParse.parseExpression(AsParse.java:1899)

        at macromedia.asc.AsParse.parseStatement(AsParse.java:6215)

        at macromedia.asc.AsParse.parseBlock(AsParse.java:2294)

        at macromedia.asc.AsParse.parseFunction(AsParse.java:3140)

        at macromedia.asc.AsParse.parseStatement(AsParse.java:6091)

        at macromedia.asc.AsParse.parseToplevelStatements(AsParse.java:2187)

        at macromedia.asc.AsParse.parseStatements(AsParse.java:2162)

        at macromedia.asc.AsParse.compileScript(AsParse.java:7126)

        at macromedia.asc.AsCompiler.parseString(AsCompiler.java:1083)

        at macromedia.asc.ActionScriptCompiler.parseOnly(ActionScriptCompiler.ja va:76)

        at flex.compiler.CodeGenContext.parseOnly(CodeGenContext.java:135)

        at flex.compiler.WatcherVisitor.compileAndReplaceFunctions(WatcherVisito r.java:172)

        at flex.compiler.WatcherVisitor.execute(WatcherVisitor.java:150)

        at flex.compiler.WatcherVisitor.doWatcherPass(WatcherVisitor.java:1562)

        at flex.compiler.SwcBuilder.createClass(SwcBuilder.java:646)

        at flex.compiler.SwcBuilder.createSwc(SwcBuilder.java:80)

        at flex.compiler.MxmlLoader.compileComponent(MxmlLoader.java:410)

        at flex.compiler.J2eeMxmlCompiler.compileSourceEntry(J2eeMxmlCompiler.ja va:60)

        at coldfusion.tagext.html.form.CFFlexApi$CompileMxml.run(CFFlexApi.java: 412)

        at java.security.AccessController.doPrivileged(Native Method)

        at coldfusion.tagext.html.form.CFFlexApi.compile(CFFlexApi.java:226)

        at coldfusion.tagext.html.form.MxmlOutput.compileMXML(MxmlOutput.java:65 )

        at coldfusion.tagext.html.form.MxmlOutput.compile(MxmlOutput.java:53)

        at coldfusion.tagext.html.form.FormTag.doEndTag(FormTag.java:878)

        at cfadmin2ecfm274100841._factor49(D:\webserve\intraweb\fobs\wfas\admin. cfm:2010)

        at cfadmin2ecfm274100841._factor50(D:\webserve\intraweb\fobs\wfas\admin. cfm:35)

        at cfadmin2ecfm274100841.runPage(D:\webserve\intraweb\fobs\wfas\admin.cf m:1)

        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:244)

        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:444)

        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)

        at coldfusion.filter.IpFilter.invoke(IpFilter.java:64)

        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:443 )

        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

        at coldfusion.filter.PathFilter.invoke(PathFilter.java:112)

        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)

        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:7 9)

        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePers istenceFilter.java:28)

        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)

        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)

        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)

        at coldfusion.CfmServlet.service(CfmServlet.java:204)

        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:8 9)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl icationFilterChain.java:305)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF ilterChain.java:210)

        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringS ervletFilter.java:42)

        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46 )

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl icationFilterChain.java:243)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF ilterChain.java:210)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV alve.java:224)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV alve.java:169)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authentica torBase.java:472)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j ava:168)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j ava:98)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java: 928)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal ve.java:118)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav a:414)

        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:203)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process( AbstractProtocol.java:539)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoin t.java:298)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:908)

        at java.lang.Thread.run(Thread.java:662)

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 11:13 AM   in reply to AlHolden

    Upon further review of the logs here, it's possible that these events could be triggered by CF's invocation of flash forms (ie CFFORM type='flash'). They seem to coincide in close proximity.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 11:15 AM   in reply to AlHolden

    Everyone should "vote" for the bug that was entered for this.  That will help show Adobe that more than one user is experiencing the issue.

     

    https://bugbase.adobe.com/index.cfm?event=bug&id=3318104

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 11:20 AM   in reply to Miguel-F

    Miguel,

     

    I added my vote.

     

    Ryan

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 11:23 AM   in reply to AlHolden

    AlHolden,

     

    I've noticed that the connector definately crashes when there is a http request with a large content payload.  If I do a HTML5 file upload the server will crash immediately.  Do your forms submit a large amount of data in the post?

     

    Ryan

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 11:40 AM   in reply to @bittersweetryan

    Yes Ryan, it seems to be fairly consistent with a large request.

     

    I earlier stated that the issue might have been related to flash form invocation/compilation - but it could also be due to the fact that these forms are starting off with some grids that are pretty heavily populated with data on first load.

     

    Al

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 9:23 PM   in reply to AlHolden

    We are working on a fix for this issue via

    https://bugbase.adobe.com/index.cfm?event=bug&id=3222748

     

    Thanks,

    Hemant

    Adobe ColdFusion

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 7, 2012 12:59 PM   in reply to hemant_k

    I just posted on the other thread that I had tried the an alternate connector (tomcatiis.riaforge.org).

    It seems to work well with ASP.net and does not require any ISAPI support, though it does require a little more work to install since it is 3rd party.

    I used instructions from this blog post:

    http://boncode.blogspot.com/2012/06/cf-coldfusion-10-experimenting-wit h.html

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 18, 2012 4:30 PM   in reply to Ryan Luo

    Eact same issue here.  Our CF10 rollout to production was a disaster.

     

    Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

    Faulting module name: isapi_redirect.dll, version: 1.2.32.0, time stamp: 0x4f745a78

    Exception code: 0xc0000005

    Fault offset: 0x0000000000018b93

    Faulting process id: 0x11f0

    Faulting application start time: 0x01cd950afc876c76

    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

    Faulting module path: C:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll

    Report Id: 98760e12-00ff-11e2-9e13-00155d015102

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 26, 2012 7:01 AM   in reply to Ryan Luo

    its obvious that your server is unable to handle the request due to maintenance or a temporary overload of the server; try to find what makes it overloaded!

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 26, 2012 8:08 AM   in reply to Ryan Luo

    With my experience, 503's are caused by slow pages eating up the worker threads. I recently installed CF10 and noticed right off the bat that it is slower that CF9 so the combination of the two either caused or exacerbated your problem. Now I just installed CF10 and have not done any fine tuning yet. When I upgraded to CF9 I remember similar problems that fine tuning solved -- but the problems did not show up right out-of-the-box like they have with CF10.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 26, 2012 10:18 AM   in reply to TALAL MANAA

    Talal/Steve,

     

    It's obvious that you did not read the bug report linked to above.  This has nothing to do with server load.  You can crash a server with zero load instantly with this bug.  ANY CF10 server on IIS.  It's not a configuration issue or a capacity issue.  It's a crashing bug, plain and simple.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 26, 2012 10:28 AM   in reply to roland.collins

    I was going by the initial question and my experience. No, I did not follow the bug link and there are several links in this thread and I didn't have time to read this thread and the associated links in detail. Sorry if I offended anyone.

     
    |
    Mark as:
  • Currently Being Moderated
    Jun 4, 2013 1:07 AM   in reply to Ryan Luo

    Wow, this is still not fixed? Incredible.

     

    Things like xhr file uploads crash the server as stated in this thread.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 27, 2013 3:32 AM   in reply to fi_jj

    Got this error today - thousands of SEVERE log entries. Is it still a known issue?

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 27, 2013 10:05 AM   in reply to tribule

    Are you running the latest update?

     

    Does it help if you enable "Cache Web Server Paths" setting in the Caching page of the ColdFusion Administrator?

     

    The bug report for 3222748 claims that as a workaround. If not, I would go here and vote up the bug, assuming your issue is producing the same error messages as the ones stated there.

    https://bugbase.adobe.com/index.cfm?event=bug&id=3222748

     

    HTH, Al

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 27, 2013 12:13 PM   in reply to AlHolden

    AlHolden, I've enabled that option (we run a single server) so I'll see if that helps. The problem is intermittent and with no regular frequency, so hard to tell when it will happen next. We run update 11.

     

    Someone also mentioned changing the application pools to run as "LocalService" - not sure if that will help either. It's a bit like black magic trying to get CF to work properly

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 27, 2013 1:09 PM   in reply to tribule

    Perhaps it could be stated "It's a bit like black magic trying to get CF to work properly with IIS"

     

    CF is basically a Java J2EE application. IIS is a Microsoft application. I think it's more a game of cat and mouse, than black magic.

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (1)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points