38 Replies Latest reply: Dec 29, 2014 11:52 AM by Raige@ncsu RSS

    Please Help - Randomly HTTP 503 Internal server error

    Ryan Luo Community Member

      Hi all,

       

       

      I have a Coldfusion 10 standard Edition installation with MSSQL 2005 on Windows Server 2008 R2 server about 2 week old.

      It is one of my live server and it is dealing with medium to high volume of traffic.

       

       

      Since deploying this server, it has been getting random HTTP 503 internal server errors,

      which require the entire server restart (as coldfusion application server, IIS restart will not able

      to resolve the problem). Within this two weeks, sometimes it can last for couple of days without problem,

      and sometimes it require 4-5 times server restart within one day.

       

       

      I can't replicate the error on demand, but I could paste one of the windows server event log here:

       

       

                Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

                Faulting module name: isapi_redirect.dll, version: 1.2.32.0, time stamp: 0x4f745a78

                Exception code: 0xc0000005

                Fault offset: 0x000000000000cd68

                Faulting process id: 0xfd4

                Faulting application start time: 0x01cd745ad58e3393

                Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

                Faulting module path: C:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll

                Report Id: 144b3231-e04e-11e1-8e6d-00505687001e

       

       

      And I started suspecting the problem is because the connector comes with coldfusion 10, based on one of the bug reports from apache

      https://issues.apache.org/bugzilla/show_bug.cgi?id=48826.

      Based on the information in log file, they are pretty similar.

       

       

      Any suggestions on how to fix this random server error would be greatly appreciated!

       

       

      Many thanks

       

       

      Ryan

        • 1. Re: Please Help - Randomly HTTP 503 Internal server error
          Ryan Luo Community Member

          Please, is there anyone could help? or confirm this at least?

          Thanks in advance

          • 2. Re: Please Help - Randomly HTTP 503 Internal server error
            Adam Cameron. Community Member

            Is there anything in the CF or Tomcat logs?

             

            --

            Adam

            • 3. Re: Please Help - Randomly HTTP 503 Internal server error
              Ryan Luo Community Member

              Hi Adam

               

              Thanks for your reply, I couldn't find anything useful in tomcat log, beside the system event log I have supplied in my original post, here is the cf log when 503 error happened.

              07/08/2012 9:50:11 PM org.apache.catalina.core.StandardWrapperValve invoke

              SEVERE: Servlet.service() for servlet [CFCServlet] in context with path [/] threw exception [coldfusion.filter.AxisFilter$WebServiceInvocationException: The web service operation caused an invocation exception.] with root cause

              java.net.SocketException: Connection reset by peer: socket write error

                        at java.net.SocketOutputStream.socketWrite0(Native Method)

                        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

                        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

                        at org.apache.coyote.ajp.AjpProcessor.output(AjpProcessor.java:299)

                        at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcesso r.java:1093)

                        at org.apache.coyote.Response.doWrite(Response.java:533)

                        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:373)

                        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:460)

                        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:324)

                        at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:301)

                        at org.apache.catalina.connector.Response.flushBuffer(Response.java:565)

                        at org.apache.catalina.connector.ResponseFacade.flushBuffer(ResponseFacade.java:307)

                        at org.apache.axis.transport.http.AxisServlet.sendResponse(AxisServlet.java:909)

                        at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:777)

                        at coldfusion.xml.rpc.CFCServlet.doAxisPost(CFCServlet.java:295)

                        at coldfusion.filter.AxisFilter.invoke(AxisFilter.java:75)

                        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:443)

                        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)

                        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

                        at coldfusion.filter.PathFilter.invoke(PathFilter.java:112)

                        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)

                        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8)

                        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

                        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

                        at coldfusion.xml.rpc.CFCServlet.invoke(CFCServlet.java:155)

                        at coldfusion.xml.rpc.CFCServlet.doPost(CFCServlet.java:331)

                        at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)

                        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

                        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

                        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:305)

                        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 )

                        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

                        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:243)

                        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

                        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

                        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

                        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

                        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

                        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

                        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

                        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

                        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:203)

                        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:539)

                        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                        at java.lang.Thread.run(Thread.java:662)

              07/08/2012 9:50:13 PM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer

              SEVERE: Error in getRealPathFromConn

              java.net.SocketException: Connection reset

                        at java.net.SocketInputStream.read(SocketInputStream.java:168)

                        at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:313)

                        at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:434)

                        at org.apache.coyote.ajp.AjpProcessor.readString(AjpProcessor.java:415)

                        at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:397)

                        at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1112)

                        at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:198)

                        at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:412)

                        at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4475)

                        at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:414)

                        at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.ja va:335)

                        at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:184)

                        at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116)

                        at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:98)

                        at coldfusion.filter.PathFilter.invoke(PathFilter.java:77)

                        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)

                        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:79)

                        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8)

                        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)

                        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)

                        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

                        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

                        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)

                        at coldfusion.CfmServlet.service(CfmServlet.java:204)

                        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

                        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:305)

                        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 )

                        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

                        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:243)

                        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

                        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

                        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

                        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

                        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

                        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

                        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

                        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

                        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:203)

                        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:539)

                        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                        at java.lang.Thread.run(Thread.java:662)

              Exception thrown by error-handling template:

              java.lang.IllegalStateException: getOutputStream() has already been called for this response

                        at org.apache.catalina.connector.Response.getWriter(Response.java:633)

                        at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:214)

                        at coldfusion.runtime.NeoJspWriter.initWriter(NeoJspWriter.java:96)

                        at coldfusion.runtime.NeoJspWriter.flush(NeoJspWriter.java:315)

                        at coldfusion.runtime.NeoPageContext.flushOutput(NeoPageContext.java:1997)

                        at coldfusion.filter.ExceptionFilter.runBuiltInHandler(ExceptionFilter.java:680)

                        at coldfusion.filter.ExceptionFilter.handleException(ExceptionFilter.java:375)

                        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:108)

                        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8)

                        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

                        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

                        at coldfusion.xml.rpc.CFCServlet.invoke(CFCServlet.java:155)

                        at coldfusion.xml.rpc.CFCServlet.doPost(CFCServlet.java:331)

                        at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)

                        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

                        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

                        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:305)

                        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 )

                        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

                        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:243)

                        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

                        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

                        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

                        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

                        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

                        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

                        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

                        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

                        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:203)

                        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:539)

                        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                        at java.lang.Thread.run(Thread.java:662)

              07/08/2012 9:50:13 PM org.apache.catalina.core.StandardWrapperValve invoke

              SEVERE: Servlet.service() for servlet [CFCServlet] in context with path [/] threw exception [coldfusion.filter.AxisFilter$WebServiceInvocationException: The web service operation caused an invocation exception.] with root cause

              java.net.SocketException: Connection reset by peer: socket write error

                        at java.net.SocketOutputStream.socketWrite0(Native Method)

                        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

                        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

                        at org.apache.coyote.ajp.AjpProcessor.output(AjpProcessor.java:299)

                        at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(AbstractAjpProcesso r.java:1093)

                        at org.apache.coyote.Response.doWrite(Response.java:533)

                        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:373)

                        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:460)

                        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:324)

                        at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:301)

                        at org.apache.catalina.connector.Response.flushBuffer(Response.java:565)

                        at org.apache.catalina.connector.ResponseFacade.flushBuffer(ResponseFacade.java:307)

                        at org.apache.axis.transport.http.AxisServlet.sendResponse(AxisServlet.java:909)

                        at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:777)

                        at coldfusion.xml.rpc.CFCServlet.doAxisPost(CFCServlet.java:295)

                        at coldfusion.filter.AxisFilter.invoke(AxisFilter.java:75)

                        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:443)

                        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)

                        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

                        at coldfusion.filter.PathFilter.invoke(PathFilter.java:112)

                        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)

                        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8)

                        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

                        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

                        at coldfusion.xml.rpc.CFCServlet.invoke(CFCServlet.java:155)

                        at coldfusion.xml.rpc.CFCServlet.doPost(CFCServlet.java:331)

                        at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)

                        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

                        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

                        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:305)

                        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 )

                        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

                        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:243)

                        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

                        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

                        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

                        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

                        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

                        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

                        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

                        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

                        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:203)

                        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:539)

                        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                        at java.lang.Thread.run(Thread.java:662)

              • 4. Re: Please Help - Randomly HTTP 503 Internal server error
                Ryan Luo Community Member

                I am the only one who having this issue?

                • 5. Re: Please Help - Randomly HTTP 503 Internal server error
                  Ryan Luo Community Member

                  sorry to ask again, but is there any thing I could confirm this issue whether because of a faulty connector delivered with cf10 or something else

                  thanks again

                  • 6. Re: Please Help - Randomly HTTP 503 Internal server error
                    van u

                    https://issues.apache.org/bugzilla/show_bug.cgi?id=45063

                     

                    Use "LocalSystem" for DefaultAppPool identity.

                     

                    Van

                    • 7. Re: Please Help - Randomly HTTP 503 Internal server error
                      djkhalif Community Member

                      Luo,

                       

                      You're not the only one having this problem. I removed CF9 after migrating to CF10 and all systems failed. I get the same results as you. I'm going back to 9.

                      • 8. Re: Please Help - Randomly HTTP 503 Internal server error
                        Ryan Luo Community Member

                        Thanks for your reply Van

                        I am not really understand the IIS setting, but the server machine serves multiple sites , each site is using a application pool and every application uses "ApplicationPoolIdentity" as their Identity. There is a "DefaultAppPool" uses "ApplicationPoolIdentity", but I don't believe it has been used though.

                        My problem is, we are using a network login for the coldfusion application service instead of "Local System Account", due to the file access permission in our network.

                        Should I change all the application pools to use "LocalSystem"? Will this affect the fille access required by the coldfusion services?

                        Thanks in advance

                        • 9. Re: Please Help - Randomly HTTP 503 Internal server error
                          Ryan Luo Community Member

                          Thanks for your reply djkhalif

                          I can not roll back to the version it was, because it was using coldfusion 6.1 and we need some new functions from later version, plus the coldfusion 10 is the only licence we have.

                          It is affecting all my production servers, and I have to baby sitting them everyday 24/7. Kind frustrating.

                          Thanks for your information again

                          • 10. Re: Please Help - Randomly HTTP 503 Internal server error
                            @bittersweetryan

                            I'm having the same issue, IIS 7.5 on Windows Server 2008 64bit.  The issue will cause the application pool to crash specifically when doing a cffile action="upload".  My logs are showing the exact same errors as yours both in ColdFusion and in the Windows Event viewer. 

                             

                            Is there a bug submission somewhere where I can vote this issue up? 

                            • 11. Re: Please Help - Randomly HTTP 503 Internal server error
                              Ryan Luo Community Member

                              Thanks for your information bittersweetryan

                              Is this happening in your development environment or it is happening in a production environment?

                              Are you able to replicate the crash yourself?

                              • 12. Re: Please Help - Randomly HTTP 503 Internal server error
                                @bittersweetryan Community Member

                                It is in a brand new server that is being tested for production.   I can replicate the crash every time, the server will work fine until I do a cffile action=upload.  The application pool will then crash causing the server to stop responding.   This is on a Windows 2008 Server 64bit, IIS 7.5, and ColdFusion 10.  

                                • 13. Re: Please Help - Randomly HTTP 503 Internal server error
                                  Ryan Luo Community Member

                                  Your problem is a bit different to mine, I am sure I do have lots cffile when action is upload in my application, and that part will be called multiple times. But the server only crash randomly and I can't replicate the error myself.

                                  It went down 4 times in an hour this morning and I am frustrating a lot and lots of pressure at the moment.

                                  • 14. Re: Please Help - Randomly HTTP 503 Internal server error
                                    @bittersweetryan Community Member

                                    I think I reduced my problem as far as I can go.  Its not the CFFILE operation itself, its with a http request with a large amount of content, such as doing an XHR file upload.   This causes the IIS connector to tomcat to throw numerous errors.  If your thread pool is set to die after x number of errors the pool will crash leaving the server unresponsive.  My solution was to do traditional form based file uploads instead of using XHR.

                                    • 15. Re: Please Help - Randomly HTTP 503 Internal server error
                                      Ryan Luo Community Member

                                      Could you please post some sample codes from your application which causing server crashes for you? There are lots of cffile in my application and I have no idea which one in what situationwill send some large amount of content  and crash the server.

                                      • 16. Re: Please Help - Randomly HTTP 503 Internal server error
                                        Ryan Luo Community Member

                                        Furthermore, you are right, IIS will stop functioning after certain number of errors been send to it, the default was 5.

                                        • 17. Re: Please Help - Randomly HTTP 503 Internal server error
                                          klevenson

                                          Ryan, I am having the same issue.  I tried the change that Van suggested above, so we'll see that it works.  I used the advice in the article in his link to change the account that the IIS 7.5 application pool runs under.  You can learn more about changing it here:  http://technet.microsoft.com/en-us/library/cc771170%28v=ws.10%29.aspx .  I would never have found it otherwise--it is tucked in application pool > advanced settings > process model - identity.  Change to LocalService and hope for the best.

                                           

                                          If you solve it though some other method, please post as I have 5 servers ready to go into production once this issue is solved.

                                          • 18. Re: Please Help - Randomly HTTP 503 Internal server error
                                            Ryan Luo Community Member

                                            Klevenson

                                            Since you have tried to solve the problem by changing the identify to LocalService, would you able to tell me whether it is working or not?

                                            I am not able to do such changes, since my servers have already into production.

                                            My temperate solution is increasing the number of error IIS can handle before shut itself off.

                                            It is in application pool > Advance setting > Rapid-Faill Protection - Maximum Failures

                                            Since I have found the IIS will shut itself off after 5 errors been sent from connector, by increasing the number here, it temperately solved my problem (but my event log still full of errors from connector)

                                            I am also trying to contact Adobe in a formal way to discuss about this issue.

                                            • 19. Re: Please Help - Randomly HTTP 503 Internal server error
                                              Ryan Luo Community Member

                                              Also there is a ticket has been sent to adobe bug report by one of our fellow cf 10 user

                                              and here is the original post

                                              http://forums.adobe.com/thread/1016323?tstart=0

                                              and the actual ticket

                                              https://bugbase.adobe.com/index.cfm?event=bug&id=3318104

                                              I couldn't try the solution suggested by JakeHand, since the servers are in production

                                              • 20. Re: Please Help - Randomly HTTP 503 Internal server error
                                                AlHolden Community Member

                                                We're getting the same issue on our CF 10 server too. It'll run fine for an hour or two, then fill up the coldfusion-error.log with about 4Mb of this error - over and over. Often the system will choke at this point.

                                                I can not say whether or not this is related to our ongoing issue with support: trying to get CF10 and ASP4 to run at the same time:

                                                 

                                                Aug 29, 2012 10:43:28 AM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer

                                                SEVERE: Error in getRealPathFromConn

                                                java.net.SocketException: Connection reset by peer: socket write error

                                                    at java.net.SocketOutputStream.socketWrite0(Native Method)

                                                    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

                                                    at java.net.SocketOutputStream.write(SocketOutputStream.java:124)

                                                    at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:391)

                                                    at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1112)

                                                    at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:198)

                                                    at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:412)

                                                    at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4475)

                                                    at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:414)

                                                    at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.ja va:335)

                                                    at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:184)

                                                    at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116)

                                                    at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:98)

                                                    at flex.util.ServletPathResolver.resolveRealPath(ServletPathResolver.java:55)

                                                    at flex.util.ServletPathResolver.deduceFile(ServletPathResolver.java:30)

                                                    at flex.util.PathResolver.resolveFile(PathResolver.java:111)

                                                    at flex.compiler.CachedComponentLoader.createSourceFileAppRoot(CachedComponentLoader.java:63 4)

                                                    at flex.compiler.AppRootASClassLoader.createSourceFile(AppRootASClassLoader.java:33)

                                                    at flex.compiler.CachedComponentLoader.hasClassName(CachedComponentLoader.java:832)

                                                    at flex.compiler.linker.Loader$1.fetch(Loader.java:406)

                                                    at macromedia.util.SoftCache.get(SoftCache.java:81)

                                                    at flex.compiler.linker.Loader.getClassName(Loader.java:395)

                                                    at flex.compiler.CompositeLoader.hasClassName(CompositeLoader.java:198)

                                                    at flex.compiler.linker.Loader$1.fetch(Loader.java:406)

                                                    at macromedia.util.SoftCache.get(SoftCache.java:81)

                                                    at flex.compiler.linker.Loader.getClassName(Loader.java:395)

                                                    at flex.compiler.linker.Loader.findClassName(Loader.java:383)

                                                    at macromedia.asc.AsPackageManager.getAsPackage(AsPackageManager.java:190)

                                                    at macromedia.asc.AsCompiler.loadSymbolTable(AsCompiler.java:1208)

                                                    at macromedia.asc.AsParse.loadSymbolTable(AsParse.java:7714)

                                                    at macromedia.asc.AsParse.loadSymbolTable(AsParse.java:7653)

                                                    at macromedia.asc.AsParse.parseMemberExpr(AsParse.java:992)

                                                    at macromedia.asc.AsParse.parseLeftHandSideExpr(AsParse.java:794)

                                                    at macromedia.asc.AsParse.parsePostfixExpr(AsParse.java:757)

                                                    at macromedia.asc.AsParse.parseUnaryExpr(AsParse.java:1472)

                                                    at macromedia.asc.AsParse.parseCastExpr(AsParse.java:1303)

                                                    at macromedia.asc.AsParse.parseMultiplicativeExpr(AsParse.java:676)

                                                    at macromedia.asc.AsParse.parseAdditiveExpr(AsParse.java:637)

                                                    at macromedia.asc.AsParse.parseShiftExpr(AsParse.java:1478)

                                                    at macromedia.asc.AsParse.parseRelationalExpr(AsParse.java:1515)

                                                    at macromedia.asc.AsParse.parseEqualityExpr(AsParse.java:1581)

                                                    at macromedia.asc.AsParse.parseBitAndExpr(AsParse.java:1679)

                                                    at macromedia.asc.AsParse.parseBitXorExpr(AsParse.java:1655)

                                                    at macromedia.asc.AsParse.parseBitOrExpr(AsParse.java:1632)

                                                    at macromedia.asc.AsParse.parseAndExpr(AsParse.java:1703)

                                                    at macromedia.asc.AsParse.parseOrExpr(AsParse.java:1738)

                                                    at macromedia.asc.AsParse.parseConditionalExpr(AsParse.java:1773)

                                                    at macromedia.asc.AsParse.parseAssignmentExpr(AsParse.java:1801)

                                                    at macromedia.asc.AsParse.parseExpression(AsParse.java:1899)

                                                    at macromedia.asc.AsParse.parseStatement(AsParse.java:6215)

                                                    at macromedia.asc.AsParse.parseBlock(AsParse.java:2294)

                                                    at macromedia.asc.AsParse.parseFunction(AsParse.java:3140)

                                                    at macromedia.asc.AsParse.parseStatement(AsParse.java:6091)

                                                    at macromedia.asc.AsParse.parseToplevelStatements(AsParse.java:2187)

                                                    at macromedia.asc.AsParse.parseStatements(AsParse.java:2162)

                                                    at macromedia.asc.AsParse.compileScript(AsParse.java:7126)

                                                    at macromedia.asc.AsCompiler.parseString(AsCompiler.java:1083)

                                                    at macromedia.asc.ActionScriptCompiler.parseOnly(ActionScriptCompiler.java:76)

                                                    at flex.compiler.CodeGenContext.parseOnly(CodeGenContext.java:135)

                                                    at flex.compiler.WatcherVisitor.compileAndReplaceFunctions(WatcherVisitor.java:172)

                                                    at flex.compiler.WatcherVisitor.execute(WatcherVisitor.java:150)

                                                    at flex.compiler.WatcherVisitor.doWatcherPass(WatcherVisitor.java:1562)

                                                    at flex.compiler.SwcBuilder.createClass(SwcBuilder.java:646)

                                                    at flex.compiler.SwcBuilder.createSwc(SwcBuilder.java:80)

                                                    at flex.compiler.MxmlLoader.compileComponent(MxmlLoader.java:410)

                                                    at flex.compiler.J2eeMxmlCompiler.compileSourceEntry(J2eeMxmlCompiler.java:60)

                                                    at coldfusion.tagext.html.form.CFFlexApi$CompileMxml.run(CFFlexApi.java:412)

                                                    at java.security.AccessController.doPrivileged(Native Method)

                                                    at coldfusion.tagext.html.form.CFFlexApi.compile(CFFlexApi.java:226)

                                                    at coldfusion.tagext.html.form.MxmlOutput.compileMXML(MxmlOutput.java:65)

                                                    at coldfusion.tagext.html.form.MxmlOutput.compile(MxmlOutput.java:53)

                                                    at coldfusion.tagext.html.form.FormTag.doEndTag(FormTag.java:878)

                                                    at cfadmin2ecfm274100841._factor49(D:\webserve\intraweb\fobs\wfas\admin.cfm:2010)

                                                    at cfadmin2ecfm274100841._factor50(D:\webserve\intraweb\fobs\wfas\admin.cfm:35)

                                                    at cfadmin2ecfm274100841.runPage(D:\webserve\intraweb\fobs\wfas\admin.cfm:1)

                                                    at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:244)

                                                    at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:444)

                                                    at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)

                                                    at coldfusion.filter.IpFilter.invoke(IpFilter.java:64)

                                                    at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:443)

                                                    at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

                                                    at coldfusion.filter.PathFilter.invoke(PathFilter.java:112)

                                                    at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)

                                                    at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:79)

                                                    at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8)

                                                    at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)

                                                    at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)

                                                    at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

                                                    at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

                                                    at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)

                                                    at coldfusion.CfmServlet.service(CfmServlet.java:204)

                                                    at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

                                                    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:305)

                                                    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                                                    at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 )

                                                    at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

                                                    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:243)

                                                    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

                                                    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

                                                    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

                                                    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

                                                    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

                                                    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

                                                    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

                                                    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

                                                    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

                                                    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:203)

                                                    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:539)

                                                    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

                                                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                                                    at java.lang.Thread.run(Thread.java:662)

                                                • 21. Re: Please Help - Randomly HTTP 503 Internal server error
                                                  AlHolden Community Member

                                                  Upon further review of the logs here, it's possible that these events could be triggered by CF's invocation of flash forms (ie CFFORM type='flash'). They seem to coincide in close proximity.

                                                  • 22. Re: Please Help - Randomly HTTP 503 Internal server error
                                                    Miguel-F Community Member

                                                    Everyone should "vote" for the bug that was entered for this.  That will help show Adobe that more than one user is experiencing the issue.

                                                     

                                                    https://bugbase.adobe.com/index.cfm?event=bug&id=3318104

                                                    • 23. Re: Please Help - Randomly HTTP 503 Internal server error
                                                      @bittersweetryan Community Member

                                                      Miguel,

                                                       

                                                      I added my vote.

                                                       

                                                      Ryan

                                                      • 24. Re: Please Help - Randomly HTTP 503 Internal server error
                                                        @bittersweetryan Community Member

                                                        AlHolden,

                                                         

                                                        I've noticed that the connector definately crashes when there is a http request with a large content payload.  If I do a HTML5 file upload the server will crash immediately.  Do your forms submit a large amount of data in the post?

                                                         

                                                        Ryan

                                                        • 25. Re: Please Help - Randomly HTTP 503 Internal server error
                                                          AlHolden Community Member

                                                          Yes Ryan, it seems to be fairly consistent with a large request.

                                                           

                                                          I earlier stated that the issue might have been related to flash form invocation/compilation - but it could also be due to the fact that these forms are starting off with some grids that are pretty heavily populated with data on first load.

                                                           

                                                          Al

                                                          • 26. Re: Please Help - Randomly HTTP 503 Internal server error
                                                            hemant_k Employee Hosts

                                                            We are working on a fix for this issue via

                                                            https://bugbase.adobe.com/index.cfm?event=bug&id=3222748

                                                             

                                                            Thanks,

                                                            Hemant

                                                            Adobe ColdFusion

                                                            • 27. Re: Please Help - Randomly HTTP 503 Internal server error
                                                              BigJ57 Community Member

                                                              I just posted on the other thread that I had tried the an alternate connector (tomcatiis.riaforge.org).

                                                              It seems to work well with ASP.net and does not require any ISAPI support, though it does require a little more work to install since it is 3rd party.

                                                              I used instructions from this blog post:

                                                              http://boncode.blogspot.com/2012/06/cf-coldfusion-10-experimenting-with.html

                                                              • 28. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                roland.collins Community Member

                                                                Eact same issue here.  Our CF10 rollout to production was a disaster.

                                                                 

                                                                Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

                                                                Faulting module name: isapi_redirect.dll, version: 1.2.32.0, time stamp: 0x4f745a78

                                                                Exception code: 0xc0000005

                                                                Fault offset: 0x0000000000018b93

                                                                Faulting process id: 0x11f0

                                                                Faulting application start time: 0x01cd950afc876c76

                                                                Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

                                                                Faulting module path: C:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll

                                                                Report Id: 98760e12-00ff-11e2-9e13-00155d015102

                                                                • 29. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                  TALAL MANAA Community Member

                                                                  its obvious that your server is unable to handle the request due to maintenance or a temporary overload of the server; try to find what makes it overloaded!

                                                                  • 30. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                    Steve Sommers Community Member

                                                                    With my experience, 503's are caused by slow pages eating up the worker threads. I recently installed CF10 and noticed right off the bat that it is slower that CF9 so the combination of the two either caused or exacerbated your problem. Now I just installed CF10 and have not done any fine tuning yet. When I upgraded to CF9 I remember similar problems that fine tuning solved -- but the problems did not show up right out-of-the-box like they have with CF10.

                                                                    • 31. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                      roland.collins Community Member

                                                                      Talal/Steve,

                                                                       

                                                                      It's obvious that you did not read the bug report linked to above.  This has nothing to do with server load.  You can crash a server with zero load instantly with this bug.  ANY CF10 server on IIS.  It's not a configuration issue or a capacity issue.  It's a crashing bug, plain and simple.

                                                                      • 32. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                        Steve Sommers Community Member

                                                                        I was going by the initial question and my experience. No, I did not follow the bug link and there are several links in this thread and I didn't have time to read this thread and the associated links in detail. Sorry if I offended anyone.

                                                                        • 33. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                          fi_jj Community Member

                                                                          Wow, this is still not fixed? Incredible.

                                                                           

                                                                          Things like xhr file uploads crash the server as stated in this thread.

                                                                          • 34. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                            tribule Community Member

                                                                            Got this error today - thousands of SEVERE log entries. Is it still a known issue?

                                                                            • 35. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                              AlHolden Community Member

                                                                              Are you running the latest update?

                                                                               

                                                                              Does it help if you enable "Cache Web Server Paths" setting in the Caching page of the ColdFusion Administrator?

                                                                               

                                                                              The bug report for 3222748 claims that as a workaround. If not, I would go here and vote up the bug, assuming your issue is producing the same error messages as the ones stated there.

                                                                              https://bugbase.adobe.com/index.cfm?event=bug&id=3222748

                                                                               

                                                                              HTH, Al

                                                                              • 36. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                                tribule Community Member

                                                                                AlHolden, I've enabled that option (we run a single server) so I'll see if that helps. The problem is intermittent and with no regular frequency, so hard to tell when it will happen next. We run update 11.

                                                                                 

                                                                                Someone also mentioned changing the application pools to run as "LocalService" - not sure if that will help either. It's a bit like black magic trying to get CF to work properly

                                                                                • 37. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                                  AlHolden Community Member

                                                                                  Perhaps it could be stated "It's a bit like black magic trying to get CF to work properly with IIS"

                                                                                   

                                                                                  CF is basically a Java J2EE application. IIS is a Microsoft application. I think it's more a game of cat and mouse, than black magic.

                                                                                  • 38. Re: Please Help - Randomly HTTP 503 Internal server error
                                                                                    Raige@ncsu Community Member

                                                                                    To help forward traffic to solutions please visit here:

                                                                                     

                                                                                    CF 10 continues to crash on Windows 2008 R2 64 bit

                                                                                     

                                                                                    (specifically look for posts by Charlie Arehart)