10 Replies Latest reply on Jun 27, 2014 10:37 AM by Phil.Cruz

    CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)

    Phil.Cruz Level 1

      I have a cluster of 2 IIS web servers on Windows Server 2008 R2. They had been running on CF10 for over a year (if not two) with no problems. I upgraded one server to CF11 and now that server is crashing while the CF10 servers is still running fine.

       

      In the Windows event log I'm seeing 4 instances of this error:

       

      "A process serving application pool '******' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4652'. The data field contains the error number."

       

      Then finally:

       

      "Application pool '******' is being automatically disabled due to a series of failures in the process(es) serving that application pool.".

       

      Looking at the application pool in the IIS admin it is stopped. Restarting will resolve but it will eventually crash.

       

      What could be the issue?

       

      -Phil

        • 1. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
          Phil.Cruz Level 1

          I'm seeing in the CF logs leading up to the crash:

           

           

           

          Jun 04, 2014 11:11:31 AM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer

          SEVERE: Error in getRealPathFromConn

          java.net.SocketException: Connection reset by peer: socket write error

            at java.net.SocketOutputStream.socketWrite0(Native Method)

            at java.net.SocketOutputStream.socketWrite(Unknown Source)

            at java.net.SocketOutputStream.write(Unknown Source)

            at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:402)

            at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1276)

            at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:202)

            at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:424)

            at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4660)

            at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:432)

            at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.ja va:333)

            at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:184)

            at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116)

            at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:98)

            at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:999)

            at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:937)

            at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:866)

            at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:420)

            at cfApplication2ecfc1835967654.runPage(C:\Data\http\domain\apps\Application.cfc:3)

            at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:246)

            at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)

            at coldfusion.runtime.TemplateProxyFactory.resolveComponentHelper(TemplateProxyFactory.java: 538)

            at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:248)

            at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:159)

            at coldfusion.runtime.TemplateProxyFactory.resolveFile(TemplateProxyFactory.java:120)

            at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:138)

            at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:84)

            at coldfusion.runtime.AppEventInvoker.<init>(AppEventInvoker.java:64)

            at coldfusion.filter.PathFilter.resolveApplicationScope(PathFilter.java:195)

            at coldfusion.filter.PathFilter.invoke(PathFilter.java:124)

            at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)

            at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8)

            at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)

            at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:58)

            at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

            at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

            at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)

            at coldfusion.CfmServlet.service(CfmServlet.java:219)

            at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:303)

            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

            at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 )

            at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:241)

            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

            at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

            at java.lang.reflect.Method.invoke(Unknown Source)

            at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterCha in.java:97)

            at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorR equestHandler.java:472)

            at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest( FusionReactorRequestHandler.java:312)

            at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(Fusio nReactorRequestHandler.java:192)

            at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorR equestHandler.java:507)

            at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCor eFilter.java:36)

            at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

            at java.lang.reflect.Method.invoke(Unknown Source)

            at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterCha in.java:79)

            at sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

            at java.lang.reflect.Method.invoke(Unknown Source)

            at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactor StaticFilter.java:53)

            at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChai nPointCut.java:41)

            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java)

            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

            at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)

            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)

            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:422)

            at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:198)

            at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:607)

            at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

            at java.lang.Thread.run(Unknown Source)

          • 2. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
            rowingpeter_s

            I see the same entries in the Event log and the same end result. This is a single server. Win2008r2 with CF11.

            • 3. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
              Phil.Cruz Level 1

              I've engaged a consultant from http://www.cfconsultant.com/ to help me troubleshoot this. We've looked at everything haven't found any code or configuration that would seem to relate and have yet to find the root cause. This is happening with very little load on the server.


              My next step is to do a full uninstall-reinstall of IIS and CF11 and see if that resolves. If it doesn't, my next move is to bail on CF11 and go back to CF10. Unless someone from Adobe might have some suggestions as to how to resolve/troubleshoot.

               

              -Phil

              • 4. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
                vishu#13 Level 3

                give a try to test a new webiste. Create a new website in IIS, configure it with CF 11.This new website will have it's own application pool. Place all of the .cfm of your original website in this new website.

                Use Apache JMeter to use load/stress testing on this new website

                 

                So, in this way you can test without the cluster setup. If everything works fine then test the behavior in a cluster environment.

                 

                Apache JMeter steps

                 

                Download jmeter from the link mentioned below

                 

                https://jmeter.apache.org/download_jmeter.cgi

                 

                NOTE : click on apache-jmeter-2.11.zip under binaries

                 

                Test plan -> Right click --> Add - Thread group

                 

                Right click on Thread group --> Add - Sampler- HTTP Request

                 

                Right click HTTP Request --> Add- Listener - View Results in Table

                 

                Click on HTTP Request  and enter the server name like localhost or ip address, then port number of your site like 80

                 

                Enter the path as : http://localhost:81/hello.cfm

                 

                Click on Thread Group in left panel. Increase the number of threads as 40 or something like that. Enter the loop count like 5000 or so

                 

                Now click on the green start icon at top and click on view results in table

                 

                If you are successful in finding the root cause, you will be able to fix it in a quick succession.

                 

                HTH

                 

                Thanks

                VJ

                • 5. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
                  Phil.Cruz Level 1

                  I have already tried creating a new website in IIS. I have taken the server out of the cluster so I can isolate it and test it alone. Out of the cluster, the only load the server sees is requests from our monitoring system. So like a couple of page requests a minute, very little. Under that "load" I'm seeing the error so there's no need to use JMeter.

                   

                  My guess, I think it's related to a bug in the connector

                   

                  Bug 54117 – access violation exception in isapi_redirect.dll

                   

                  This was adressed but I think CF11 ships with the old connector.

                   

                  Thanks,

                  Phil

                  • 6. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
                    vishu#13 Level 3

                    Recreate the connector for this new website and this time enable the verbose log

                     

                     

                    Capture.JPG

                     

                    Every request to the website will be monitored and the log will be inside isapi_redirect.log (C:\ColdFusion11\config\wsconfig\1)

                    Share the logs

                    • 7. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
                      Phil.Cruz Level 1

                      I enabled logging. Here's the log file

                       

                      Dropbox - isapi_redirect.zip

                       

                      (Though I noticed it stopped logging even though I didn't disable logging. Does it stop logging after the log file gets to a certain size?)

                       

                      Do you see anything that indicates the root cause?

                       

                      -Phil

                      • 8. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
                        vishu#13 Level 3

                        You need to disable the verbose. Change the log level to info from debug in isapi_redirect.properties (C:\ColdFusion11\config\wsconfig\1)

                         

                        Did you face the issue after enabling verbose?

                        Have you added Application under your website? If yes, is that Application running with it's own App pool or the websit's App pool?

                        Also, share the Edit Application Pool settings screenshot

                         

                        1.JPG

                        • 9. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
                          Phil.Cruz Level 1

                          I didn't disable logging but it has in fact stopped logging since around 8:30am PST, so that is strange.

                           

                          I didn't get the exact errors mentioned in the original post (yet) but I am facing the issue in that some requests are taking very long. I have Failed Request Tracing enabled and it showing requests for simple static files like .css files taking 10+ seconds.

                           

                          -Phil

                          • 10. Re: CF11 is crashing ( application pool suffered a fatal communication error with the Windows Process Activation Service)
                            Phil.Cruz Level 1

                            I've spent a lot of time trying to figure out why this server is so unstable.

                             

                            I uninstalled CF11 and re-installed CF10 and now it seems back to normal. Just a guess, but I think there's an issue in the connector perhaps related to Bug 54117 – access violation exception in isapi_redirect.dll

                             

                            -Phil