1 2 Previous Next 76 Replies Latest reply: Dec 11, 2014 8:34 AM by Duffy RSS

    CF 10 continues to crash on Windows 2008 R2 64 bit

    BoReahard Community Member

      I have had CF 10 running on my VM for about a week now. The VM is Windows 2008 Enterprise, 64 bit. CF 10 is 64 bit.  The error that I keep getting is:

       

      Service Temporary Unavailable!

      The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.





      Jakarta/ISAPI/isapi_redirector/1.2.32 ()

      To recover, I need to stop and restart all the CF services.

      Anyone else having this problem or has anyone else had to fix it?

        • 1. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
          carl type3 Community Member

          Any errors and warnings in log files?

           

          CFadmin > Debugging & Logging > Log Files > coldfusion-error.log & coldfusion-out.log

           

          Could also enable tomcat metrics logging:

           

          CFadmin > Debugging & Logging > Debug Output Settings > Enable Metrics Logging

           

          then CFadmin > Debugging & Logging > Log Files > metrics.log

           

          You could also read those logs in the file system ColdFusion10\cfusion\logs

           

          HTH, Carl.

          • 2. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
            BoReahard Community Member

            I got the error again this morning. Here is what I got from the metrics log -

             

            Jun 4, 2012 7:45 AM Information scheduler-1
            Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 310923872 Total memory: 705232896 Active Sessions: 0
            Jun 4, 2012 7:44 AM Information scheduler-1
            Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 131668984 Total memory: 485359616 Active Sessions: 0
            Jun 4, 2012 7:43 AM Information scheduler-0
            Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 131670904 Total memory: 485359616 Active Sessions: 0
            Jun 4, 2012 7:42 AM Information scheduler-1
            Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 132744016 Total memory: 485359616 Active Sessions: 0
            Jun 4, 2012 7:03 AM Information scheduler-0
            Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 732531472 Total memory: 795607040 Active Sessions: 0
            Jun 4, 2012 7:02 AM Information scheduler-1
            Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 732640128 Total memory: 795607040 Active Sessions: 0
            Jun 4, 2012 7:01 AM Information scheduler-0
            Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 732640128 Total memory: 795607040 Active Sessions: 0
            Jun 4, 2012 7:00 AM Information scheduler-1
            Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 733411944 Total memory: 795607040 Active Sessions: 0
            • 3. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
              BoReahard Community Member

              Here is what came from the coldfusion-out.log file around the time of the crash -

               

              Jun 4, 2012 06:40:58 AM Information [scheduler-2] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 472578760 Total memory: 786694144 Active Sessions: 0
              Jun 4, 2012 06:41:58 AM Information [scheduler-2] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 471010920 Total memory: 786694144 Active Sessions: 0
              Jun 4, 2012 06:42:58 AM Information [scheduler-1] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 470469360 Total memory: 786694144 Active Sessions: 0
              Jun 4, 2012 06:43:58 AM Information [scheduler-2] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 467062464 Total memory: 786694144 Active Sessions: 0
              Jun 4, 2012 06:44:58 AM Information [scheduler-1] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 466982088 Total memory: 786694144 Active Sessions: 0
              Jun 4, 2012 06:45:58 AM Information [scheduler-2] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 466960872 Total memory: 786694144 Active Sessions: 0
              Jun 4, 2012 06:46:58 AM Information [scheduler-1] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 466077504 Total memory: 786694144 Active Sessions: 0
              Jun 4, 2012 06:47:58 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 747578808 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:48:58 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 746919072 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:49:58 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 746636008 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:50:58 AM Information [scheduler-2] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 742126568 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:51:58 AM Information [scheduler-2] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 740588936 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:52:58 AM Information [scheduler-2] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 740574256 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:53:58 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 740554888 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:54:58 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 740439008 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:55:58 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 740439008 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:56:58 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 734485712 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:57:58 AM Information [scheduler-2] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 733983216 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:58:58 AM Information [scheduler-2] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 733842304 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 06:59:58 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 733842304 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 07:00:58 AM Information [scheduler-1] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 733411944 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 07:01:59 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 732640128 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 07:02:59 AM Information [scheduler-1] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 732640128 Total memory: 795607040 Active Sessions: 0
              Jun 4, 2012 07:03:59 AM Information [scheduler-0] - Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 732531472 Total memory: 795607040 Active Sessions: 0

              • 4. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                carl type3 Community Member

                Of interest here are some CF10 metrics logging I took as a sample to compare.

                 

                05/06/2012

                8:49 AM

                Information

                scheduler-1

                Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 378536 Request count: 201 Error count: 4 Bytes received: 780 Bytes sent: 6993022 Free memory: 45875240 Total memory: 175702016 Active Sessions: 0

                05/06/2012

                8:48 AM

                Information

                scheduler-2

                Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 378536 Request count: 201 Error count: 4 Bytes received: 780 Bytes sent: 6993022 Free memory: 47207152 Total memory: 175702016 Active Sessions: 0

                05/06/2012

                8:47 AM

                Information

                scheduler-0

                Max threads: 150 Current thread count: 4 Current thread busy: 1 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 52128072 Total memory: 175898624 Active Sessions: 0

                05/06/2012

                8:46 AM

                Information

                scheduler-1

                Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 59819016 Total memory: 167116800 Active Sessions: 0

                05/06/2012

                8:45 AM

                Information

                scheduler-1

                Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 61154744 Total memory: 167116800 Active Sessions: 0

                05/06/2012

                8:44 AM

                Information

                scheduler-1

                Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 62484720 Total memory: 167116800 Active Sessions: 0

                05/06/2012

                8:43 AM

                Information

                scheduler-2

                Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 63821448 Total memory: 167116800 Active Sessions: 0

                05/06/2012

                8:42 AM

                Information

                scheduler-2

                Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 65154032 Total memory: 167116800 Active Sessions: 0

                05/06/2012

                8:41 AM

                Information

                scheduler-1

                Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 66487544 Total memory: 167116800 Active Sessions: 0

                05/06/2012

                8:40 AM

                Information

                scheduler-0

                Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 67824496 Total memory: 167116800 Active Sessions: 0

                 

                The online CF10 documentation omits CF10 metrics logging details so I am not sure what to make of the log detail differing. Reference:

                 

                http://help.adobe.com/en_US/ColdFusion/10.0/Admin/WSc3ff6d0ea77859461172e0811cbf3638e6-7fe 0.html#WSc3ff6d0ea77859461172e0811cbf3638e6-7fda

                 

                I don't think the logs have been any use so far. Still interested to know why CF10 is crashing for you. Something else comes to mind. Is it possible for you to run CF10 from CMD prompt for a time, ColdFusion10\cfusion\bin\cfstart.bat ? Perhaps there is an error and having the details displayed in CMD prompt will lead to solution.

                HTH again, Carl.

                 

                • 5. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                  BoReahard Community Member

                  Should I go in to the services control panel and stop all the cf services, then run the batch file from the command line?

                  • 6. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                    carl type3 Community Member

                    Stop ColdFusion 10 Application Server service

                     

                    CMD prompt (as administrator?)

                     

                    CD ColdFusion10\cfusion\bin

                     

                    cfstart.bat

                     

                    Watch wait report back any interesting details

                     

                    • 8. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                      BoReahard Community Member

                      At 7:07 AM today, I got the error again. Here is the netrics log from that time span, I'm not sure what I can tell from these???

                       

                      "Information","scheduler-1","06/07/12","07:00:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 476862912 Total memory: 554303488 Active Sessions: 0"
                      "Information","scheduler-1","06/07/12","07:01:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 468659888 Total memory: 554303488 Active Sessions: 0"
                      "Information","scheduler-0","06/07/12","07:02:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 466452672 Total memory: 554303488 Active Sessions: 0"
                      "Information","scheduler-0","06/07/12","07:03:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 465661168 Total memory: 554303488 Active Sessions: 0"
                      "Information","scheduler-1","06/07/12","07:04:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 464591872 Total memory: 554303488 Active Sessions: 0"
                      "Information","scheduler-0","06/07/12","07:05:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 463093992 Total memory: 554303488 Active Sessions: 0"
                      "Information","scheduler-1","06/07/12","07:23:21",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 142125480 Total memory: 495845376 Active Sessions: 0"
                      "Information","scheduler-0","06/07/12","07:25:11",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 129513304 Total memory: 495845376 Active Sessions: 0"
                      "Information","scheduler-1","06/07/12","07:26:11",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 129511384 Total memory: 495845376 Active Sessions: 0"

                      • 9. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                        carl type3 Community Member

                        Odd the CF10 metric log only showing memory values. Do not know what to make of that. Any feedback on running CF10 from CMD prompt?

                        • 10. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                          BoReahard Community Member

                          Have not had a chance to try from command line yet.  I have opened a case with Adobe support and I am going to give them a couple of days to work on this for me.

                          • 11. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                            carl type3 Community Member

                            Please post back findings I am sure is of interest to other CF10 users.

                             

                            There are other things can do eg CFSTAT, JVM logging, however not having a clear idea of where to look I do not want to send you down a path that is going to waste your time.

                             

                            • 12. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                              BoReahard Community Member

                              If I get something definitive from Adobe support, I will post it back here. Don't hold you breath though, when I tried to send them the server settings and log files, my email was blocked by their spam filter and they are trying now to get it released.

                              • 13. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                xxrxndxvxs

                                I cannot even get it to accept my admin password. After resets, etc. Windows 2008 Web R2 this is such a bummer.

                                • 14. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                  BoReahard Community Member

                                  My case with Adobe is still open. I opened a case with Microsoft becuase I found that simply restarting IIS 7.5 would restore the operation of the server.  I was fortunate enough to has a crash while Microsoft was on the phone with me last night and we got a memory dump of the process in IIS that was hung. The analysis of that memory dump is shown below. I have forwarded the dump to Adobe for their analysis.

                                   

                                  Hi Ralph,

                                   

                                  Today we captured 4 set of dumps while page was in a hung state.

                                  Please find analysis with this email :-

                                   

                                  Looking at Dumps: w3wp.dmp ,

                                                                        w3wp (2) .dmp

                                                                        w3wp (3) .dmp

                                                       w3wp (4).dmp

                                  èThe following threads in W3WP(4~1.DMP are waiting on data to be returned from another server via WinSock.

                                  The call to WinSock originated from isapi_redirect+10ba2

                                  Threads ( 4 5 6 7 15 16 17 18 19 20 22 24 )

                                  48.00% of threads blocked

                                  Number of Threads being blocked are increasing .

                                   

                                  èLet’s see what is happening at Thread 4.

                                  This thread is waiting on data to be returned from another server via WinSock.

                                  The call to WinSock originated from isapi_redirect+10ba2

                                  CallStack

                                   

                                  ntdll!ZwWaitForSingleObject+a

                                  mswsock!SockWaitForSingleObject+cc25

                                  mswsock!WSPRecv+1166e

                                  ws2_32!recv+13c

                                  isapi_redirect+10ba2

                                  isapi_redirect+8f82

                                  isapi_redirect+a520

                                  isapi_redirect+d4d3

                                  isapi_redirect!HttpExtensionProc+4f5

                                  isapi!ProcessIsapiRequest+13c

                                  isapi!W3_ISAPI_HANDLER::DoWork+4fb

                                  isapi!RequestDoWork+40d

                                  isapi!CIISHttpModule::OnExecuteRequestHandler+1c

                                  iiscore!NOTIFICATION_CONTEXT::RequestDoWork+233

                                  iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+17f

                                  iiscore!NOTIFICATION_CONTEXT::CallModules+25

                                  iiscore!W3_CONTEXT::DoWork+3a6

                                  iiscore!W3_MAIN_CONTEXT::StartNotificationLoop+5d

                                  iiscore!W3_CONTEXT::ExecuteRequest+2bf

                                  filter!RequestDoWork+40be

                                  filter!CIISHttpModule::OnPostAuthenticateRequest+20

                                  iiscore!NOTIFICATION_CONTEXT::RequestDoWork+20a

                                  iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+17f

                                  iiscore!NOTIFICATION_CONTEXT::CallModules+25

                                  iiscore!W3_CONTEXT::DoWork+326

                                  iiscore!W3_MAIN_CONTEXT::OnNewRequest+1ea

                                  w3dt!UL_NATIVE_REQUEST::DoWork+126

                                  w3dt!OverlappedCompletionRoutine+1c

                                  w3tp!THREAD_POOL_DATA::ThreadPoolThread+7a

                                  w3tp!THREAD_POOL_DATA::ThreadPoolThread+34

                                  w3tp!THREAD_MANAGER::ThreadManagerThread+61

                                  kernel32!BaseThreadInitThunk+d

                                  ntdll!RtlUserThreadStart+1d

                                  è Request is currently in IsapiFilterModule as shown below :-

                                  !pipeline 0000000001372990

                                  BEGIN_REQUEST( 0x1 ):

                                      cachhttp-HttpCacheModule

                                      modrqflt-RequestFilteringModule

                                      validcfg-ConfigurationValidationModule

                                      rewrite-RewriteModule

                                  AUTHENTICATE_REQUEST( 0x2 ):

                                      filter-IsapiFilterModule

                                      authanon-AnonymousAuthenticationModule

                                    POST:

                                  --> filter-IsapiFilterModule

                                  RESOLVE_REQUEST_CACHE( 0x8 ):

                                      cachhttp-HttpCacheModule

                                  MAP_REQUEST_HANDLER( 0x10 ):

                                      compstat-StaticCompressionModule

                                  EXECUTE_REQUEST_HANDLER( 0x80 ):

                                      defdoc-DefaultDocumentModule

                                      dirlist-DirectoryListingModule

                                      protsup-ProtocolSupportModule

                                      static-StaticFileModule

                                      isapi-IsapiModule

                                      iis_ssi-ServerSideIncludeModule

                                      cgi-CgiModule

                                      iisfcgi-FastCgiModule

                                  RELEASE_REQUEST_STATE( 0x100 ):

                                      compstat-StaticCompressionModule

                                    POST:

                                      rewrite-RewriteModule

                                  UPDATE_REQUEST_CACHE( 0x200 ):

                                      cachhttp-HttpCacheModule

                                  SEND_RESPONSE( 0x20000000 ):

                                      loghttp-HttpLoggingModule

                                      isapi-IsapiModule

                                      protsup-ProtocolSupportModule

                                      filter-IsapiFilterModule

                                      cachhttp-HttpCacheModule

                                      rewrite-RewriteModule

                                      custerr-CustomErrorModule

                                   

                                  è0:000> !connections

                                  64 Bit IIS 7.5 Detected

                                  RequestListHead at 000007fefc939088

                                  NUM     NATIVEREQ                        MAIN_CONTEXT                              IIS_REQMON_CONTEXT               NREQ_STATE     VERB                VERSION              SERVER ADDRESS                             CLIENT ADDRESS                              REQUEST ACTIVE                                              URL

                                  ------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------ -----------------

                                  1              0000000001352dd0          00000000013539f0                           1354398                                                PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 208.104.28.202(3592)                      0 days: 00:05:49.578                        /jakarta/isapi_redirect.dll

                                  2              0000000001355f60           0000000001356b80                          1357528                                                PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 208.104.28.202(3654)                      0 days: 00:05:26.422                        /jakarta/isapi_redirect.dll

                                  3              00000000013590f0           0000000001359d10                          135a6b8                                               PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 208.104.28.202(3930)                      0 days: 00:05:21.516                        /jakarta/isapi_redirect.dll

                                  4              000000000135c280           000000000135cea0                           135d848                                               PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 67.54.20.62(55208)                          0 days: 00:04:46.328                        /jakarta/isapi_redirect.dll

                                  5              000000000135f410           0000000001360030                           13609d8                                               PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 67.54.20.62(55221)                          0 days: 00:02:49.516                        /jakarta/isapi_redirect.dll

                                  6              00000000013625a0           00000000013631c0                           1363b68                                               PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 67.54.20.62(55222)                          0 days: 00:02:37.750                        /jakarta/isapi_redirect.dll

                                  7              0000000001365730           0000000001366350                           1366cf8                                                PROCESS               GET                HTTP/1.1              67.54.20.62(80)                                 67.54.20.62(55225)                          0 days: 00:02:35.188                        /jakarta/isapi_redirect.dll

                                  8              00000000013688c0           00000000013694e0                          1369e88                                               PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 67.54.20.62(55227)                          0 days: 00:02:33.0                             /jakarta/isapi_redirect.dll

                                  9              000000000136ba50          000000000136c670                           136d018                                               PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 67.54.20.62(55230)                          0 days: 00:00:50.532                        /jakarta/isapi_redirect.dll

                                  10           000000000136ebe0          000000000136f800                           13701a8                                                PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 67.54.20.62(55231)                          0 days: 00:00:26.500                        /jakarta/isapi_redirect.dll

                                  11           0000000001371d70          0000000001372990                           1373338                                                PROCESS              GET                HTTP/1.1              67.54.20.62(80)                                 67.54.20.62(55234)                          0 days: 00:00:25.938                        /jakarta/isapi_redirect.dll

                                   

                                  è

                                  Lmvm  isapi_redirect

                                  start             end                 module name

                                  00000000`6a6b0000 00000000`6a714000   isapi_redirect   (deferred)            

                                      Image path: D:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll

                                      Image name: isapi_redirect.dll

                                      Timestamp:        Thu Mar 29 07:50:00 2012 (4F745A78)

                                      CheckSum:         0006CC8A

                                      ImageSize:        00064000

                                      File version:     1.2.32.0

                                      Product version:  1.2.32.0

                                      File flags:       0 (Mask 3F)

                                      File OS:          40004 NT Win32

                                      File type:        1.0 App

                                      File date:        00000000.00000000

                                      Translations:     0409.04b0

                                      CompanyName:      Apache Software Foundation

                                      ProductName:      Apache Tomcat isapi_redirector Connector

                                      InternalName:     isapi_redirector

                                      OriginalFilename: isapi_redirector.dll

                                      ProductVersion:   1.2.32

                                      FileVersion:      1.2.32

                                      FileDescription:  Apache Tomcat Connector

                                      LegalCopyright:   Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements.  See the NOTICE file distributed with this work for additional information regarding copyright ownership.

                                      Comments:         The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.  You may obtain a copy of the License at

                                  Action Plan:

                                  To follow up with the vendor for  “isapi_redirect .dll“ for further troubleshooting.

                                   

                                  Regards,

                                  Anjum


                                  Microsoft is committed to protecting your privacy. Please read the Microsoft Privacy Statement for more information.
                                  The above is an email for a support case from Microsoft Corp.
                                  REPLY ALL TO THIS MESSAGE or INCLUDE
                                  casemail@microsoft.com IN YOUR REPLY if you want your response added to the case automatically.  For technical assistance, please include the Support Engineer on the TO: line.
                                  Thank you.

                                  • 15. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                    BoReahard Community Member

                                    Issue was never resolved for CF10 so I down graded to CF9 and the server is running just fine. Adobe support helped me get through the process of moving back from CF10 to CF9

                                    • 16. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                      JakeHand

                                      One option you have when receiving those errors is to change your IIS connector to something like the Boncode IIS to Tomcat connector (which is currently used by Railo's Windows installer). From what I can tell, the Adobe connector just can't handle high load and gives out at a certain (undetermined) threshold.

                                       

                                      Bilal (the main developer of the project) has been very helpful with making the Boncode connector very compatible with CF 10, and I've seen good results from it so far. Here's the link to the project: http://tomcatiis.riaforge.org/

                                      • 17. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                        BoReahard Community Member

                                        Thanks for the thoughts.  I have given up on CF10 for now and reverted to CF9

                                        • 18. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                          Ryan Luo Community Member

                                          I have tried the boncode connector since I am having http://forums.adobe.com/message/4619002#4619002, and I have tried different configuration setting for the connector, but the page always has issue with displaying. I will have to keep refreshing the page, until the contents have been showed correctly and in full. It not only happens to my application, but also the cfadmin page. The header part of the admin page sometimes shown as blank. In order to make it display correctly, I will need to refresh the page couple of times.

                                          • 19. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                            Aquitaine Community Member

                                            We have this happen about once every 2-3 weeks on a site with a few thousand active users. Moreso when we are undergoing are quarterly PCI tests and we get blitzed by SecuritMetrics' automated tests.

                                             

                                            I'd prefer not to monkey around with a 3rd-party connector. Is Adobe aware of this?

                                            • 20. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                              xxrxndxvxs Community Member

                                              I have just given up on it at this point. Feeling like a wasted purchase. I have reverted to using Railo. I hope their is some response from Adobe.

                                               

                                              Aaron

                                              • 21. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                Ryan Luo Community Member

                                                We are on the same boat, my server randomally crashes when there are few thounsands active users as well.

                                                I did spend some time on a alternative connecter myself, but I couldn't get it to work perfectly. And I do agree with you that we shouldn't play around with some 3rd party thing, since the cf10 is using some customized version of tomcat connector and there are lots of things we are not sure at this stage

                                                • 23. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                  JakeHand Community Member

                                                  We've been looking into this further and have found that tuning the connector timeout and the worker.properties can really help with this issue. I'm still a big fan of the BonCode connector though, considering how fast Bilal's turnaround time on adapting to CF 10 has been (and that it does not run as a funky ISAPI redirect filter). With that said, you may want to open up your server.xml in "cfusion\runtime\conf" and find the section for the AJP connector (~line 100), and set the maxThreads, connectionTimeout, and acceptCount attributes. Should look similar to this:

                                                   

                                                  <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="300" connectionTimeout="60000"  acceptCount="50" />

                                                   

                                                  Those attributes help prevent Tomcat from spooling up too many or not enough threads for your application. The acceptCount also queues things up temporarily if you reach the max number of threads.

                                                   

                                                  Now you'll want to edit workers.properties at "C:\ColdFusion10\config\wsconfig\1".

                                                   

                                                  By default Adobe does not have any connection pool limitations, so things can get out of hand when an application gets busy. If your server hosts a lot of IIS sites, you may want to consider adding these lines to your workers.properties file:

                                                   

                                                  worker.cfusion.connection_pool_size=10

                                                  worker.cfusion.connection_pool_minsize=1

                                                  worker.cfusion.connection_pool_timeout=60

                                                   

                                                  More info on this here: http://tomcat.apache.org/connectors-doc/reference/workers.html

                                                   

                                                  Those adjustments have helped tame this issue for sites that need to use the Adobe connector. Your mileage will vary, and you'll want to tweak those settings to meet your environment, but good luck!

                                                   

                                                  Also, Ryan, the issue you mentioned with the connector is very odd. Was it perhaps set as a Wildcard and trying to process every request, even the static files? If you do feel like continuing with the connector though, you may want to upgrade to the latest version. I know Bilal's fixed a few bugs in the last month.

                                                  • 24. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                    Ryan Luo Community Member

                                                    Thank you for your reply JakeHand

                                                    I would like to try the alternative connector in my VM, but not production at the moment, but can you teach me how to correctly setup the Boncode?

                                                    I tried different configuration myself (none of them uses wildcat), but just doesn't work for me.

                                                     

                                                    Regards

                                                    • 25. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                      lg84

                                                      We are experiencing a similar issue.  We have CF10 running on VM (32 bit Windows Server 2008).  We've been live almost a week and the issue just started last night.  We saw the message "Service Temporarily Unavailable" and "Service Unavailable - Zero size Object."  We restarted the server multiple times. It would work initialy after the restart but would then fail again, usually within 10 minutes.  Looking at the IIS log, we did have a lot of hits from bots so I'm guessing the increased traffic made the issue occur.  We have moved back to CF9 to prevent the issue and probably have to place a support call with Adobe.

                                                      • 26. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                        Miguel-F Community Member

                                                        Everyone should "vote" for the bug that was entered for this.  That will help show Adobe that more than one user is experiencing the issue.

                                                         

                                                        https://bugbase.adobe.com/index.cfm?event=bug&id=3318104

                                                        • 27. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                          hemant_k Adobe Employee

                                                          Is this issue similar to

                                                          https://bugbase.adobe.com/index.cfm?event=bug&id=3222748?

                                                           

                                                          We are investigating the above issue.

                                                          • 28. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                            Aquitaine Community Member

                                                            No, I don't think so.

                                                             

                                                            The connector breaks in both cases but we don't have any xml being submitted in our app.

                                                            • 29. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                              lg84 Community Member

                                                              There is a new Coldfusion 10 update. It mentions a bug fix for Connector. Has anyone installed this and has it fixed the issue?

                                                              • 30. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                JakeHand Community Member

                                                                We applied the hotfix to our CF 10 servers earlier this week, and so far our uptime has drastically improved their uptime (and we haven't been seeing the Service Unavailable errors).

                                                                 

                                                                Keep in mind though, we also added the tuning parameters to our server.xml files as mentioned above, and specifying an appropriate maxThreads is important. So far we've gotten away with increasing that value to 1500, but our hardware is pretty hefty so you want to make sure the value is tuned for your environement. Also, we ended up removing the acceptCount and instead added a  keepAliveTimeout to help make sure client connections aren't prematurely timed out. By default this is set to the same value as connectionTimeout, and we wanted a much higher keepAliveTimeout than connectionTimeout. This page has a description of all the attributes you can add to the AJP connector in your ColdFusion (Tomcat) server.xml file.

                                                                • 31. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                  BigJ57 Community Member

                                                                  The alternate connector worked for me.

                                                                  I used directions in this blog:

                                                                  http://boncode.blogspot.com/2012/06/cf-coldfusion-10-experimenting-with.html

                                                                   

                                                                  The trick for me was to add the folders in section d) to IIS site and assign wildcard mappings to each folder.

                                                                  Here is the setting file I used:

                                                                   

                                                                  <!-- CF10 connection -->
                                                                  <Settings>
                                                                    <Port>8012</Port>
                                                                    <Server>localhost</Server>  
                                                                    <MaxConnections>200</MaxConnections>
                                                                    <FlushThreshold>0</FlushThreshold>
                                                                    <EnableAdobeMode>True</EnableAdobeMode>
                                                                  </Settings>

                                                                   

                                                                  • 32. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                    roland.collins Community Member

                                                                    This error has led to our production CF10 rollout being an absolute disaster.  The error never occurred once during our development and staging cycles, and a load tester has been unable to reproduce the issue in either environment, so we can't tell if it's due to high load, memry leaks, etc.  Our production servers have been a mess, and our clients are more than a bit angry.

                                                                     

                                                                    Our application event log is littered with these:

                                                                     

                                                                    Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

                                                                    Faulting module name: isapi_redirect.dll, version: 1.2.32.0, time stamp: 0x4f745a78

                                                                    Exception code: 0xc0000005

                                                                    Fault offset: 0x0000000000018b93

                                                                    Faulting process id: 0x11f0

                                                                    Faulting application start time: 0x01cd950afc876c76

                                                                    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

                                                                    Faulting module path: C:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll

                                                                    Report Id: 98760e12-00ff-11e2-9e13-00155d015102

                                                                    • 33. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                      charlie@carehart.org Community Member

                                                                      To future readers of this thread, updater 3 for CF10 is supposed to address some of these very problems, so Adobe is indeed on the case.

                                                                       

                                                                      Unfortunately, many people have found there to be new, different problems introduced by it (most importantly, loss of scheduled tasks, both on restart of CF after applying the update, and then also on any subsequent updates.) Adobe has also acknowledged that and are working on a correction.

                                                                       

                                                                      You can read the discussion of people having problems (and Adobe's reply) at http://blogs.coldfusion.com/post.cfm/coldfusion-10-update-3-released.

                                                                       

                                                                      And the technote for the update is at http://helpx.adobe.com/coldfusion/kb/coldfusion-10-update-3.html. Perhaps tomorrow or in coming days it may be updated if they fix whatever was wrong that was introduced in the updater itself.

                                                                       

                                                                      But again, as for the issues in this thread, hopefully there will be resolution soon when this updater 3 is sorted out.

                                                                       

                                                                      Hope that's helpful to someone here.

                                                                      • 34. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                        *** Tyson *** Community Member

                                                                        Same exact issue for me with one of my biggest clients.  Constant crashes on our ColdFusion 10 server multiple times per day.  Event viewer is riddled with constant errors (thousands per day) from the w3wp.exe service and the ISAPI connector.  I was able to seemingly relieve the situation somewhat by disabling IIS 7.5's "Rapid-Fail Protection" on the application pools (which I can't imagine is advisable for a production environment), but that only seems to have decreased the frequency of the crashes slightly - from a dozen times each day, down to only a few times per day.  Completely unacceptable.

                                                                         

                                                                        It absolutely astounds me that Adobe has not halted in their tracks and prioritized every single reputatble development resource with developing and releasing an effective resolution to this problem.  Honestly, it gives me cause (for the first time in 17 years of ColdFusion development and support) to consider migrating away from ColdFusion to a different platform.  Updater 3 was apparently a disaster (thank God I checked for community feedback before installing it), but I've not heard a single follow up about the fix to this problem since the Updater 3 was recalled by Adobe - and that was a couple weeks ago.

                                                                         

                                                                        How on earth can an absolutely critical bug like this remain open and unaddressed for so long without any resolution!?  Adobe, you've -completely- dropped the ball on this and disappointed droves upon droves of loyal customers, developers, and community members.  Our clients are angry.  Not only are they losing money and credibility every single day that this bug remains unfixed, but their confidence and support of ColdFusion as a platform is dwindling.  What in the world is holding up the completion and release of a prioritized hot fix for this issue!?  Forget about bundling it up with a bunch of other fixes at this point.  This bug is critical enough to stand on its own and be addressed immediately!

                                                                         

                                                                        Unbelievable.

                                                                        • 35. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                          hemant_k Adobe Employee

                                                                          Hi,

                                                                           

                                                                          We will be most likely releasing update 4 (after resolving issues with update 3) by end of this week. We are done with our internal testing and private testing from other community members and this process took us some time. Thursday is a holiday and hence most likely date will be Friday if last sanity check are all ok.

                                                                           

                                                                          Thanks for your support and feedback on this issue.

                                                                           

                                                                          Hemant

                                                                          • 36. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                            roland.collins Community Member

                                                                            Wow.  Did you really just say that you guys are going to take a day off and wait to release ths due to a holiday?  And then you only have a "likely" release date?

                                                                             

                                                                            Do you have any idea how much time, money, and goodwill this bug has cost you already?

                                                                             

                                                                            Do you have any idea how much pain this has caused your user base?

                                                                             

                                                                            Do you have any idea what customer service is?

                                                                             

                                                                            You have clearly demonstrated that you just don't care about your users and customers any more.  This product has taken a swift downward decline in the last two years, and it is now time for everyone to start looking at alternatives.

                                                                            • 37. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                              hemant_k Adobe Employee

                                                                              Hi Roland,

                                                                               

                                                                              Thanks for your feedback. We certainly understand the pain that this issue has caused and want to be careful before we release the patch. I realize that holiday reference was unnecessary and misleading and sorry about that. We have shared this patch to many community members early last week, by quickly trying to fix all issues, since we are committed to fix and release the update quickly. As I mentioned ealrier, we have one last issue being discussed with one of the members and that we are expecting to close everything on Friday.

                                                                               

                                                                              If you need the private fix and help us verify the same, please let me know at hkhandel a t adobe . com

                                                                               

                                                                              Thanks,

                                                                              Hemant

                                                                              • 38. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                                Adam Cameron. Community Member

                                                                                roland.collins wrote:

                                                                                 

                                                                                Do you have any idea how much time, money, and goodwill this bug has cost you already?

                                                                                 

                                                                                Do you have any idea how much pain this has caused your user base?

                                                                                 

                                                                                 

                                                                                Roland: steady on mate.

                                                                                 

                                                                                I agree that the bug in updater 3 which cleared out scheduled tasks was a bit annoying, and there's no excuse for not catching that during testing.  I understand there were other bugs with it too (I dunno what though... do you?).

                                                                                 

                                                                                But so you backed-out the updater, restored your tasks (from the back-up of your config you made before upgrading, right?), and you were back to updater 2. And that shoulda taken about 5min.

                                                                                 

                                                                                Updater 2 is stable.  What of the bug fixes that updater 3 was going to deliver are sooooo important to you that wait a coupla weeks is that major a problem? Serious question.  Were you clamouring for one of those bugs it fixed to be fixed as it was causing you problems in some way?  In some way you can't just hold on for a bit before it gets fixed and tested?

                                                                                 

                                                                                Look, I'm not defending Adobe (anyone who knows me will point out that I'm not exactly their favourite person): it's taken an age to get this bug in the updater fixed, and I do question why. And that it went out the door with such an obvious bug in it is a bit of a let down. But they're fixing it: cool.  They're testing it: cool. There's clearly a public holiday in India today (it's Thurs as I type), and it's not like the problem with the updater is really the end of the world.  Bear in mind that the bods in the Adobe ColdFusion team are people, and they have families and plans and stuff to do.  Seriously, if you have a non-urgent bug to fix (and this is a non-urgent bug, in the bigger scheme of things), do you cancel your weekend or your bank holiday plans just to fix it?  Well bravo if you do, but I think the bulk of us would just resolve ourselves to the fact that it'll still be there on Monday, and that's OK.  That's all Hemant said they were doing.

                                                                                 

                                                                                Let's just hope they get it out the door tomorrow. Although if I had production boxes waiting to be patched... I'd not be doing it on a Fri, to be honest. Well: I'd be testing this thing on a test box first, TBH, well before thinking of putting it live.

                                                                                 

                                                                                --

                                                                                Adam

                                                                                • 39. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                                  charlie@carehart.org Community Member

                                                                                  Well, I’ll offer a thought to defend at least some of the angst--though I do think there can often be an overreaction by some as to what Adobe does or doesn’t do. (I trust that they’re smart people and want always to rectify things as soon as they can, and they DO appreciate how this has hurt many, and they would have fixed this sooner if they could have.)

                                                                                   

                                                                                  In the case of the “need” for updater 3, there were in fact a couple of really important problems that it was meant to fix, ones that for some people have CF10 (even with updater 1 or 2) not performing at all well, and often even failing, under load. And it was THAT fix that many desperately wanted to get with Updater 3.

                                                                                   

                                                                                  So the fact that it then caused still other problems (some severe) did leave move to back it out. More important though, Adobe pulled it, so that people couldn’t even get it if they were ok with those “problems” it added, in order to get the fixes that it DID add.

                                                                                   

                                                                                  So yes, some really are desperate to get the fix and can’t really just be satisfied (as it were) that “updater 2 is stable”, because it’s not solving the more important problems that some folks have hit.

                                                                                   

                                                                                  It is indeed a bummer that the fix and testing has taken this long. Let’s hope that part of the reason it’s taking longer is that they’re doing more testing so that this sort of thing doesn’t happen again. (Again, I think that they’re smart people who may make a mistake but would learn from it, and I don’t feel as some do that they’ve proven entirely otherwise.)

                                                                                   

                                                                                   

                                                                                   

                                                                                  /charlie

                                                                                  1 2 Previous Next