1 2 Previous Next 76 Replies Latest reply on Dec 11, 2014 8:34 AM by Duffy Go to original post
      • 40. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
        Adam Cameron. Level 5

        hemant_k wrote:

         

        Hi,

         

        We will be most likely releasing update 4 (after resolving issues with update 3) by end of this week.

         

        Right.  So here we are then...

         

        --

        Adam

        • 41. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
          roland.collins Level 1

          Adam,

           

          If I seem angry, it's because I am!!!

           

          This issue isn't the only fix we've been waiting on. In fact, it's not the important one, as far as I'm concerned.  https://bugbase.adobe.com/index.cfm?event=bug&id=3222748 is the bad one.

           

          I still can't believe it hasn't sunk in for Adobe. Because of that issue, I can DOS any CF10 server at will.  It's not just about a broken connector - if I know that your server is running CF10, I can crash it instantly, and there is nothing you can do about it. That seems to me like the kind of thing that should be addressed in days, not months.

           

          It's also because every single customer support incident I have had in the last 3 years has been an utter disaster.  We pay many MANY thousands of dollars for a bunch of CF licenses AND pay for support incidents. Yet the last time I had to get a critical bug fixed, I actually had to email Ben Forta (god bless him for still being a truly awesome guy), and ask HIM to escalate my issue because Adobe was ignoring the issue in the product I paid for, after I again paid them just so I could submit a bug report. 

           

          So yes, I'm pissed.  I've been championing CF as a platform since 1997, and recently it seems like it has gone way downhill.  For five months there has been a bug open that will let me crash any CF10 server at will.  It keeps XML webservices from being reliable too.   And instead of communicating the issue clearly and openly, Adobe says "we will release a fix when we can".  Then the fix they release breaks every CF server it was applied to. 

           

          Yeah, remind me again where my thousands upon thousands of dollars went?

          • 42. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
            Adam Cameron. Level 5

            I can't fault anything you say Roland (and, yes, I know you have previously been a champion for CF, so I was surprised @ your "outburst" before).

             

            My one support ticket with Adobe has been much the same as yours. This was some years ago now, and hardly anyone involved in CF now would be the same as the people I was dealing with.

             

            And it's now after close-of-business in India, so I'm guessing we're not getting the update this week?

             

            --

            Adam

            • 43. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
              Evelin Varghese Level 2

              Hi Roland ,

              ColdFusion 10 Update 4 is released today.Please find more details here - http://helpx.adobe.com/coldfusion/kb/coldfusion-10-update-4.html

              Please apply the update and reconfigure connector .We have fixed this isse in this update.

              Please let us know if you are still facing the problem.

              Thanks

              Evelin Varghese

              ColdFusion Team

              • 44. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                Tim Cunningham-ACP Level 2

                Please note if you receive a "Failed Signature" message you need this manual update: http://cfml.us/Ob

                 

                Thanks,

                 

                Tim Cunningham

                • 45. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                  conanbarb Level 1

                  Are there more details on how to update the connector and how to verify? 

                   

                  In my case, the Remove button is greyed out and only the Add button is available.  I click on the Add button, then hit OK.  When it says the web server is currently running and must be restarted to add this configuration I click Yes.  Is this the correct procedure?

                   

                  Also, how do I verify that the connector has been updated successfully?

                   

                  Thanks,

                  Christian

                  • 46. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                    Evelin Varghese Level 2

                    @Christian, You need to click the entry for your webserver connection for ex:


                    [ localhost:cfusion ] Internet Information Server (IIS) : All


                    Once you click the entry Remove button will get enabled.

                    • 47. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                      *** Tyson ***

                      I'm sad to report that I have recently (last night) applied update 4 to our ColdFusion 10 production server, yet I am still seeing problems under the hood that remain unaddressed with this update.  Specifically, I'm still seeing constant faults from IIS/w3wp.exe/isapi_redirect.dll within the Application Log in the Windows Event Viewer.

                       

                      Allow me to give a brief bit of background for any Adobe folks who may be listening out there.

                       

                      My client was previously running a 32-bit server environment with ColdFusion 8.  We skipped ColdFusion 9 and were attempting to upgrade directly to ColdFusion 10.  Presently, my client is now running a newly launched 64-bit production environment with Windows Server 2008 R2 SP1 and IIS 7.5.7600.16385.  Prior to installing update 4 last night at midnight, my client had briefly been running ColdFusion 10 update 2.  We had numerous stability problems with ColdFusion 10 update 2 - most involving various types of crashes, service inavailabilities, 503 errors, etc (bugs 3222748, 3216317, 3318104, and 3300889).  Though we were tempted to jump on update 3 when it was released, we held patient to wait for community feedback from people brave enough to install the update.  When we heard the community feedback and ultimate recall of update 3, we patiently remained on update 2 awaiting a reliable fix.

                       

                      In the meantime, our server was crashing multiple times a day.  This was a direct result of internal faults being generated by w3wp.exe/isapi_redirect.dll triggering IIS's "Rapid-Fail Protection" which, in turn, was shutting down application pools within IIS.  As a temporary solution to keep our server from crashing every 15-20 minutes, I completely disabled IIS's Rapid-Fail Protection.  So, IIS is no longer shutting down the application pool automatically when it detects frequent faults from within IIS and/or the connector to ColdFusion.  With Rapid-Fail Protection disabled, the frequency of these crashes dropped off significantly (though we have still experienced a few of them with this disabled).  However, the frequency of reports in the Application Log within the Windows Event Viewer has not diminished at all.  To quantify that just a bit, our production server logged roughly 26,000 of these fault errors within the past 30 days.

                       

                      On Tuesday, November 13, 2012 (technically last night/this morning), we proceeded cynically with the installation of update 4 on our production server.  The installation was successful without any errors.  And yes, we also reconfigured the web server connectors via wsconfig.exe (executed with Administrator permissions, of course) once update 4 installation was complete.  Presently, our ColdFusion server reports a version number of 10,0,4,283281.

                       

                      Today, here's what our Application Log looks like.  Remember - this is with update 4 installed.

                       

                      Screen Shot 2012-11-13 at 1.39.52 PM.jpg

                      You can see that every few minutes, we're getting a group of 3 entries - one "Error" entry and two "Information" entries.

                       

                      The "Error" entries describe a fault from w3wp.exe (IIS) and a faulting module of isapi_redirect.dll (the one within the ColdFusion10\config\wsconfig folder structure).

                       

                      Here's what the "Error" entry contains.

                       

                      ---[ SNIP ]---

                      Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

                      Faulting module name: isapi_redirect.dll, version: 1.2.32.0, time stamp: 0x50850ee6

                      Exception code: 0xc0000005

                      Fault offset: 0x00000000000118c1

                      Faulting process id: 0x1414

                      Faulting application start time: 0x01cdc1c716356825

                      Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

                      Faulting module path: D:\ColdFusion10\config\wsconfig\2\isapi_redirect.dll

                      Report Id: b93bdec9-2dbb-11e2-9b05-0050568a6da8

                      ---[ SNIP ]---

                       

                      And here's what the two accompanying "Information" events contain.

                       

                      ---[ SNIP ]---

                      Fault bucket , type 0

                      Event Name: APPCRASH

                      Response: Not available

                      Cab Id: 0

                       

                      Problem signature:

                      P1: w3wp.exe

                      P2: 7.5.7601.17514

                      P3: 4ce7afa2

                      P4: isapi_redirect.dll

                      P5: 1.2.32.0

                      P6: 50850ee6

                      P7: c0000005

                      P8: 00000000000118c1

                      P9:

                      P10:

                       

                      Attached files:

                       

                      These files may be available here:

                      C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_c7987f849945058256ab720 f046114e117b593e_3d793644

                       

                      Analysis symbol:

                      Rechecking for solution: 0

                      Report Id: b93bdec9-2dbb-11e2-9b05-0050568a6da8

                      Report Status: 4

                      ---[ SNIP ]---

                       

                      ---[ SNIP ]---

                      Fault bucket , type 0

                      Event Name: APPCRASH

                      Response: Not available

                      Cab Id: 0

                       

                      Problem signature:

                      P1: w3wp.exe

                      P2: 7.5.7601.17514

                      P3: 4ce7afa2

                      P4: isapi_redirect.dll

                      P5: 1.2.32.0

                      P6: 50850ee6

                      P7: c0000005

                      P8: 00000000000118c1

                      P9:

                      P10:

                       

                      Attached files:

                       

                      These files may be available here:

                      C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_c7987f849945058256ab720 f046114e117b593e_3d793644

                       

                      Analysis symbol:

                      Rechecking for solution: 0

                      Report Id: b93bdec9-2dbb-11e2-9b05-0050568a6da8

                      Report Status: 0

                      ---[ SNIP ]---

                       

                      If update 4 had truly resolved the problems plaguing IIS and the web connector, then I would not expect to see these Application Log entries continuing to amass with the update applied.  The fact that these faults persist post update is, to say the least, unsettling.

                       

                      As a contingency, I wound up calling Adobe to insist that they grant my client a backward license for ColdFusion 9 (which they technically never owned or installed since they were upgrading from ColdFusion 8) in the event that we'd like to install it until ColdFusion 10 proves (if ever) to be a stable build on Windows/IIS configurations.  After 58 minutes on the phone being passed from Customer Support to a Customer Support Supervisor to the Volume Licensing Department and finally to the ColdFusion Support/Tech team re-explaining the same scenario and circumstance over and over and over and over again, Adobe finally agreed to grant a ColdFusion 9 license to my client at no additional cost.  Presently, I am drafting a plan to degrade our production server from ColdFusion 10 back down to ColdFusion 9 since I've lost all confidence in ColdFusion 10 at this point.

                       

                      Soapbox time.

                       

                      Adobe... seriously... for the sake of the development community that has supported and assisted in helping to march this product from mom-and-pop shops all the way into the ranks of enterprise-class businesses around the world over the 17-year history of this platform because they -believe- in this platform, PLEASE get your act together.  I know it must be a nuisance for you to have to continue to support "legacy server products" that came attached as a sidecar during your acquisition of Macromedia when your primary focus and emphasis is constantly prioritized on refining and selling your bread-and-butter client-side and design tools and your subscription-model offerings.  Your silent depriotization of ColdFusion has become more and more obvious over the years.  ColdFusion is always buried under some "All Other Products" option on the Adobe web site, and when you dial in to Support and ColdFusion is buried 4 prompts deep beneath options including "a list of additional products" and "a product you did not hear listed," it doesn't exactly instill confidence in the remaining community that this product bears any importance to Adobe.  But there are dedicated people sitting out here in this community just like me who have built a successful career thanks to ColdFusion.  We work circles around PHP and .NET and C# and Java developers because of the powerful platform that ColdFusion provides without forcing us to adhere to the lengthy development times associated with other languages or platforms.  We continue to optimistically serve clients who have invested in the software, the staffing, and the time/costs of custom software development despite the fact that they get less support and attention from Adobe than any other time in the history of the product.  I am personally contacted by significantly more of these clients than I have time to assist.  There's still an opportunity and demand for this product, despite the fact that it is given constant backseat placement within the ranks of Adobe's other solution offerings.  If you need a first-hand testimony from a Fortune 200 client who is ready to drop ColdFusion entirely as a result of their headaches with ColdFusion 10 and it's "updates," I'd be happy to arrange a conference call if I thought it would light a fire under this situation and get it resolved immediately, effectively, and assuredly.  ColdFusion is not dead... but Adobe certainly seems to be letting it die... and that's a real shame.

                      • 48. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                        *** Tyson *** Level 1

                        Point of fact: we received 96 additional errors/warnings of this nature on our production server in the time it took me to author my last post.

                        • 49. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                          conanbarb Level 1

                          For those of you still having issues, you may want to double check that the connectors were reconfigured properly.  The instructions in the tech note say to use the wsconfig tool, but no details on how to use it.  My first go around I did not do it properly.  My second go around I hit the "Remove" button, but the "Add" button did not configure everything correctly when I selected "All" IIS websites.  In fact, all it did was create a C:\ColdFusion10\config\wsconfig\2\ folder and then a C:\ColdFusion10\config\wsconfig\3\ folder when I ran it again. 

                           

                          Within the /2/ and /3/ folders I noticed that the isapi_redirect.dll had a different date modified(10/22/12) and file size even though both are version 1.2.32.0, but the /1/ folder still had the "old" file with date modified 3/29/12.

                           

                          I then stumbled on the Update_all_connectors.bat file which "upgraded" the connector automatically for me

                           

                          C:\ColdFusion10/cfusion\bin\connectors>Upgrade_all_connectors.bat

                          WARNING!  This will upgrade ALL ColdFusion MX web server connectors.

                          Make sure ColdFusion MX Application Server and Microsoft IIS are not running before running the upgrade.

                          Press Control+C to abort.

                          Press any key to continue . . .

                          command line: -upgrade -v

                          Created file C:\ColdFusion10\config\wsconfig\cfwin32.dll

                          Created file C:\Users\XXXXXX~1\AppData\Local\Temp\1\\ExecuteAppCmd\ExecuteAppCmd.exe

                          Stopped "World Wide Web Publishing Service" service

                          Created file C:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll

                          Started "World Wide Web Publishing Service" service

                          The Internet Information Server (IIS) connector was upgraded in All

                           

                          C:\ColdFusion10\cfusion\bin\connectors>

                           

                           

                          I don't even know if this process is correct since there aren't any instructions and I'm no Tomcat expert, but that is one place I'd look.

                           

                          "old" isapi_redirect.dll - 388,608 bytes

                          "new" isapi_redirect.dll - 389,120 bytes

                          • 50. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                            carl type3 Level 4

                            >isapi_redirect.dll size

                            Would the file size differ for Windows 32 bit?

                            Regards, Carl.

                            • 51. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                              markm75g

                              Has anyone else confirmed re-running the ugprade_all_connectors.bat file has fixed the original issue at this time?

                              conanbarb wrote:

                               

                              For those of you still having issues, you may want to double check that the connectors were reconfigured properly.  The instructions in the tech note say to use the wsconfig tool, but no details on how to use it.  My first go around I did not do it properly.  My second go around I hit the "Remove" button, but the "Add" button did not configure everything correctly when I selected "All" IIS websites.  In fact, all it did was create a C:\ColdFusion10\config\wsconfig\2\ folder and then a C:\ColdFusion10\config\wsconfig\3\ folder when I ran it again. 

                               

                              Within the /2/ and /3/ folders I noticed that the isapi_redirect.dll had a different date modified(10/22/12) and file size even though both are version 1.2.32.0, but the /1/ folder still had the "old" file with date modified 3/29/12.

                               

                              I then stumbled on the Update_all_connectors.bat file which "upgraded" the connector automatically for me

                               

                              C:\ColdFusion10/cfusion\bin\connectors>Upgrade_all_connectors.bat

                              WARNING!  This will upgrade ALL ColdFusion MX web server connectors.

                              Make sure ColdFusion MX Application Server and Microsoft IIS are not running before running the upgrade.

                              Press Control+C to abort.

                              Press any key to continue . . .

                              command line: -upgrade -v

                              Created file C:\ColdFusion10\config\wsconfig\cfwin32.dll

                              Created file C:\Users\XXXXXX~1\AppData\Local\Temp\1\\ExecuteAppCmd\ExecuteAppCmd.e xe

                              Stopped "World Wide Web Publishing Service" service

                              Created file C:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll

                              Started "World Wide Web Publishing Service" service

                              The Internet Information Server (IIS) connector was upgraded in All

                               

                              C:\ColdFusion10\cfusion\bin\connectors>

                               

                               

                              I don't even know if this process is correct since there aren't any instructions and I'm no Tomcat expert, but that is one place I'd look.

                               

                              "old" isapi_redirect.dll - 388,608 bytes

                              "new" isapi_redirect.dll - 389,120 bytes

                              • 52. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                etnevels1

                                Exact server here. Only have this with CF10 - have installed all updates. What I notice is this only happens on sites that have several bad page requests. For example if the client is new to us and has some links out there to non-existent pages, then that site will crash several times a day after a few of these bad GET page requests are made.

                                 

                                Here is a recent log entry that killed the site:

                                2013-02-19 16:06:41 W3SVC26 FULLPARTNER5 76.12.79.46 GET /wp-login.php - 91.207.7.238 Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.9.1.3)+Gecko/20090824+Firefox/3.5.3+ GTB5 - - 200 0 235 93

                                Notice the page is /wp-login.php,obviously a page from the previous site that does not exist on the new site.

                                Tragic issue - CF10 is a step backwards from CF8 or CF9.  Please Adobe, fix this issue!

                                • 53. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                  GaryFunk

                                  I am running a .gov server farm and we are seeingh the exact same problem on Windows 2006 64 bit machines. We are not seeing the problem on Windows 2003 32 but machines.

                                   

                                  All servers are running CF 10 either on 2003 32 but or 2006 64 bit.

                                   

                                  Here is strange item that stopped one server from crashing.  I removed all the scheduled tasks from the 2008 64 bit server and put them on a 2003 32 bit server and it's been 10 days since the 2008 64 bit machine failed. Before it would fail 2 to 3 times a day.

                                   

                                  Adobe, it would rteally be nice to get an idea on why this is happenng.

                                   

                                  Gary Funk

                                  • 54. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                    etnevels1 Level 1

                                    Gary

                                     

                                    We noticed that there were GET requests for /autodiscover/autodiscover.xml on all the sites that were going down. This page did not exist on the server. So I added the folder and file (it is blank) and that greatly reduced the number of crashes. Basically anytime a site is hit more than 10 or 15 times for GET requests of non-existent pages, the site will crash.

                                     

                                    Ted

                                    • 55. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                      GaryFunk Level 1

                                      A GET request for a non CF page would not cause ColdFusion to fail. That would be the sole property of the web server engine. I don't get many requests or a non-existent page, so I don't think that has much to do with ColdFusion failing.

                                      • 56. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                        conanbarb Level 1

                                        etnevels1 wrote:

                                         

                                        Gary

                                         

                                        We noticed that there were GET requests for /autodiscover/autodiscover.xml on all the sites that were going down. This page did not exist on the server. So I added the folder and file (it is blank) and that greatly reduced the number of crashes. Basically anytime a site is hit more than 10 or 15 times for GET requests of non-existent pages, the site will crash.

                                         

                                        Ted

                                         

                                        Looks like /autodiscover/autodiscover.xml is related to Exchange and not necessarily anything to do with CF

                                         

                                        http://technet.microsoft.com/en-us/library/bb124251(v=exchg.150).aspx

                                        • 57. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                          etnevels1 Level 1

                                          The point is this - CF10 has added the Jakarta connector on every website, it errors out and stalls the site when too many redirects to 404 pages are submitted (which are handled by this connector). These bad page requests can be any type of page, not just cfm. This is the flaw in CF10 that I am struggling with. Until I created a blank folder/file to stop the 404 error, these sites were going down several times a day. The error was always a jakarta redirect DLL failure. Now that I have added this blank folder the sites go down 4-5 times a week and always the error is a jakart redirect DLL failure. I also inspected logs and added a few other blank folder/pages to popular non-existent pages and that helped to reduce the failures.

                                           

                                          Ted

                                          • 58. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                            GaryFunk Level 1

                                            Interesitng, but that still does not solve the problem. 32 bit does not fail and 64 but does.

                                             

                                            Can 32 bit CF 10 be installed on Windows 64 bit?

                                            • 59. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                              conanbarb Level 1

                                              etnevels1, did you also upgrade all of the connectors(as administrator) along with the updates?  I don't want to just assume that you did.  Without knowing more about your environment, it is hard to tell what is going on.  Can you post the actual Jakarta error you are getting and also the size of your Jakarta DLL?

                                              • 60. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                etnevels1 Level 1

                                                One of the sites has Jakarta mapped to C:\ColdFusion10\config\wsconfig\24 and within that folder is isapi_redirect.dll which is 389,120 bytes

                                                 

                                                I do not have the exact error - need to have a site go down to see it - but it is an error that isapi_redirect.dll cannot start

                                                 

                                                Next time I have a site go down I will record the exact error phrase

                                                • 61. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                  carl type3 Level 4

                                                  @etnevels1 CF10\config\wsconfig\24\isapi_redirect.log have any errors at the time or leading up to system failure?

                                                  • 62. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                    Anit_Kumar Adobe Employee

                                                    Yes, 32-bit CF can be installed on Windows 64-bit, provided the bitness of Webserver is also 32-bit.

                                                     

                                                    Regards,

                                                    Anit Kumar

                                                    • 63. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                      Charlie Arehart Adobe Community Professional & MVP

                                                      @etnevels1, note that one of the many things that CF10 update 11 addressed was an issue related to when CF was processing 404s for IIS. I know your note here was from March, but someone pointed me to this thread today, so I wanted to offer that if you'd not tried it.

                                                       

                                                      And if it does not work, can you please confirm that the date on the isapi_redirect.dll in your [cf10]\config\wsconfig\[n]\ folder(s) is from May 2013? If it's from March 2012, that too is a severe cause of problems. CF10 update 5 fixed that (for IIS and Apache), but anyone who applied that or a later update then NEEDS to rebuild the web server connectors (run the CF web server config tool, and remove and re-add all connections). And don't forget to run that tool by right-clicking and choosing "run as administrator".

                                                       

                                                      To you or others with issues in this thread, please to clarify if you've updated to the latest build (11, as I write) and (more important) checked the date of that isapi_redirect.dll (in ALL folders under wsconfig), before reporting any seeming ongoing issues?

                                                       

                                                      Either really could be the simple solution to so many problems report with CF10 (on IIS and Apache). Hope that's helpful.

                                                       

                                                      Message was edited by: Charlie Arehart: changed my mistaken reference of aspnet_isapi.dll to isapi_redirect.dll.

                                                      1 person found this helpful
                                                      • 64. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                        vbrouss181250

                                                        side issue carl;

                                                        can you point me in the right direction?

                                                        "

                                                        I have just installed ColdFusion 10 on a development windows 2008 box. when I try to connect to our gateway application on the server I get the following error:

                                                        Error Type : coldfusion.runtime.CfErrorWrapper
                                                        Error Content : Object Instantiation Exception. Class not found: com.sap.mw.jco.JCO
                                                        The error occurred on line 121.

                                                        There is no information on line 121 or near there that would show us this error.
                                                        Adobe tech support has me looking for the following class paths on the server:
                                                        com.sap.mw.jco.JCO
                                                        com.sap.mw.jco.JCO$Client

                                                        but they cannot be found. But I also cannot locate those specific paths on our working CF8 or 7 servers. Is there a method to add these to my ColdFusion class paths?
                                                        thanks"

                                                        • 65. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                          Charlie Arehart Adobe Community Professional & MVP

                                                          As a correction to my last note, where I referenced the file to check (for updates) as aspnet_isapi.dll, I meant instead isapi_redirect.dll. (There is an aspnet_isapi.dll, but that's a file within the .NET framework locations, and I just confused the two in my mind when I wrote that note.) I have edited the messsage to make the proper reference (which updated the edit date for the message.)

                                                          • 66. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                            nealb

                                                            Charlie -

                                                             

                                                            <<note that one of the many things that CF10 update 11 addressed was an issue related to when CF was processing 404s for IIS>>

                                                             

                                                            Are we all crazy?

                                                             

                                                            From the bug base for the 404 handler issue, https://bugbase.adobe.com/index.cfm?event=bug&id=3488063, multiple people have installed Update 11, have confirmed that isapi_redirect.dll has the latest update date, have rebuilt the web connector, and still are getting random disconnects - at least 50% of the time.  Meanwhile, CF9 has no connection problems - ever.

                                                             

                                                            This is incredibly easy to reproduce, is very critical and is preventing many folks from switching to CF10, but yet has been broken for over 6 months, with no fix in sight.

                                                             

                                                            Is the fix really there, but we're just not figuring it out?

                                                             

                                                            Please set me straight.  And if it's really still a problem, please push Adobe to get this fixed.

                                                             

                                                            Thanks,

                                                            Neal

                                                            • 67. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                              DCwebGuy Level 1

                                                              I also work for a large enterprise that has spent $10,000's on ColdFusion over the years, and I am beginning to get the same dreaded feeling as many here that the operational aspects (not "features") of CF10 are a real disaster compared to 9 or any of the previous versions.

                                                               

                                                              We also recently upgraded from CF9 on 32-bit Windows 2003/IIS6 to CF10 on 64-bit Windows 2008/IIS7.5, and despite having boatloads of extra capacity now hardware-wise, and despite carefully reseaching and implementing all the connector "tips" and recommendations and having the latest CF10 Update 11 installed, etc., I am now seeing more and more intermittent 503 issues on our production machine that have me very worried.

                                                               

                                                              Add me to the list of people thinking about reverting back to CF9, which was rock solid as far as I'm concerned.  And although this one is the most critical issue I'm facing right now, there are plenty of other buggy things in CF10 that have me dismayed, but that is for other topics (you can see my separate posts about these other items).

                                                               

                                                              As roland.collins points out in post #41 above, there seems to be an inherent flaw that allows anyone anywhere to launch a DoS with guaranteed results.  He points out the XML flaw (Bug 3222748) but I am beginning to think it also has to do with the connector having a static pool size. Why did CF9 not have a pool size I had to monkey with, and how do I even know what the correct pool size should be?  Of course you are going to say "well, stress test your system you fool!" but I never, ever had to do this in the past, and never had any problems.  Now I feel like I'm just guessing what my connector settings should be.  Last time I had an issue, I doubled my connector pool sizes from the recommendations set forth in Adobe's tuning blog, but this seemed to make no difference (http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connector-configuration).

                                                               

                                                              I am going to go out on the limb here and suggest that most, if not all, of the unpredictable issues surrounding these server crash/503/unavailable messages has to do with these connector settings.  There is such a incredible plethora of spiders and bots crawling and scanning our sites (the collective "we") that it's no wonder the connections will sometimes spike and surpass the Tomcat connector limits, which I'm guessing acts like a circuit breaker preventing anything else from happening regardless if there is the server capacity (CPU/RAM) to back it up.  Correct me if I'm wrong about the circuit breaker analogy.

                                                               

                                                              So how were previous versions of ColdFusion handling this?  I am genuinely interested in the technical aspects here, in hopes there is something I can do short of spending a ton of time reverting all my servers back to CF9.  I have searched hell and high water through all my logs and settings and can't find a single clue, and I've been doing this (ColdFusion development on Win/IIS systems) for 16 years straight.

                                                               

                                                              Personally, I am adding my vote to this bug, as it decribes exactly my problem, and I see that others are still having this besides me even after all the connector tuning business: https://bugbase.adobe.com/index.cfm?event=bug&id=3318104

                                                               

                                                              Thanks everyone for listening and helping out if you can.  A few more technical details of my production setup in case it helps:

                                                               

                                                              • Win2008 R2 running on VMware
                                                              • Dual AMD CPUs
                                                              • 8Gb RAM
                                                              • JMV set to 2048 (min) and 4096 (max).  Usually hovers around 800mb used.
                                                              • Running Java version 1.7.0_15 (default install), single server, connector configured for all IIS web sites
                                                              • ISAPI_redirect.dll timestamp: May 23, 2013
                                                              • Tomcat worker.cfusion.max_reuse_connections=500, worker.cfusion.connection_pool_size=1000, timeout=60
                                                              • CFAdmin: Max queries to cache set at 30,000.  Usually hovers around 22k.
                                                              • CFAdmin: Max templates set to 2,000, usually hovers around 700.
                                                              • Server monitoring enabled, but not as a separate server on port #
                                                              • Profiling and Memory Tracking sometimes enabled, sometimes not, does not seem to make a difference
                                                              • Windows Perfmon shows "web service current connections" usually in the 200-300 range.  Max can be anywhere.
                                                              • CFAdmin: Max number of simultaneous template requests: 85
                                                              • Everything else is pretty standard default setup. 
                                                              • No scheduled tasks are running during the 503 events.  Logs files don't indicate anything unusual.
                                                              • 68. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                carl type3 Level 4

                                                                >Tomcat worker.cfusion.max_reuse_connections=500, worker.cfusion.connection_pool_size=1000, timeout=60


                                                                Some tomcat references suggest timeout=600 ie 10 minutes rather than 1 minute.


                                                                Like java the minimum values can just as important as maximum. Seems you do not specify a minimum tomcat value. You could try set this EG: worker.cfusion.connection_pool_minsize = 100.


                                                                Have you adjusted the server end of tomcat connector to have corresponding maximum minimum and timeout values for AJP section  in ColdFusion10\cfusion\runtime\conf\server.xml? EG:
                                                                <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="1000" minSpareThreads="100" connectionTimeout="600000" />

                                                                 

                                                                >JMV set to 2048 (min) and 4096 (max).  Usually hovers around 800mb used


                                                                Is Permanent Generation memory use the issue? If so then CF will fail. On 64 bit MaxPermSize=192m can be inadequate.


                                                                Suggest perform some JVM logging to know how well the heap and non heap parts of the "800Mb used" are functioning.

                                                                HTH, Carl.

                                                                • 69. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                  DCwebGuy Level 1

                                                                  Thanks, Carl.  Your are a very dependable helper on this forum!  To answer your questions:

                                                                   

                                                                  1) Can you provide the reference to set Tomcat timeout to 600 instead of 60?  Wonder why Adobe would be off by a factor.

                                                                  2) I have never seen advice to set a minimum pool size, so that is a good tip, I will have to try it out if problems persist.

                                                                  3) Yes, I had/have configured both the workers.properties and server.xml per Adobe article.

                                                                  4) I use -XX:MaxPermSize=356m.  I do not have a regular "PermSize" set, however.  Does that matter?  I really don't think my issue (or any of these 503 issues) are JVM related honestly.

                                                                   

                                                                  I don't know how to perform JVM logging exactly.  Can you provide advice?

                                                                   

                                                                  Finally, tonight I did two things so I will have to see if these help or have any effect:

                                                                   

                                                                  1) Patched from Update 11 to Update 12.  Re-ran the connector using wsconfig tool per Adobe's instructions and re-updated all the connector params in workers.properties and server.xml, and reconfigured all my IIS virtual dirs (PITA!) as per usual.  Noticed isapi_redirect.dll is now timestamped at 11/2/2013, which is encouraging.  Used to be 5/23/2013.

                                                                   

                                                                  2) Updated the default JRE from Java 1.7.0_15 to 1.7.0_45.  I installed the "Server JRE" which includes tools for JVM monitoring so maybe that will come in handy.  Here are Ben's instructions, for replacing the Java engine, which worked great.  I put the jdk1.7.0_45 under the default c:/ColdFusion10 directory on Windows and simply pointed the JVM to it (you can put it anywhere).

                                                                   

                                                                  OBSERVATIONS:

                                                                   

                                                                  I had absolutely no trouble installing either of the above.  My only advice is to wait a minute between the CFAdmin's ColdFusion updater and the manual removing and adding of your sites using the wsconfig tool.  Things seem to take a little longer since more memory has to be released and re-allocated (keeping Task Manager open during this process helps you see what's going on).

                                                                   

                                                                  Another thing I noticed is that upon rebooting or manually stopping and restarting the CF service, I get a constant 503 error while ColdFusion allocates the minimum JVM heap.  I would think allocating RAM for itself would be pretty instantaneous but it's NOT.  On my old server with 512mb min heap I simply never noticed this.  Now that min heap is 2048mb it takes longer for CF to start up.

                                                                   

                                                                  Coincidentally, this also creates a problem when trying to restart the service via Windows control panel...it will usually timeout, and not gracefully.  Furthermore, If you try to use CF's built in cfstop/cfstart.bat files the service will start under the username you are currently logged into your server with, and if that is not the "system" then CF will stop the second you log off, as is frequently the case if you are using Remote Desktop.  So for me the only way to reliably stop and restart CF is to do a full reboot.  Not great for production but what choice to do I have?

                                                                   

                                                                  I performed this update on all 3 of my machines (prod/dev/staging) and they are all identical, and all behaved exactly the same.

                                                                   

                                                                  For now, attached are my current "core" settings.  I will report back if any of this makes a difference.  Since the CF update 12 did not make any specific reference to resolving stability problems, I am not all that hopeful.  But maybe, just maybe, updating the JRE helps.

                                                                   

                                                                  Now I'm going to finish my bottle of wine and watch some Top Gear to get my mind of all this B.S.!  Good night!

                                                                   

                                                                  Capture.JPG

                                                                  • 70. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                    carl type3 Level 4

                                                                    >Can you provide the reference to set Tomcat timeout to 600 instead of 60?

                                                                     

                                                                    Refer:

                                                                     

                                                                    http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html

                                                                     

                                                                    Parts say:

                                                                     

                                                                    Do not set timeouts to extreme values. Very small timeouts will likely be counterproductive.

                                                                     

                                                                    We generally recommend values around 10 minutes, so setting connection_pool_timeout to 600 (seconds). If you use this attribute, please also set the attribute connectionTimeout in the AJP Connector element of your Tomcat server.xml configuration file to an analogous value. Caution: connectionTimeout is in milliseconds. So if you set JK connection_pool_timeout to 600, you should set Tomcat connectionTimeout to 600000.

                                                                     

                                                                    >I have never seen advice to set a minimum pool size, so that is a good tip, I will have to try it out if problems persist

                                                                     

                                                                    Worth trying since you can appreciate the default value of 10 can be under specified for some load.

                                                                     

                                                                    >I don't know how to perform JVM logging exactly.  Can you provide advice?

                                                                     

                                                                    With MaxPermSize set to 356Mb I would not bother for now spending time with JVM logging as you say Java is probably not the issue and tho it maybe tuned for some  performance gains you could end up consuming time and not be treating the immediate problem. If tho there more reason to be concerned with Java then sure I can get you information to understand JVM logging.

                                                                     

                                                                    HTH again, Carl.

                                                                    • 71. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                      DCwebGuy Level 1

                                                                      Thanks again, Carl.  I read the timeouts link you provided and noticed it says:

                                                                       

                                                                      "The JK attribute connection_pool_minsize defines, how many idle connections remain when the pool gets shrunken. By default this is half of the maximum pool size."

                                                                       

                                                                      So it would appear I do not need to set a min, as the min is automatically set at 1/2 the max.  Does this sound right to you?  So my min would be 500?

                                                                       

                                                                      I am going to be looking at JMeter and ways to stress test my system so I know what these numbers should actually be.  In reading further, I see that it says "For Apache httpd we automatically detect the number of threads per process and set the maximum pool size to this value."  I guess if one were running CF10 on Apache instead of IIS then the pools are automatically sized, which was one of my questions.  It sucks this is not the case for IIS and we have to guess or stress test to find out what our real limits are as per the instructions:

                                                                       

                                                                      "We strongly recommend adjusting this value for IIS and the Sun Web Server to the number of requests one web server process should be able to send to a backend in parallel. You should measure how many connections you need during peak hours without performance problems, and then add some percentage depending on your growth rate etc. Finally you should check, whether your web server processes are able to use at least as many threads, as you configured as the pool size."

                                                                       

                                                                      I know, I know, once in a while I really should RTFM.

                                                                       

                                                                       

                                                                      UPDATE:  Sadly, I got another random 503 this morning for no reason whatsoever that I can tell.  It goes back to normal after a few mins, but this is really frustrating and unacceptable.  I don't know what else to look at, but I will carry on somehow.  Probably contact Adobe next, then revert to CF9 if problems continue.

                                                                      • 72. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                        carl type3 Level 4

                                                                        >So it would appear I do not need to set a min, as the min is automatically set at 1/2 the max.  Does this sound right to you?  So my min would be 500?

                                                                         

                                                                        Via JMX and Jconsole monitoring I find that not to be the case.  You would have to set connection_pool_minsize=500 and minSpareThreads="500" for that to happen.

                                                                         

                                                                        >UPDATE:  Sadly, I got another random 503 this morning for no reason whatsoever that I can tell.

                                                                         

                                                                        Don't give up with a bit of monitoring and some minimum settings you are probably close to a stable system.

                                                                         

                                                                        Regards, Carl.

                                                                        • 73. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                          Charlie Arehart Adobe Community Professional & MVP

                                                                          Hey Neal (going back to discussion in this thread from a few months ago), on your issue with IIS 404 handling passing to CF10, have you seen the recent discussion on that in another thread in this forum? There’s a new suggested solution from Ben Holland, which is working for many folks. I point out and discuss more about his suggestion in a blog entry I posted last night:

                                                                          http://www.carehart.org/blog/client/index.cfm/2014/3/12/Interesting_solution_to_CF10_IIS_4 04_handlers_problem


                                                                          Just thought it may help you of some others in this thread.

                                                                          /charlie

                                                                          • 74. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                            pdrich

                                                                            We are continuing to have this problem in CF10 with the latest updates as of May 2014. We have tried most of the workarounds, some of them potentially destructive.  We restart CF nightly to try to mitigate the issue but still encounter it. We just upgraded to 10 but might do another upgrade to 11 if this issue isn't resolved. We have separate IIS application pools, and only some of them go down this way, but CF has to be restarted for the box, taking down all sites. Unfortunately, CF doesn't stop properly and takes about 5 minutes to restart.  Our next step is to try the IIS rewrite workaround mentioned by Charlie, but as this problem has been around for 2 years, we shouldn't have to spend so much time working around it.  Is Adobe still working on the issue?

                                                                            • 75. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                              Priyank Shrivastava. Adobe Employee

                                                                              Hi,


                                                                              Can you please contact us at cf.install@adobe.com

                                                                               

                                                                              Also did you try the settings which is mentioned in this blog?

                                                                               

                                                                              http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connector-configuration

                                                                               

                                                                              Thanks,

                                                                              Priyank

                                                                              • 76. Re: CF 10 continues to crash on Windows 2008 R2 64 bit
                                                                                Duffy

                                                                                ,

                                                                                 

                                                                                I think the issue you have described is similar to an issue I had to resolve with Coldfusion 10. When I use a custom 404 page my content was randomly getting truncated or not displaying. What I soon realized is that the content length was not getting passed back to IIS from tomcat and this was causing my page to get truncated or not show anything. To resolve this issue I added a little hack to the custom 404 page. I captured all content that was going to be displayed on the page as a variable, I then check the length of the content and use a cfheader tag to force the content length within the browser. This simple fix seems to have resolved my issues.

                                                                                 

                                                                                I hope this helps!

                                                                                1 2 Previous Next