Skip navigation
Currently Being Moderated

CF 10 continues to crash on Windows 2008 R2 64 bit

Jun 2, 2012 12:06 PM

I have had CF 10 running on my VM for about a week now. The VM is Windows 2008 Enterprise, 64 bit. CF 10 is 64 bit.  The error that I keep getting is:

 

Service Temporary Unavailable!

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.





Jakarta/ISAPI/isapi_redirector/1.2.32 ()

To recover, I need to stop and restart all the CF services.

Anyone else having this problem or has anyone else had to fix it?

 
Replies 1 2 Previous Next
  • Currently Being Moderated
    Jun 3, 2012 3:34 PM   in reply to BoReahard

    Any errors and warnings in log files?

     

    CFadmin > Debugging & Logging > Log Files > coldfusion-error.log & coldfusion-out.log

     

    Could also enable tomcat metrics logging:

     

    CFadmin > Debugging & Logging > Debug Output Settings > Enable Metrics Logging

     

    then CFadmin > Debugging & Logging > Log Files > metrics.log

     

    You could also read those logs in the file system ColdFusion10\cfusion\logs

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Jun 4, 2012 4:25 PM   in reply to BoReahard

    Of interest here are some CF10 metrics logging I took as a sample to compare.

    05/06/2012

    8:49 AM

    Information

    scheduler-1

    Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 378536 Request count: 201 Error count: 4 Bytes received: 780 Bytes sent: 6993022 Free memory: 45875240 Total memory: 175702016 Active Sessions: 0

    05/06/2012

    8:48 AM

    Information

    scheduler-2

    Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 378536 Request count: 201 Error count: 4 Bytes received: 780 Bytes sent: 6993022 Free memory: 47207152 Total memory: 175702016 Active Sessions: 0

    05/06/2012

    8:47 AM

    Information

    scheduler-0

    Max threads: 150 Current thread count: 4 Current thread busy: 1 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 52128072 Total memory: 175898624 Active Sessions: 0

    05/06/2012

    8:46 AM

    Information

    scheduler-1

    Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 59819016 Total memory: 167116800 Active Sessions: 0

    05/06/2012

    8:45 AM

    Information

    scheduler-1

    Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 61154744 Total memory: 167116800 Active Sessions: 0

    05/06/2012

    8:44 AM

    Information

    scheduler-1

    Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 62484720 Total memory: 167116800 Active Sessions: 0

    05/06/2012

    8:43 AM

    Information

    scheduler-2

    Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 63821448 Total memory: 167116800 Active Sessions: 0

    05/06/2012

    8:42 AM

    Information

    scheduler-2

    Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 65154032 Total memory: 167116800 Active Sessions: 0

    05/06/2012

    8:41 AM

    Information

    scheduler-1

    Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 66487544 Total memory: 167116800 Active Sessions: 0

    05/06/2012

    8:40 AM

    Information

    scheduler-0

    Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 67824496 Total memory: 167116800 Active Sessions: 0

    The online CF10 documentation omits CF10 metrics logging details so I am not sure what to make of the log detail differing. Reference:

    http://help.adobe.com/en_US/ColdFusion/10.0/Admin/WSc3ff6d0ea778594611 72e0811cbf3638e6-7fe0.html#WSc3ff6d0ea77859461172e0811cbf3638e6-7fda

    I don't think the logs have been any use so far. Still interested to know why CF10 is crashing for you. Something else comes to mind. Is it possible for you to run CF10 from CMD prompt for a time, ColdFusion10\cfusion\bin\cfstart.bat ? Perhaps there is an error and having the details displayed in CMD prompt will lead to solution.

    

    HTH again, Carl.

     

    
     
    |
    Mark as:
  • Currently Being Moderated
    Jun 4, 2012 4:44 PM   in reply to BoReahard

    Stop ColdFusion 10 Application Server service

    CMD prompt (as administrator?)

    CD ColdFusion10\cfusion\bin

    cfstart.bat

    Watch wait report back any interesting details

     

    
     
    |
    Mark as:
  • Currently Being Moderated
    Jun 7, 2012 12:10 AM   in reply to carl type3

    Here is a link to some detail on CF10 metric logging:

    http://helpx.adobe.com/coldfusion/kb/coldfusion10-enable-metric-loggin g.html

    
     
    |
    Mark as:
  • Currently Being Moderated
    Jun 7, 2012 2:56 PM   in reply to BoReahard

    Odd the CF10 metric log only showing memory values. Do not know what to make of that. Any feedback on running CF10 from CMD prompt?

     
    |
    Mark as:
  • Currently Being Moderated
    Jun 7, 2012 4:33 PM   in reply to BoReahard

    Please post back findings I am sure is of interest to other CF10 users.

    There are other things can do eg CFSTAT, JVM logging, however not having a clear idea of where to look I do not want to send you down a path that is going to waste your time.

     

    
     
    |
    Mark as:
  • Currently Being Moderated
    Jul 5, 2012 9:06 PM   in reply to BoReahard

    I cannot even get it to accept my admin password. After resets, etc. Windows 2008 Web R2 this is such a bummer.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 13, 2012 12:29 PM   in reply to BoReahard

    One option you have when receiving those errors is to change your IIS connector to something like the Boncode IIS to Tomcat connector (which is currently used by Railo's Windows installer). From what I can tell, the Adobe connector just can't handle high load and gives out at a certain (undetermined) threshold.

     

    Bilal (the main developer of the project) has been very helpful with making the Boncode connector very compatible with CF 10, and I've seen good results from it so far. Here's the link to the project: http://tomcatiis.riaforge.org/

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 13, 2012 10:28 PM   in reply to JakeHand

    I have tried the boncode connector since I am having http://forums.adobe.com/message/4619002#4619002, and I have tried different configuration setting for the connector, but the page always has issue with displaying. I will have to keep refreshing the page, until the contents have been showed correctly and in full. It not only happens to my application, but also the cfadmin page. The header part of the admin page sometimes shown as blank. In order to make it display correctly, I will need to refresh the page couple of times.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 25, 2012 2:34 PM   in reply to JakeHand

    We have this happen about once every 2-3 weeks on a site with a few thousand active users. Moreso when we are undergoing are quarterly PCI tests and we get blitzed by SecuritMetrics' automated tests.

     

    I'd prefer not to monkey around with a 3rd-party connector. Is Adobe aware of this?

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 25, 2012 3:56 PM   in reply to Aquitaine

    I have just given up on it at this point. Feeling like a wasted purchase. I have reverted to using Railo. I hope their is some response from Adobe.

     

    Aaron

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 26, 2012 5:26 PM   in reply to Aquitaine

    We are on the same boat, my server randomally crashes when there are few thounsands active users as well.

    I did spend some time on a alternative connecter myself, but I couldn't get it to work perfectly. And I do agree with you that we shouldn't play around with some 3rd party thing, since the cf10 is using some customized version of tomcat connector and there are lots of things we are not sure at this stage

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 26, 2012 8:13 PM   in reply to Ryan Luo
     
    |
    Mark as:
  • Currently Being Moderated
    Aug 26, 2012 11:14 PM   in reply to Aquitaine

    We've been looking into this further and have found that tuning the connector timeout and the worker.properties can really help with this issue. I'm still a big fan of the BonCode connector though, considering how fast Bilal's turnaround time on adapting to CF 10 has been (and that it does not run as a funky ISAPI redirect filter). With that said, you may want to open up your server.xml in "cfusion\runtime\conf" and find the section for the AJP connector (~line 100), and set the maxThreads, connectionTimeout, and acceptCount attributes. Should look similar to this:

     

    <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="300" connectionTimeout="60000"  acceptCount="50" />

     

    Those attributes help prevent Tomcat from spooling up too many or not enough threads for your application. The acceptCount also queues things up temporarily if you reach the max number of threads.

     

    Now you'll want to edit workers.properties at "C:\ColdFusion10\config\wsconfig\1".

     

    By default Adobe does not have any connection pool limitations, so things can get out of hand when an application gets busy. If your server hosts a lot of IIS sites, you may want to consider adding these lines to your workers.properties file:

     

    worker.cfusion.connection_pool_size=10

    worker.cfusion.connection_pool_minsize=1

    worker.cfusion.connection_pool_timeout=60

     

    More info on this here: http://tomcat.apache.org/connectors-doc/reference/workers.html

     

    Those adjustments have helped tame this issue for sites that need to use the Adobe connector. Your mileage will vary, and you'll want to tweak those settings to meet your environment, but good luck!

     

    Also, Ryan, the issue you mentioned with the connector is very odd. Was it perhaps set as a Wildcard and trying to process every request, even the static files? If you do feel like continuing with the connector though, you may want to upgrade to the latest version. I know Bilal's fixed a few bugs in the last month.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 28, 2012 11:09 PM   in reply to JakeHand

    Thank you for your reply JakeHand

    I would like to try the alternative connector in my VM, but not production at the moment, but can you teach me how to correctly setup the Boncode?

    I tried different configuration myself (none of them uses wildcat), but just doesn't work for me.

     

    Regards

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 8:18 AM   in reply to BoReahard

    We are experiencing a similar issue.  We have CF10 running on VM (32 bit Windows Server 2008).  We've been live almost a week and the issue just started last night.  We saw the message "Service Temporarily Unavailable" and "Service Unavailable - Zero size Object."  We restarted the server multiple times. It would work initialy after the restart but would then fail again, usually within 10 minutes.  Looking at the IIS log, we did have a lot of hits from bots so I'm guessing the increased traffic made the issue occur.  We have moved back to CF9 to prevent the issue and probably have to place a support call with Adobe.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 11:14 AM   in reply to lg84

    Everyone should "vote" for the bug that was entered for this.  That will help show Adobe that more than one user is experiencing the issue.

     

    https://bugbase.adobe.com/index.cfm?event=bug&id=3318104

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 29, 2012 9:23 PM   in reply to Miguel-F

    Is this issue similar to

    https://bugbase.adobe.com/index.cfm?event=bug&id=3222748?

     

    We are investigating the above issue.

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 30, 2012 4:54 AM   in reply to hemant_k

    No, I don't think so.

     

    The connector breaks in both cases but we don't have any xml being submitted in our app.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 4, 2012 6:15 AM   in reply to Aquitaine

    There is a new Coldfusion 10 update. It mentions a bug fix for Connector. Has anyone installed this and has it fixed the issue?

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 7, 2012 9:51 AM   in reply to lg84

    We applied the hotfix to our CF 10 servers earlier this week, and so far our uptime has drastically improved their uptime (and we haven't been seeing the Service Unavailable errors).

     

    Keep in mind though, we also added the tuning parameters to our server.xml files as mentioned above, and specifying an appropriate maxThreads is important. So far we've gotten away with increasing that value to 1500, but our hardware is pretty hefty so you want to make sure the value is tuned for your environement. Also, we ended up removing the acceptCount and instead added a  keepAliveTimeout to help make sure client connections aren't prematurely timed out. By default this is set to the same value as connectionTimeout, and we wanted a much higher keepAliveTimeout than connectionTimeout. This page has a description of all the attributes you can add to the AJP connector in your ColdFusion (Tomcat) server.xml file.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 7, 2012 12:45 PM   in reply to JakeHand

    The alternate connector worked for me.

    I used directions in this blog:

    http://boncode.blogspot.com/2012/06/cf-coldfusion-10-experimenting-wit h.html

     

    The trick for me was to add the folders in section d) to IIS site and assign wildcard mappings to each folder.

    Here is the setting file I used:

     

    <!-- CF10 connection -->
    <Settings>
      <Port>8012</Port>
      <Server>localhost</Server>  
      <MaxConnections>200</MaxConnections>
      <FlushThreshold>0</FlushThreshold>
      <EnableAdobeMode>True</EnableAdobeMode>
    </Settings>

     

    
     
    |
    Mark as:
  • Currently Being Moderated
    Sep 18, 2012 4:30 PM   in reply to BoReahard

    This error has led to our production CF10 rollout being an absolute disaster.  The error never occurred once during our development and staging cycles, and a load tester has been unable to reproduce the issue in either environment, so we can't tell if it's due to high load, memry leaks, etc.  Our production servers have been a mess, and our clients are more than a bit angry.

     

    Our application event log is littered with these:

     

    Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

    Faulting module name: isapi_redirect.dll, version: 1.2.32.0, time stamp: 0x4f745a78

    Exception code: 0xc0000005

    Fault offset: 0x0000000000018b93

    Faulting process id: 0x11f0

    Faulting application start time: 0x01cd950afc876c76

    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

    Faulting module path: C:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll

    Report Id: 98760e12-00ff-11e2-9e13-00155d015102

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 17, 2012 7:32 PM   in reply to roland.collins

    To future readers of this thread, updater 3 for CF10 is supposed to address some of these very problems, so Adobe is indeed on the case.

     

    Unfortunately, many people have found there to be new, different problems introduced by it (most importantly, loss of scheduled tasks, both on restart of CF after applying the update, and then also on any subsequent updates.) Adobe has also acknowledged that and are working on a correction.

     

    You can read the discussion of people having problems (and Adobe's reply) at http://blogs.coldfusion.com/post.cfm/coldfusion-10-update-3-released.

     

    And the technote for the update is at http://helpx.adobe.com/coldfusion/kb/coldfusion-10-update-3.html. Perhaps tomorrow or in coming days it may be updated if they fix whatever was wrong that was introduced in the updater itself.

     

    But again, as for the issues in this thread, hopefully there will be resolution soon when this updater 3 is sorted out.

     

    Hope that's helpful to someone here.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 31, 2012 7:03 AM   in reply to roland.collins

    Same exact issue for me with one of my biggest clients.  Constant crashes on our ColdFusion 10 server multiple times per day.  Event viewer is riddled with constant errors (thousands per day) from the w3wp.exe service and the ISAPI connector.  I was able to seemingly relieve the situation somewhat by disabling IIS 7.5's "Rapid-Fail Protection" on the application pools (which I can't imagine is advisable for a production environment), but that only seems to have decreased the frequency of the crashes slightly - from a dozen times each day, down to only a few times per day.  Completely unacceptable.

     

    It absolutely astounds me that Adobe has not halted in their tracks and prioritized every single reputatble development resource with developing and releasing an effective resolution to this problem.  Honestly, it gives me cause (for the first time in 17 years of ColdFusion development and support) to consider migrating away from ColdFusion to a different platform.  Updater 3 was apparently a disaster (thank God I checked for community feedback before installing it), but I've not heard a single follow up about the fix to this problem since the Updater 3 was recalled by Adobe - and that was a couple weeks ago.

     

    How on earth can an absolutely critical bug like this remain open and unaddressed for so long without any resolution!?  Adobe, you've -completely- dropped the ball on this and disappointed droves upon droves of loyal customers, developers, and community members.  Our clients are angry.  Not only are they losing money and credibility every single day that this bug remains unfixed, but their confidence and support of ColdFusion as a platform is dwindling.  What in the world is holding up the completion and release of a prioritized hot fix for this issue!?  Forget about bundling it up with a bunch of other fixes at this point.  This bug is critical enough to stand on its own and be addressed immediately!

     

    Unbelievable.

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 31, 2012 9:47 AM   in reply to *** Tyson ***

    Hi,

     

    We will be most likely releasing update 4 (after resolving issues with update 3) by end of this week. We are done with our internal testing and private testing from other community members and this process took us some time. Thursday is a holiday and hence most likely date will be Friday if last sanity check are all ok.

     

    Thanks for your support and feedback on this issue.

     

    Hemant

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 31, 2012 10:41 AM   in reply to hemant_k

    Wow.  Did you really just say that you guys are going to take a day off and wait to release ths due to a holiday?  And then you only have a "likely" release date?

     

    Do you have any idea how much time, money, and goodwill this bug has cost you already?

     

    Do you have any idea how much pain this has caused your user base?

     

    Do you have any idea what customer service is?

     

    You have clearly demonstrated that you just don't care about your users and customers any more.  This product has taken a swift downward decline in the last two years, and it is now time for everyone to start looking at alternatives.

    
     
    |
    Mark as:
  • Currently Being Moderated
    Oct 31, 2012 12:09 PM   in reply to roland.collins

    Hi Roland,

     

    Thanks for your feedback. We certainly understand the pain that this issue has caused and want to be careful before we release the patch. I realize that holiday reference was unnecessary and misleading and sorry about that. We have shared this patch to many community members early last week, by quickly trying to fix all issues, since we are committed to fix and release the update quickly. As I mentioned ealrier, we have one last issue being discussed with one of the members and that we are expecting to close everything on Friday.

     

    If you need the private fix and help us verify the same, please let me know at hkhandel a t adobe . com

     

    Thanks,

    Hemant

     
    |
    Mark as:
  • Currently Being Moderated
    Nov 1, 2012 12:48 PM   in reply to roland.collins

    roland.collins wrote:

     

    Do you have any idea how much time, money, and goodwill this bug has cost you already?

     

    Do you have any idea how much pain this has caused your user base?

     

     

    Roland: steady on mate.

     

    I agree that the bug in updater 3 which cleared out scheduled tasks was a bit annoying, and there's no excuse for not catching that during testing.  I understand there were other bugs with it too (I dunno what though... do you?).

     

    But so you backed-out the updater, restored your tasks (from the back-up of your config you made before upgrading, right?), and you were back to updater 2. And that shoulda taken about 5min.

     

    Updater 2 is stable.  What of the bug fixes that updater 3 was going to deliver are sooooo important to you that wait a coupla weeks is that major a problem? Serious question.  Were you clamouring for one of those bugs it fixed to be fixed as it was causing you problems in some way?  In some way you can't just hold on for a bit before it gets fixed and tested?

     

    Look, I'm not defending Adobe (anyone who knows me will point out that I'm not exactly their favourite person): it's taken an age to get this bug in the updater fixed, and I do question why. And that it went out the door with such an obvious bug in it is a bit of a let down. But they're fixing it: cool.  They're testing it: cool. There's clearly a public holiday in India today (it's Thurs as I type), and it's not like the problem with the updater is really the end of the world.  Bear in mind that the bods in the Adobe ColdFusion team are people, and they have families and plans and stuff to do.  Seriously, if you have a non-urgent bug to fix (and this is a non-urgent bug, in the bigger scheme of things), do you cancel your weekend or your bank holiday plans just to fix it?  Well bravo if you do, but I think the bulk of us would just resolve ourselves to the fact that it'll still be there on Monday, and that's OK.  That's all Hemant said they were doing.

     

    Let's just hope they get it out the door tomorrow. Although if I had production boxes waiting to be patched... I'd not be doing it on a Fri, to be honest. Well: I'd be testing this thing on a test box first, TBH, well before thinking of putting it live.

     

    --

    Adam

     
    |
    Mark as:
  • Currently Being Moderated
    Nov 1, 2012 1:57 PM   in reply to Adam Cameron.

    Well, I’ll offer a thought to defend at least some of the angst--though I do think there can often be an overreaction by some as to what Adobe does or doesn’t do. (I trust that they’re smart people and want always to rectify things as soon as they can, and they DO appreciate how this has hurt many, and they would have fixed this sooner if they could have.)

     

    In the case of the “need” for updater 3, there were in fact a couple of really important problems that it was meant to fix, ones that for some people have CF10 (even with updater 1 or 2) not performing at all well, and often even failing, under load. And it was THAT fix that many desperately wanted to get with Updater 3.

     

    So the fact that it then caused still other problems (some severe) did leave move to back it out. More important though, Adobe pulled it, so that people couldn’t even get it if they were ok with those “problems” it added, in order to get the fixes that it DID add.

     

    So yes, some really are desperate to get the fix and can’t really just be satisfied (as it were) that “updater 2 is stable”, because it’s not solving the more important problems that some folks have hit.

     

    It is indeed a bummer that the fix and testing has taken this long. Let’s hope that part of the reason it’s taking longer is that they’re doing more testing so that this sort of thing doesn’t happen again. (Again, I think that they’re smart people who may make a mistake but would learn from it, and I don’t feel as some do that they’ve proven entirely otherwise.)

     

     

     

    /charlie

     
    |
    Mark as:
1 2 Previous Next
Actions

More Like This

  • Retrieving data ...

Bookmarked By (1)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points