Copy link to clipboard
Copied
I have had CF 10 running on my VM for about a week now. The VM is Windows 2008 Enterprise, 64 bit. CF 10 is 64 bit. The error that I keep getting is:
Service Temporary Unavailable!
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Jakarta/ISAPI/isapi_redirector/1.2.32 ()
To recover, I need to stop and restart all the CF services.
Anyone else having this problem or has anyone else had to fix it?
Copy link to clipboard
Copied
Any errors and warnings in log files?
CFadmin > Debugging & Logging > Log Files > coldfusion-error.log & coldfusion-out.log
Could also enable tomcat metrics logging:
CFadmin > Debugging & Logging > Debug Output Settings > Enable Metrics Logging
then CFadmin > Debugging & Logging > Log Files > metrics.log
You could also read those logs in the file system ColdFusion10\cfusion\logs
HTH, Carl.
Copy link to clipboard
Copied
I got the error again this morning. Here is what I got from the metrics log -
Jun 4, 2012 | 7:45 AM | Information | scheduler-1 | |
Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 310923872 Total memory: 705232896 Active Sessions: 0 | ||||
Jun 4, 2012 | 7:44 AM | Information | scheduler-1 | |
Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 131668984 Total memory: 485359616 Active Sessions: 0 | ||||
Jun 4, 2012 | 7:43 AM | Information | scheduler-0 | |
Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 131670904 Total memory: 485359616 Active Sessions: 0 | ||||
Jun 4, 2012 | 7:42 AM | Information | scheduler-1 | |
Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 132744016 Total memory: 485359616 Active Sessions: 0 | ||||
Jun 4, 2012 | 7:03 AM | Information | scheduler-0 | |
Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 732531472 Total memory: 795607040 Active Sessions: 0 | ||||
Jun 4, 2012 | 7:02 AM | Information | scheduler-1 | |
Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 732640128 Total memory: 795607040 Active Sessions: 0 | ||||
Jun 4, 2012 | 7:01 AM | Information | scheduler-0 | |
Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 732640128 Total memory: 795607040 Active Sessions: 0 | ||||
Jun 4, 2012 | 7:00 AM | Information | scheduler-1 | |
Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 733411944 Total memory: 795607040 Active Sessions: 0 |
Copy link to clipboard
Copied
Of interest here are some CF10 metrics logging I took as a sample to compare.
05/06/2012 8:49 AM Information scheduler-1 Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 378536 Request count: 201 Error count: 4 Bytes received: 780 Bytes sent: 6993022 Free memory: 45875240 Total memory: 175702016 Active Sessions: 0 05/06/2012 8:48 AM Information scheduler-2 Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 378536 Request count: 201 Error count: 4 Bytes received: 780 Bytes sent: 6993022 Free memory: 47207152 Total memory: 175702016 Active Sessions: 0 05/06/2012 8:47 AM Information scheduler-0 Max threads: 150 Current thread count: 4 Current thread busy: 1 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 52128072 Total memory: 175898624 Active Sessions: 0 05/06/2012 8:46 AM Information scheduler-1 Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 59819016 Total memory: 167116800 Active Sessions: 0 05/06/2012 8:45 AM Information scheduler-1 Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 61154744 Total memory: 167116800 Active Sessions: 0 05/06/2012 8:44 AM Information scheduler-1 Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 62484720 Total memory: 167116800 Active Sessions: 0 05/06/2012 8:43 AM Information scheduler-2 Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 63821448 Total memory: 167116800 Active Sessions: 0 05/06/2012 8:42 AM Information scheduler-2 Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 65154032 Total memory: 167116800 Active Sessions: 0 05/06/2012 8:41 AM Information scheduler-1 Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 66487544 Total memory: 167116800 Active Sessions: 0 05/06/2012 8:40 AM Information scheduler-0 Max threads: 150 Current thread count: 4 Current thread busy: 0 Max processing time: 333509 Request count: 200 Error count: 4 Bytes received: 780 Bytes sent: 5695004 Free memory: 67824496 Total memory: 167116800 Active Sessions: 0
The online CF10 documentation omits CF10 metrics logging details so I am not sure what to make of the log detail differing. Reference:
I don't think the logs have been any use so far. Still interested to know why CF10 is crashing for you. Something else comes to mind. Is it possible for you to run CF10 from CMD prompt for a time, ColdFusion10\cfusion\bin\cfstart.bat ? Perhaps there is an error and having the details displayed in CMD prompt will lead to solution.
HTH again, Carl.
Copy link to clipboard
Copied
Should I go in to the services control panel and stop all the cf services, then run the batch file from the command line?
Copy link to clipboard
Copied
Stop ColdFusion 10 Application Server service
CMD prompt (as administrator?)
CD ColdFusion10\cfusion\bin
cfstart.bat
Watch wait report back any interesting details
Copy link to clipboard
Copied
Here is a link to some detail on CF10 metric logging:
http://helpx.adobe.com/coldfusion/kb/coldfusion10-enable-metric-logging.html
Copy link to clipboard
Copied
At 7:07 AM today, I got the error again. Here is the netrics log from that time span, I'm not sure what I can tell from these???
"Information","scheduler-1","06/07/12","07:00:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 476862912 Total memory: 554303488 Active Sessions: 0"
"Information","scheduler-1","06/07/12","07:01:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 468659888 Total memory: 554303488 Active Sessions: 0"
"Information","scheduler-0","06/07/12","07:02:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 466452672 Total memory: 554303488 Active Sessions: 0"
"Information","scheduler-0","06/07/12","07:03:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 465661168 Total memory: 554303488 Active Sessions: 0"
"Information","scheduler-1","06/07/12","07:04:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 464591872 Total memory: 554303488 Active Sessions: 0"
"Information","scheduler-0","06/07/12","07:05:31",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 463093992 Total memory: 554303488 Active Sessions: 0"
"Information","scheduler-1","06/07/12","07:23:21",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 142125480 Total memory: 495845376 Active Sessions: 0"
"Information","scheduler-0","06/07/12","07:25:11",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 129513304 Total memory: 495845376 Active Sessions: 0"
"Information","scheduler-1","06/07/12","07:26:11",,"Max threads: null Current thread count: null Current thread busy: null Max processing time: null Request count: null Error count: null Bytes received: null Bytes sent: null Free memory: 129511384 Total memory: 495845376 Active Sessions: 0"
Copy link to clipboard
Copied
Odd the CF10 metric log only showing memory values. Do not know what to make of that. Any feedback on running CF10 from CMD prompt?
Copy link to clipboard
Copied
Have not had a chance to try from command line yet. I have opened a case with Adobe support and I am going to give them a couple of days to work on this for me.
Copy link to clipboard
Copied
Please post back findings I am sure is of interest to other CF10 users.
There are other things can do eg CFSTAT, JVM logging, however not having a clear idea of where to look I do not want to send you down a path that is going to waste your time.
Copy link to clipboard
Copied
If I get something definitive from Adobe support, I will post it back here. Don't hold you breath though, when I tried to send them the server settings and log files, my email was blocked by their spam filter and they are trying now to get it released.
Copy link to clipboard
Copied
I cannot even get it to accept my admin password. After resets, etc. Windows 2008 Web R2 this is such a bummer.
Copy link to clipboard
Copied
My case with Adobe is still open. I opened a case with Microsoft becuase I found that simply restarting IIS 7.5 would restore the operation of the server. I was fortunate enough to has a crash while Microsoft was on the phone with me last night and we got a memory dump of the process in IIS that was hung. The analysis of that memory dump is shown below. I have forwarded the dump to Adobe for their analysis.
Hi Ralph,
Today we captured 4 set of dumps while page was in a hung state.
Please find analysis with this email :-
Looking at Dumps: w3wp.dmp ,
w3wp (2) .dmp
w3wp (3) .dmp
w3wp (4).dmp
èThe following threads in W3WP(4~1.DMP are waiting on data to be returned from another server via WinSock.
The call to WinSock originated from isapi_redirect+10ba2
Threads ( 4 5 6 7 15 16 17 18 19 20 22 24 )
48.00% of threads blocked
Number of Threads being blocked are increasing .
èLet’s see what is happening at Thread 4.
This thread is waiting on data to be returned from another server via WinSock.
The call to WinSock originated from isapi_redirect+10ba2
CallStack
ntdll!ZwWaitForSingleObject+a
mswsock!SockWaitForSingleObject+cc25
mswsock!WSPRecv+1166e
ws2_32!recv+13c
isapi_redirect+10ba2
isapi_redirect+8f82
isapi_redirect+a520
isapi_redirect+d4d3
isapi_redirect!HttpExtensionProc+4f5
isapi!ProcessIsapiRequest+13c
isapi!W3_ISAPI_HANDLER::DoWork+4fb
isapi!RequestDoWork+40d
isapi!CIISHttpModule::OnExecuteRequestHandler+1c
iiscore!NOTIFICATION_CONTEXT::RequestDoWork+233
iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+17f
iiscore!NOTIFICATION_CONTEXT::CallModules+25
iiscore!W3_CONTEXT::DoWork+3a6
iiscore!W3_MAIN_CONTEXT::StartNotificationLoop+5d
iiscore!W3_CONTEXT::ExecuteRequest+2bf
filter!RequestDoWork+40be
filter!CIISHttpModule::OnPostAuthenticateRequest+20
iiscore!NOTIFICATION_CONTEXT::RequestDoWork+20a
iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+17f
iiscore!NOTIFICATION_CONTEXT::CallModules+25
iiscore!W3_CONTEXT::DoWork+326
iiscore!W3_MAIN_CONTEXT::OnNewRequest+1ea
w3dt!UL_NATIVE_REQUEST::DoWork+126
w3dt!OverlappedCompletionRoutine+1c
w3tp!THREAD_POOL_DATA::ThreadPoolThread+7a
w3tp!THREAD_POOL_DATA::ThreadPoolThread+34
w3tp!THREAD_MANAGER::ThreadManagerThread+61
kernel32!BaseThreadInitThunk+d
ntdll!RtlUserThreadStart+1d
è Request is currently in IsapiFilterModule as shown below :-
!pipeline 0000000001372990
BEGIN_REQUEST( 0x1 😞
cachhttp-HttpCacheModule
modrqflt-RequestFilteringModule
validcfg-ConfigurationValidationModule
rewrite-RewriteModule
AUTHENTICATE_REQUEST( 0x2 😞
filter-IsapiFilterModule
authanon-AnonymousAuthenticationModule
POST:
--> filter-IsapiFilterModule
RESOLVE_REQUEST_CACHE( 0x8 😞
cachhttp-HttpCacheModule
MAP_REQUEST_HANDLER( 0x10 😞
compstat-StaticCompressionModule
EXECUTE_REQUEST_HANDLER( 0x80 😞
defdoc-DefaultDocumentModule
dirlist-DirectoryListingModule
protsup-ProtocolSupportModule
static-StaticFileModule
isapi-IsapiModule
iis_ssi-ServerSideIncludeModule
cgi-CgiModule
iisfcgi-FastCgiModule
RELEASE_REQUEST_STATE( 0x100 😞
compstat-StaticCompressionModule
POST:
rewrite-RewriteModule
UPDATE_REQUEST_CACHE( 0x200 😞
cachhttp-HttpCacheModule
SEND_RESPONSE( 0x20000000 😞
loghttp-HttpLoggingModule
isapi-IsapiModule
protsup-ProtocolSupportModule
filter-IsapiFilterModule
cachhttp-HttpCacheModule
rewrite-RewriteModule
custerr-CustomErrorModule
è0:000> !connections
64 Bit IIS 7.5 Detected
RequestListHead at 000007fefc939088
NUM NATIVEREQ MAIN_CONTEXT IIS_REQMON_CONTEXT NREQ_STATE VERB VERSION SERVER ADDRESS CLIENT ADDRESS REQUEST ACTIVE URL
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 0000000001352dd0 00000000013539f0 1354398 PROCESS GET HTTP/1.1 67.54.20.62(80) 208.104.28.202(3592) 0 days: 00:05:49.578 /jakarta/isapi_redirect.dll
2 0000000001355f60 0000000001356b80 1357528 PROCESS GET HTTP/1.1 67.54.20.62(80) 208.104.28.202(3654) 0 days: 00:05:26.422 /jakarta/isapi_redirect.dll
3 00000000013590f0 0000000001359d10 135a6b8 PROCESS GET HTTP/1.1 67.54.20.62(80) 208.104.28.202(3930) 0 days: 00:05:21.516 /jakarta/isapi_redirect.dll
4 000000000135c280 000000000135cea0 135d848 PROCESS GET HTTP/1.1 67.54.20.62(80) 67.54.20.62(55208) 0 days: 00:04:46.328 /jakarta/isapi_redirect.dll
5 000000000135f410 0000000001360030 13609d8 PROCESS GET HTTP/1.1 67.54.20.62(80) 67.54.20.62(55221) 0 days: 00:02:49.516 /jakarta/isapi_redirect.dll
6 00000000013625a0 00000000013631c0 1363b68 PROCESS GET HTTP/1.1 67.54.20.62(80) 67.54.20.62(55222) 0 days: 00:02:37.750 /jakarta/isapi_redirect.dll
7 0000000001365730 0000000001366350 1366cf8 PROCESS GET HTTP/1.1 67.54.20.62(80) 67.54.20.62(55225) 0 days: 00:02:35.188 /jakarta/isapi_redirect.dll
8 00000000013688c0 00000000013694e0 1369e88 PROCESS GET HTTP/1.1 67.54.20.62(80) 67.54.20.62(55227) 0 days: 00:02:33.0 /jakarta/isapi_redirect.dll
9 000000000136ba50 000000000136c670 136d018 PROCESS GET HTTP/1.1 67.54.20.62(80) 67.54.20.62(55230) 0 days: 00:00:50.532 /jakarta/isapi_redirect.dll
10 000000000136ebe0 000000000136f800 13701a8 PROCESS GET HTTP/1.1 67.54.20.62(80) 67.54.20.62(55231) 0 days: 00:00:26.500 /jakarta/isapi_redirect.dll
11 0000000001371d70 0000000001372990 1373338 PROCESS GET HTTP/1.1 67.54.20.62(80) 67.54.20.62(55234) 0 days: 00:00:25.938 /jakarta/isapi_redirect.dll
è
Lmvm isapi_redirect
start end module name
00000000`6a6b0000 00000000`6a714000 isapi_redirect (deferred)
Image path: D:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll
Image name: isapi_redirect.dll
Timestamp: Thu Mar 29 07:50:00 2012 (4F745A78)
CheckSum: 0006CC8A
ImageSize: 00064000
File version: 1.2.32.0
Product version: 1.2.32.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Apache Software Foundation
ProductName: Apache Tomcat isapi_redirector Connector
InternalName: isapi_redirector
OriginalFilename: isapi_redirector.dll
ProductVersion: 1.2.32
FileVersion: 1.2.32
FileDescription: Apache Tomcat Connector
LegalCopyright: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership.
Comments: The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Action Plan:
To follow up with the vendor for “isapi_redirect .dll“ for further troubleshooting.
Regards,
Anjum
Microsoft is committed to protecting your privacy. Please read the Microsoft Privacy Statement for more information.
The above is an email for a support case from Microsoft Corp.
REPLY ALL TO THIS MESSAGE or INCLUDE casemail@microsoft.com IN YOUR REPLY if you want your response added to the case automatically. For technical assistance, please include the Support Engineer on the TO: line.
Thank you.
Copy link to clipboard
Copied
Issue was never resolved for CF10 so I down graded to CF9 and the server is running just fine. Adobe support helped me get through the process of moving back from CF10 to CF9
Copy link to clipboard
Copied
One option you have when receiving those errors is to change your IIS connector to something like the Boncode IIS to Tomcat connector (which is currently used by Railo's Windows installer). From what I can tell, the Adobe connector just can't handle high load and gives out at a certain (undetermined) threshold.
Bilal (the main developer of the project) has been very helpful with making the Boncode connector very compatible with CF 10, and I've seen good results from it so far. Here's the link to the project: http://tomcatiis.riaforge.org/
Copy link to clipboard
Copied
Thanks for the thoughts. I have given up on CF10 for now and reverted to CF9
Copy link to clipboard
Copied
I have tried the boncode connector since I am having http://forums.adobe.com/message/4619002#4619002, and I have tried different configuration setting for the connector, but the page always has issue with displaying. I will have to keep refreshing the page, until the contents have been showed correctly and in full. It not only happens to my application, but also the cfadmin page. The header part of the admin page sometimes shown as blank. In order to make it display correctly, I will need to refresh the page couple of times.
Copy link to clipboard
Copied
I think the issue you have described is similar to an issue I had to resolve with Coldfusion 10. When I use a custom 404 page my content was randomly getting truncated or not displaying. What I soon realized is that the content length was not getting passed back to IIS from tomcat and this was causing my page to get truncated or not show anything. To resolve this issue I added a little hack to the custom 404 page. I captured all content that was going to be displayed on the page as a variable, I then check the length of the content and use a cfheader tag to force the content length within the browser. This simple fix seems to have resolved my issues.
I hope this helps!
Copy link to clipboard
Copied
We have this happen about once every 2-3 weeks on a site with a few thousand active users. Moreso when we are undergoing are quarterly PCI tests and we get blitzed by SecuritMetrics' automated tests.
I'd prefer not to monkey around with a 3rd-party connector. Is Adobe aware of this?
Copy link to clipboard
Copied
I have just given up on it at this point. Feeling like a wasted purchase. I have reverted to using Railo. I hope their is some response from Adobe.
Aaron
Copy link to clipboard
Copied
We are on the same boat, my server randomally crashes when there are few thounsands active users as well.
I did spend some time on a alternative connecter myself, but I couldn't get it to work perfectly. And I do agree with you that we shouldn't play around with some 3rd party thing, since the cf10 is using some customized version of tomcat connector and there are lots of things we are not sure at this stage
Copy link to clipboard
Copied
Copy link to clipboard
Copied
We've been looking into this further and have found that tuning the connector timeout and the worker.properties can really help with this issue. I'm still a big fan of the BonCode connector though, considering how fast Bilal's turnaround time on adapting to CF 10 has been (and that it does not run as a funky ISAPI redirect filter). With that said, you may want to open up your server.xml in "cfusion\runtime\conf" and find the section for the AJP connector (~line 100), and set the maxThreads, connectionTimeout, and acceptCount attributes. Should look similar to this:
<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="300" connectionTimeout="60000" acceptCount="50" />
Those attributes help prevent Tomcat from spooling up too many or not enough threads for your application. The acceptCount also queues things up temporarily if you reach the max number of threads.
Now you'll want to edit workers.properties at "C:\ColdFusion10\config\wsconfig\1".
By default Adobe does not have any connection pool limitations, so things can get out of hand when an application gets busy. If your server hosts a lot of IIS sites, you may want to consider adding these lines to your workers.properties file:
worker.cfusion.connection_pool_size=10
worker.cfusion.connection_pool_minsize=1
worker.cfusion.connection_pool_timeout=60
More info on this here: http://tomcat.apache.org/connectors-doc/reference/workers.html
Those adjustments have helped tame this issue for sites that need to use the Adobe connector. Your mileage will vary, and you'll want to tweak those settings to meet your environment, but good luck!
Also, Ryan, the issue you mentioned with the connector is very odd. Was it perhaps set as a Wildcard and trying to process every request, even the static files? If you do feel like continuing with the connector though, you may want to upgrade to the latest version. I know Bilal's fixed a few bugs in the last month.
Copy link to clipboard
Copied
Thank you for your reply JakeHand
I would like to try the alternative connector in my VM, but not production at the moment, but can you teach me how to correctly setup the Boncode?
I tried different configuration myself (none of them uses wildcat), but just doesn't work for me.
Regards