CF 10 CFADMIN loading issues

[tribule]

Hi,

We run CF10 standard on IIS8 Win 2012 and our CF Admin is going haywire. It frequently will not answer on 127.0.0.1 and when it does load it sometimes is missing information in the left nav bar or right hand pane.

Today we see "Service Temporary Unavailable!The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.   Jakarta/ISAPI/isapi_redirector/1.2.32 ()"

and yet the web site on the server is running fine.

We run FusionReactor and when the CF Admin is trying to load there are no running requests to stack trace. How can we debug this?

Sometimes, if we press CTRL-R repeatedly, it will load. Individual frames of the admin can be loaded if their address is entered, but the whole thing doesn't load. Most times it just refuses to load in its entirety and if we are lucky enough to get to a login screen, the system will not load after that screen.

On Chrome browser we got the Jakarta error, but in MSIE we get in and see a JavaScript alert "Error invoking CFC /CFIDE/Administrator/updates/download.cfc - Service unavailable"

Appreciate some help - we want to downgrade back to CF 7 which was rock solid for us

Regards,

Mark

[carl type3]

Hi Mark,

I did a talk about CF tomcat IIS AJP connector matters. Here:
http://experts.adobeconnect.com/p8l51p4s9m4/
The PDF slide deck for that talk is still present:
http://www.tassweb.com.au/cf10tomcat.pdf

I think details in talk will assist you getting CF10 running well.

There is more to performance of system than the tomcat parts. Could be CF Java JVM is having issues with heap settings. You did not post how much RAM the Windows has. These days a server with 8Gb is typical. Given typical RAM is available CF10admin > Server Settings > Java and JVM settings of Minimum JVM Heap Size 256 (in MB)  Maximum JVM Heap Size 512 (in MB) plus > JVM Arguments > -XX:MaxPermSize=192m  can be inadequate. It is a long way from Java tuning and one should do some JVM logging to know but you could do well to increase those.

HTH, Carl.

[Aegis_Kleais]

Wild shot in the dark here, but by chance, do you have any form of request throttling enabled in IIS?

We had an issue a while back, not sure if it was "X amount of requests per second" or what not, and it was causing CF10's Admin page to load up sporadically.  I guess the page   Also, ensure that you are not using any IIS-based request filtering that may be preventing some URLs from executing.

[tribule]

Thanks very much for the replies.

Our (64 bit) server has 8 MB of memory and maximum heap is set to 8192 MB. Our -XX:MaxPermSize is 256m. Our server was tuned independently by a knowledgeable company.

We are running update 11. Did that version require a connector rebuild?

We do have request filtering enabled on our main public domain in IIS, in order to deny access to the sensitive /CFIDE folders from the outside world, but the internal local server allows access to them so that we can access the CF Admin etc. We also have authentication enabled on /CFIDE for the main domain, so if it was accessed externally it would not allow public access.

Regarding throttling in IIS, I checked the default pool (which the local server uses) and it has a "queue length" entry, which I guess is the number of requests, and it has the default value "1000".

I had a look at the isapi_redirect.log in ~wsconfig/1 and see quite a few of these errors:

[Thu Oct 03 20:32:59.983 2013] [56160:74964] [error] ajp_send_request::jk_ajp_common.c (1669): (cfusion) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)

[Thu Oct 03 20:44:50.798 2013] [56160:21584] [error] HttpExtensionProc::jk_isapi_plugin.c (2309): service() failed with http error 503

but I am not sure if those are related to the CFAdmin accesses. Are these errors to be expected?

My workers.properties file looks like:

worker.list=cfusion

worker.cfusion.type=ajp13
worker.cfusion.host=localhost
worker.cfusion.port=8012
worker.cfusion.max_reuse_connections=250

I see that we are missing some of the connection_pool commands from your PDF, Carl (size, minsize, and timeout.)

It is so difficult to debug this when it works for a couple of weeks and then stops for no reason. When it works, surely the config is ok?

The only way we could get in to CFAdmin last night was to restart CF from the services panel.

Regards,

Mark

[carl_bus]

>We are running update 11. Did that version require a connector rebuild?

Yes post running WSCONFIG your ISAPI.DLL should be dated March 2013

>I had a look at the isapi_redirect.log ... Are these errors to be expected?

Not expected unless there is an issue. Some part of CF or tomcat has stopped responding.

Generally does not indicate a tomcat connector problem at IIS end. Can be connector problem at CF end of tomcat.

>I see that we are missing some of the connection_pool commands from your PDF, Carl (size, minsize, and timeout.)

Those are manual settings you can choose to apply. Tho without applying similar settings to CF\runtime\conf\server.xml AJP section, changes to workers.properties alone will not help.

>It is so difficult to debug this when it works for a couple of weeks and then stops for no reason. When it works, surely the config is ok?

That's the nature of it. Probably because something Java or tomcat is running close to bottleneck then given certain condition reaches full.

HTH, Carl

PS my normal login is not working so this is the same Carl as carl type3. Long weekend for me so I can't be bothered to login to work.

[tribule]

Carl, the isap_redirect.dll file is dated 25 May 2013.

Btw your login may have changed after the Adobe hack where the source code for ColdFusion was leaked

[carl type3]

Mark, Yes your correct, update 11 isap_redirect.dll is dated May 2013.

[tribule]

Do you think a connector rebuild would be helpful?

[carl type3]

No I do not think rebuild connector will help. I find the isap_redirect.dll connector file is OK for CF10 update 5 to 10 and 11.

I think some edit's to works.properties and server.xml may help (requires IIS and CF restart). I figure it's broken and if changes to configuration files do not help then you can revert back to previous syntax easily.

I also think some monitoring via JVM logging and JMX will help to resolve where the problem lays but monitoring logging alone will not resolve matter.
 

[tribule]

Thanks for the advice. Our ISP had a look over the server and one thing they mentioned was the webconfig.xml permissions, which they say said did not seem correct. Do you know what permissions this should have under IIS?

[carl type3]

Mark I can not find webconfig.xml file on a couple Windows CF10 servers (2008r2 & 2012) that I searched.

[tribule]

Sorry, it was /web.config - the IIS permissions file. Our permissions on the server have been having some issues after an install went wrong

[carl type3]

I got numerous web.config files in published virtual directory paths, but none under cf10\wwwroot\CFIDE. web.config files permissions:

System Full, Administrators Full, Users Read & Execute plus Read.

[tribule]

Thanks, I've got the same now.