• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
0

How to eradicate the 'Server is busy' errors?

Participant ,
Jan 16, 2012 Jan 16, 2012

Copy link to clipboard

Copied

We use Coldfusion 9.01 on Linux Redhat 64bit with Apache. Our users regularly get the folllowing error message:

Application server is busy.  Either there are too many concurrent requests or server still starting up, errorMsg: Application server is busy.  Either there are too many concurrent requests or the server still is starting up

There is a lot of info on the web about this, with advice how to solve it, but we can not seem to get rid of this problem.

We have enabled metrics logging and nothing indicates that we are running out of threads at the time we get the error. Our thread config is as follows, in the instrance's JRUN.XML file

  <service class="jrun.servlet.jrpp.JRunProxyService" name="ProxyService">

    <attribute name="activeHandlerThreads">10</attribute>

    <attribute name="backlog">500</attribute>

    <attribute name="deactivated">false</attribute>

    <attribute name="interface">*</attribute>

    <attribute name="maxHandlerThreads">390</attribute>

    <attribute name="minHandlerThreads">5</attribute>

    <attribute name="port">51003</attribute>

    <attribute name="threadWaitTimeout">300</attribute>

    <attribute name="timeout">300</attribute>

When the error happens, our metrics reporting shows the following:

01/16 11:05:12 metrics Web threads (busy/total): 0/25 Sessions: 416 Total Memory=1046592 Free=597788

So the nr of requests (presumably 25, the nr of threads, not 100% sure)  is higher than the activeHandlerThreads value (10), but not as high as the maxHandlerThreads value. The error is returned immediately, and not after the specified timeout of 5 minutes.

That is the puzzling part...

Is there any other set of configuration parameters to tweak as well?

Thanks for your help!

Views

7.9K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 16, 2012 Jan 16, 2012

Copy link to clipboard

Copied

Suggest add other CFMetrics counters to get more details:

{listenTh},{idleTh},{delayTh},{busyTh},{totalTh},{delayRq},{droppedRq},{handledRq},{handledMs},{delayMs},{freeMemory},{totalMemory},{sessions},{sessionsInMem}

Looks like your CFMetrics is set to log these details, perhaps the issue is not with one of those counters:

{busyTh}/{totalTh}, {sessions}, {totalMemory}, {freeMemory}

While the memory looks OK from your CFMetrics results there are other java memory buffers to consider. Perhaps check in the java logs for "out of memory" type messages. What java logs? I have not used Linux / Unix for a long time so fail to recall. If I was using windows the log files to read would be in ColdFusion9\runtime\logs or \Jrun4\logs (pending Standard or Enterprise).

HTH, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Jan 16, 2012 Jan 16, 2012

Copy link to clipboard

Copied

Thanks, the extra logging has been put in place, waiting for a re-occurrence of the 'server busy' problem..

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 16, 2012 Jan 16, 2012

Copy link to clipboard

Copied

Something else that may help understand the CFMetrics output would be this graphical tool

http://www.adobe.com/cfusion/exchange/index.cfm?event=extensionDetail&extid=1029407

HTH, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Jan 17, 2012 Jan 17, 2012

Copy link to clipboard

Copied

The problem occurred again, but there is no indication from the metrics what the problem could be. One promising avenue is the setup parameters for the JRun-Apache connector. It has settings in the vhost.conf file such as

    #JRunConfig ConnectTimeout 15 <number of seconds to wait on a socket connect to a jrun server>

    #JRunConfig RecvTimeout 300 <number of seconds to wait on a socket receive to a jrun server>

    #JRunConfig SendTimeout 15 <number of seconds to wait on a socket send to a jrun server>

So will experiment with the following settings:

    JRunConfig ConnectTimeout 60

    JRunConfig RecvTimeout 300

    JRunConfig SendTimeout 60

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 17, 2012 Jan 17, 2012

Copy link to clipboard

Copied

Think CF java logs would still be interesting. Is there any warnings in java log? Not sure where those are on Li/U nix tho could be defined about the same place where you configure CFMetrics - this type of syntax:

<attribute name="filename">{jrun.rootdir}/logs/{jrun.server.name}-out.log</attribute>

Or defined in the CF daemon startup eg:

start default 1>$CF_DIR/runtime/logs/default-out.log

Is CF Enterprise or Standard licence? If Enterprise CF Monitor is available and could help.

CF9 so Server Manager should be able to be used. While it is fairly limited perhaps the CF Server Manager > Details View > JVM memory items will be of interest?

Else there are people who recommend FusionReactor or seeFusion. I have not used either since CF7.0.0 days (long time - about the same time have not used unix).

HTH, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Jan 18, 2012 Jan 18, 2012

Copy link to clipboard

Copied

Thanks Carl,

We use the Enterprise version. So far we have not run into Java errors, just Apache connector errors.

Interestingly enough, after we increased the timeouts for the connector, we now get the following error instead:

returning error page for JRun too busy or out of memory

The original 'Application Server is Busy'...." has disappeared.

Does this ring a bell? The metrics do not show anything odd as far as I can tell.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 18, 2012 Jan 18, 2012

Copy link to clipboard

Copied

While CFmetric will be showing you heap used and free the out of memory could be one of the other memory buffers Perminant Generation, Code Cache or other. There are a few things you can do to measure if that is the case or not however takes time. Seems you are willing to give things a go so could make a guess at it.

Suggest increase the size of your PermGen in JVM arguments. Alter -XX:MaxPermSize=192m to say -XX:MaxPermSize=512m then restart CF. Insert note here to backup (copy) JVM configuration file before applying any changes.

That change made is well short of JVM tunning but if it solves your problem who cares. If it does not help well we on forum making many assumptions so can be wrong.

HTH, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Jan 20, 2012 Jan 20, 2012

Copy link to clipboard

Copied

I'd be careful on cranking up MaxPermSize without knowing what your heap sizes are configured for.

While this won't directly impact your problem - this should be 6-8 threads per core in your server:

<attribute name="activeHandlerThreads">10</attribute>

Also, you may want to consider turning up logging on the connector.  Should be in the cfusion/lib/wsconfig directory if it is anything like Windows.

You may also want to test out Fusion Reactor to see if it'll give any insight.

I've not messed with Apache much as a stand-alone product, but I do have some extensive work with it in WebSphere.  Quote often we'll adjust the Thread limit and Threadsperchild in httpd.conf to help with httpd performance.  I don't know if they'd be aplicable to your situation.

Also, you might want to validate that the timeout setting in httpd.conf matches that in the connector and in your CF admin settings.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Jan 28, 2012 Jan 28, 2012

Copy link to clipboard

Copied

Tried lots of things with JVM tuning but that did not really solve the problem.

Now thanks to the suggestion of ke4pym we changed the MaxClients and ServerLimit in the Apache prefork module (/etc/httpd/conf/httpd.conf). The values were set at 512. The JRUN maxHandlerThreads value is 400, i.e. lower than the number of Apache clients, which could cause trouble. So now we set the MaxClients and ServerLimit to 250 and the errors are not occurring anymore.

Originally the JRun maxHandlerThreads value had been at 1000, but supposedly Linux can not create so many threads? So it was lowered to 400.

We'll do some more experimenting but if this solves the issues then it will be great.

It just makes me wonder why this is not documented anywhere.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Jan 30, 2012 Jan 30, 2012

Copy link to clipboard

Copied

Never Mind, the problem is STILL occurring. So we still have not found the answer. Things tried to date

- JRun Proxy thread counts, maxHandlerThread, activeHandlerThreads

- JRun Apache connector timeout settings

- JVM memory and GC settings

- Apache threads in the prefork module

This is a mystery

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 30, 2012 Jan 30, 2012

Copy link to clipboard

Copied

To be sure you can tick off JVM you could enable some logging. Examine log details then you would have an extra level of confidence that it indeed is not the problem. Reply the thread if you want need more details how to enable JVM log or read log.

The graphical tools FR seeF have been good to others. Are you going to buy CF9 for another project, if so Adobe have a special buy CF9 get free FR.

Just trying to help, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
May 30, 2012 May 30, 2012

Copy link to clipboard

Copied

This problem is still not solved. We found that if you enable request debugging the problem occurs very often, even with JSP or servlets (non-CF).

Is the only way out to get rid of JRUN? There is no real support left it seems.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
May 30, 2012 May 30, 2012

Copy link to clipboard

Copied

jkaashoek, I appreciate that you just want the problem to go away, but some problems are simply not answered easily (or well) via email or forums. Sometims there's just no substitute for having an experienced troubleshooter looking "over your shoulder" at your server with you, over the web (not remote desktop, but using a tool like Adobe Connect, GotoMeeting, Join.me, etc.)

If you may be interested in that, there are several folks (including myself) who offer such services. I list them in a category of my CF411.com site, specifically http://www.cf411.com/cfconsult.

I'll add that I myself have no minimum, and can solve many problems very quickly (hours/minutes, not days/weeks) and even offer a satisfaction guarantee, so you won’t pay if you don’t feel my time is helpful. I have helped many people solve such “server hung/busy” problems, but there is not always one obvious answer. Instead, what I would propose is that we work together, looking at your server, understanding available diagnostics (some of which you may have missed, or not known how to interpret).

In most such cases, the problem is in configuration rather than code, so it’s often surprisingly easy to implement a solution (based on the diagnostics) and wait to see if it helps. Then again, sometimes you fix one problem and another surfaces which was hiding behind it. But it rarely is more than a couple of these, so generally not a matter of days or work. And in the end you get more than a solution, you’ve gotten an education, to better understand and resolve other problems that will inevitably arise.

All that said, I appreciate that you likely prefer a free answer here, and it may seem (to some) that this is garish hawking of services. Note that I am pointing to others as well, so this is not for my benefit so much as yours, if you want a solution perhaps sooner than later. Sometimes community interaction is just tedious and time-consuming (and not all of us can be available all the time to interact on every problem even in the forums we may watch occasionally.)

This is a sincere offer to help, not a sales pitch. Anyone with that concern can look at how many questions I do answer in these forums, without mentioning my services. I only bring this up when it seems especially well-suited to the problem at hand, and I then also mention the other service providers.

Bottom line, no, “only way out” is not “to get rid of JRUN”, and while you may have helped there was “no real support left it seems”, I hope this shows that there is in fact some, when needed, and when these forums aren’t suited to particular problems.

Hope that’s helpful.

/charlie arehart

charlie@carehart.org

Providing fast, remote, on-demand troubleshooting services for CF (and CFBuilder)

More at http://www.carehart.org/consulting

See also http://www.cf911.com for more on CF troubleshooting resources


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jun 25, 2012 Jun 25, 2012

Copy link to clipboard

Copied

We found the problem, with the help of Adobe's CF Support team. It was not a tuning problem. It was a specific code problem. One of our JSP's was creating and downloading a PDF so it was using a call to 'response.getOutputStream()' to write the binary content to the browser. But JSP's are not supposed to do this, they are meant to do text based output using getWriter(). There was a stack dump from the JSP page cleanup code (out.close()) showing an IllegalStateException, but we overlooked it thinking it was caused by connectivity problems with the browser. And the exception was not visible to the page, it happened at cleanup behind the scenes.  But this occurrence was apparently crippling the JRun thread so that it could not be used the next time and would cause our problem.

So JRun does not recover from application errors.

Something to be aware of when trouble-shooting. Make sure all errors in the <instance>error.log file are accounted for.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Jun 26, 2012 Jun 26, 2012

Copy link to clipboard

Copied

Hello,

After reading your topic, your problem looks a bit like mine. (http://forums.adobe.com/thread/772471 and https://bugbase.adobe.com/index.cfm?event=bug&id=3043033).

The big difference was my problem occured with IIS, I've never check on Apache...

For now we are still using CF8 since this problem has not been solved.

Are you able to enable the verbose option for the apache connector ? In my case, the problem did not occured when verbose was enable. But this don't solve anything...

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Participant ,
Jun 26, 2012 Jun 26, 2012

Copy link to clipboard

Copied

LATEST

I replied in your listed thread. The O'Reilly utilities should not be used with CF9/JRun, they should be replaced with the Apache Common FileUpload utilities. It looks like any exception, caused by any (application) code, can cause the JRun threads to be degraded and caused the 'busy or timeout' error. So it is imperative to avoid causing any exceptions. This is quite a revelation after months and months of trying to eradicate the 'busy or timeout' problem. There is no documentation on how the JRun server gets degraded by exceptions.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Resources
Documentation