Web Connector Settings

@Carl - Yes, I raised it there first hoping for a response. When I got none I thought I'd spread the coverage by repeating it here :-)

@Carl V - Yes, of course you're right, I'm forgetting my net-etiquette :-)

I'll surely feedback here what I find.

@Carl M - If you see the last half dozen comments on that ColdFusion blog you'll see me having a "conversation" with Anit Kumar Panda of Adobe. Anit has been very supportive and has offered to call me to discuss my issue. I'll feedback the results here.

I have watched your Connect session several times. Most helpful, thanks. It gave me the insight, via JConsole, to see the results of changes to connection_pool_size / maxConnections and connection_pool_timeout / keepAliveTimeout. I'm guessing that I won't see any entry in JConsole for max_reuse_connections as my understanding is that this setting is part of Adobe's customisation of mod_jk?

My issues still stands. I now have lots of information (metrics.log and JConsole output) but no knowledge of how to use it to inform worker.properties / server.xml settings.

I'll capture a day's worth of currentThreadCount / currentThreadBusy chart data and post it back here with my worker.properties / server.xml settings later today to aid the debate.

As promised, here are some screen grabs from a good period of activity...

The worker.properties and server.xml files for the instance are unchanged from the default install (max_reuse_connections=250 only).

My interpretation of this is that thread count reached a maximum of around 50 which is well within the maximum allowed of 200.

The result I can't explain is that the thread busy count seems to be dropping as soon as the thread is no longer in use. The Tomcat Connector docs (http://tomcat.apache.org/connectors-doc/reference/workers.html) state that the default value is indefinite (zero). I've not added a value for connection_pool_timeout so I'm guess the Adobe mod_jk is adding the -1 for keepAliveTimeout which I'm also guessing mean "drop as soon as unused"?

I hope this helps inform someone who might be able to make some sense out of all this! :-)

Richard

Hi Carl,

Thanks for your time with this.

Your observations and suggestions on connection_pool_minsize, minSpareThreads and connection_pool_timeout, connectionTimeout raises a question I've had in my mind for some time.

Why release the threads at all? Why not wack them up high and forget about it?

My justification for this thought is that, assumably, when a request has finished with the thread it's marked as unused and can be adopted by a new request that needs a thread. Why drop it as, in time, if it's going to be opened again by the process when it has run out of free threads. Or have I misunderstood the technology?

Your thoughts on that would be appreciated.

One thing I should mention, which I should have noted at the beginning, is that this instance is supporting about 30 sites with one site being dominant. If that has any bearing?

You've read my mind about the size of the mod_jk.log file. I had noticed earlier in the week that it had reached > 200MB since Nov '13. I'd considered that it was only a write only file and that the connector had no interest in reading the content and therefore wasn't a performance issue. That said I'm trialling log rotation in our pre-production environments as a way to make the files easier to inspect and host on our logging drive. This is what I've added to our Apache config...

# mod_jk logging level [trace/debug/error/warn/info] and log file location with rotation

JkLogLevel info

JkLogFile "|/path/to/rotatelogs.exe -f /path/to/mod_jk.%Y-%m-%d.log 86400"

...which overwrites the entry in the /path/to/Apache/conf/mod_jk.conf file...

# Where to put jk logs

JkLogFile "\path\to\coldfusion10\config\wsconfig\1\mod_jk.log"

Anyway, as you say, now that we have the connection to JConsole established we're better placed to review the threads when we have another slow down. I'll post back here when I've got some slow site data.

I know the peak topped out at 50 the other day but why stop there?

Why not 100, why not 150, why not match the connection_pool_size value? What are the consequences of a large number? These are the questions I'm struggling to find documented answers to.

We've decide to add the config to rotation the mod_jk.log  file now and then wait until we get our next slowdown event and then review the JConsole thread count at that time and then consider making changes at that time.

Can you point to a resource or define that does "good tolerances" look like?

Regards,

Richard

So we had an event this morning and this time we were ready!

Around 10am I pushed the Apache change to rotate the mod_jk log files, as described earlier, to production on all three boxes and restarted Apache.

About 20 minutes later all the servers experienced a slowdown. A quick look at the threads didn't seem to indicate any issues...

We left it for a while, sometimes it rights itself but not on this occasion.

Around 10:28 we restarted Apache again on all three servers. In the past this has helped resolve any slowdown issue.

No joy :-(

More discussion and examination of log files. Nothing looked out of order.

So around 11:15 we started ColdFusion on all three servers. The penultimate sanction.

Still no improvement.

At 11:30 we decided to take the ultimate option and restart all three boxes.

After while, when all the servers had settled down response times returned to normal and the thread chart now looks like this...

Conclusions...

???????????????

You'd think that restarting Apache would solve any thread issue there or with mod_jk, not that we were seeing anything but no.

Restarting ColdFusion would have cleared any long running requests or memory issues but we weren't seeing anything to believe that was the problem.

Only restarting all the actual servers cleared the problem. We have experienced that before as the only solution to clear an increasing slowdown event.

I'll lob this record of events out there in the hope that some enlightened person will be able to say, "Ah, yes, I know what that is, you need to........"

In desperation and frustration,

Richard

@Carl - Yes, not ideal but like I said, our issues are normally resolved by an Apache restart, occasionally a ColdFusion restart and on the rare occasion a server restart.

The database is on it's own cluster, not on the web/ColdFusion servers.

Task Manager never seems to show any undue stress in these situations.

Can't see anything untoward in Event Viewer but I'll try and get our server guys to give them the once over.

Having recently read Russ Michael's insightful blog post http://www.michaels.me.uk/post.cfm/why-coldfusion-railo-are-not-suited-to-shared-hosting we're thinking about moving our primary domain onto it's own instance and leaving these secondary domains on a "shared hosting" instance.

That way we might be able isolate any coding issues that might be the root cause for the problem.

Richard

Hi Russ,

Humm, like I said, for whatever problem we have, more often restarting Apache works for us, occasionally we have to add a ColdFusion restart to get things working again and rarely we have to go as far as restarting the server itself.

I'm guessing you mean the ColdFusion service when you refer to killing "tomcat" but I can't see a service listed that looks like Java on it's own. Can you advise what I should be looking for?

Richard

@Carl V - Yes, of course you're right. I was "having a moment" there and was looking in the list of "Services" rather than "Task Manager" for "processes". D'oh!

@Russ - Yes, we're on Windows. We're enjoying the flexibility and features of Apache but there has been talk about considering IIS. First we have to prove that Apache is the issue before considering the effort involved in changing.

This is outside of my experience but I assume than BonCode is IIS only? Is there an alternative to the Adobe mod_jk?

Richard