Intermittent hang with CF10, apache 2.2, RHEL6

[GuitsBoy]

Hello -

We recently purchased four ColdFusion 10 Enterprise Licenses to refresh our production webservers.  We are currently testing our buildout, and one of the two initial servers has been giving us intermittent issues.  It seems that occasionally apache hands off the request to the coldfusion back end, but it never receives a response from the CF service, and the connection just hangs in a "W" Sending Reply state, with no obvious timeout.  I do not see any active threads in the coldfusion Server Monitor.  I see connections on the Apache server-status page that have been in the same state for 54,000 seconds and counting.  The end result is that the browser just hangs waiting for a response, but never timing out.  We estimate between 1% and 2% of all requests wind up in this hung state.

We have ensured that the /etc/hosts and /etc/resolve.conf files are correct.  I see no obvious errors in with the apache logs or coldfusion logs.

Details:

Coldfusion 10 Enterprise

Apache 2.2.15

RedHat Enterprise Linux 6.5 - 64 bit

MySQL 5.6.19

Roughly 200 virtual hosts, each with a duplicate JkMountFile "/opt/coldfusion10/config/wsconfig/1/uriworkermap.properties"

Any help troubleshooting this installation would be greatly appreciated.

[GuitsBoy · Accepted solution]

The problem was related to the well documented issue regarding the randomizer CF10 uses.   We had a large number or cfhttp calls, and each one requires calling /dev/urandom to generate a random number.  The default randomizer requires about 5 seconds of CPU time to gather enough entropy to create a truly unique random number.  Unfortunately this delay compounds upon itself as you get more and more calls.  If you have 100 concurrent cfhttp calls, requests would take 500 seconds.

The fix is editing the jvm.config to: -Djava.security.egd=file:/dev/./urandom”.

The default /dev/urandom is overridden / mapped to /dev/random, which causes the delay.  Editing the jvm.config exactly as above will supersede the mapping.

For more info on the subject:

Sameeksha: Random number generation in Unix & Performance