JRun Connector Errors

[-_MM_-]

We've migrated from ColdFusion MX 6.1 to 8 recently and we're seeing an alarming number of error messages like these:

Server Error
The server encountered an internal error and was unable to complete your request.
JRun Connector Protocol Error.

We're running:
Cold Fusion 8,0,1,195765 64-Bit
Solaris 10
Apache 2.2.10
JRun Connector mod_jrun22.so
JDK 1.5.0_16 (we downgraded the JVM as a potential fix)
Oracle JDBC drivers 11.1.0.6.0 for JDK 5 (we need Oracle 11g database support)

We've spent several days investigating this now. The only patterns seem to be that:
-happens a little bit more after a server restart
-can return the error very quickly on random pages
-but can also return the error on pages that are running big queries and take a long time to run
-very inconsistent, we can call the same page multiple times at once - some requests fail, others succeed
-we are also seeing a lot of java.io.EOFException errors in our cfserver.log with stack traces of some kind
-the logs don't point in any parcticular direction, nor do the JRun Connector errors seem to coincide with any logs entries

What we've tried to do to fix this:
-Tried the built-in Adobe Oracle database drivers
-Tried the JDK 5 version of the Oracle 11.1.0.6.0 database drivers
-Downgraded the JVM used by Cold Fusion 8 to JDK 5.
-Adjusted the data source connection pools timeouts; turned off Maintain Connections altogether just to see if it made a difference
-Adjusted template caching settings
-Tweaked the 3 or 4 JRun Connector timeout settings in Apache. Also turned on verbose mode on the JRun Connector to see if that revealed anything. Saw a few suspicious entries, such as:
jrApache[init] 127.0.0.1:51800 destroying/closing socket [35] error 0
jrApache[init] jrCloseSocket [35]
But these did not parallel all of the JRun connector errors we've been seeing.
-Tried connecting to only one database node at a time (we have an Oracle cluster)
-Verified our JDBC connection string
-Tried reproducing this error under previous Cold Fusion 6.1 production servers and in development environment, but didn't have any luck
-Even tried cleaning up the linebreak characters in some of our code after vi complained about "premature end of file" or "premature end of line" - no luck.

We're pulling our hair out at this point. The only other thing we've suspected is a firewall or loadbalancer issue causing us to lose connections. HELP!!!!

[BrienMalone]

I hate seeing complex technical questions like these go unanswered in the forums. I just posted one and don't hold out much hope for resolution.

It looks like you posted this about 2 weeks ago -- have you had any luck?

I had bizarre problems where threads would lock up until the entire server would freeze. Turns out our network admins caused the problem by blocking communications for certain types of packets on port 80... what kind, I'm not sure. Point being -- you might try to replicate the problem off your company network if possible.

--B

[-_MM_-]

BrienMalone,
Thanks for the words of support. Unfortunately, in the month since I've posted this I have not heard a peep otherwise, nor have we resolved the issue.

I spent a fair amount of time adjusting the JVM to see if that had an impact. Given that I still saw the error, I don't think it made a significant positive impact.

We are suspicious of network components like firewalls causing issues here. We tried running Cold Fusion 8 for linux under a VMWare environment on a seperate network and could not reproduce this issue. Moving one of the production servers to a different network location is the next step (a big one).

[Charlie Arehart]

MM, have you looked at other than the [cf]/logs? There are also [cf]/runtime/logs/ which often have far more (and more useful) info for solving serious problems like this. If you're running the multiserver (multi-instance) deployment of CF, then look for them in the [jrun4]/logs/ directory instead.

Also, you don't say if you're using CF8 Enterprise or Standard. If the former, you may get value from the CF8 Server Monitor, by enabling "start monitoring" to view what requests are running when things seem to go bad. You may have some hung requests, and the connection error could be from the web server not getting an answer from some pages because they're hung up. I don't use Apache much so can't tell from the error message if that may be an issue. It's just a guess.

Let us know what you see.

[-_MM_-]

Two additional comments:

We are using Cold Fusion 8 Enterprise.

One trend that we have picked up on is that these JRun Connector errors seem to coincide with java.lang.NullPointerException errors that do appear under Errors in the Monitoring application and in the logs. The line number seems to always coincide with a CFC being instantiated (different CFC's) via a createObject() call.

[derCris77]

Hello,

we also have got the same error and also very sporadic.

We're running:
Cold Fusion 8,0,1 32-Bit
Window 2003 Server
Apache 2.0.63
JRun Connector mod_jrun20.so
JDK 1.6.0_04
ms sqlserver 2005
on VMWare

The error occures on different installations and different machines. allways on vmware.
when it happens nothing helps any more.
- restarting apache - no effect
- restarting jrun - no effect
- restarting both - no effect
- killing all apache and jrun processes and restart - no effect

and the system does not recover by itself.

only restarting the vmware helps...

sometimes even the jdbc connection to our mssqlserver 2005 ist not available anymore.
message: socket write error.

seems to be something very deep in the system, but we could not figure out what.
we got the same
"destroying/closing socket"
error in our apache log files.

seems to happen more often if cpu is on load, or if more threads are running but we are not able to reproduce it at all...

help please!

[Guest]

Hello,

I've first identified this exact same problem back in October to Adobe under our support contract. We've offered large amounts of support and even helped by working with Tier 3 engineers at Adobe. As it stands right now though, there is still no resolution, our ticket remains open at Adobe. 6 months have gone by! This is pretty bad for a peice of software that many have invested large sums of money into to have a pretty significate amount of these errors is alarming.  We also have log files for internal systems that show this problem occurs daily, and multiples times daily on our servers!

I hope Adobe have a Hotfix on it's way, they sure need to own up to one!