CPU spinning at 50%

A Cameron wrote:

 

You've not just restarted the CF service in the hope it was just a temporary aberation?

Yes many times, and sooner or later the spinning has returned.

Have you run FusionReactor (or CF's own server monitoring, I guess... never used it...) on it to see what might be causing it?  You should just be able to kill that thread.

 

FusionReactor, from memory, will let you cross reference threads to actual code.

I'll have to check out FusionReactor to see if it truely does cross reference threads to actual code.  That is something I have not been able to do for the threads that hang around.  With the jrpp threads that have the cfml reference as part of the thread dump it is pretty simple, but none of those threads seem to be the problem as far as I can tell from the thread dumps I have gotten from the ColdFusion built in monitoring tools.

BKBK wrote:

 

Sticks out like the proverbial sore thumb: "Flash Remoting Running requests"

Except that the only Flash Remoting running is the ColdFusion Monitor used to get the thread dump, and it was only turned on after the CPU started spinning.  So I have not put much weight into that being part of the problem.

Chiwi8888 wrote:

 

Im wondering if it could be the GC going nuts?

It was one of the first things we tried to look at.  But the last time the CPU started spinning, there was plently of memory available.  So unless GC can start going nuts even when less then 1/5 of the memory is being used.

BKBK wrote:

Fair enough. You get just too many waiting and blocked threads. Race conditions or even deadlock, perhaps?

Not that we have found yet.  But I am not all that experience at diagnosing thread issues.  But using the jConsloe tool seems to show no deadlocks.  The only thing that I can see that may be out of order is the above mentioned print thread.  But it has shown up after restarts of the CF service and is present even when the CPU is not spinning.  But, I don't know if it is supposed to be there or not.

In any case, you seem to be on Java 1.6.0_04, which is known to have a notorious  classloader problem. I wonder whether that could be the cause of your  woes. There is only one way to find out: upgrade to a higher JVM.

I am hopfully getting a lab setup next week where I can start testing upgrades on our system.  We may be able to upgrade to CF9 at that time.

BKBK wrote:

 

Are you sure it can wait? My point is that the spinning might be caused by the 1.6.0_04 classloader bug.

Well this is a State Government web server, so even a wild fire rush job can take a couple of weeks to work through the bureaucracy.

But, a quick restart the the ColdFusion instance with the spinning, resets the CPU problem for upwords of a week or two, and we have been limping along for almost two months now with this last resort solution.

The classloader bug is actually a bit of a hopefully sign in that it might explain some of the symptons, such as when the CPU is spinning there are actually zero threads running CFML code as best as I can tell in the dumps.  Apparently, even when the one of the CPU's is maxed out, the second CPU is capable of responding to all the normal trafic load our box receives on a normal day, so our users have not yet noticed a preformance problem with their applications.

I have no idea if this is a related problem or not,

But I have notices that our {serverName}-out logs are filling up with messages like this.

02/28 18:31:27 Information [cfthread-112] - BrowserFontProvider: Can't find font file for @Arial Unicode MS
02/28 18:31:27 Information [cfthread-112] - BrowserFontProvider: Can't find font file for @Arial Unicode MS
02/28 18:31:27 Information [cfthread-112] - BrowserFontProvider: Can't find font file for @Arial Unicode MS

Thousands of them, filling up dozens of log files every day.

Even if this is not related to the 50% cpu problem, this is not a good thing is it?

A Cameron wrote:

 

Well it's less than ideal ;-)

 

Is the font actually listed in CFAdmin?

I don't know if it is in CFAdmin or not, I guess I should look.

But I have already determined that it does not exist on the server.  Apparently this is a "special" MS version of Arial that would be installed if you have installed any of the common office productivity tools from the MS Office suite.   Something, of course, we have not done on our server.  If you don't, you are supposed to buy an Arial Unicode MS license to install it on a system without a licensed version of one of these Office tools.

I think I will just recomend that the application be reworked to use one of the other six flavors of Arial that are already on the server.  I am 99.99999% sure that this PDF generation code is not using anything that is only available in the Arial Unicode MS version of the font.

Well, silly me for searching for the Microsoft font only in the Microsoft directories on my server.

So now the question is why does ColdFusion not find it?  I've never done anything with the ColdFusion font settings, I would have thought that would have been one of the default locations it looks for a font.

Ok, I see an ArialUnicodeMS.fft file in my E:\JRun4\servers\general\cfusion.ear\cfusion.war\WEB-INF\cfusion\charting\fonts\ directory.

When tried to add that directory I got this error message:

• Either directory "E:\JRun4\servers\general\cfusion.ear\cfusion.war\WEB-INF\cfusion\charting\fonts\" does not contain usable font files, or all the font files in the directory have already been registered with the server. Acceptable font extensions are:
  
  TTF (True Type Font)
  TTC (True Type Collection)
  AFM (Adobe Font Metrics)

What is the difference between a fft and a ttf file?

Jochem van Dieten wrote:

I have no idea if not finding that font is even the cause of your problem. But why not copy that font to the Windows font dir, restart Cf and only if it helps start investigating why CF would need that font and why CF can not find it?

 

BTW, this is not the more recent full version of MS Arual Unicode. This is the old one that had a more liberal redistribution license.

I doubt this is the cause of the problem either as the times do not sync at all.  This is an effort to remove ~400,000 lines of log data being generated every day so that these log files are much easier to parse through for other information.

I guess I could try to move this to the font directory, but I am concerned that message only said it will take ttf files and this is a fft file.  Is that some type of compresed font file or something?  I have a low feeling that just moving the file would do much good.

NEW INFORMATION

Using process explore from Microsoft's sysinternals website.  We narrowed down the CPU spinning to a thread running Msvcr71.dll inside of the Jrun process inside the Jrunsvc process.  When we suspended this thread that was using 50% of the servers CPU, the problem resolved itself.  As best as I can tell this has not introduced any instability on the server.  But it is scheduled for a reboot soon anyway, so it should get a clean start at that time.

Searching for information on this dll I have never been aware of before, I found Jochem's web blog here:

http://jochem.vandieten.net/tag/msvcr71dll/

Does it seem resonable to try this fix for our problem, just grabbing the latest version of the Msvc71.dll file and putting it into the Jrun folder?  I presume this is the root Jrun folder or should it be somewhere deeper?  Does having a multi-server configuration matter?  Should we have the same Msvcr71.dll file in each Jrun server folder?

I'm uncertain this is a related problem as only one of our four JRun, aka ColdFusion, servers in our multi-home configuration has experienced this spinning problem.  But I am grasping at straws here.