Can you attach stack trace etc. that can provide more details to us?
Hi, please see attached.
Thread Name WorkManager.LCDSWorkManager : 2-rtmp-server-Reactor3 State Waiting on condition Java Stack at java/util/HashMap$AbstractMapIterator.hasNext(HashMap.java:112(Compiled Code))
at flex/messaging/socketserver/Reactor$ReactorImpl.run(Reactor.java:427(Compiled Code))
at flex/messaging/util/concurrent/AsynchBeansWorkManagerExecutor$WorkCommandWrapper.run(Asyn chBeansWorkManagerExecutor.java:186)
at java/security/AccessController.doPrivileged(AccessController.java:251(Compiled Code))
at com/ibm/ws/asynchbeans/WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java: 222)
Native Stack (0x00002AAAAB06D7A2 [libj9prt24.so+0xe7a2])
pthread_cond_wait+0xb9 (0x0000003849C0AEE9 [libpthread.so.0+0xaee9])
So no clues on this?
We have recently checked in a fix to the Reactor to work around a JDK bug. The symptom of that bug is also the same i.e. CPU usage spike, however the stack is different as shown below. Also, I don't think a waiting thread (as in your stack above) can consume CPU resources therefore please look for threads that are alive and running (not blocked or waiting) in your stack trace. The only place in ReactorImpl.run() that has an iterator is when we loop over the selected keys of the selector. That call is smack in the middle of the loop that calls select() in an infinite loop (due to the JDK bug), so maybe your profiler is showing the iterator as a hotspot. This is just a guess.
Take a look at the stack that's been reported to us:
"neo-nio-server-Reactor1" id=157 idx=0x264 tid=22394 prio=5 alive, in native, daemon
at sun/nio/ch/EPollArrayWrapper.epollWait(JIJI)I(Native Method)
^-- Holding lock: sun/nio/ch/Util$2@0x1157a6600[biased lock]
^-- Holding lock: java/util/Collections$UnmodifiableSet@0x1157a65f0[biased lock]
^-- Holding lock: sun/nio/ch/EPollSelectorImpl@0x1157a58b0[biased lock]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
Turns out it's an infamous JDK bug in epollWait() where the code doesn't block when it's supposed to.
The following links help to better understand the problem and the workaround:
http://bugs.sun.com/view_bug.do?bug_id=6403933 (official Oracle JDK bug)
http://www.java.net/node/670383 (another app [orbd] affected by the same issue)
If you wish to obtain this patch, please open a ticket with support. We haven't qualified this patch yet so I won't completely rule out changes to the workaround or vouch for its efficacy.
We are just evaluating this product, can you please provide the patch here?
Not sure I can. Please send me a PM with your email and company name and I'll have someone from Product Management follow up with you.