We are doing some load testing on the new LCDS4.6 with RTMP channel and we noticed some pretty high CPU usage here from this method
The cause seems to be a call to HashMap$AbstractMapIterator.hasNext().
PS: LCDS3.1 did not have this problem.
Hi, please see attached.
|Thread Name||WorkManager.LCDSWorkManager : 2-rtmp-server-Reactor3|
|State||Waiting on condition|
|Java Stack|| at java/util/HashMap$AbstractMapIterator.hasNext(HashMap.java:112(Compil ed Code))|
at flex/messaging/socketserver/Reactor$ReactorImpl.run(Reactor.java:427( Compiled Code))
at flex/messaging/util/concurrent/AsynchBeansWorkManagerExecutor$WorkCom mandWrapper.run(AsynchBeansWorkManagerExecutor.java:186)
at java/security/AccessController.doPrivileged(AccessController.java:224 )
at com/ibm/ws/asynchbeans/J2EEContext$DoAsProxy.run(J2EEContext.java:335 )
at java/security/AccessController.doPrivileged(AccessController.java:251 (Compiled Code))
at com/ibm/ws/asynchbeans/WorkWithExecutionContextImpl.go(WorkWithExecut ionContextImpl.java:222)
|Native Stack|| (0x00002AAAAB06D7A2 [libj9prt24.so+0xe7a2])|
pthread_cond_wait+0xb9 (0x0000003849C0AEE9 [libpthread.so.0+0xaee9])
We have recently checked in a fix to the Reactor to work around a JDK bug. The symptom of that bug is also the same i.e. CPU usage spike, however the stack is different as shown below. Also, I don't think a waiting thread (as in your stack above) can consume CPU resources therefore please look for threads that are alive and running (not blocked or waiting) in your stack trace. The only place in ReactorImpl.run() that has an iterator is when we loop over the selected keys of the selector. That call is smack in the middle of the loop that calls select() in an infinite loop (due to the JDK bug), so maybe your profiler is showing the iterator as a hotspot. This is just a guess.
Take a look at the stack that's been reported to us:
"neo-nio-server-Reactor1" id=157 idx=0x264 tid=22394 prio=5 alive, in native, daemon
at sun/nio/ch/EPollArrayWrapper.epollWait(JIJI)I(Native Method)
at sun/nio/ch/EPollArrayWrapper.poll(EPollArrayWrapper.java:210)[inlined ]
at sun/nio/ch/EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)[inli ned]
at sun/nio/ch/SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)[inlined ]
^-- Holding lock: sun/nio/ch/Util$2@0x1157a6600[biased lock]
^-- Holding lock: java/util/Collections$UnmodifiableSet@0x1157a65f0[biased lock]
^-- Holding lock: sun/nio/ch/EPollSelectorImpl@0x1157a58b0[biased lock]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
Turns out it's an infamous JDK bug in epollWait() where the code doesn't block when it's supposed to.
The following links help to better understand the problem and the workaround:
http://bugs.sun.com/view_bug.do?bug_id=6403933 (official Oracle JDK bug)
http://www.java.net/node/670383 (another app [orbd] affected by the same issue)
If you wish to obtain this patch, please open a ticket with support. We haven't qualified this patch yet so I won't completely rule out changes to the workaround or vouch for its efficacy.