We have recently checked in a fix to the Reactor to work around a JDK bug. The symptom of that bug is also the same i.e. CPU usage spike, however the stack is different as shown below. Also, I don't think a waiting thread (as in your stack above) can consume CPU resources therefore please look for threads that are alive and running (not blocked or waiting) in your stack trace. The only place in ReactorImpl.run() that has an iterator is when we loop over the selected keys of the selector. That call is smack in the middle of the loop that calls select() in an infinite loop (due to the JDK bug), so maybe your profiler is showing the iterator as a hotspot. This is just a guess.
Take a look at the stack that's been reported to us:
"neo-nio-server-Reactor1" id=157 idx=0x264 tid=22394 prio=5 alive, in native, daemon
at sun/nio/ch/EPollArrayWrapper.epollWait(JIJI)I(Native Method)
at sun/nio/ch/EPollArrayWrapper.poll(EPollArrayWrapper.java:210)[inlined]
at sun/nio/ch/EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)[inlined]
at sun/nio/ch/SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)[inlined]
at sun/nio/ch/SelectorImpl.select(SelectorImpl.java:80)[inlined]
at sun/nio/ch/SelectorImpl.select(SelectorImpl.java:84)[optimized]
^-- Holding lock: sun/nio/ch/Util$2@0x1157a6600[biased lock]
^-- Holding lock: java/util/Collections$UnmodifiableSet@0x1157a65f0[biased lock]
^-- Holding lock: sun/nio/ch/EPollSelectorImpl@0x1157a58b0[biased lock]
at flex/messaging/socketserver/Reactor$ReactorImpl.run(Reactor.java:523)
at java/lang/Thread.run(Thread.java:662)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
-- end of trace
Turns out it's an infamous JDK bug in epollWait() where the code doesn't block when it's supposed to.
The following links help to better understand the problem and the workaround:
https://issues.apache.org/jira/browse/DIRMINA-678
https://github.com/netty/netty/issues/327
https://github.com/netty/netty/pull/565 (workaround)
http://bugs.sun.com/view_bug.do?bug_id=6403933 (official Oracle JDK bug)
https://forums.oracle.com/forums/thread.jspa?threadID=2424401&tstart=0 (unanswered)
http://www.java.net/node/670383 (another app [orbd] affected by the same issue)
If you wish to obtain this patch, please open a ticket with support. We haven't qualified this patch yet so I won't completely rule out changes to the workaround or vouch for its efficacy.
