Scaling JMS Adapter connecting to Remove JMS Provider

[radnz]

Hello,

We have many Flex clients (SDK 4.5 connecting to ADEP Data Services 4.6) creating consumers to receive pushed Server messages on a particular Topic. On several topics the consumers each set a unique selector that identifies the client connected (we use selectors rather than sub-topics as the JMS Adapter does not support sub-topics) - so effectively for these topics we have point-to-point messaging. This means we don't benefit from any multiplexing in the MessageBroker - each subscription gets passed through to the backend JMSAdapter.

It seems with our current configuration that this causes the JMS Adapter to create a new TopicConnection and TopicSession per client consumer – meaning that we rapidly run out of Server threads as new clients are added.

With 140 clients connected we see everything grind to a halt in ADEP/LCDS and the following thread counts - each client is causing individual ActiveMQ connections - even though we only have a handful of different Topics, but with different selectors on these topics per Flex client.

1935 ActiveMQ

1572 ActiveMQConnection

1 AsyncBatchingAppender

2 C

2 com

1 Concurrent

1 DestroyJavaVM

2 EmailAlertAppender

1 Finalizer

2 Gang

1 HashSessionScavenger-

1 HtmlAdaptorServer

17 InactivityMonitor

1 JmsLeafDestination

1 lcdsLive-Pool-

1 Low

1 nio-server-Acceptor

1 nio-server-IdleConnectionTimeoutThread-

8 nio-server-Reactor

16 nio-server-WorkerThread-

1 MetricExporter-

1 MultiThreadedHttpConnectionManager

1 NIOHTTPFlexSessionTimeoutThread-

8 qtp

1 RedeployManager

1 Reference

1 Signal

1 StreamingEndpoint-nio-amf-stream-ConnectionHeartbeatTimer

1 Surrogate

1 Thread-

1 Timer-

2 VM

1 Wrapper-Connection

1 Wrapper-Control-Event-Monitor

Ideally the JMS Adapter would have a single connection (TCP) to the external JMS Provider for each Topic and then de-multiplex messages for individual Flex Clients based upon the selectors set by the clients subscribed to that topic. Is there a configuration setting we can use to achieve this?

Configuration of our messaging destinations (example – they are all like this):

<destination id="abcd-response">

                        <adapter ref="jms"/>

                        <properties>

                                                <jms>

                                                                        <destination-type>topic</destination-type>

                                                                        <message-type>javax.jms.ObjectMessage</message-type>

                                                                        <connection-factory>java:comp/env/jms/flex/ClientRegionalBroker</connection-factory>

                                                                        <destination-jndi-name>java:comp/env/jms/topic/flex/ABCD_RESPONSE</destination-jndi-name>

                                                                        <delivery-mode>NON_PERSISTENT</delivery-mode>

                                                                        <message-priority>DEFAULT_PRIORITY</message-priority>

                                                                        <acknowledge-mode>AUTO_ACKNOWLEDGE</acknowledge-mode>

                                                                        <transacted-sessions>false</transacted-sessions>

                                                                        <delivery-settings>

                                                                                                <mode>async</mode>

                                                                        </delivery-settings>

                                                </jms>

                                                <network>

                                                                        <reliable>true</reliable>

                                                </network>

                        </properties>

</destination>

These destinations are using NIO End Points and Channels.

We first used the default “sync” mode JMS delivery setting and found that an additional thread was created for each client consumer topic (with this configuration we were getting "OutOfMemoryError: Unable to create native thread" errors due to our nproc limit of 8192).

Changing to use async mode solved this issue and reduced the number of threads dramatically. We can see in the LCDS logs that we are using the async message receiver.

[LCDS] 02/19/2012 23:03:29.841 [Service.Message.JMS] JMS consumer for JMS destination 'java:comp/env/jms/topic/flex/ABCD_RESPONSE is starting.

2012-02-19 23:03:30,174 INFO ActiveMQ Task-4 o.a.a.t.f.FailoverTransport Successfully connected to tcp://abcd.blah.com:9501?socket.tcpNoDelay=true&soTimeout=60000

[LCDS] 02/19/2012 23:03:31.505 [Service.Message.JMS] JMS consumer for JMS destination 'java:comp/env/jms/topic/flex/ ABCD_RESPONSE ' is using async message receiver.

However, we still see one ActiveMQ thread created for each Flex client consumer per topic and cannot scale.

We have tried replacing the ActiveMQ TopicConnectionFactory with a PooledConnection Factory and this has made a difference in the thread counts - but now we start to see timeouts.

Any configuration suggestions? From looking at the BlazeDS source, it looks like JMSTopicConsumer will always create a connection/session per topic subscription. This seems wasteful as sessions and particularly connections are expensive. Is there any mechanism to multiplex multiple subscriptions over a single session/connection out of LCDS?

Or perhaps there is a better way we could achieve the point-to-point connections from the external JMS Provider to the Flex Clients?

[paulvitic]

Could you resolve this issue. We are having similar porblems.