6 Replies Latest reply on Mar 28, 2016 11:21 AM by GuitsBoy

    Coldfusion 10 update 18 cluster instance restarts in hung state while under load

    GuitsBoy Level 1

      So this problem has been driving me completely bonkers.

       

      CF10 update 18, RedHat 6.7 64 bit.  Java 1.7.093

       

      We noticed a similar problem with coldfusion 10 update 14 last year, and rolled back to update 13.  Recently we pushed out update 18, which fixed some issues, but did not fix CF restarting into a hung or dead state.

       

      I have gone as far as uninstalling CF10 completely, reinstalling a fresh CF10, doing the mandatory update, and then doing update 18.  Creating two fresh new instances, a new cluster, and configuring apache connector to use said cluster.  Its about as out of the box as I can make it.

       

      With this physical box quiesced in my hardware load balancer (not taking requests) I can happily restart the two CF instances all day long without much problem (maybe the occasional hang, but rare).   The trouble starts when I put the box back into production and allow it to start taking requests, and then restart a CF instance.   At least one CF instance will always come up in a hung or dead state.  I cannot get to tomcat under port 8501, and mod_jk shows the worker as down.  The coldfusion_out log shows "PM Information [localhost-startStop-2] - ColdFusion stopped" as the last entry.  Running a ps aux shows that the instance is indeed running.  Issuing a kill -3 <pid> does not log anything to std err or to coldfusion-out, or anywhere else for that matter.  When I try to stop or restart the instance again, I get the following message, and again, the instance starts in a hung state

       

      [root]# /opt/coldfusion10/web4cf1/bin/coldfusion restart               

      Restarting ColdFusion 10 server instance named web4cf1 ...

      Stopping ColdFusion 10 server instance named web4cf1, please wait

      Mar 22, 2016 2:24:09 PM com.adobe.coldfusion.launcher.Launcher stopServer

      SEVERE: Shutdown Port 8008is not active. Stop the server only after it is started.

      ColdFusion 10 server instance named web4cf1 has been stopped

      Starting ColdFusion 10 server instance named web4cf1 ...

      The ColdFusion 10 server instance named web4cf1 is starting up and will be available shortly.

      nohup: appending output to `nohup.out'

      ======================================================================

      ColdFusion 10 server instance named web4cf1 has been started.

      ColdFusion 10 will write logs to /opt/coldfusion10/web4cf1/logs/coldfusion-out.log

      ======================================================================

       

      I have tried various timeouts and settings in the server.xml files as well as workers.properties with no luck.   Here are my current files:

       

      worker.list=web4cluster,web4cf1,web4cf2

      worker.web4cluster.type=lb

      worker.web4cluster.balance_workers=web4cf1,web4cf2

      worker.web4cluster.sticky_session=true

      worker.web4cf1.type=ajp13

      worker.web4cf1.host=localhost

      worker.web4cf1.port=8041

      worker.web4cf1.connect_timeout=250

      worker.web4cf1.max_reuse_connections=250

      worker.web4cf1.connection_pool_size = 400

      worker.web4cf1.connection_pool_minsize= 200

      worker.web4cf1.connection_pool_timeout = 600

      worker.web4cf1.lbfactor=1

      worker.web4cf1.route=web4cf1

      worker.web4cf2.type=ajp13

      worker.web4cf2.host=localhost

      worker.web4cf2.port=8042

      worker.web4cf2.connect_timeout=250

      worker.web4cf2.max_reuse_connections=250

      worker.web4cf2.connection_pool_size = 400

      worker.web4cf2.connection_pool_minsize= 200

      worker.web4cf2.connection_pool_timeout = 600

      worker.web4cf2.lbfactor=1

      worker.web4cf2.route=web4cf2

       

       

      <Server port="8008" shutdown="SHUTDOWN">

        <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on">

        </Listener>

        <Listener className="org.apache.catalina.core.JasperListener">

        </Listener>

        <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener">

        </Listener>

        <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener">

        </Listener>

        <GlobalNamingResources>

          <Resource description="User database that can be updated and saved" name="UserDatabase" pathname="conf/tomcat-users.xml" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" type="org.apache.catalina.UserDatabase" auth="Container">

          </Resource>

        </GlobalNamingResources>

        <Service name="Catalina">

          <Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="150" namePrefix="catalina-exec-">

          </Executor>

          <Connector port="8501" protocol="org.apache.coyote.http11.Http11Protocol" connectionTimeout="2000" redirectPort="8446" executor="tomcatThreadPool" maxThreads="50">

          </Connector>

          <Connector port="8041" protocol="AJP/1.3" redirectPort="8446" tomcatAuthentication="false">

          </Connector>

          <Engine jvmRoute="web4cf1" name="Catalina" defaultHost="localhost">

            <Realm className="org.apache.catalina.realm.LockOutRealm">

              <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase">

              </Realm>

            </Realm>

            <Host name="localhost" autoDeploy="false" unpackWARs="true" appBase="webapps">

              <Valve pattern="%h %l %u %t &quot;%r&quot; %s %b" directory="logs" prefix="localhost_access_log." className="org.apache.catalina.valves.AccessLogValve" suffix=".txt" resolveHosts="false">

              </Valve>

            </Host>

            <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">

              <Manager notifyListenersOnReplication="true" expireSessionsOnShutdown="false" className="org.apache.catalina.ha.session.DeltaManager">

              </Manager>

              <Channel className="org.apache.catalina.tribes.group.GroupChannel">

                <Membership port="45564" dropTime="3000" address="228.0.240.104" className="org.apache.catalina.tribes.membership.McastService" frequency="500">

                </Membership>

                <Receiver port="4001" autoBind="100" address="auto" selectorTimeout="2000" maxThreads="6" className="org.apache.catalina.tribes.transport.nio.NioReceiver">

                </Receiver>

                <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">

                  <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender">

                  </Transport>

                </Sender>

                <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector">

                </Interceptor>

                <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor">

                </Interceptor>

              </Channel>

              <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="">

              </Valve>

              <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve">

              </Valve>

              <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener">

              </ClusterListener>

              <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener">

              </ClusterListener>

            </Cluster>

          </Engine>

        </Service>

      </Server>

       

       

      Now if I stop apache, or if I remove the load by removing the physical box from the hardware load balancer, the coldfusion instance will happily restart (after the port 8008 not active error).

       

       

       

      If I start the coldfusion instance manually from the command line, I do get some error messages when it fails:

       

      [root]# /opt/coldfusion10/jre/bin/java -classpath /opt/coldfusion10/web4cf2/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/web4cf2/bin/cf-bo otstrap.jar:/opt/coldfusion10/web4cf2/lib/oosdk/lib:/opt/coldfusion10/web4cf2/lib/oosdk/li b/*:/opt/coldfusion10/web4cf2/lib/oosdk/classes:/opt/coldfusion10/web4cf2/lib/oosdk/classe s/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/web4cf2 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/web4cf2 -Dcoldfusion.libPath=/opt/coldfusion10/web4cf2/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/web4cf2/lib/updates,/opt/coldfusion10/web4cf2/li b,/opt/coldfusion10/web4cf2/lib/axis2,/opt/coldfusion10/web4cf2/gateway/lib/,/opt/coldfusi on10/web4cf2/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/web4cf2/wwwroot/WEB-INF/cfform/ja rs,/var/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start

      Mar 22, 2016 3:45:36 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

      INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/coldfusion10/jre/lib/amd64/server:/opt/coldfusion10/jre/lib/amd64:/opt/coldfusion10/ jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

      Mar 22, 2016 3:45:36 PM org.apache.coyote.AbstractProtocol init

      INFO: Initializing ProtocolHandler ["http-bio-8502"]

      Mar 22, 2016 3:45:36 PM org.apache.coyote.AbstractProtocol init

      INFO: Initializing ProtocolHandler ["ajp-bio-8042"]

      Mar 22, 2016 3:45:36 PM org.apache.catalina.core.StandardService startInternal

      INFO: Starting service Catalina

      Mar 22, 2016 3:45:36 PM org.apache.catalina.core.StandardEngine startInternal

      INFO: Starting Servlet Engine: Apache Tomcat/7.0.64

      Mar 22, 2016 3:45:36 PM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal

      INFO: Cluster is about to start

      Mar 22, 2016 3:45:36 PM org.apache.catalina.tribes.transport.ReceiverBase bind

      INFO: Receiver Server Socket bound to:/10.10.240.104:4002

      Mar 22, 2016 3:45:36 PM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket

      INFO: Setting cluster mcast soTimeout to 500

      Mar 22, 2016 3:45:36 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

      INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:4

      Mar 22, 2016 3:45:36 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded

      INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 104}:4001,{10, 10, 240, 104},4001, alive=3337299, securePort=-1, UDP Port=-1, id={-65 -3 47 75 8 -4 74 2 -123 -42 -41 -22 84 88 60 82 }, payload={}, command={}, domain={}, ]

      Mar 22, 2016 3:45:37 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

      INFO: Done sleeping, membership established, start level:4

      Mar 22, 2016 3:45:37 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

      INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:8

      Mar 22, 2016 3:45:37 PM org.apache.catalina.tribes.io.BufferPool getBufferPool

      INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl

      Mar 22, 2016 3:45:38 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:38 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:38 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:38 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

      INFO: Done sleeping, membership established, start level:8

      Mar 22, 2016 3:45:38 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

      WARNING: Context manager doesn't exist:localhost#

      Mar 22, 2016 3:45:40 PM org.apache.catalina.startup.TldConfig execute

      INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

      Mar 22, 2016 3:45:41 PM org.apache.catalina.session.StandardSession tellNew

      SEVERE: Session event listener threw exception

      java.lang.NullPointerException

              at coldfusion.bootstrap.HttpFlexSessionBootstrap.getListener(HttpFlexSessionBootstrap.java:1 54)

              at coldfusion.bootstrap.HttpFlexSessionBootstrap.sessionCreated(HttpFlexSessionBootstrap.jav a:69)

              at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:422)

              at org.apache.catalina.session.StandardSession.setId(StandardSession.java:394)

              at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:275)

              at org.apache.catalina.ha.session.DeltaManager.handleSESSION_CREATED(DeltaManager.java:1317)

              at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1195)

              at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)

              at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListe ner.java:91)

              at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)

              at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)

              at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)

              at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

              at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailu reDetector.java:117)

              at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

              at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

              at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.ja va:252)

              at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:2 87)

              at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTa sk.java:210)

              at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:9 9)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:662)

      Mar 22, 2016 3:45:41 PM org.apache.catalina.session.StandardSession tellNew

      SEVERE: Session event listener threw exception

      java.lang.NullPointerException

              at coldfusion.bootstrap.HttpFlexSessionBootstrap.getListener(HttpFlexSessionBootstrap.java:1 54)

              at coldfusion.bootstrap.HttpFlexSessionBootstrap.sessionCreated(HttpFlexSessionBootstrap.jav a:69)

              at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:422)

              at org.apache.catalina.session.StandardSession.setId(StandardSession.java:394)

              at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:275)

              at org.apache.catalina.ha.session.DeltaManager.handleSESSION_CREATED(DeltaManager.java:1317)

              at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1195)

              at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)

              at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListe ner.java:91)

              at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)

              at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)

              at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)

              at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

              at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailu reDetector.java:117)

              at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

              at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

              at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.ja va:252)

              at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:2 87)

              at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTa sk.java:210)

              at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:9 9)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:662)

      Mar 22, 2016 3:45:41 PM org.apache.catalina.ha.session.DeltaManager startInternal

      INFO: Register manager localhost# to cluster element Engine with name Catalina

      Mar 22, 2016 3:45:41 PM org.apache.catalina.ha.session.DeltaManager startInternal

      INFO: Starting clustering manager at localhost#

      Mar 22, 2016 3:45:41 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions

      INFO: Manager [localhost#], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 104}:4001,{10, 10, 240, 104},4001, alive=3341800, securePort=-1, UDP Port=-1, id={-65 -3 47 75 8 -4 74 2 -123 -42 -41 -22 84 88 60 82 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds.

      Mar 22, 2016 3:45:47 PM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions

      INFO: Manager [localhost#]; session state send at 3/22/16 3:45 PM received in 5,614 ms.

       

       

       

       

       

       

       

      Does anyone have any suggestions on what I could try?

        • 1. Re: Coldfusion 10 update 18 cluster instance restarts in hung state while under load
          carl type3 Level 4

          Hi,

           

          I don't really know plus is not my particular environment but case it helps putting some idea's out for your consideration.

           

          Some of my Windows CF clusters had to be on Amazon EC2. Amazon does not support tomcat clustering using multicast traffic so those are setup using tomcat static membership. Since I had a good experience using static membership method I prefer to use that even in non Amazon situations. So perhaps you might like to alter your cluster method.

           

          One other that might be worth a try. Put your cluster settings in HOST section of server.xml. I don't know why but sometimes I have seen folks cluster problems solve but moving that. So server.xml would end like </Cluster></Host></Engine></Service></Server>

           

          I know that last option trends against common knowledge or documentation but this is a forum, it is just a passing idea to a poster who has had no responses yet, please don't flame me.

           

          HTH, Carl.

          • 2. Re: Coldfusion 10 update 18 cluster instance restarts in hung state while under load
            GuitsBoy Level 1

            Hi Carl -

             

            Thanks for the response.  Anyone willing to lend a helping hand or few ideas is certainly undeserving of being flamed no matter how against convention it may be, IMO.

             

            I started to try static clusters, but didnt get it to work in the hour or so I tried, so I really didnt give it a solid effort yet.  I also tried the blocking Bio instead of Nio listeners/responders, as well as changing to backup manager but no dice there.  Perhaps tomorrow Ill try static members again.  Ill also rearrange server.xml to see if that gets us anywhere.

             

            I was able to reproduce this error on a brand new fresh install of RHEL6.7 on a spare desktop.  Updated to CF 10 update 18, and created a new cluster as I always had.  I was able  to generate load using jmeter to hit a page that makes a large cfhttp call about 10 times a second.  This was enough load to get an instance to lockup on restart.  It restarts fine without the load but, once its working, it chokes on sending/receiving sessions.  I have increased all my timeouts up to 120 seconds, and still no good under any moderate load.

             

            I found some additional information by calling java directly from the shell to run CF.  This way standard error is spit out onto the screen, and I can see where its choking.  Ill post my latest info tomorrow when I get back to work.

             

            Anyway, Thanks again for the suggestions.

            -T

            • 3. Re: Coldfusion 10 update 18 cluster instance restarts in hung state while under load
              GuitsBoy Level 1

              When I start CF manually via command line, here's what I see.

               

              This is instance 1:

              Instance 1

              [root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf1/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf1/bin/cf-boot strap.jar:/opt/coldfusion10/newcf1/lib/oosdk/lib:/opt/coldfusion10/newcf1/lib/oosdk/lib/*: /opt/coldfusion10/newcf1/lib/oosdk/classes:/opt/coldfusion10/newcf1/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf1 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf1 -Dcoldfusion.libPath=/opt/coldfusion10/newcf1/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf1/lib/updates,/opt/coldfusion10/newcf1/lib, /opt/coldfusion10/newcf1/lib/axis2,/opt/coldfusion10/newcf1/gateway/lib/,/opt/coldfusion10 /newcf1/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf1/wwwroot/WEB-INF/cfform/jars,/va r/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start

              Mar 24, 2016 3:30:18 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

              INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

              Mar 24, 2016 3:30:18 AM org.apache.coyote.AbstractProtocol init

              INFO: Initializing ProtocolHandler ["http-bio-8503"]

              Mar 24, 2016 3:30:18 AM org.apache.coyote.AbstractProtocol init

              INFO: Initializing ProtocolHandler ["ajp-bio-8015"]

              Mar 24, 2016 3:30:19 AM org.apache.catalina.core.StandardService startInternal

              INFO: Starting service Catalina

              Mar 24, 2016 3:30:19 AM org.apache.catalina.core.StandardEngine startInternal

              INFO: Starting Servlet Engine: Apache Tomcat/7.0.64

              Mar 24, 2016 3:30:19 AM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal

              INFO: Cluster is about to start

              Mar 24, 2016 3:30:19 AM org.apache.catalina.tribes.transport.ReceiverBase bind

              INFO: Receiver Server Socket bound to:/10.10.2.84:4001

              Mar 24, 2016 3:30:19 AM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket

              INFO: Setting cluster mcast soTimeout to 5000

              Mar 24, 2016 3:30:19 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

              INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:4

              Mar 24, 2016 3:30:29 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

              INFO: Done sleeping, membership established, start level:4

              Mar 24, 2016 3:30:29 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

              INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:8

              Mar 24, 2016 3:30:39 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

              INFO: Done sleeping, membership established, start level:8

              Mar 24, 2016 3:30:41 AM org.apache.catalina.startup.TldConfig execute

              INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

              Mar 24, 2016 3:30:41 AM org.apache.catalina.ha.session.DeltaManager startInternal

              INFO: Register manager localhost# to cluster element Engine with name Catalina

              Mar 24, 2016 3:30:41 AM org.apache.catalina.ha.session.DeltaManager startInternal

              INFO: Starting clustering manager at localhost#

              Mar 24, 2016 3:30:41 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions

              INFO: Manager [localhost#]: skipping state transfer. No members active in cluster group.

              Mar 24, 2016 3:30:42 AM org.apache.catalina.core.ApplicationContext log

              INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services

              Mar 24, 2016 3:30:42 AM org.apache.catalina.core.ApplicationContext log

              INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 24.95-b01

              Mar 24, 2016 3:30:48 AM org.apache.catalina.core.ApplicationContext log

              INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646

              Mar 24, 2016 3:30:52 AM org.apache.catalina.ha.session.JvmRouteBinderValve startInternal

              INFO: JvmRouteBinderValve started

              Mar 24, 2016 3:30:52 AM org.apache.coyote.AbstractProtocol start

              INFO: Starting ProtocolHandler ["http-bio-8503"]

              Mar 24, 2016 3:30:52 AM org.apache.coyote.AbstractProtocol start

              INFO: Starting ProtocolHandler ["ajp-bio-8015"]

              Mar 24, 2016 3:30:52 AM com.adobe.coldfusion.launcher.Launcher run

              INFO: Server startup in 35420 ms

              Mar 24, 2016 3:31:19 AM org.apache.catalina.tribes.io.BufferPool getBufferPool

              INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded

              INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=10027, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]

              Mar 24, 2016 3:32:09 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=20035, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:09 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=20035, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:09 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:837)

                      at org.apache.catalina.ha.session.DeltaManager.sendSessions(DeltaManager.java:1429)

                      at org.apache.catalina.ha.session.DeltaManager.handleGET_ALL_SESSIONS(DeltaManager.java:1357 )

                      at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1183)

                      at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)

                      at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListe ner.java:91)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)

                      at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailu reDetector.java:117)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.ja va:252)

                      at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:2 87)

                      at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTa sk.java:210)

                      at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:9 9)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:497)

                      at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:487)

                      at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:463)

                      at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:450)

                      at org.apache.catalina.connector.Request.doGetSession(Request.java:3029)

                      at org.apache.catalina.connector.Request.getSession(Request.java:2386)

                      at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:897)

                      at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:974)

                      at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1067)

                      at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:361)

                      at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)

                      at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

                      at coldfusion.filter.PathFilter.invoke(PathFilter.java:112)

                      at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)

                      at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8)

                      at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)

                      at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:58)

                      at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

                      at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

                      at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)

                      at coldfusion.CfmServlet.service(CfmServlet.java:219)

                      at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

                      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:303)

                      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

                      at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 )

                      at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

                      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:241)

                      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

                      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

                      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

                      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)

                      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)

                      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

                      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)

                      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

                      at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:218)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:333)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:44 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=85037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:44 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=85037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:44 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:51 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:51 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:51 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:52 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:52 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:52 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:32:52 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify.

              Mar 24, 2016 3:32:52 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

              INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]]

              Mar 24, 2016 3:32:52 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

              SEVERE: Unable to send message through cluster sender.

              org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                      at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                      at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                      at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                      at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)

                      at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValv e.java:506)

                      at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java: 419)

                      at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)

                      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

                      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:625)

                      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

                      at java.lang.Thread.run(Thread.java:745)

               

              Instance 2

              [root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf2/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf2/bin/cf-boot strap.jar:/opt/coldfusion10/newcf2/lib/oosdk/lib:/opt/coldfusion10/newcf2/lib/oosdk/lib/*: /opt/coldfusion10/newcf2/lib/oosdk/classes:/opt/coldfusion10/newcf2/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf2 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf2 -Dcoldfusion.libPath=/opt/coldfusion10/newcf2/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf2/lib/updates,/opt/coldfusion10/newcf2/lib, /opt/coldfusion10/newcf2/lib/axis2,/opt/coldfusion10/newcf2/gateway/lib/,/opt/coldfusion10 /newcf2/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf2/wwwroot/WEB-INF/cfform/jars,/va r/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start

              Mar 24, 2016 3:31:15 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

              INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

              Mar 24, 2016 3:31:15 AM org.apache.coyote.AbstractProtocol init

              INFO: Initializing ProtocolHandler ["http-bio-8504"]

              Mar 24, 2016 3:31:15 AM org.apache.coyote.AbstractProtocol init

              INFO: Initializing ProtocolHandler ["ajp-bio-8016"]

              Mar 24, 2016 3:31:16 AM org.apache.catalina.core.StandardService startInternal

              INFO: Starting service Catalina

              Mar 24, 2016 3:31:16 AM org.apache.catalina.core.StandardEngine startInternal

              INFO: Starting Servlet Engine: Apache Tomcat/7.0.64

              Mar 24, 2016 3:31:16 AM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal

              INFO: Cluster is about to start

              Mar 24, 2016 3:31:16 AM org.apache.catalina.tribes.transport.ReceiverBase bind

              INFO: Receiver Server Socket bound to:/10.10.2.84:4002

              Mar 24, 2016 3:31:16 AM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket

              INFO: Setting cluster mcast soTimeout to 5000

              Mar 24, 2016 3:31:16 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

              INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:4

              Mar 24, 2016 3:31:19 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded

              INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4001,{10, 10, 2, 84},4001, alive=60017, securePort=-1, UDP Port=-1, id={66 70 35 114 47 105 70 -54 -66 113 -45 -14 -100 109 -50 -63 }, payload={}, command={}, domain={}, ]

              Mar 24, 2016 3:31:26 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

              INFO: Done sleeping, membership established, start level:4

              Mar 24, 2016 3:31:26 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

              INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:8

              Mar 24, 2016 3:31:26 AM org.apache.catalina.tribes.io.BufferPool getBufferPool

              INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

              INFO: Done sleeping, membership established, start level:8

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

              WARNING: Context manager doesn't exist:localhost#

              Mar 24, 2016 3:31:39 AM org.apache.catalina.startup.TldConfig execute

              INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

              Mar 24, 2016 3:31:39 AM org.apache.catalina.session.StandardSession tellNew

              SEVERE: Session event listener threw exception

              java.lang.NullPointerException

                      at coldfusion.bootstrap.HttpFlexSessionBootstrap.getListener(HttpFlexSessionBootstrap.java:1 54)

                      at coldfusion.bootstrap.HttpFlexSessionBootstrap.sessionCreated(HttpFlexSessionBootstrap.jav a:69)

                      at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:422)

                      at org.apache.catalina.session.StandardSession.setId(StandardSession.java:394)

                      at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:275)

                      at org.apache.catalina.ha.session.DeltaManager.handleSESSION_CREATED(DeltaManager.java:1317)

                      at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1195)

                      at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)

                      at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListe ner.java:91)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)

                      at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)

                      at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                      at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailu reDetector.java:117)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                      at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                      at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.ja va:252)

                      at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:2 87)

                      at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTa sk.java:210)

                      at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:9 9)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                      at java.lang.Thread.run(Thread.java:745)

               

               

              Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.DeltaManager startInternal

              INFO: Register manager localhost# to cluster element Engine with name Catalina

              Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.DeltaManager startInternal

              INFO: Starting clustering manager at localhost#

              Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions

              INFO: Manager [localhost#], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4001,{10, 10, 2, 84},4001, alive=80018, securePort=-1, UDP Port=-1, id={66 70 35 114 47 105 70 -54 -66 113 -45 -14 -100 109 -50 -63 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds.

              Mar 24, 2016 3:32:09 AM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions

              INFO: Manager [localhost#]; session state send at 3/24/16 3:31 AM received in 30,167 ms.

              • 4. Re: Coldfusion 10 update 18 cluster instance restarts in hung state while under load
                GuitsBoy Level 1

                I'm running into similar issues with static clusters as well, but it does seem more likely to continue starting up when encountering an error:

                 

                Instance 1

                [root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf1/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf1/bin/cf-boot strap.jar:/opt/coldfusion10/newcf1/lib/oosdk/lib:/opt/coldfusion10/newcf1/lib/oosdk/lib/*: /opt/coldfusion10/newcf1/lib/oosdk/classes:/opt/coldfusion10/newcf1/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf1 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf1 -Dcoldfusion.libPath=/opt/coldfusion10/newcf1/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf1/lib/updates,/opt/coldfusion10/newcf1/lib, /opt/coldfusion10/newcf1/lib/axis2,/opt/coldfusion10/newcf1/gateway/lib/,/opt/coldfusion10 /newcf1/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf1/wwwroot/WEB-INF/cfform/jars,/va r/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start

                Mar 24, 2016 6:02:36 AM org.apache.tomcat.util.IntrospectionUtils setProperty

                WARNING: IntrospectionUtils: InvocationTargetException for class org.apache.catalina.tribes.membership.StaticMember uniqueId={1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,0})

                java.lang.reflect.InvocationTargetException

                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                        at java.lang.reflect.Method.invoke(Method.java:606)

                        at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:303)

                        at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:282)

                        at org.apache.tomcat.util.digester.SetPropertiesRule.begin(SetPropertiesRule.java:215)

                        at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1282)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXPars er.java:509)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(Abstrac tXMLDocumentParser.java:182)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(X MLDocumentFragmentScannerImpl.java:1344)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDri ver.next(XMLDocumentFragmentScannerImpl.java:2787)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImp l.java:606)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDo cumentFragmentScannerImpl.java:510)

                        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.ja va:848)

                        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.ja va:777)

                        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java :1213)

                        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.j ava:642)

                        at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1555)

                        at com.adobe.coldfusion.launcher.Launcher.run(Launcher.java:200)

                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                        at java.lang.reflect.Method.invoke(Method.java:606)

                        at com.adobe.coldfusion.bootstrap.Bootstrap.init(Bootstrap.java:86)

                        at com.adobe.coldfusion.bootstrap.Bootstrap.main(Bootstrap.java:161)

                Caused by: java.lang.NullPointerException

                        at org.apache.catalina.tribes.membership.MemberImpl.getData(MemberImpl.java:250)

                        at org.apache.catalina.tribes.membership.MemberImpl.setUniqueId(MemberImpl.java:616)

                        at org.apache.catalina.tribes.membership.StaticMember.setUniqueId(StaticMember.java:74)

                        ... 27 more

                 

                 

                Mar 24, 2016 6:02:36 AM org.apache.tomcat.util.digester.SetPropertiesRule begin

                WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Cluster/Channel/Interceptor/Member} Setting property 'uniqueId' to '{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,0}' did not find a matching property.

                Mar 24, 2016 6:02:36 AM org.apache.tomcat.util.IntrospectionUtils setProperty

                WARNING: IntrospectionUtils: InvocationTargetException for class org.apache.catalina.tribes.membership.StaticMember domain=static-cluster)

                java.lang.reflect.InvocationTargetException

                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                        at java.lang.reflect.Method.invoke(Method.java:606)

                        at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:303)

                        at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:282)

                        at org.apache.tomcat.util.digester.SetPropertiesRule.begin(SetPropertiesRule.java:215)

                        at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1282)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXPars er.java:509)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(Abstrac tXMLDocumentParser.java:182)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(X MLDocumentFragmentScannerImpl.java:1344)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDri ver.next(XMLDocumentFragmentScannerImpl.java:2787)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImp l.java:606)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDo cumentFragmentScannerImpl.java:510)

                        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.ja va:848)

                        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.ja va:777)

                        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java :1213)

                        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.j ava:642)

                        at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1555)

                        at com.adobe.coldfusion.launcher.Launcher.run(Launcher.java:200)

                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                        at java.lang.reflect.Method.invoke(Method.java:606)

                        at com.adobe.coldfusion.bootstrap.Bootstrap.init(Bootstrap.java:86)

                        at com.adobe.coldfusion.bootstrap.Bootstrap.main(Bootstrap.java:161)

                Caused by: java.lang.NullPointerException

                        at org.apache.catalina.tribes.membership.MemberImpl.getData(MemberImpl.java:250)

                        at org.apache.catalina.tribes.membership.MemberImpl.setDomain(MemberImpl.java:636)

                        at org.apache.catalina.tribes.membership.StaticMember.setDomain(StaticMember.java:65)

                        ... 27 more

                 

                 

                Mar 24, 2016 6:02:36 AM org.apache.tomcat.util.digester.SetPropertiesRule begin

                WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Cluster/Channel/Interceptor/Member} Setting property 'domain' to 'static-cluster' did not find a matching property.

                Mar 24, 2016 6:02:36 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

                INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

                Mar 24, 2016 6:02:36 AM org.apache.coyote.AbstractProtocol init

                INFO: Initializing ProtocolHandler ["http-bio-8503"]

                Mar 24, 2016 6:02:36 AM org.apache.coyote.AbstractProtocol init

                INFO: Initializing ProtocolHandler ["ajp-bio-8015"]

                Mar 24, 2016 6:02:36 AM org.apache.catalina.core.StandardService startInternal

                INFO: Starting service Catalina

                Mar 24, 2016 6:02:36 AM org.apache.catalina.core.StandardEngine startInternal

                INFO: Starting Servlet Engine: Apache Tomcat/7.0.64

                Mar 24, 2016 6:02:36 AM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal

                INFO: Cluster is about to start

                Mar 24, 2016 6:02:36 AM org.apache.catalina.tribes.transport.ReceiverBase bind

                INFO: Receiver Server Socket bound to:/10.10.2.84:4001

                Mar 24, 2016 6:02:38 AM org.apache.catalina.startup.TldConfig execute

                INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

                Mar 24, 2016 6:02:38 AM org.apache.catalina.ha.session.DeltaManager startInternal

                INFO: Register manager  to cluster element Host with name localhost

                Mar 24, 2016 6:02:38 AM org.apache.catalina.ha.session.DeltaManager startInternal

                INFO: Starting clustering manager at

                Mar 24, 2016 6:02:38 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions

                INFO: Manager []: skipping state transfer. No members active in cluster group.

                Mar 24, 2016 6:02:40 AM org.apache.catalina.core.ApplicationContext log

                INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services

                Mar 24, 2016 6:02:40 AM org.apache.catalina.core.ApplicationContext log

                INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 24.95-b01

                Mar 24, 2016 6:02:45 AM org.apache.catalina.core.ApplicationContext log

                INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646

                Mar 24, 2016 6:02:49 AM org.apache.catalina.ha.session.JvmRouteBinderValve startInternal

                INFO: JvmRouteBinderValve started

                Mar 24, 2016 6:02:49 AM org.apache.coyote.AbstractProtocol start

                INFO: Starting ProtocolHandler ["http-bio-8503"]

                Mar 24, 2016 6:02:49 AM org.apache.coyote.AbstractProtocol start

                INFO: Starting ProtocolHandler ["ajp-bio-8015"]

                Mar 24, 2016 6:02:49 AM com.adobe.coldfusion.launcher.Launcher run

                INFO: Server startup in 15105 ms

                Mar 24, 2016 6:03:06 AM org.apache.catalina.tribes.io.BufferPool getBufferPool

                INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl

                Mar 24, 2016 6:03:06 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded

                INFO: Replication member added:org.apache.catalina.tribes.membership.StaticMember[tcp://10.10.2.84:4002,10.10.2.84 ,4002, alive=0, securePort=-1, UDP Port=-1, id={1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 }, payload={}, command={}, domain={115 116 97 116 105 99 45 99 108 ...(14)}, ]

                Mar 24, 2016 6:03:06 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector performBasicCheck

                INFO: Suspect member, confirmed alive.[org.apache.catalina.tribes.membership.StaticMember[tcp://10.10.2.84:4002,10.10.2.8 4,4002, alive=0, securePort=-1, UDP Port=-1, id={1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 }, payload={}, command={}, domain={115 116 97 116 105 99 45 99 108 ...(14)}, ]]

                Mar 24, 2016 6:03:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

                INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=1458813786146, securePort=-1, UDP Port=-1, id={-41 123 58 104 -22 -27 72 81 -103 -22 29 91 -58 -82 -98 23 }, payload={}, command={}, domain={}, ]] message. Will verify.

                Mar 24, 2016 6:03:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared

                INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=1458813786146, securePort=-1, UDP Port=-1, id={-41 123 58 104 -22 -27 72 81 -103 -22 29 91 -58 -82 -98 23 }, payload={}, command={}, domain={}, ]]

                Mar 24, 2016 6:03:13 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

                SEVERE: Unable to send message through cluster sender.

                org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:837)

                        at org.apache.catalina.ha.session.DeltaManager.sendSessions(DeltaManager.java:1429)

                        at org.apache.catalina.ha.session.DeltaManager.handleGET_ALL_SESSIONS(DeltaManager.java:1357 )

                        at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1183)

                        at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)

                        at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListe ner.java:91)

                        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)

                        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)

                        at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)

                        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailu reDetector.java:117)

                        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                        at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.ja va:252)

                        at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:2 87)

                        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTa sk.java:210)

                        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:9 9)

                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                        at java.lang.Thread.run(Thread.java:745)

                 

                Instance 2

                [root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf2/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf2/bin/cf-boot strap.jar:/opt/coldfusion10/newcf2/lib/oosdk/lib:/opt/coldfusion10/newcf2/lib/oosdk/lib/*: /opt/coldfusion10/newcf2/lib/oosdk/classes:/opt/coldfusion10/newcf2/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf2 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf2 -Dcoldfusion.libPath=/opt/coldfusion10/newcf2/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf2/lib/updates,/opt/coldfusion10/newcf2/lib, /opt/coldfusion10/newcf2/lib/axis2,/opt/coldfusion10/newcf2/gateway/lib/,/opt/coldfusion10 /newcf2/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf2/wwwroot/WEB-INF/cfform/jars,/va r/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start

                Mar 24, 2016 6:03:05 AM org.apache.tomcat.util.IntrospectionUtils setProperty

                WARNING: IntrospectionUtils: InvocationTargetException for class org.apache.catalina.tribes.membership.StaticMember uniqueId={0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15})

                java.lang.reflect.InvocationTargetException

                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                        at java.lang.reflect.Method.invoke(Method.java:606)

                        at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:303)

                        at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:282)

                        at org.apache.tomcat.util.digester.SetPropertiesRule.begin(SetPropertiesRule.java:215)

                        at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1282)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXPars er.java:509)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(Abstrac tXMLDocumentParser.java:182)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(X MLDocumentFragmentScannerImpl.java:1344)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDri ver.next(XMLDocumentFragmentScannerImpl.java:2787)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImp l.java:606)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDo cumentFragmentScannerImpl.java:510)

                        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.ja va:848)

                        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.ja va:777)

                        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java :1213)

                        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.j ava:642)

                        at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1555)

                        at com.adobe.coldfusion.launcher.Launcher.run(Launcher.java:200)

                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                        at java.lang.reflect.Method.invoke(Method.java:606)

                        at com.adobe.coldfusion.bootstrap.Bootstrap.init(Bootstrap.java:86)

                        at com.adobe.coldfusion.bootstrap.Bootstrap.main(Bootstrap.java:161)

                Caused by: java.lang.NullPointerException

                        at org.apache.catalina.tribes.membership.MemberImpl.getData(MemberImpl.java:250)

                        at org.apache.catalina.tribes.membership.MemberImpl.setUniqueId(MemberImpl.java:616)

                        at org.apache.catalina.tribes.membership.StaticMember.setUniqueId(StaticMember.java:74)

                        ... 27 more

                 

                 

                Mar 24, 2016 6:03:05 AM org.apache.tomcat.util.digester.SetPropertiesRule begin

                WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Cluster/Channel/Interceptor/Member} Setting property 'uniqueId' to '{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}' did not find a matching property.

                Mar 24, 2016 6:03:05 AM org.apache.tomcat.util.IntrospectionUtils setProperty

                WARNING: IntrospectionUtils: InvocationTargetException for class org.apache.catalina.tribes.membership.StaticMember domain=static-cluster)

                java.lang.reflect.InvocationTargetException

                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                        at java.lang.reflect.Method.invoke(Method.java:606)

                        at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:303)

                        at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:282)

                        at org.apache.tomcat.util.digester.SetPropertiesRule.begin(SetPropertiesRule.java:215)

                        at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1282)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXPars er.java:509)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(Abstrac tXMLDocumentParser.java:182)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(X MLDocumentFragmentScannerImpl.java:1344)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDri ver.next(XMLDocumentFragmentScannerImpl.java:2787)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImp l.java:606)

                        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDo cumentFragmentScannerImpl.java:510)

                        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.ja va:848)

                        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.ja va:777)

                        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)

                        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java :1213)

                        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.j ava:642)

                        at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1555)

                        at com.adobe.coldfusion.launcher.Launcher.run(Launcher.java:200)

                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                        at java.lang.reflect.Method.invoke(Method.java:606)

                        at com.adobe.coldfusion.bootstrap.Bootstrap.init(Bootstrap.java:86)

                        at com.adobe.coldfusion.bootstrap.Bootstrap.main(Bootstrap.java:161)

                Caused by: java.lang.NullPointerException

                        at org.apache.catalina.tribes.membership.MemberImpl.getData(MemberImpl.java:250)

                        at org.apache.catalina.tribes.membership.MemberImpl.setDomain(MemberImpl.java:636)

                        at org.apache.catalina.tribes.membership.StaticMember.setDomain(StaticMember.java:65)

                        ... 27 more

                 

                 

                Mar 24, 2016 6:03:05 AM org.apache.tomcat.util.digester.SetPropertiesRule begin

                WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Cluster/Channel/Interceptor/Member} Setting property 'domain' to 'static-cluster' did not find a matching property.

                Mar 24, 2016 6:03:05 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

                INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

                Mar 24, 2016 6:03:05 AM org.apache.coyote.AbstractProtocol init

                INFO: Initializing ProtocolHandler ["http-bio-8504"]

                Mar 24, 2016 6:03:05 AM org.apache.coyote.AbstractProtocol init

                INFO: Initializing ProtocolHandler ["ajp-bio-8016"]

                Mar 24, 2016 6:03:06 AM org.apache.catalina.core.StandardService startInternal

                INFO: Starting service Catalina

                Mar 24, 2016 6:03:06 AM org.apache.catalina.core.StandardEngine startInternal

                INFO: Starting Servlet Engine: Apache Tomcat/7.0.64

                Mar 24, 2016 6:03:06 AM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal

                INFO: Cluster is about to start

                Mar 24, 2016 6:03:06 AM org.apache.catalina.tribes.transport.ReceiverBase bind

                INFO: Receiver Server Socket bound to:/10.10.2.84:4002

                Mar 24, 2016 6:03:06 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded

                INFO: Replication member added:org.apache.catalina.tribes.membership.StaticMember[tcp://10.10.2.84:4001,10.10.2.84 ,4001, alive=0, securePort=-1, UDP Port=-1, id={0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 }, payload={}, command={}, domain={115 116 97 116 105 99 45 99 108 ...(14)}, ]

                Mar 24, 2016 6:03:06 AM org.apache.catalina.tribes.io.BufferPool getBufferPool

                INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl

                Mar 24, 2016 6:03:08 AM org.apache.catalina.startup.TldConfig execute

                INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

                Mar 24, 2016 6:03:08 AM org.apache.catalina.ha.session.DeltaManager startInternal

                INFO: Register manager  to cluster element Host with name localhost

                Mar 24, 2016 6:03:08 AM org.apache.catalina.ha.session.DeltaManager startInternal

                INFO: Starting clustering manager at

                Mar 24, 2016 6:03:08 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions

                INFO: Manager [], requesting session state from org.apache.catalina.tribes.membership.StaticMember[tcp://10.10.2.84:4001,10.10.2.84,4001, alive=0, securePort=-1, UDP Port=-1, id={0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 }, payload={}, command={}, domain={115 116 97 116 105 99 45 99 108 ...(14)}, ]. This operation will timeout if no session state has been received within 60 seconds.

                Mar 24, 2016 6:03:13 AM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions

                INFO: Manager []; session state send at 3/24/16 6:03 AM received in 5,118 ms.

                Mar 24, 2016 6:03:14 AM org.apache.catalina.tribes.transport.nio.NioReplicationTask sendAck

                WARNING: Unable to send ACK back through channel, channel disconnected?: null

                Mar 24, 2016 6:03:14 AM org.apache.catalina.core.ApplicationContext log

                INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services

                Mar 24, 2016 6:03:14 AM org.apache.catalina.core.ApplicationContext log

                INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 24.95-b01

                Mar 24, 2016 6:03:19 AM org.apache.catalina.core.ApplicationContext log

                INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646

                Mar 24, 2016 6:03:24 AM org.apache.catalina.ha.session.JvmRouteBinderValve startInternal

                INFO: JvmRouteBinderValve started

                Mar 24, 2016 6:03:24 AM org.apache.coyote.AbstractProtocol start

                INFO: Starting ProtocolHandler ["http-bio-8504"]

                Mar 24, 2016 6:03:24 AM org.apache.coyote.AbstractProtocol start

                INFO: Starting ProtocolHandler ["ajp-bio-8016"]

                Mar 24, 2016 6:03:24 AM com.adobe.coldfusion.launcher.Launcher run

                INFO: Server startup in 20150 ms

                 

                I cant seem to figure out how to get rid of that error about the static member's uniqueID.  The syntax looks good, I dont get it.

                 

                 

                Header 1

                <Server port="8010" shutdown="SHUTDOWN">

                  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on">

                  </Listener>

                  <Listener className="org.apache.catalina.core.JasperListener">

                  </Listener>

                  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener">

                  </Listener>

                  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener">

                  </Listener>

                  <GlobalNamingResources>

                    <Resource description="User database that can be updated and saved" name="UserDatabase" pathname="conf/tomcat-users.xml" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" type="org.apache.catalina.UserDatabase" auth="Container">

                    </Resource>

                  </GlobalNamingResources>

                  <Service name="Catalina">

                    <Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="150" namePrefix="catalina-exec-">

                    </Executor>

                    <Connector port="8503" protocol="org.apache.coyote.http11.Http11Protocol" connectionTimeout="20000" redirectPort="8448" executor="tomcatThreadPool" maxThreads="50">

                    </Connector>

                    <Connector port="8015" protocol="AJP/1.3" redirectPort="8448" tomcatAuthentication="false">

                    </Connector>

                    <Engine jvmRoute="newcf1" name="Catalina" defaultHost="localhost">

                      <Realm className="org.apache.catalina.realm.LockOutRealm">

                        <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase">

                        </Realm>

                      </Realm>

                      <Host name="localhost" autoDeploy="false" unpackWARs="true" appBase="webapps">

                        <Valve pattern="%h %l %u %t &quot;%r&quot; %s %b" directory="logs" prefix="localhost_access_log." className="org.apache.catalina.valves.AccessLogValve" suffix=".txt" resolveHosts="false">

                        </Valve>

                      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelStartOptions="3" channelSendOptions="6">

                        <Manager notifyListenersOnReplication="true" expireSessionsOnShutdown="false" className="org.apache.catalina.ha.session.DeltaManager">

                        </Manager>

                        <Channel className="org.apache.catalina.tribes.group.GroupChannel">

                          <!--<Membership port="45564" dropTime="10000" address="228.0.0.4" className="org.apache.catalina.tribes.membership.McastService" frequency="5000">

                          </Membership>-->

                          <Receiver port="4001" autoBind="100" address="10.10.2.84" selectorTimeout="5000" maxThreads="6" className="org.apache.catalina.tribes.transport.nio.NioReceiver" timeout="30000"></Receiver>

                          <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">

                            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"></Transport>

                          </Sender>

                          <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"></Intercepto r>

                          <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"></ Interceptor>

                          <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">

                <!--        <LocalMember port="4001" className="org.apache.catalina.tribes.membership.StaticMember" securePort="-1" uniqueId="{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}" domain="static-cluster" host="10.10.2.84"/>-->

                            <Member      port="4002" className="org.apache.catalina.tribes.membership.StaticMember" securePort="-1" uniqueId="{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,0}" domain="static-cluster" host="10.10.2.84"/>

                          </Interceptor>

                        </Channel>

                        <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""></Valve>

                        <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"></Valve>

                        <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"></ClusterListe ner>

                        <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"></ClusterListener>

                      </Cluster>

                      </Host>

                    </Engine>

                  </Service>

                </Server>

                <Server port="8011" shutdown="SHUTDOWN">

                  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on">

                  </Listener>

                  <Listener className="org.apache.catalina.core.JasperListener">

                  </Listener>

                  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener">

                  </Listener>

                  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener">

                  </Listener>

                  <GlobalNamingResources>

                    <Resource description="User database that can be updated and saved" name="UserDatabase" pathname="conf/tomcat-users.xml" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" type="org.apache.catalina.UserDatabase" auth="Container">

                    </Resource>

                  </GlobalNamingResources>

                  <Service name="Catalina">

                    <Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="150" namePrefix="catalina-exec-">

                    </Executor>

                    <Connector port="8504" protocol="org.apache.coyote.http11.Http11Protocol" connectionTimeout="20000" redirectPort="8449" executor="tomcatThreadPool" maxThreads="50">

                    </Connector>

                    <Connector port="8016" protocol="AJP/1.3" redirectPort="8449" tomcatAuthentication="false">

                    </Connector>

                    <Engine jvmRoute="newcf2" name="Catalina" defaultHost="localhost">

                      <Realm className="org.apache.catalina.realm.LockOutRealm">

                        <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase">

                        </Realm>

                      </Realm>

                      <Host name="localhost" autoDeploy="false" unpackWARs="true" appBase="webapps">

                        <Valve pattern="%h %l %u %t &quot;%r&quot; %s %b" directory="logs" prefix="localhost_access_log." className="org.apache.catalina.valves.AccessLogValve" suffix=".txt" resolveHosts="false">

                        </Valve>

                      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelStartOptions="3" channelSendOptions="8">

                        <Manager notifyListenersOnReplication="true" expireSessionsOnShutdown="false" className="org.apache.catalina.ha.session.DeltaManager">

                        </Manager>

                        <Channel className="org.apache.catalina.tribes.group.GroupChannel">

                          <!--<Membership port="45564" dropTime="10000" address="228.0.0.4" className="org.apache.catalina.tribes.membership.McastService" frequency="5000">

                          </Membership>-->

                          <Receiver port="4002" autoBind="100" address="10.10.2.84" selectorTimeout="5000" maxThreads="6" className="org.apache.catalina.tribes.transport.nio.NioReceiver" timeout="30000"></Receiver>

                          <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">

                            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"></Transport>

                          </Sender>

                          <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"></Intercepto r>

                          <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"></ Interceptor>

                          <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">     

                <!--        <LocalMember port="4002" className="org.apache.catalina.tribes.membership.StaticMember" securePort="-1" uniqueId="{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,0}" domain="static-cluster" host="10.10.2.84"/>-->

                            <Member      port="4001" className="org.apache.catalina.tribes.membership.StaticMember" securePort="-1" uniqueId="{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}" domain="static-cluster" host="10.10.2.84"/>

                          </Interceptor>

                        </Channel>

                        <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""></Valve>

                        <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"></Valve>

                        <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"></ClusterListe ner>

                        <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"></ClusterListener>

                      </Cluster>

                      </Host>

                    </Engine>

                  </Service>

                </Server>

                • 5. Re: Coldfusion 10 update 18 cluster instance restarts in hung state while under load
                  GuitsBoy Level 1

                  Well, I spent all day playing around with static clusters, and unfortunately I'm in exactly the same place as with the multicast cluster.  I even stripped down the config and put Carl's syntax in there verbatim, save for IP addresses.  Still craps out under load.   Looks like somehow its just not communicating when a node recovers (ie. restarted).

                   

                  Its still choking on these two errors primarily:

                   

                   

                  Mar 24, 2016 10:15:58 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

                  SEVERE: Unable to send message through cluster sender.

                  org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

                          at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender. java:109)

                          at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelS ender.java:54)

                          at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransm itter.java:78)

                          at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:7 7)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                          at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(Mess ageDispatchInterceptor.java:77)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                          at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDe tector.java:93)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBas e.java:77)

                          at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

                          at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

                          at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:837)

                          at org.apache.catalina.ha.session.DeltaManager.sendSessions(DeltaManager.java:1429)

                          at org.apache.catalina.ha.session.DeltaManager.handleGET_ALL_SESSIONS(DeltaManager.java:1357 )

                          at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1183)

                          at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)

                          at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListe ner.java:91)

                          at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)

                          at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)

                          at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                          at org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor.messageReceived(TcpPingI nterceptor.java:170)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                          at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailu reDetector.java:117)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                          at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelIntercepto rBase.java:82)

                          at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.ja va:252)

                          at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:2 87)

                          at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTa sk.java:210)

                          at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:9 9)

                          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                          at java.lang.Thread.run(Thread.java:745)

                   

                   

                   

                  Mar 24, 2016 10:15:58 AM org.apache.catalina.session.StandardSession tellNew

                  SEVERE: Session event listener threw exception

                  java.lang.NullPointerException

                          at coldfusion.bootstrap.HttpFlexSessionBootstrap.getListener(HttpFlexSessionBootstrap.java:1 54)

                          at coldfusion.bootstrap.HttpFlexSessionBootstrap.sessionCreated(HttpFlexSessionBootstrap.jav a:69)

                          at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:422)

                          at org.apache.catalina.session.StandardSession.setId(StandardSession.java:394)

                          at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:275)

                          at org.apache.catalina.ha.session.DeltaManager.handleSESSION_CREATED(DeltaManager.java:1317)

                          at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1195)

                          at org.apache.catalina.ha.session.DeltaManager.getAllClusterSessions(DeltaManager.java:803)

                          at org.apache.catalina.ha.session.DeltaManager.startInternal(DeltaManager.java:756)

                          at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

                          at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5526)

                          at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

                          at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1574)

                          at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1564)

                          at java.util.concurrent.FutureTask.run(FutureTask.java:262)

                          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                          at java.lang.Thread.run(Thread.java:745)

                  • 6. Re: Coldfusion 10 update 18 cluster instance restarts in hung state while under load
                    GuitsBoy Level 1

                    After playing with static clustering a few more days, I still ran headlong into a wall with this.

                     

                    I decided to go back to multicast, and try BackupManager instead of DeltaManager, and it now seems to be restarting correctly, as long as my timout was high enough for the 15+ second AbstractMapReplication init.  Overall it takes about 65 seconds for server startup compared to 15 sec for static deltaManager and 30 sec for multicast deltaManager.  But the important part is that it eventually does start the server while under load, and session failover seems to work as expected.

                     

                    Unfortunately using backupManager is not my preferred solution, but it may be the only option until I can figure out why DeltaManager is broken in our environment.  I would still love to hear if anyone has any suggestions as to why backup manager is working but delta manager is broken.

                     

                    Thanks,

                    -Tony

                     

                     

                     

                     

                     

                    [root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf1/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf1/bin/cf-boot strap.jar:/opt/coldfusion10/newcf1/lib/oosdk/lib:/opt/coldfusion10/newcf1/lib/oosdk/lib/*: /opt/coldfusion10/newcf1/lib/oosdk/classes:/opt/coldfusion10/newcf1/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf1 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf1 -Dcoldfusion.libPath=/opt/coldfusion10/newcf1/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf1/lib/updates,/opt/coldfusion10/newcf1/lib, /opt/coldfusion10/newcf1/lib/axis2,/opt/coldfusion10/newcf1/gateway/lib/,/opt/coldfusion10 /newcf1/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf1/wwwroot/WEB-INF/cfform/jars,/va r/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start

                    Mar 28, 2016 2:17:34 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

                    INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

                    Mar 28, 2016 2:17:34 PM org.apache.coyote.AbstractProtocol init

                    INFO: Initializing ProtocolHandler ["http-bio-8503"]

                    Mar 28, 2016 2:17:34 PM org.apache.coyote.AbstractProtocol init

                    INFO: Initializing ProtocolHandler ["ajp-bio-8015"]

                    Mar 28, 2016 2:17:35 PM org.apache.catalina.core.StandardService startInternal

                    INFO: Starting service Catalina

                    Mar 28, 2016 2:17:35 PM org.apache.catalina.core.StandardEngine startInternal

                    INFO: Starting Servlet Engine: Apache Tomcat/7.0.64

                    Mar 28, 2016 2:17:35 PM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal

                    INFO: Cluster is about to start

                    Mar 28, 2016 2:17:35 PM org.apache.catalina.tribes.transport.ReceiverBase bind

                    INFO: Receiver Server Socket bound to:/10.10.2.84:4101

                    Mar 28, 2016 2:17:35 PM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket

                    INFO: Setting cluster mcast soTimeout to 5000

                    Mar 28, 2016 2:17:35 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

                    INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:4

                    Mar 28, 2016 2:17:37 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded

                    INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4202,{10, 10, 2, 84},4202, alive=655064, securePort=-1, UDP Port=-1, id={-18 29 64 7 95 -64 78 -124 -71 122 -116 113 -74 -10 31 -66 }, payload={}, command={}, domain={}, ]

                    Mar 28, 2016 2:17:45 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

                    INFO: Done sleeping, membership established, start level:4

                    Mar 28, 2016 2:17:45 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

                    INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:8

                    Mar 28, 2016 2:17:45 PM org.apache.catalina.tribes.io.BufferPool getBufferPool

                    INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl

                    Mar 28, 2016 2:17:55 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

                    INFO: Done sleeping, membership established, start level:8

                    Mar 28, 2016 2:17:58 PM org.apache.catalina.startup.TldConfig execute

                    INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

                    Mar 28, 2016 2:17:58 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap init

                    INFO: Initializing AbstractReplicatedMap with context name:-map

                    Mar 28, 2016 2:18:10 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap memberDisappeared

                    INFO: Member[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4202,{10, 10, 2, 84},4202, alive=676062, securePort=-1, UDP Port=-1, id={-18 29 64 7 95 -64 78 -124 -71 122 -116 113 -74 -10 31 -66 }, payload={}, command={}, domain={}, ]] disappeared. Related map entries will be relocated to the new node.

                    Mar 28, 2016 2:18:10 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap memberDisappeared

                    INFO: Relocation of map entries was complete in 20 ms.

                    Mar 28, 2016 2:18:13 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap broadcast

                    WARNING: broadcast received 0 replies, probably a timeout.

                    Mar 28, 2016 2:18:13 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap init

                    INFO: AbstractReplicatedMap[-map] initialization was completed in 15300 ms.

                    Mar 28, 2016 2:18:17 PM org.apache.catalina.core.ApplicationContext log

                    INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services

                    Mar 28, 2016 2:18:17 PM org.apache.catalina.core.ApplicationContext log

                    INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 24.95-b01

                    Mar 28, 2016 2:18:30 PM org.apache.catalina.core.ApplicationContext log

                    INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646

                    Mar 28, 2016 2:18:37 PM org.apache.catalina.ha.session.JvmRouteBinderValve startInternal

                    INFO: JvmRouteBinderValve started

                    Mar 28, 2016 2:18:37 PM org.apache.coyote.AbstractProtocol start

                    INFO: Starting ProtocolHandler ["http-bio-8503"]

                    Mar 28, 2016 2:18:37 PM org.apache.coyote.AbstractProtocol start

                    INFO: Starting ProtocolHandler ["ajp-bio-8015"]

                    Mar 28, 2016 2:18:37 PM com.adobe.coldfusion.launcher.Launcher run

                    INFO: Server startup in 64767 ms