9 Replies Latest reply: Sep 11, 2012 11:27 AM by lincoln311219 RSS

    server instances starts automatically

    lincoln311219 Community Member

      my server instances are starting automatically  ( I have multi server cf envronment on j-run)

       

      The server log file has following logs messages

       

      Primary

       

       

      "Information","scheduler-20","05/01/12","10:00:41",,"Starting logging..."

      "Information","scheduler-5","05/01/12","10:43:11",,"Starting logging..."

      nformation","scheduler-1","05/01/12","11:37:14",,"Starting logging..."

       

       

      Seconday

       

       

      "Information","scheduler-20","05/01/12","10:00:55",,"Starting logging..."

       

       

      The Primary service was restarted 3 times  and secondary once . Did not find much in the logs to find the reason.

       

      Can anyboy have reasons why this might happen?

        • 1. Re: server instances starts automatically
          lincoln311219 Community Member

          further this is CF 9.0.1

          • 2. Re: server instances starts automatically
            carl type3 Community Member

            Are there any error or warning messages in the \JRun4\logs cfusion-event.log cfusion-out.log or "instance"-event and -out log?

             

            HTH, Carl.

            • 3. Re: server instances starts automatically
              charlie@carehart.org CommunityMVP

              Well, as to why a service would start automatically, that’s a built-in feature. Notice there is a jrunsvc.exe in task manager, in addition to jrun.exe. It’s job is to watch and see if a CF instance goes down and if so, it brings it back up.

               

              But as for why the service (instance) in question went down, you’d want to look at the messages in the logs BEFORE it went down, as they may give you a clue, though not always. I would recommend that rather than look in the server.log within the instance (which you can view with the CF Admin like other CF logs), you should instead look in the jrun4\logs directory and the out.log’s for your instance in question (as Carl also suggested), as they often have more valuable confirmation about overall server health.

               

              You can also see if you see messages indicating that CF was brought down (the service was “stopped”), as there would be messages before the restart about that. If there are none, then instead it may have died or been killed (from someone killing the process in task manager, typically.)

               

              Finally, there are indeed sometimes no messages at all in the out logs for why a service went down. Again, if it was killed, someone may have done that because they felt it was unresponsive. That, too, may offer no message. If you’re sure no one killed it (or stopped it), you should also look in the jrun4\bin directory to see if there are any pid.log files. They are hotspot compiler crashes, and can often offer helpful clues (though not always).

               

              Finally, if you remain stumped and need to resolve this, there are folks you can hire (myself included) who help resolve such problems by watching over your shoulders remotely while helping guide you through a process of investigation. But if you may prefer to stick with email for now, you’ll find there are many also willing to help here.

               

              /charlie arehart

               

              charlie@carehart.org

               

              Providing fast, remote, on-demand troubleshooting services for CF (and CFBuilder)

               

              More at http://www.carehart.org/consulting

               

              See also http://www.cf911.com for more on CF troubleshooting resources

              • 4. Re: server instances starts automatically
                lincoln311219 Community Member

                Hi Charlie,

                Server re-starting severals times this mornig..

                the PID log is as following

                #
                # A fatal error has been detected by the Java Runtime Environment:
                #
                # java.lang.OutOfMemoryError: requested 99562752 bytes for jbyte in C:\BUILD_AREA\jdk6_17\hotspot\src\share\vm\prims\jni.cpp. Out of swap space?
                #
                #  Internal Error (allocation.inline.hpp:39), pid=7492, tid=8840
                #  Error: jbyte in C:\BUILD_AREA\jdk6_17\hotspot\src\share\vm\prims\jni.cpp
                #
                # JRE version: 6.0_17-b04
                # Java VM: Java HotSpot(TM) Server VM (14.3-b01 mixed mode windows-x86 )
                # If you would like to submit a bug report, please visit:
                #   http://java.sun.com/webapps/bugreport/crash.jsp
                #

                ---------------  T H R E A D  ---------------

                Current thread (0x51091c00):  JavaThread "jrpp-1" [_thread_in_vm, id=8840, stack(0x625d0000,0x626d0000)]

                Stack: [0x625d0000,0x626d0000],  sp=0x626cec48,  free space=1019k
                Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
                V  [jvm.dll+0x3067f7]
                V  [jvm.dll+0xf81ec]
                V  [jvm.dll+0x198b49]

                Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
                j  com.sun.medialib.codec.png.Encoder.encode(Ljava/io/OutputStream;Lcom/sun/medialib/codec/j iio/mediaLibImage;JI)V+0
                j  com.sun.medialib.codec.png.Encoder.encode(Ljava/io/OutputStream;)V+14
                j  com.sun.media.imageioimpl.plugins.png.CLibPNGImageWriter.write(Ljavax/imageio/metadata/II OMetadata;Ljavax/imageio/IIOImage;Ljavax/imageio/ImageWriteParam;)V+796
                j  javax.imageio.ImageWriter.write(Ljava/awt/image/RenderedImage;)V+13
                j  javax.imageio.ImageIO.write(Ljava/awt/image/RenderedImage;Ljava/lang/String;Ljavax/imagei o/stream/ImageOutputStream;)Z+92
                j  javax.imageio.ImageIO.write(Ljava/awt/image/RenderedImage;Ljava/lang/String;Ljava/io/Outp utStream;)Z+41
                j  coldfusion.image.ImageWriter.writeImagetoStream(Ljava/lang/String;Ljava/awt/image/Rendere dImage;Lcoldfusion/image/Image;FLjava/io/OutputStream;Ljavax/imageio/metadata/IIOMetadata; )V+140
                j  coldfusion.image.ImageWriter.writeImage(Ljava/awt/image/RenderedImage;Lcoldfusion/image/I mage;Ljava/lang/String;Ljava/lang/String;FLjava/lang/String;)V+108
                j  coldfusion.image.Image.write(Ljava/lang/String;FZLjavax/servlet/jsp/PageContext;)V+464
                j  coldfusion.tagext.io.ImageTag.performWrite(Ljava/lang/String;Ljava/lang/String;Ljavax/ser vlet/jsp/PageContext;)V+35
                j  coldfusion.tagext.io.ImageTag.writeTempImageFile(Ljava/io/File;)V+19
                j  coldfusion.tagext.io.ImageTag.writeImageToBrowser(Ljava/lang/String;)V+8
                j  coldfusion.tagext.io.ImageTag.doStartTag()I+918
                j  cfdspSSCProcessing22ecfm1546786630._factor8(Ljavax/servlet/jsp/tagext/Tag;Ljavax/servlet/ jsp/JspWriter;)Ljava/lang/Object;+1007
                j  cfdspSSCProcessing22ecfm1546786630.runPage()Ljava/lang/Object;+22
                j  coldfusion.runtime.CfJspPage.invoke(Lcoldfusion/filter/FusionContext;)V+84
                j  coldfusion.tagext.lang.IncludeTag.doStartTag()I+90
                j  coldfusion.runtime.CfJspPage._emptyTcfTag(Ljavax/servlet/jsp/tagext/Tag;)Z+6
                j  cfindex2ecfm493090355.runPage()Ljava/lang/Object;+2267
                j  coldfusion.runtime.CfJspPage.invoke(Lcoldfusion/filter/FusionContext;)V+84
                j  coldfusion.tagext.lang.IncludeTag.doStartTag()I+90
                j  coldfusion.filter.CfincludeFilter.invoke(Lcoldfusion/filter/FusionContext;)V+75
                j  coldfusion.filter.ApplicationFilter.invoke(Lcoldfusion/filter/FusionContext;)V+1039
                j  coldfusion.filter.RequestMonitorFilter.invoke(Lcoldfusion/filter/FusionContext;)V+56
                j  coldfusion.filter.MonitoringFilter.invoke(Lcoldfusion/filter/FusionContext;)V+12
                j  coldfusion.filter.PathFilter.invoke(Lcoldfusion/filter/FusionContext;)V+146
                j  coldfusion.filter.ExceptionFilter.invoke(Lcoldfusion/filter/FusionContext;)V+13
                j  coldfusion.filter.BrowserDebugFilter.invoke(Lcoldfusion/filter/FusionContext;)V+193
                j  coldfusion.filter.ClientScopePersistenceFilter.invoke(Lcoldfusion/filter/FusionContext;)V +5
                j  coldfusion.filter.BrowserFilter.invoke(Lcoldfusion/filter/FusionContext;)V+35
                j  coldfusion.filter.NoCacheFilter.invoke(Lcoldfusion/filter/FusionContext;)V+120
                j  coldfusion.filter.GlobalsFilter.invoke(Lcoldfusion/filter/FusionContext;)V+13
                j  coldfusion.filter.DatasourceFilter.invoke(Lcoldfusion/filter/FusionContext;)V+5
                j  coldfusion.filter.CachingFilter.invoke(Lcoldfusion/filter/FusionContext;)V+36
                j  coldfusion.CfmServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletRespons e;)V+182
                j  coldfusion.bootstrap.BootstrapServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servle t/ServletResponse;)V+30
                j  jrun.servlet.FilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletRes ponse;)V+53
                j  coldfusion.monitor.event.MonitoringServletFilter.doFilter(Ljavax/servlet/ServletRequest;L javax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+9
                j  coldfusion.bootstrap.BootstrapFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servle t/ServletResponse;Ljavax/servlet/FilterChain;)V+25
                j  jrun.servlet.FilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletRes ponse;)V+80
                j  jrun.servlet.FilterChain.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResp onse;)V+3
                j  jrun.servlet.ServletInvoker.invoke(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/ http/HttpServletResponse;Ljrun/servlet/InvokerChain;)V+183
                j  jrun.servlet.JRunInvokerChain.invokeNext(Ljavax/servlet/http/HttpServletRequest;Ljavax/se rvlet/http/HttpServletResponse;)V+55
                j  jrun.servlet.JRunRequestDispatcher.invoke(Ljrun/servlet/ServletConnection;)V+249
                j  jrun.servlet.ServletEngineService.dispatch(Ljrun/servlet/ServletConnection;)V+74
                j  jrun.servlet.jrpp.JRunProxyService.invokeRunnable(Ljava/lang/Runnable;)V+24
                j  jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(Ljava/lang/Runnable;)V+16
                j  jrunx.scheduler.WorkerThread.run()V+25
                v  ~StubRoutines::call_stub

                ---------------  P R O C E S S  ---------------

                Java Threads: ( => current thread )
                  0x51742400 JavaThread "Timer-4" [_thread_blocked, id=864, stack(0x5be30000,0x5bf30000)]
                  0x510c9400 JavaThread "cfthread-4" [_thread_blocked, id=9620, stack(0x5bd30000,0x5be30000)]
                  0x5543a800 JavaThread "cfthread-3" [_thread_blocked, id=1664, stack(0x5ba30000,0x5bb30000)]
                  0x50f4d400 JavaThread "cfthread-2" [_thread_blocked, id=8884, stack(0x59dc0000,0x59ec0000)]
                  0x5160c400 JavaThread "task" daemon [_thread_blocked, id=8016, stack(0x5b6c0000,0x5b7c0000)]
                  0x553f9800 JavaThread "task" daemon [_thread_blocked, id=728, stack(0x5b5c0000,0x5b6c0000)]
                  0x5541d800 JavaThread "task" daemon [_thread_blocked, id=1744, stack(0x5b4c0000,0x5b5c0000)]
                  0x5197e400 JavaThread "task" daemon [_thread_blocked, id=9044, stack(0x5b3c0000,0x5b4c0000)]
                  0x5150c800 JavaThread "task" daemon [_thread_blocked, id=1248, stack(0x5b2c0000,0x5b3c0000)]
                  0x50b33000 JavaThread "RMI TCP Connection(43)-161.80.200.121" daemon [_thread_in_native, id=8948, stack(0x564c0000,0x565c0000)]
                  0x559f6800 JavaThread "RMI TCP Connection(42)-161.80.200.121" daemon [_thread_in_native, id=8368, stack(0x561c0000,0x562c0000)]
                  0x5fd57c00 JavaThread "task" daemon [_thread_blocked, id=10176, stack(0x5b1c0000,0x5b2c0000)]
                  0x50c50400 JavaThread "task" daemon [_thread_blocked, id=2044, stack(0x5abc0000,0x5acc0000)]
                  0x552f7400 JavaThread "task" daemon [_thread_blocked, id=5288, stack(0x5aac0000,0x5abc0000)]
                  0x5f4f3400 JavaThread "task" daemon [_thread_blocked, id=3416, stack(0x5a9c0000,0x5aac0000)]
                  0x5152bc00 JavaThread "task" daemon [_thread_blocked, id=7416, stack(0x5a8c0000,0x5a9c0000)]
                  0x51395400 JavaThread "RMI TCP Connection(40)-161.80.200.121" daemon [_thread_in_native, id=5564, stack(0x5a7c0000,0x5a8c0000)]
                  0x51355400 JavaThread "RMI TCP Connection(41)-161.80.200.121" daemon [_thread_in_native, id=8316, stack(0x5a6c0000,0x5a7c0000)]
                  0x50be3400 JavaThread "RMI TCP Connection(39)-161.80.200.121" daemon [_thread_in_native, id=7480, stack(0x5a5c0000,0x5a6c0000)]
                  0x513f3800 JavaThread "task" daemon [_thread_blocked, id=5344, stack(0x5a3c0000,0x5a4c0000)]
                  0x5144a400 JavaThread "task" daemon [_thread_blocked, id=5440, stack(0x59ec0000,0x59fc0000)]
                  0x55850400 JavaThread "task" daemon [_thread_blocked, id=8928, stack(0x560c0000,0x561c0000)]
                  0x554b9800 JavaThread "task" daemon [_thread_blocked, id=10112, stack(0x563c0000,0x564c0000)]
                  0x512f3400 JavaThread "task" daemon [_thread_blocked, id=620, stack(0x580c0000,0x581c0000)]
                  0x559a5000 JavaThread "task" daemon [_thread_blocked, id=9900, stack(0x574c0000,0x575c0000)]
                  0x51501400 JavaThread "task" daemon [_thread_blocked, id=3948, stack(0x573c0000,0x574c0000)]
                  0x51863000 JavaThread "RMI RenewClean-[161.80.200.121:2108]" daemon [_thread_blocked, id=8816, stack(0x546c0000,0x547c0000)]
                  0x50f4a400 JavaThread "RMI TCP Connection(35)-161.80.200.121" daemon [_thread_in_native, id=748, stack(0x537d0000,0x538d0000)]
                  0x554c9400 JavaThread "RMI TCP Connection(37)-161.80.200.121" daemon [_thread_in_native, id=7752, stack(0x533c0000,0x534c0000)]
                  0x50b6c800 JavaThread "RMI TCP Connection(32)-161.80.200.121" daemon [_thread_in_native, id=10100, stack(0x531c0000,0x532c0000)]
                  0x513f2400 JavaThread "jrpp-6" [_thread_in_native, id=7452, stack(0x52ec0000,0x52fc0000)]
                  0x554b9400 JavaThread "RMI TCP Connection(36)-161.80.200.121" daemon [_thread_in_native, id=5384, stack(0x52cc0000,0x52dc0000)]
                  0x5f4f4c00 JavaThread "jrpp-5" [_thread_in_native, id=9804, stack(0x528c0000,0x529c0000)]
                  0x5144c800 JavaThread "AWT-Windows" daemon [_thread_in_native, id=8640, stack(0x53bc0000,0x53cc0000)]
                  0x5587b800 JavaThread "jrpp-4" [_thread_in_native, id=8984, stack(0x536d0000,0x537d0000)]
                  0x5148c400 JavaThread "cfthread-1" [_thread_blocked, id=5264, stack(0x535d0000,0x536d0000)]
                  0x50daac00 JavaThread "cfthread-0" [_thread_blocked, id=220, stack(0x534d0000,0x535d0000)]
                  0x556dd400 JavaThread "jrpp-3" [_thread_in_native, id=9468, stack(0x52dc0000,0x52ec0000)]
                  0x51356c00 JavaThread "scheduler-1" [_thread_blocked, id=3220, stack(0x52bc0000,0x52cc0000)]
                  0x50ce7000 JavaThread "jrpp-2" [_thread_in_native, id=5528, stack(0x623d0000,0x624d0000)]
                  0x003f7000 JavaThread "DestroyJavaVM" [_thread_blocked, id=416, stack(0x00030000,0x00130000)]
                  0x50fe1800 JavaThread "HashSessionScavenger-0" daemon [_thread_blocked, id=5272, stack(0x668d0000,0x669d0000)]
                  0x55996800 JavaThread "qtp25678348-329 - Acceptor1 SelectChannelConnector@0.0.0.0:5501" [_thread_in_native, id=4548, stack(0x674c0000,0x675c0000)]
                  0x51417400 JavaThread "qtp25678348-328 - Acceptor0 SelectChannelConnector@0.0.0.0:5501" [_thread_in_native, id=6272, stack(0x667d0000,0x668d0000)]
                  0x5f56a800 JavaThread "worker #4" daemon [_thread_blocked, id=5096, stack(0x673c0000,0x674c0000)]
                  0x51935400 JavaThread "worker #3" daemon [_thread_blocked, id=4916, stack(0x672c0000,0x673c0000)]
                  0x51625000 JavaThread "worker #2" daemon [_thread_blocked, id=8744, stack(0x671c0000,0x672c0000)]
                  0x5fd4e400 JavaThread "worker #1" daemon [_thread_blocked, id=4716, stack(0x670c0000,0x671c0000)]
                  0x5f5c2400 JavaThread "worker #0" daemon [_thread_blocked, id=10016, stack(0x66fc0000,0x670c0000)]
                  0x5f7e0400 JavaThread "Thread-34" [_thread_blocked, id=9600, stack(0x66e40000,0x66f40000)]
                  0x50fd7c00 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=5780, stack(0x66bd0000,0x66cd0000)]
                  0x512b0400 JavaThread "obj-skimmer-changeRequestProd" daemon [_thread_blocked, id=8420, stack(0x666d0000,0x667d0000)]
                  0x51628400 JavaThread "obj-skimmer-atsprod_stg" daemon [_thread_blocked, id=2928, stack(0x665d0000,0x666d0000)]
                  0x50f75400 JavaThread "obj-skimmer-PDLibrary" daemon [_thread_blocked, id=1160, stack(0x664d0000,0x665d0000)]
                  0x5572c400 JavaThread "obj-skimmer-PARKING" daemon [_thread_blocked, id=8296, stack(0x663d0000,0x664d0000)]
                  0x51a3a800 JavaThread "obj-skimmer-CMS_OLD" daemon [_thread_blocked, id=9028, stack(0x662d0000,0x663d0000)]
                  0x5f60b400 JavaThread "obj-skimmer-adminKANSAS" daemon [_thread_blocked, id=968, stack(0x661d0000,0x662d0000)]
                  0x512c8800 JavaThread "obj-skimmer-rolemanagement" daemon [_thread_blocked, id=5060, stack(0x660d0000,0x661d0000)]
                  0x5570c800 JavaThread "obj-skimmer-adminWFMS_WASHDC" daemon [_thread_blocked, id=8288, stack(0x65fd0000,0x660d0000)]
                  0x5597b800 JavaThread "obj-skimmer-PSB_EPASS_REPORT_CMSP" daemon [_thread_blocked, id=3024, stack(0x65ed0000,0x65fd0000)]
                  0x512d8c00 JavaThread "obj-skimmer-PSB_EPASS_REPORT_CMST" daemon [_thread_blocked, id=1112, stack(0x65dd0000,0x65ed0000)]
                  0x5f7b2800 JavaThread "obj-skimmer-ps_stg" daemon [_thread_blocked, id=4588, stack(0x65cd0000,0x65dd0000)]
                  0x556dc400 JavaThread "obj-skimmer-oasis_shared" daemon [_thread_blocked, id=8032, stack(0x65bd0000,0x65cd0000)]
                  0x5f5cbc00 JavaThread "obj-skimmer-WMMpd" daemon [_thread_blocked, id=5168, stack(0x65ad0000,0x65bd0000)]
                  0x50b9d400 JavaThread "obj-skimmer-CATS_PROD" daemon [_thread_blocked, id=8612, stack(0x659d0000,0x65ad0000)]
                  0x510e6400 JavaThread "obj-skimmer-EPASSfailover" daemon [_thread_blocked, id=1204, stack(0x658d0000,0x659d0000)]
                  0x5563f400 JavaThread "obj-skimmer-adminWASHDCdev" daemon [_thread_blocked, id=6256, stack(0x657d0000,0x658d0000)]
                  0x5f51a400 JavaThread "obj-skimmer-cfr2_stg" daemon [_thread_blocked, id=9180, stack(0x656d0000,0x657d0000)]
                  0x557c9400 JavaThread "obj-skimmer-conman_stg" daemon [_thread_blocked, id=7648, stack(0x655d0000,0x656d0000)]
                  0x5f7a6400 JavaThread "obj-skimmer-oarm_workflow_stg" daemon [_thread_blocked, id=7852, stack(0x654d0000,0x655d0000)]
                  0x5f5c8400 JavaThread "obj-skimmer-changereq_stg" daemon [_thread_blocked, id=9684, stack(0x653d0000,0x654d0000)]
                  0x5f510800 JavaThread "obj-skimmer-CoopProd" daemon [_thread_blocked, id=10120, stack(0x652d0000,0x653d0000)]
                  0x55779800 JavaThread "obj-skimmer-cdx_epass" daemon [_thread_blocked, id=2280, stack(0x651d0000,0x652d0000)]
                  0x55630400 JavaThread "obj-skimmer-persec_stg" daemon [_thread_blocked, id=3008, stack(0x650d0000,0x651d0000)]
                  0x5193d400 JavaThread "obj-skimmer-meteringProd_namp" daemon [_thread_blocked, id=4368, stack(0x64fd0000,0x650d0000)]
                  0x5f9ba800 JavaThread "obj-skimmer-PRINTJOB_OAPD" daemon [_thread_blocked, id=9184, stack(0x64ed0000,0x64fd0000)]
                  0x552de800 JavaThread "obj-skimmer-coop_stg" daemon [_thread_blocked, id=3004, stack(0x64dd0000,0x64ed0000)]
                  0x552de400 JavaThread "obj-skimmer-dt_stg" daemon [_thread_blocked, id=7064, stack(0x64cd0000,0x64dd0000)]
                  0x517f8800 JavaThread "obj-skimmer-adminWFMS_RTP" daemon [_thread_blocked, id=7116, stack(0x64bd0000,0x64cd0000)]
                  0x5f6ad400 JavaThread "obj-skimmer-persec_oadv" daemon [_thread_blocked, id=8792, stack(0x64ad0000,0x64bd0000)]
                  0x517a9800 JavaThread "obj-skimmer-persec_oapd" daemon [_thread_blocked, id=9320, stack(0x649d0000,0x64ad0000)]
                  0x5602f800 JavaThread "obj-skimmer-oarm_workflow" daemon [_thread_blocked, id=9216, stack(0x648d0000,0x649d0000)]
                  0x5589b400 JavaThread "obj-skimmer-IDMSProd" daemon [_thread_blocked, id=5740, stack(0x647d0000,0x648d0000)]
                  0x50c45c00 JavaThread "obj-skimmer-PSBProd" daemon [_thread_blocked, id=6008, stack(0x646d0000,0x647d0000)]
                  0x5f6cf400 JavaThread "obj-skimmer-adminWASHDC" daemon [_thread_blocked, id=8108, stack(0x645d0000,0x646d0000)]
                  0x5531b400 JavaThread "obj-skimmer-action_jands_stg" daemon [_thread_blocked, id=4656, stack(0x644d0000,0x645d0000)]
                  0x518bd400 JavaThread "obj-skimmer-PSB_EPASS_APPS_CMSP" daemon [_thread_blocked, id=4552, stack(0x643d0000,0x644d0000)]
                  0x5187c800 JavaThread "obj-skimmer-ATSProd" daemon [_thread_blocked, id=7400, stack(0x642d0000,0x643d0000)]
                  0x50c09c00 JavaThread "obj-skimmer-PSB_EPASS_APPS_CMST" daemon [_thread_blocked, id=6248, stack(0x641d0000,0x642d0000)]
                  0x50c09800 JavaThread "obj-skimmer-oasis_shared_stg" daemon [_thread_blocked, id=6444, stack(0x640d0000,0x641d0000)]
                  0x5519c400 JavaThread "obj-skimmer-cfrv2" daemon [_thread_blocked, id=9012, stack(0x63fd0000,0x640d0000)]
                  0x50f70800 JavaThread "obj-skimmer-cats_stg" daemon [_thread_blocked, id=5864, stack(0x63ed0000,0x63fd0000)]
                  0x51926c00 JavaThread "obj-skimmer-pdlibrary_stg" daemon [_thread_blocked, id=6956, stack(0x63dd0000,0x63ed0000)]
                  0x50ce9800 JavaThread "obj-skimmer-bsd_stg" daemon [_thread_blocked, id=3960, stack(0x63cd0000,0x63dd0000)]
                  0x50fa8800 JavaThread "obj-skimmer-metering_stg" daemon [_thread_blocked, id=7732, stack(0x63bd0000,0x63cd0000)]
                  0x5106f800 JavaThread "obj-skimmer-parktrans_stg" daemon [_thread_blocked, id=420, stack(0x63ad0000,0x63bd0000)]
                  0x50fd0800 JavaThread "obj-skimmer-clicks_stg" daemon [_thread_blocked, id=8920, stack(0x639d0000,0x63ad0000)]
                  0x51060c00 JavaThread "obj-skimmer-psoapd" daemon [_thread_blocked, id=4464, stack(0x638d0000,0x639d0000)]
                  0x519e5400 JavaThread "obj-skimmer-dtoapd" daemon [_thread_blocked, id=4348, stack(0x637d0000,0x638d0000)]
                  0x55cf4400 JavaThread "obj-skimmer-BSD" daemon [_thread_blocked, id=8380, stack(0x636d0000,0x637d0000)]
                  0x50eba400 JavaThread "obj-skimmer-CredBadgeProd" daemon [_thread_blocked, id=8636, stack(0x635d0000,0x636d0000)]
                  0x51804800 JavaThread "obj-skimmer-ticket_stg" daemon [_thread_blocked, id=8952, stack(0x634d0000,0x635d0000)]
                  0x50b40800 JavaThread "obj-skimmer-contract_management" daemon [_thread_blocked, id=9340, stack(0x633d0000,0x634d0000)]
                  0x5515b400 JavaThread "obj-skimmer-EPASSprod" daemon [_thread_blocked, id=4364, stack(0x632d0000,0x633d0000)]
                  0x51957400 JavaThread "obj-skimmer-psb_stg" daemon [_thread_blocked, id=8536, stack(0x631d0000,0x632d0000)]
                  0x51427800 JavaThread "obj-skimmer-Ticket_Prod" daemon [_thread_blocked, id=5832, stack(0x630d0000,0x631d0000)]
                  0x55262800 JavaThread "obj-skimmer-PERSECOPM" daemon [_thread_blocked, id=664, stack(0x62fd0000,0x630d0000)]
                  0x559bc400 JavaThread "obj-skimmer-oa_workflow" daemon [_thread_blocked, id=8348, stack(0x62ed0000,0x62fd0000)]
                  0x55f95800 JavaThread "obj-skimmer-action_tracking_jands" daemon [_thread_blocked, id=9816, stack(0x62dd0000,0x62ed0000)]
                  0x5148c800 JavaThread "obj-skimmer-MCMpd" daemon [_thread_blocked, id=8904, stack(0x62cd0000,0x62dd0000)]
                  0x4fe65c00 JavaThread "obj-skimmer-printjob_stg" daemon [_thread_blocked, id=7724, stack(0x62bd0000,0x62cd0000)]
                  0x50bec800 JavaThread "obj-skimmer-CredBadge_stg" daemon [_thread_blocked, id=7168, stack(0x62ad0000,0x62bd0000)]
                  0x5138f800 JavaThread "obj-skimmer-mcm_stg" daemon [_thread_blocked, id=8624, stack(0x629d0000,0x62ad0000)]
                  0x5158a400 JavaThread "obj-skimmer-wmm_stg" daemon [_thread_blocked, id=9644, stack(0x628d0000,0x629d0000)]
                  0x5137f400 JavaThread "obj-skimmer-oapd_stg" daemon [_thread_blocked, id=5388, stack(0x627d0000,0x628d0000)]
                  0x51397400 JavaThread "obj-skimmer-oa_workflow_stg" daemon [_thread_blocked, id=7084, stack(0x626d0000,0x627d0000)]
                =>0x51091c00 JavaThread "jrpp-1" [_thread_in_vm, id=8840, stack(0x625d0000,0x626d0000)]
                  0x517fac00 JavaThread "scheduler-0" [_thread_blocked, id=8808, stack(0x624d0000,0x625d0000)]
                  0x55bfc400 JavaThread "Thread-27" daemon [_thread_blocked, id=9980, stack(0x620d0000,0x621d0000)]
                  0x5f513400 JavaThread "scheduler-22" [_thread_blocked, id=4792, stack(0x61fd0000,0x620d0000)]
                  0x51580c00 JavaThread "task" daemon [_thread_blocked, id=7196, stack(0x61dd0000,0x61ed0000)]
                  0x519dc000 JavaThread "task" daemon [_thread_blocked, id=7176, stack(0x5ca30000,0x5cb30000)]
                  0x55c94c00 JavaThread "task" daemon [_thread_blocked, id=8696, stack(0x618d0000,0x619d0000)]
                  0x55ef0800 JavaThread "jndi-5" [_thread_blocked, id=5376, stack(0x5ed30000,0x5ee30000)]
                  0x560bf800 JavaThread "task" daemon [_thread_blocked, id=9616, stack(0x617d0000,0x618d0000)]
                  0x55be5400 JavaThread "Timer-2" daemon [_thread_blocked, id=3840, stack(0x616d0000,0x617d0000)]
                  0x51a9a400 JavaThread "Timer-1" [_thread_blocked, id=9396, stack(0x615d0000,0x616d0000)]
                  0x5f51dc00 JavaThread "Timer-0" [_thread_blocked, id=2912, stack(0x614d0000,0x615d0000)]
                  0x50e3d800 JavaThread "task" daemon [_thread_blocked, id=7728, stack(0x5f130000,0x5f230000)]
                  0x5179e800 JavaThread "task" daemon [_thread_blocked, id=5284, stack(0x5f2d0000,0x5f3d0000)]
                  0x50da4400 JavaThread "task" daemon [_thread_blocked, id=7736, stack(0x5f030000,0x5f130000)]
                  0x51660000 JavaThread "task" daemon [_thread_blocked, id=7224, stack(0x5ee30000,0x5ef30000)]
                  0x5603a800 JavaThread "task" daemon [_thread_blocked, id=9084, stack(0x57bc0000,0x57cc0000)]
                  0x55f5bc00 JavaThread "task" daemon [_thread_blocked, id=8228, stack(0x5b7c0000,0x5b8c0000)]
                  0x519d6800 JavaThread "scheduler-21" [_thread_blocked, id=6876, stack(0x5d930000,0x5da30000)]
                  0x5152c400 JavaThread "task" daemon [_thread_blocked, id=8400, stack(0x5d530000,0x5d630000)]
                  0x5152fc00 JavaThread "Thread-23" daemon [_thread_blocked, id=4336, stack(0x5d230000,0x5d330000)]
                  0x513a3400 JavaThread "Thread-22" daemon [_thread_blocked, id=8544, stack(0x5d130000,0x5d230000)]
                  0x51530800 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=8656, stack(0x5d030000,0x5d130000)]
                  0x5172f400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=5128, stack(0x5cf30000,0x5d030000)]
                  0x51526000 JavaThread "jrpp-0" [_thread_in_native, id=9672, stack(0x5cc30000,0x5cd30000)]
                  0x519dd400 JavaThread "web-0" [_thread_in_native, id=9652, stack(0x5cb30000,0x5cc30000)]
                  0x55860c00 JavaThread "Thread-21" daemon [_thread_blocked, id=8940, stack(0x5c730000,0x5c830000)]
                  0x550c0c00 JavaThread "Thread-20" daemon [_thread_blocked, id=8412, stack(0x5c630000,0x5c730000)]
                  0x551fd400 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=7288, stack(0x5c530000,0x5c630000)]
                  0x51523800 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=4480, stack(0x5c430000,0x5c530000)]
                  0x515a1000 JavaThread "task" daemon [_thread_blocked, id=3444, stack(0x5c130000,0x5c230000)]
                  0x556e0800 JavaThread "task" daemon [_thread_blocked, id=4744, stack(0x5b8c0000,0x5b9c0000)]
                  0x55883400 JavaThread "Thread-19" [_thread_blocked, id=8604, stack(0x5bc30000,0x5bd30000)]
                  0x51780c00 JavaThread "task" daemon [_thread_blocked, id=6252, stack(0x5bb30000,0x5bc30000)]
                  0x51413800 JavaThread "scheduler-20" [_thread_blocked, id=5808, stack(0x572c0000,0x573c0000)]
                  0x55835800 JavaThread "Thread-17" daemon [_thread_blocked, id=7080, stack(0x5b0c0000,0x5b1c0000)]
                  0x5590c400 JavaThread "Thread-16" daemon [_thread_blocked, id=8684, stack(0x5afc0000,0x5b0c0000)]
                  0x556b1800 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=7764, stack(0x5aec0000,0x5afc0000)]
                  0x5170dc00 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=3108, stack(0x5adc0000,0x5aec0000)]
                  0x5170c800 JavaThread "RMI RenewClean-[161.80.200.121:4518]" daemon [_thread_blocked, id=9944, stack(0x5acc0000,0x5adc0000)]
                  0x51818400 JavaThread "Thread-15" daemon [_thread_blocked, id=4900, stack(0x5a2c0000,0x5a3c0000)]
                  0x55825000 JavaThread "Thread-14" daemon [_thread_blocked, id=9632, stack(0x5a1c0000,0x5a2c0000)]
                  0x55823c00 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=1740, stack(0x5a0c0000,0x5a1c0000)]
                  0x5574e400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=9532, stack(0x59fc0000,0x5a0c0000)]
                  0x55840000 JavaThread "Thread-13" daemon [_thread_blocked, id=9008, stack(0x59cc0000,0x59dc0000)]
                  0x5583ec00 JavaThread "Thread-12" daemon [_thread_blocked, id=9964, stack(0x59bc0000,0x59cc0000)]
                  0x556c3000 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=5276, stack(0x59ac0000,0x59bc0000)]
                  0x556c2400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=7060, stack(0x599c0000,0x59ac0000)]
                  0x518a2c00 JavaThread "scheduler-19" [_thread_blocked, id=4708, stack(0x596c0000,0x597c0000)]
                  0x518a1800 JavaThread "scheduler-18" [_thread_blocked, id=2924, stack(0x595c0000,0x596c0000)]
                  0x518a0000 JavaThread "scheduler-17" [_thread_blocked, id=4452, stack(0x594c0000,0x595c0000)]
                  0x5189ec00 JavaThread "scheduler-16" [_thread_blocked, id=8024, stack(0x593c0000,0x594c0000)]
                  0x5189d400 JavaThread "scheduler-15" [_thread_blocked, id=7304, stack(0x592c0000,0x593c0000)]
                  0x51617400 JavaThread "scheduler-14" [_thread_blocked, id=6204, stack(0x591c0000,0x592c0000)]
                  0x51616000 JavaThread "scheduler-13" [_thread_blocked, id=7696, stack(0x590c0000,0x591c0000)]
                  0x51614800 JavaThread "scheduler-12" [_thread_blocked, id=2748, stack(0x58fc0000,0x590c0000)]
                  0x51613400 JavaThread "scheduler-11" [_thread_blocked, id=6996, stack(0x58ec0000,0x58fc0000)]
                  0x51612400 JavaThread "scheduler-10" [_thread_blocked, id=6112, stack(0x58dc0000,0x58ec0000)]
                  0x51841400 JavaThread "scheduler-9" [_thread_blocked, id=6720, stack(0x58cc0000,0x58dc0000)]
                  0x51840000 JavaThread "scheduler-8" [_thread_blocked, id=7760, stack(0x58bc0000,0x58cc0000)]
                  0x51a12000 JavaThread "scheduler-7" [_thread_blocked, id=10064, stack(0x58ac0000,0x58bc0000)]
                  0x51a10c00 JavaThread "scheduler-6" [_thread_blocked, id=8912, stack(0x589c0000,0x58ac0000)]
                  0x51a0fc00 JavaThread "scheduler-5" [_thread_blocked, id=4804, stack(0x588c0000,0x589c0000)]
                  0x517aa400 JavaThread "scheduler-4" [_thread_blocked, id=10184, stack(0x587c0000,0x588c0000)]
                  0x51629c00 JavaThread "scheduler-3" [_thread_blocked, id=4768, stack(0x586c0000,0x587c0000)]
                  0x55681400 JavaThread "scheduler-2" [_thread_blocked, id=5368, stack(0x585c0000,0x586c0000)]
                  0x519e6800 JavaThread "scheduler-1" [_thread_blocked, id=9032, stack(0x584c0000,0x585c0000)]
                  0x51503000 JavaThread "scheduler-0" [_thread_blocked, id=712, stack(0x583c0000,0x584c0000)]
                  0x51770800 JavaThread "Transaction Domain PrivateJmsDomain" [_thread_blocked, id=5756, stack(0x582c0000,0x583c0000)]
                  0x55461800 JavaThread "clock" daemon [_thread_blocked, id=4800, stack(0x581c0000,0x582c0000)]
                  0x5158bc00 JavaThread "task" daemon [_thread_blocked, id=7132, stack(0x57fc0000,0x580c0000)]
                  0x551cf400 JavaThread "task" daemon [_thread_blocked, id=5072, stack(0x57ec0000,0x57fc0000)]
                  0x51581800 JavaThread "Transaction Domain DefaultDomain" [_thread_blocked, id=4732, stack(0x57dc0000,0x57ec0000)]
                  0x5193e400 JavaThread "tyrex.util.daemonMaster" daemon [_thread_blocked, id=5200, stack(0x57cc0000,0x57dc0000)]
                  0x517d7800 JavaThread "Thread-10" daemon [_thread_blocked, id=10108, stack(0x578c0000,0x579c0000)]
                  0x517d6400 JavaThread "Thread-9" daemon [_thread_blocked, id=9728, stack(0x577c0000,0x578c0000)]
                  0x517d4c00 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=5088, stack(0x576c0000,0x577c0000)]
                  0x515cf400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=3424, stack(0x575c0000,0x576c0000)]
                  0x51a9f000 JavaThread "Thread-8" daemon [_thread_blocked, id=196, stack(0x56fc0000,0x570c0000)]
                  0x51a9d800 JavaThread "Thread-7" daemon [_thread_blocked, id=5908, stack(0x56ec0000,0x56fc0000)]
                  0x51a9c400 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=1164, stack(0x56dc0000,0x56ec0000)]
                  0x5144b000 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=9200, stack(0x56cc0000,0x56dc0000)]
                  0x5140dc00 JavaThread "jndi-4" [_thread_in_native, id=9576, stack(0x569c0000,0x56ac0000)]
                  0x51987800 JavaThread "jndi-3" [_thread_blocked, id=1168, stack(0x568c0000,0x569c0000)]
                  0x551d0400 JavaThread "jndi-2" [_thread_blocked, id=5044, stack(0x567c0000,0x568c0000)]
                  0x51988c00 JavaThread "jndi-1" [_thread_blocked, id=7500, stack(0x566c0000,0x567c0000)]
                  0x515cd000 JavaThread "jndi-0" [_thread_blocked, id=7376, stack(0x565c0000,0x566c0000)]
                  0x5158c400 JavaThread "Thread-6" daemon [_thread_blocked, id=8832, stack(0x54ec0000,0x54fc0000)]
                  0x519b6800 JavaThread "Thread-5" daemon [_thread_blocked, id=7952, stack(0x54cc0000,0x54dc0000)]
                  0x518d0400 JavaThread "RMI RenewClean-[161.80.200.121:1393]" daemon [_thread_blocked, id=9284, stack(0x54dc0000,0x54ec0000)]
                  0x513a9400 JavaThread "Thread-4" daemon [_thread_blocked, id=6216, stack(0x54bc0000,0x54cc0000)]
                  0x5110c400 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=4796, stack(0x548c0000,0x549c0000)]
                  0x50f98400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=8452, stack(0x547c0000,0x548c0000)]
                  0x513b1c00 JavaThread "jms-fifo-5" [_thread_blocked, id=10192, stack(0x544c0000,0x545c0000)]
                  0x513b1400 JavaThread "jms-fifo-4" [_thread_blocked, id=7680, stack(0x543c0000,0x544c0000)]
                  0x51416800 JavaThread "jms-fifo-3" [_thread_blocked, id=4936, stack(0x542c0000,0x543c0000)]
                  0x5177a400 JavaThread "jms-fifo-2" [_thread_blocked, id=1000, stack(0x541c0000,0x542c0000)]
                  0x5110a000 JavaThread "jms-fifo-1" [_thread_blocked, id=5252, stack(0x52ac0000,0x52bc0000)]
                  0x51106800 JavaThread "RMI RenewClean-[161.80.200.121:1220]" daemon [_thread_blocked, id=6288, stack(0x53ac0000,0x53bc0000)]
                  0x50f94000 JavaThread "RMI TCP Connection(38)-161.80.200.121" daemon [_thread_in_native, id=7124, stack(0x539c0000,0x53ac0000)]
                  0x50f49000 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=9704, stack(0x529c0000,0x52ac0000)]
                  0x50d18800 JavaThread "snapshot thread" daemon [_thread_blocked, id=1772, stack(0x527c0000,0x528c0000)]
                  0x50e90000 JavaThread "discovery announcement" daemon [_thread_blocked, id=5188, stack(0x526c0000,0x527c0000)]
                  0x50eaa400 JavaThread "multicast request" daemon [_thread_in_native, id=9024, stack(0x525c0000,0x526c0000)]
                  0x50e6d400 JavaThread "unicast request" daemon [_thread_in_native, id=8956, stack(0x524c0000,0x525c0000)]
                  0x50d0c400 JavaThread "event expire" daemon [_thread_blocked, id=8324, stack(0x523c0000,0x524c0000)]
                  0x50db8400 JavaThread "service expire" daemon [_thread_blocked, id=484, stack(0x522c0000,0x523c0000)]
                  0x50e98000 JavaThread "Thread-3" daemon [_thread_blocked, id=1784, stack(0x520d0000,0x521d0000)]
                  0x50f52400 JavaThread "Thread-2" daemon [_thread_blocked, id=7020, stack(0x51fd0000,0x520d0000)]
                  0x50d7c400 JavaThread "multicast announcement timer" daemon [_thread_blocked, id=9140, stack(0x51ed0000,0x51fd0000)]
                  0x50e51400 JavaThread "multicast discovery announcement listener" daemon [_thread_in_native, id=10084, stack(0x51dd0000,0x51ed0000)]
                  0x51449800 JavaThread "GC Daemon" daemon [_thread_blocked, id=4884, stack(0x51cd0000,0x51dd0000)]
                  0x50eea400 JavaThread "RMI Reaper" [_thread_blocked, id=9880, stack(0x51bd0000,0x51cd0000)]
                  0x51407c00 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=8548, stack(0x51ad0000,0x51bd0000)]
                  0x4fe7a400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=5872, stack(0x50890000,0x50990000)]
                  0x4fe77400 JavaThread "CompilerThread1" daemon [_thread_blocked, id=9976, stack(0x50790000,0x50890000)]
                  0x4fe72400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=1140, stack(0x50690000,0x50790000)]
                  0x4fe70800 JavaThread "Attach Listener" daemon [_thread_blocked, id=1804, stack(0x50590000,0x50690000)]
                  0x4fe6f400 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5488, stack(0x50490000,0x50590000)]
                  0x4fe5d000 JavaThread "Finalizer" daemon [_thread_blocked, id=1116, stack(0x50390000,0x50490000)]
                  0x4fe5b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=8892, stack(0x50290000,0x50390000)]

                Other Threads:
                  0x4fe57800 VMThread [stack: 0x50190000,0x50290000] [id=5304]
                  0x4fe7c000 WatcherThread [stack: 0x50990000,0x50a90000] [id=6708]

                VM state:not at safepoint (normal execution)

                VM Mutex/Monitor currently owned by a thread: None

                Heap
                PSYoungGen      total 77696K, used 52443K [0x48630000, 0x4f7f0000, 0x4f7f0000)
                  eden space 38912K, 35% used [0x48630000,0x49389070,0x4ac30000)
                  from space 38784K, 99% used [0x4d210000,0x4f7edc50,0x4f7f0000)
                  to   space 38784K, 0% used [0x4ac30000,0x4ac30000,0x4d210000)
                PSOldGen        total 291584K, used 244182K [0x0f7f0000, 0x214b0000, 0x48630000)
                  object space 291584K, 83% used [0x0f7f0000,0x1e665810,0x214b0000)
                PSPermGen       total 89600K, used 71687K [0x037f0000, 0x08f70000, 0x0f7f0000)
                  object space 89600K, 80% used [0x037f0000,0x07df1ce8,0x08f70000)

                Dynamic libraries:
                0x00400000 - 0x00411000  D:\JRun4\bin\jrun.exe
                0x7c800000 - 0x7c8c3000  C:\WINDOWS\system32\ntdll.dll
                0x77e40000 - 0x77f42000  C:\WINDOWS\system32\kernel32.dll
                0x7d1e0000 - 0x7d27c000  C:\WINDOWS\system32\ADVAPI32.dll
                0x77c50000 - 0x77cf0000  C:\WINDOWS\system32\RPCRT4.dll
                0x76f50000 - 0x76f63000  C:\WINDOWS\system32\Secur32.dll
                0x78520000 - 0x785c3000  C:\WINDOWS\WinSxS\x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.21022.8_x-ww_D08D0375\MSVCR 90.dll
                0x6db60000 - 0x6df6a000  D:\JRun4\jre\bin\server\jvm.dll
                0x77380000 - 0x77411000  C:\WINDOWS\system32\USER32.dll
                0x77c00000 - 0x77c49000  C:\WINDOWS\system32\GDI32.dll
                0x76aa0000 - 0x76acd000  C:\WINDOWS\system32\WINMM.dll
                0x7c340000 - 0x7c396000  D:\JRun4\bin\MSVCR71.dll
                0x76290000 - 0x762ad000  C:\WINDOWS\system32\IMM32.DLL
                0x6d860000 - 0x6d86c000  D:\JRun4\jre\bin\verify.dll
                0x6d3e0000 - 0x6d3ff000  D:\JRun4\jre\bin\java.dll
                0x6d340000 - 0x6d348000  D:\JRun4\jre\bin\hpi.dll
                0x76b70000 - 0x76b7b000  C:\WINDOWS\system32\PSAPI.DLL
                0x6d8a0000 - 0x6d8af000  D:\JRun4\jre\bin\zip.dll
                0x6d6c0000 - 0x6d6d3000  D:\JRun4\jre\bin\net.dll
                0x71c00000 - 0x71c17000  C:\WINDOWS\system32\WS2_32.dll
                0x77ba0000 - 0x77bfa000  C:\WINDOWS\system32\msvcrt.dll
                0x71bf0000 - 0x71bf8000  C:\WINDOWS\system32\WS2HELP.dll
                0x71b20000 - 0x71b61000  C:\WINDOWS\System32\mswsock.dll
                0x76ed0000 - 0x76efa000  C:\WINDOWS\system32\DNSAPI.dll
                0x76f70000 - 0x76f77000  C:\WINDOWS\System32\winrnr.dll
                0x76f10000 - 0x76f3e000  C:\WINDOWS\system32\WLDAP32.dll
                0x76f80000 - 0x76f85000  C:\WINDOWS\system32\rasadhlp.dll
                0x51150000 - 0x5115c000  D:\JRun4\bin\portscan.dll
                0x5f270000 - 0x5f2ca000  C:\WINDOWS\system32\hnetcfg.dll
                0x71ae0000 - 0x71ae8000  C:\WINDOWS\System32\wshtcpip.dll
                0x68000000 - 0x68035000  C:\WINDOWS\system32\rsaenh.dll
                0x7c8d0000 - 0x7d0cf000  C:\WINDOWS\system32\SHELL32.dll
                0x7d180000 - 0x7d1d2000  C:\WINDOWS\system32\SHLWAPI.dll
                0x77420000 - 0x77523000  C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.4770_x- ww_05FDF087\comctl32.dll
                0x6d7b0000 - 0x6d7b6000  D:\JRun4\jre\bin\rmi.dll
                0x6d840000 - 0x6d848000  D:\JRun4\jre\bin\sunmscapi.dll
                0x761b0000 - 0x76243000  C:\WINDOWS\system32\CRYPT32.dll
                0x76190000 - 0x761a2000  C:\WINDOWS\system32\MSASN1.dll
                0x521d0000 - 0x52250000  D:\JRun4\servers\CF9Primary\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\izmjniado.dll
                0x77670000 - 0x777a9000  C:\WINDOWS\system32\ole32.dll
                0x7d0e0000 - 0x7d16b000  C:\WINDOWS\system32\OLEAUT32.dll
                0x76cf0000 - 0x76d0a000  C:\WINDOWS\system32\iphlpapi.dll
                0x76cd0000 - 0x76ce9000  C:\WINDOWS\system32\MPRAPI.dll
                0x76df0000 - 0x76e24000  C:\WINDOWS\system32\ACTIVEDS.dll
                0x76dc0000 - 0x76de8000  C:\WINDOWS\system32\adsldpc.dll
                0x71c40000 - 0x71c97000  C:\WINDOWS\system32\NETAPI32.dll
                0x76b80000 - 0x76bae000  C:\WINDOWS\system32\credui.dll
                0x76a80000 - 0x76a92000  C:\WINDOWS\system32\ATL.DLL
                0x76e30000 - 0x76e3c000  C:\WINDOWS\system32\rtutils.dll
                0x7e020000 - 0x7e02f000  C:\WINDOWS\system32\SAMLIB.dll
                0x770e0000 - 0x771e8000  C:\WINDOWS\system32\SETUPAPI.dll
                0x52280000 - 0x522a3000  D:\JRun4\servers\CF9Primary\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\CFXNeo.dll
                0x5b9d0000 - 0x5b9e3000  D:\JRun4\servers\cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\lib\cfregistry.dll
                0x5ba10000 - 0x5ba23000  D:\JRun4\servers\CF9Primary\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\cfregistry.dll
                0x6d0b0000 - 0x6d1fa000  D:\JRun4\jre\bin\awt.dll
                0x73070000 - 0x73097000  C:\WINDOWS\system32\WINSPOOL.DRV
                0x77530000 - 0x775c7000  C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_5.82.3790.4770_x -ww_A689AB02\comctl32.dll
                0x6d2e0000 - 0x6d334000  D:\JRun4\jre\bin\fontmanager.dll
                0x6d6e0000 - 0x6d6e9000  D:\JRun4\jre\bin\nio.dll
                0x6d610000 - 0x6d619000  D:\JRun4\jre\bin\management.dll
                0x538d0000 - 0x538de000  D:\JRun4\servers\CF9Primary\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\NeoUUID.dll
                0x53900000 - 0x5390b000  D:\JRun4\servers\CF9Primary\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\mlib_jai_util.dll
                0x5da30000 - 0x5de7e000  D:\JRun4\servers\CF9Primary\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\mlib_jai_mmx.dll
                0x52fc0000 - 0x52fcb000  D:\JRun4\servers\CF9Primary\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\clib_jiio_util.dl l
                0x52fe0000 - 0x530ec000  D:\JRun4\servers\CF9Primary\cfusion.ear\cfusion.war\WEB-INF\cfusion\lib\clib_jiio_sse2.dl l

                VM Arguments:
                jvm_args: -Xmx1G -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -Dcoldfusion.rootDir=D:\JRun4/ -Djmx.invoke.getters=true
                java_command: <unknown>
                Launcher Type: generic

                Environment Variables:
                CLASSPATH=D:\Documentum\dctm.jar;D:\Documentum\config
                PATH=D:\JRun4\servers\cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\lib;D:\JRun4\servers \cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\jintegra\bin;D:\JRun4\servers\cfusion\cfu sion-ear\cfusion-war\WEB-INF\cfusion\jintegra\bin\international;D:\JRun4\servers\cfusion\c fusion-ear\cfusion-war\WEB-INF\cfusion\lib\oosdk\classes\win;D:\Documentum\Shared;D:\Docum entumRuntime\Shared;D:\JRun4\verity\k2\_nti40\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOW S\System32\Wbem;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Program Files\Intel\DMIX;D:\Program Files\Dell\SysMgt\RAC5;D:\Program Files\Dell\SysMgt\oma\bin
                OS=Windows_NT
                PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 6, GenuineIntel

                 

                ---------------  S Y S T E M  ---------------

                OS: Windows Server 2003 family Build 3790 Service Pack 2

                CPU:total 4 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

                Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)

                vm_info: Java HotSpot(TM) Server VM (14.3-b01) for windows-x86 JRE (1.6.0_17-b04), built on Oct 11 2009 00:46:21 by "java_re" with MS VC++ 7.1

                time: Fri Aug 31 12:50:50 2012
                elapsed time: 300 seconds

                • 5. Re: server instances starts automatically
                  carl type3 Community Member

                  java.lang.OutOfMemoryError: Out of swap space - Generally means some resource is full (or close to it). Sometimes hard disk that CF Java is installed on is full or even a policy may only allow limited space allocated.

                  HTH, Carl.

                  • 6. Re: server instances starts automatically
                    charlie@carehart.org CommunityMVP

                    As for OutOfMemory Swap Space errors, in my experience these are an indication rather that CF was unable to obtain the memory that it needed. (I appreciate why Carl would propose that it was related to disk space. The message surely suggests that, and indeed others on the web including Java folks will propose that’s it, but it really is not, in my experience.)

                     

                    Typically this is inability for CF to “get the memory it needs” is because something else is running on the server (along with CF) and using memory, so that when CF tries to use memory, it cannot. A common example is when one has a database on the same server (such as MySQL, SQL Server, etc.), or it could be a virus scanner, or the web server, or the sum of all these things.

                     

                    And it could be that things all “fit” fine together most of the time, but when memory used by those other things grows too high, or perhaps when CF memory use goes high at times (or at the same time), it just becomes a problem of “too many sheriffs in one town”.  

                     

                    Now, you can’t look at things at some one point in time (like now, as you read this and check your server’s memory use), and say “it looks like they should all fit fine right now”, because things could change later. For instance, consider that a virus scan may only use lots of memory when it runs. And if you have a database on the same server, it could use a lot more at a later time (than you see now), such as when a bunch of requests come in, and/or many large queries are run, and they push the database to load up a lot of buffers (using memory in the database address space). A burst of requests could also drive up the memory used for the web server process(es).

                     

                    So given the above, there are many “solutions” to this problem.

                    1) You could increase the memory on the box (so more things can co-exist), which may require moving to 64-bit if you’re on 32-bit for now and maxed out in memory.

                    2) You could do something to temper the other things that are using memory (move a DB server to another machine, or set it to not be able to use more than a given max memory size), or

                    3) (a real “outside the box” solution to most people) you may be need to LOWER the max heap size for CF, so that it doesn’t try to use so much memory. Sometimes people set CF to use more memory than they really need. Let’s say you have a 4GB (64-bit) box and set CF to use 2GB. Maybe CF would work fine with the max heap set to 1.5GB, or even 1GB.

                     

                    Of course, the natural next question is “so what should I set the CF max heap size to?” No one can tell you. It depends on too many variables. But I would propose (if you can’t increase memory on the box, or lower the memory used by other things) that you should set it lower than it is now, and then watch your CF server logs (\runtime\logs or \logs) to watch out for any outofmemory heap space errors. Those happen when in fact the heap fills. (The “swap space”, though also an “outofmemory” error, happens in my experience BEFORE the heap fills, because of this contention.)

                     

                     

                    Let us know if this makes sense, or what you find.

                     

                     

                     

                    /charlie

                    • 7. Re: server instances starts automatically
                      lincoln311219 Community Member

                      I reduced the  max heap size for CF to 512 MB . It creashed again.

                      • 8. Re: server instances starts automatically
                        charlie@carehart.org CommunityMVP

                        The key question is, did it crash with the exact same reason? Before you say yes, can you tell us how you're observing that? Which log file? And also, are you looking at just the last message in the log before it crashes? Look instead at seconds or minutes before, to see if there may be some other error as well.

                         

                        Second, since you're running multiserver, are you sure that the jvm.config you're changing is the one used by the instance in question? How are you confirming that? I've seen people who did not realize that they had multiple jvm.configs and different instances using different ones. I have a blog entry explaining that: http://www.carehart.org/blog/client/index.cfm/2012/6/26/identifying_what_instance_uses_a_g iven_jvm_config

                         

                        Please don't take offense in my asking these questions and sharing all the detail above. I can't know what you do and don't know, so am simply trying to help you (and any other readers who may find this thread in the future).

                         

                        Message was edited by: Charlie Arehart Corrected a couple of minor typos

                        • 9. Re: server instances starts automatically
                          lincoln311219 Community Member

                          am checking logs in JRun4\bin

                          we are using only one JVM.cofig file

                          As of now the server instances are stable

                          the changes we made are

                           

                          1. Disable debugging in production environment

                          2. Change Client Variable storage to Cookies or Database , right now it is Registry.

                          3. Update JDK version , currently we are using 1.6.17 , we should find out the latest      version      supported      for      ColdFuion 9.0.1 (1.6.24?)

                          4. Tune Garbage Collection parameters in jvm.config

                          5. Set minimum heap size to 512 MB

                          6. Optimize the pages which take long time to process , then reduce the "Time out request time " , we have set it to 210 seconds