4 Replies Latest reply on Jun 5, 2012 2:49 PM by azimuthinternet

    JRun Thread Pool Issue

    azimuthinternet Level 1

      I'm running CF 9.0.1 on Ubuntu on an "Medium" Amazon EC2 instance. CF has been crashing intermittently (several times per day). At such times, running top gets me this (or something similar):


      15855wwwrun2001762m730m20mS99.319.413:22.96 coldfusion9


      So, it's obviously consuming most of the server resources. The following error has been showing up in my cfserver.log in the leadup to each crash:


      java.lang.RuntimeException: Request timed out waiting for an available thread to run. You may want to consider increasing the number of active threads in the thread pool.


      If I run /opt/coldfusion9/bin/coldfusion status, I get:


      Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes

      Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec

      0   0   0   0   -1  -1  150   25    0     0      -1352560      0      0


      In the administrator, under Server Settings > Request Tuning, the setting for Maximum number of simultaneous Template requests is 25. So this makes sense so far. I could just increase the thread pool to cover these sort of load spikes. I could make it 200. (Which I did just now as a test.)


      However, there's also this file /opt/coldfusion9/runtime/servers/coldfusion/SERVER-INF/jrun.xml. And some of the settings in there appear to conflict. For example, it reads:


      <service class="jrunx.scheduler.SchedulerService" name="SchedulerService">

        <attribute name="bindToJNDI">true</attribute>

        <attribute name="activeHandlerThreads">25</attribute>

        <attribute name="maxHandlerThreads">1000</attribute>

        <attribute name="minHandlerThreads">20</attribute>

        <attribute name="threadWaitTimeout">180</attribute>

        <attribute name="timeout">600</attribute>



      Which a) has fewer active threads (what does this mean?), and b) has a max threads that exceed the simultaneous request limit set in the admin. So, I'm not sure. Are these independent configs that need to be made to match manually? Or is the jrun.xml file supposed to be written by the CF Admin when changes are made there? Hmm. But maybe this is different because presumably the CF Scheduler should only use a subset of all available threads, right...so we'd always have some threads for real live users. We also have this in there:


      <service class="jrun.servlet.http.WebService" name="WebService">

        <attribute name="port">8500</attribute>

        <attribute name="interface">*</attribute>

        <attribute name="deactivated">true</attribute>

        <attribute name="activeHandlerThreads">200</attribute>

        <attribute name="minHandlerThreads">1</attribute>

        <attribute name="maxHandlerThreads">1000</attribute>

        <attribute name="mapCheck">0</attribute>

        <attribute name="threadWaitTimeout">300</attribute>

        <attribute name="backlog">500</attribute>

        <attribute name="timeout">300</attribute>



      This appears to have changed when I changed the CF Admin setting...maybe...but it's the activeHandlerThreads that matches my new maximum simulataneous requests setting...rather than the maxHandlerThreads, which again exceeds it. Finally, we have this:


      <service class="jrun.servlet.jrpp.JRunProxyService" name="ProxyService">

        <attribute name="activeHandlerThreads">200</attribute>

        <attribute name="minHandlerThreads">1</attribute>

        <attribute name="maxHandlerThreads">1000</attribute>

        <attribute name="mapCheck">0</attribute>

        <attribute name="threadWaitTimeout">300</attribute>

        <attribute name="backlog">500</attribute>

        <attribute name="deactivated">false</attribute>

        <attribute name="interface">*</attribute>

        <attribute name="port">51800</attribute>

        <attribute name="timeout">300</attribute>

        <attribute name="cacheRealPath">true</attribute>



      So, I'm not certain which (if any) of these I should change and what exactly the relationship is between maximum requests and maximum threads. Also, since several of these list the maxHandlerThreads as 1000, I'm wondering if I should just set the maximum simultaneous requests to 1000. There must be some upper limit that depends on available server resources...but I'm not sure what it is and I don't really want to play around with it since it's a production environment.


      I'm not sure if it pertains to this issue at all, but when I run a ps aux | grep coldfusion I get the following:


      wwwrun   15853  0.0  0.0   8704   760 pts/1S

      20:22   0:00 /opt/coldfusion9/runtime/bin/coldfusion9 -jar jrun.jar -autorestart -start coldfusion

      wwwrun   15855  5.4 18.2 1678552 701932 pts/1   Sl20:22   1:38 /opt/coldfusion9/runtime/bin/coldfusion9 -jar jrun.jar -start coldfusion


      There are always these two and never more than these two processes. So there does not appear to be a one-to-one relationship between processes and threads. I recall from an MX 6.1 install I maintained for many years that additional CF processes were visible in the process list. It seemed to me at the time like I had a process for each thread...so either I was wrong or something is quite different in version 9 since it's reporting 25 running requests and only showing these two processes. If a single process can have multiple threads in the background, then I'm given to wonder why I have two processes instead of one...just curious.


      So, anyway, I've been experimenting while composing this post. As noted above I adjusted the maximum simulataneous requests up to 200. I was hoping this would solve my problem, but CF just crashed again (rather it slogged down and requests started timing out...so effectively "crashed"). This time, top looked similar (still consuming more than 99% of the CPU), but CF status looked different:


      Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes

      Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec

      0   0   0   0   -1  -1  0     150   0     0      0      0      0      0


      Obviously, since I'd increased the maximum simultaneous requests, it was allowing more requests to run simultaneously...but it was still maxing out the server resources.


      Further experiments (after restarting CF) showed me that the server became unusably slogged after about 30-35 "Reqs Run'g", with all additional requests headed for an inevitible timeout:


      Pg/Sec  DB/Sec  CP/Sec  Reqs  Reqs  Reqs  AvgQ   AvgReq AvgDB  Bytes  Bytes

      Now Hi  Now Hi  Now Hi  Q'ed  Run'g TO'ed Time   Time   Time   In/Sec Out/Sec

      0   0   0   0   -1  -1  0     33    0     0      -492   0      0      0


      So, it's clear that increasing the maximum simultaneous requests has not helped. I guess what it comes down to is this: What is it having such a hard time with? Where are these spikes coming from? Bursts of traffic? On what pages? What requests are running at any given time? I guess I simply need more information to continue troubleshooting. If there are long-running requests, or other issues, I'm not seeing it in the logs (although I do have that option checked in the admin). I need to know which requests exactly are those responsible for these spikes. Any help would be much appreciated. Thanks.



        • 1. Re: JRun Thread Pool Issue

          In the admin you can enable the server monitor which will give you more metrics than the logs. You can view long running threads/queries, active threads, request statistics, etc... Do not enable memory tracking unless specifically needed as it will affect performance. You may also look into fusion reactor for more in-depth metrics. Ideally, having server monitoring enabled and open, you can view which requests are getting hung up and address it appropriately.

          • 2. Re: JRun Thread Pool Issue
            azimuthinternet Level 1

            Thanks for your response. I tried what you suggested, but unfortunately I received the message, "This feature is not available in this edition of the ColdFusion server." I'm running Standard Edition. I'm guessing this feature is only enabled in Enterprise. Bummer. I will look into FusionReactor too. Thanks again.

            • 3. Re: JRun Thread Pool Issue
              j_caito Level 1

              In the administrator, there are a few more settings you can explore:


              Maximum number of running JRun threads

              This is how many threads your JEE server can execute at any time. If there are 25 threads running and a new thread spawns, then the thread gets put into a queue.


              Maximum number of queued JRun Threads

              This is the size of the JEE thread queue. Once this get's full, JRun can no longer process threads and your JVM will usually crash.


              This is the setting you changed. This is how many clients can simultaneously make a request to ColdFusion. This is not the setting you wanted.


              Most of the settings that are changed in the admin get persisted to a bunch of neo*.xml files stored in (depending on your installation method) C:\JRun4\servers\cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\lib


              I would adjust the number of simultaneous JRun threads up to see if that helps and definitely check the error logs to see if there is a particular template / cfc that is timing out. You can also up your JVM size in your jvm.config file.


              1 person found this helpful
              • 4. Re: JRun Thread Pool Issue
                azimuthinternet Level 1

                I really appreciate your help. However, I haven't been able to find the JRun Thread settings you describe above.


                Under Request Tuning, I see:




                Request Limits

                  Restricts the number of simultaneously processed requests. Use this setting to increase overall system performance for heavy load applications. Requests beyond the specified limit are queued. On Standard Edition, you must restart ColdFusion to enable this setting. 

                  The number of Flash Remoting requests that can be processed concurrently.

                  The number of Web Service requests that can be processed concurrently.

                  The number of ColdFusion Component methods that can be processed concurrently via HTTP. This does not affect invocation of CFC methods from within CFML, only methods requested via an HTTP request.



                Tag Limit Settings

                  The maximum number of ColdFusion reports that can be processed concurrently.

                  The maximum number of threads created by CFTHREAD that will be run concurrently. Threads created by CFTHREAD in excess of this are queued.  On Standard Edition, the maximum limit is 10. 


                And under Java and JVM, I see:



                    Java and JVM settings control the way ColdFusion starts the Java Virtual Machine when it starts.  You can control settings like what classpaths are used and how memory is allocated as well as add custom command line arguments.  Changing these settings requires restarting ColdFusion.  If you enter an incorrect setting, ColdFusion may not restart properly. 


                   Backups of the jvm.config file are created when you hit the submit button. You can use this backup to restore from a critical change. 




                  Specifies the location of the Java Virtual Machine.



                   The Memory Size settings determine the amount of memory that the JVM can use for programs and data. 




                  Specifies any additional class paths for the JVM, with multiple directories separated by  commas.




                  Specifies any specific JVM initialization options, separated by spaces.


                I did go take a look at FusionReactor and found it's not free (which would be fine, of course, if it would actually help). It looks like there's a fully functional demo, which is cool...but I've haven't been able to get it to install yet, so we'll see.


                Thanks again!




                (By the way, I've cross-posted this inquiry on StackOverflow. So if you're able to help me arrive at a solution you might want to answer there as well.)