Skip navigation
Currently Being Moderated

Help: CF10 (all hotfixes) on Win2K3 starts to consume up to 100% cpu (coldfusion.exe)

Jan 3, 2013 3:14 PM

Tags: #windows #cpu #load #100% #cf10

hi

 

We have a CF9 latest updates running on a Win2K3 server  under Apache 2.2.22 and JRun4 for years. Now, I wanted to upgrade to CF10.

 

I first updated to Apache 2.2.23 with ssl 1.0.1c

 

For reassurance, I just disabled all JRUN CF9 services as seen in the Task Manager.

 

Then I installed CF10. The installer mentioend different  NET Ports. I accepted them.

 

Install continued without a problem, CF10 was setup and I configured it. I then ran our main ap which also worked fine.

 

Then, after some few hours while I was checking the health of it, I figured that  COLDFUSION.EXE takes up all CPU time.

 

Server response  is still there but sluggish.I have not seen any reason why the cpu usage increases.

 

I have to kill coldfusion.exe to cut cpu load. however, a few minutes after that (about 10-20 minutes) coldfusion.exe again begins to climb up. it takes about 5-10 minutes to hit 100% ...

 

I have no idea how to deal with that (except going back to CF9)

 

On my developer CF10 (same config) I have not seen that behavior. Howewer, this server is not exposed to the net, the productive one is ... there might be some deep links pointing somewhere ... however, our server is not a busy server, so it should not face such a cpu load .. as said it ran under CF9 for years without such problems.

 

any hints how to detect the cause of this? Can I use any CF10 tools to figure out this problem?

 

any other clues to that?

 

it's urging ... as usual ...

 

Thank

 

Martin

 
Replies
  • Currently Being Moderated
    Jan 6, 2013 7:36 PM   in reply to tinu8805

    Hi Martin. Any errors or warnings in log files CF10\cfusion\logs\ coldfusion-error.log coldfusion-out.log ?

     

    How about enable CF10 Metrics logging, CF10admin > Debugging & Logging > Debug Output Settings > then check metrics.log after CF restart to enable?

     

    I know post CF10 update 5 WSCONFIG had some webserver connector updates in case of Windows and IIS. I guess with Apache you would be using a connector / extension / mod_jk but I have not been following CF10 update matters that relate to Apache webserver. Could be worth knowing if you need to apply a CF10 WSCONFIG webserver update with Apache in use?

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 16, 2013 5:35 AM   in reply to tinu8805

    Hi Martin,

     

    we just migrated last week and ran into a similar issue, having to restart CF every couple of hours.  We thought we had a memory leak

    but.....  it turns out that we were running server monitor and it was sucking all the resources.  Go into cf admin >> server monitor>>monitor settings  and if they are enabled, uncheck them.

     

    We did this and it solved our problems.

     

    hope this helps.

     

    JB

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 17, 2013 2:57 AM   in reply to tinu8805

    AJP is the protocol "CF10 / java / tomcat" uses to the mod_jk connector which inturn talks to your apache webserver.
    If it was IIS I would suggest check CF10\config\wsconfig\N\isapi_redirect.log however I have not done CF10 with apache so hard for me to say. Is there a log that goes with mod_jk (it is late for me so tomorrow perhaps I will search detail)? I still think CF10\cfusion\logs\coldfusion-error.log warning or error messages would be interesting.


    Your server.xml AJP section would be interesting to know eg:
    \CF10\cfusion\runtime\conf\server.xml
    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8012" protocol="AJP/1.3"
    redirectPort="8445"
    tomcatAuthentication="false"
    maxThreads= etc
    />

     

    Also Martin you can monitor AJP with JMX settings, JDK tools Jconsole and Jvisualvm. If your are familiar with JMX and Jconsole
    use that to monitor MBeans, Catalina (aka tomcat) threadpool AJP particularly currentthreadcount currenthreadbusy. Get back to me if you are interested in more details on any of that.

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 17, 2013 4:25 PM   in reply to tinu8805

    I think the AJP + mod_jk needs tuning for production environment. Some of the default tomcat AJP + mod_jk settings may not meet your load.
    While I have not done any (until today) CF10 + Windows + Apache I have become familiar with CF10 + Windows + IIS which uses an isapi connector. Tuning the AJP + isapi connector is important for performance and stability. Likely AJP + mod_jk similarly needs tuning.


    What values to tune for AJP and connector?

     

    Refer to some paste from tomcat doc:
    maxThreads
    The maximum number of request processing threads to be created by
    this Connector, which therefore determines the maximum number of
    simultaneous requests that can be handled. If not specified, this
    attribute is set to 200.
    minSpareThreads
    The number of request processing threads that will be created when
    this Connector is first started. The connector will also make sure it
    has the specified number of idle processing threads available. This
    attribute should be set to a value smaller than that set for
    maxThreads. The default value is 10.
    ConnectionTimeout
    The number of milliseconds this Connector will wait, after accepting a
    connection, for the request URI line to be presented. The default
    value for AJP protocol connectors is -1 (i.e. infinite).

     

    How to apply those settings?

     

    Edit (take a copy of first so you got a backup) CF10\cfusion\runtime\conf\server.xml AJP part -
    Add values for maxthreads, minspare, timeout 

     

    Edit (backup copy first) CF10\config\wsconfig\N\workers.properties -
    Add values for connection_pool_size, connection_pool_timeout, connection_pool_minsize .

     

    Restart Apache and CF10 service to apply changes.


    EG

     

    Server.xml part:

    <Connector port="8012"
    protocol="AJP/1.3"
    redirectPort="8445"
    tomcatAuthentication="false"
    maxThreads="n00"
    minSpareThreads="n00/2"
    connectionTimeout="n0000"
    />

     

    workers.properties:

     

    worker.list=cfusion


    worker.cfusion.type=ajp13
    worker.cfusion.host=localhost
    worker.cfusion.port=8012
    worker.cfusion.max_reuse_connections=250
    worker.cfusion.connection_pool_size = n00
    worker.cfusion.connection_pool_timeout = n0
    worker.cfusion.connection_pool_minsize=n00/3

     

     

    But what values?

     

    It is hard for me to say since I have not seen problem server or have any monitoring details (eg JXM + Jconsole).
    Note server.xml AJP timeout part is in milliseconds where as workers.properties timeout value is in seconds.
    I do not want to say one set of values is a cure all  however as an initial change I would double the default values for maximum, more than double minimum (like Java JVM matters sometimes the minimum settings are the most important) and add timeout. EG -

     

    Server.xml AJP part:

     

    <Connector port="8012"
    protocol="AJP/1.3"
    redirectPort="8445"
    tomcatAuthentication="false"
    maxThreads="400"
    minSpareThreads="80"
    connectionTimeout="60000"
    />

     

    workers.properties:

     

    worker.list=cfusion


    worker.cfusion.type=ajp13
    worker.cfusion.host=localhost
    worker.cfusion.port=8012
    worker.cfusion.max_reuse_connections=500
    worker.cfusion.connection_pool_size = 400
    worker.cfusion.connection_pool_timeout = 60
    worker.cfusion.connection_pool_minsize=80

     

     

     

    Well I typed all that before your update with CFSTAT and lack of PerfMon. I will try comment on that soon.

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 17, 2013 5:13 PM   in reply to tinu8805

    Hi Martin,

     

    I do not think the stopped CF9 would effect things other than a possible issue with the Webserver (apache) having a Jrun connector on it as well as a tomcat connector. Seems to me since you are managing to change CF10 to CF9 versa visa you have your apache change to tomcat or jrun connector versa visa sorted.

     

    Could the CPU load climb after a period due to tomcat connector default timeout settings are infinite? Guessing over time all threads are used none handed back to the queue. Harking on here about AJP (server.xml) and mod_jk (workers.properties) timeout settings.

     

    I find CF10 CFSTAT odd say in comparison with CFSTAT on CF9 and earlier. There is probably more I can say on that but will not just now.
    CF10admin > Debugging & Logging > Debug Output Settings > What is your value for "Connector Port"?

     

    Cheers, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 18, 2013 1:23 PM   in reply to tinu8805

    Some closing thoughts from me. Following DB updates if you want to touch on this again just touch the thread.

     

    Tomcat connector logs [info] messages are normal tho [error] and [warning] are matters of concern. The mod_jk log is showing some [error] so stop these from happening and system should work without failing.

     

    For CFSTAT what is your CF10admin > Debugging & Logging > Debug Output Settings > "Connector Port" value?

     

    Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 20, 2013 2:50 PM   in reply to tinu8805

    Regarding CFSTAT. Your server.xml and workers.properties define connector port or AJP protocol on port 8012.

    Debugging & Logging > Debug Output Settings > Connector Port should also be that value 8012 (not 8500) to provide counters for the other statistics. CF may need a restart to apply.

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 20, 2013 4:10 PM   in reply to tinu8805

    Restart CF10 every 15 minutes that’s got to be awful for all concerned.

     

    I think I have made a fairly good suggestion to backup and apply a change for tomcat parameters. I figure you got everything to gain and nothing lost by trying a change and given a backup copy you can always copy back to original configuration.

     

    Certainly applying a change without having some benchmark logs or monitoring to refer, before and after, can leave you unaware as to outcome. Some may offer changing many parameters at once cumbersome and I agree but I am not remotely connected to the server to be hands on and have diagnosis to refer upon. If however a change is applied mod_jk.log stops reporting [error] and CF10 does not need restart  you have a good idea that changes have been beneficial.

     

    Again, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 21, 2013 8:59 PM   in reply to tinu8805

    Re -  cf10\coldfusion-error.log (of a restarted service) lists this first ... something to be worried?

     

    Apache Tomcat Native Library - What is it? Some paste from documents say-
    The Apache Tomcat Native Library provides portable API for features not found in contemporary JDK's. It uses Apache Portable Runtime as operating system abstraction layer and OpenSSL for SSL networking and allows optimal performance in production environments.

     

    Download tomcat-native-1.1.23-win32-bin

     

    note \bin contains tcnative-1.dll ie 32 bit and bin\x64 has the 64 bit tcnative-1.dll

     

    Copy correct bit dll to CF10\cfusion\lib

     

    Restart CF10 to apply

     

    CF10 coldfusion-error.log will now report:

     

    org.apache.coyote.AbstractProtocol init
    INFO: Initializing ProtocolHandler ["ajp-apr-8012"]
    org.apache.catalina.core.StandardService startInternal
    INFO: Starting service Catalina
    org.apache.catalina.core.StandardEngine startInternal
    INFO: Starting Servlet Engine: Apache Tomcat/7.0.23
    etc

     

    You may well not be using "Apache Tomcat Native Library" however if you like to be rid of the INFO optimal performance message – that’s how.

     

    Once more, Carl.

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points