1 Reply Latest reply: Sep 19, 2014 5:37 AM by sbleon RSS

    Coldfusion 10 Enterprise with Tomcat + mod_jk and Apache2 experiencing child process hangups

    jbrock777 Community Member

      I am experiencing the most bizarre thing that so far I am unable to reproduce with my own visits to the site.

       

      After restarting Apache2 my cacti graphs show that the child processes increment consistently over the course of a day without dropping back down during off hours.  This behavior eventually leaves the website inaccessible...

       

      Looking at server-status it is filled with Ws (Sending Reply) and GET calls to my cfm applications :


      Current Time: Tuesday, 22-Jul-2014 16:33:00 PDTRestart Time: Monday, 21-Jul-2014 22:51:12 PDTParent Server Generation: 0Server uptime: 17 hours 41 minutes 48 secondsTotal accesses: 194844 - Total Traffic: 3.8 GBCPU Usage: u201.55 s34.46 cu0 cs0 - .37% CPU load3.06 requests/sec - 63.2 kB/second - 20.6 kB/request73 requests currently being processed, 4 idle workers

      WWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWCWWWWW_WWWWWWCWWW_WWW _WKW...KWW.W_KWW....W........................................... ................................................................ ................................................................ 

      Scoreboard Key:
      "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
      "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
      "C" Closing connection, "L" Logging, "G" Gracefully finishing,
      "I" Idle cleanup of worker, "." Open slot with no current process

       

      SrvPIDAccMCPUSSReqConnChildSlotClientVHostRequest
      0-0150740/46/1370W7.394615800.00.4423.89192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=7000 HTTP/1.1
      1-0115630/47/468W2.755886700.04.6913.64192.168.1.10www.mysite.eduGET /catalog/index.cfm?courselist=list&dept=&searchc=PEHW%20148
      2-0129060/65/884W7.305453600.00.8014.62192.168.1.10www.mysite.eduGET /athletics/resources/nwaacc-athlete-of-the-week/ HTTP/1.1
      3-0138400/41/1085W4.015116200.00.5620.57192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=17001 HTTP/1.1
      4-0159280/20/1635W5.404371500.00.0641.37192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=37000 HTTP/1.1
      5-0187740/19/2387W0.333456400.00.2452.70192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=19009 HTTP/1.1
      6-043210/36/6612W3.611320000.00.28129.74192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=department&deptexp=28011 HTT
      7-0130770/0/808W0.425438300.00.0024.81192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=department&deptexp=6005 HTTP
      8-0164880/118/1673W12.394069200.01.3035.44192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=31003 HTTP/1.1
      9-0107260/15/110W0.586196300.00.051.83192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=All&index=Q HTTP/1.1
      10-0131540/1/688W0.005416500.00.0016.83192.168.1.10www.mysite.eduGET /directory/?directory=All&firstname=Patrick&lastname=Murphy
      11-0125900/25/516W4.455585100.00.7611.19192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=4000 HTTP/1.1
      12-0125510/13/454W1.845605500.00.3810.00192.168.1.10www.mysite.eduGET /directory/index.cfm?directory=department&deptexp=20001 HTT
      13-0133330/23/626W3.865318900.00.5711.66192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=31005 HTTP/1.1
      14-0124100/13/387W2.705648400.00.4210.55192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=6003 HTTP/1.1
      15-0131620/70/389W10.815311400.00.865.60192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=6005 HTTP/1.1
      16-0123090/22/275W2.235687800.00.433.91192.168.1.10www.mysite.eduGET /directory/?directory=department&deptexp=20005 HTTP/1.1
      17-0131630/57/341W11.855312000.01.386.49192.168.1.10www.mysite.eduGET /catalog/index.cfm?courselist=list&dept=&searchc=ENGR%26%20

       

       

      I have straced a hung process to only find the following :

      strace -p 6472

      Process 6472 attached - interrupt to quit

      read(23,

       

       

      Another interesting bit of info, none of these GET requests make it into my access.log file which I find very peculiar as well.

       

      Here are my CF Specs

      Server Details

      Server Product ColdFusion

      Version 10,0,13,287689

      Tomcat Version 7.0.23.0

      Edition Enterprise 

      Serial Number

      Operating System UNIX 

      OS Version 3.2.0-65-generic 

      Update Level /opt/coldfusion10/cfusion/lib/updates/chf10000013.jar 

      Adobe Driver Version 4.1 (Build 0001) 

      JVM Details

      Java Version 1.6.0_29 

      Java Vendor Sun Microsystems Inc. 

       

       

      Here are my Apache2 Specs

      Server version: Apache/2.2.22 (Ubuntu)

      Server built:   Apr 17 2014 21:49:25

      Server's Module Magic Number: 20051115:30

      Server loaded:  APR 1.4.6, APR-Util 1.3.12

      Compiled using: APR 1.4.6, APR-Util 1.3.12

      Architecture:   64-bit

      Server MPM:     Prefork

        threaded:     no

          forked:     yes (variable process count)

      Server compiled with....

      -D APACHE_MPM_DIR="server/mpm/prefork"

      -D APR_HAS_SENDFILE

      -D APR_HAS_MMAP

      -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)

      -D APR_USE_SYSVSEM_SERIALIZE

      -D APR_USE_PTHREAD_SERIALIZE

      -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT

      -D APR_HAS_OTHER_CHILD

      -D AP_HAVE_RELIABLE_PIPED_LOGS

      -D DYNAMIC_MODULE_LIMIT=128

      -D HTTPD_ROOT="/etc/apache2"

      -D SUEXEC_BIN="/usr/lib/apache2/suexec"

      -D DEFAULT_PIDLOG="/var/run/apache2.pid"

      -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"

      -D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"

      -D DEFAULT_ERRORLOG="logs/error_log"

      -D AP_TYPES_CONFIG_FILE="mime.types"

      -D SERVER_CONFIG_FILE="apache2.conf"

       

       

      I am hoping this is no normal behavior for Coldfusion 10.

       

      Many thanks in advance.