7 Replies Latest reply: Oct 6, 2014 9:32 AM by PhilBro RSS

    Memory leak after upgrading to ColdFusion 10

    DFresh4130 Community Member
    DFresh4130 Sep 24, 2014 1:53 PM

      We recently upgraded from CF8 to CF10 and we're running into some issues.  We started off getting a lot of OutOfMemory errors with the default heap settings. Chaning them to 768MB/1280MB which has helped, but we're still running into occasional OOM issues.  We took a heapdump as the instance was having problems today with blocked threads and noticed the heap usage was near the max so it was probably going to run out of memory soon too.  Below is what the eclipse memory analyzer tool found as a possible leak suspect.  Does anyone have any suggestions on where to go from here?  Not sure if it's an issue with CF itself or something in the app.  Thanks

       

      Description

       

      One instance of "coldfusion.runtime.ClientScopeServiceImpl" loaded by "coldfusion.bootstrap.BootstrapClassLoader @ 0x120bbaf0" occupies 838,847,880 (88.33%) bytes. The memory is accumulated in one instance of "java.util.HashMap$Entry[]" loaded by "<system class loader>".

       

      Keywords
      coldfusion.runtime.ClientScopeServiceImpl
      java.util.HashMap$Entry[]
      coldfusion.bootstrap.BootstrapClassLoader @ 0x120bbaf0

       

      Shortest Paths To the Accumulation Point
      Class NameShallow HeapRetained Heap
      2,097,168838,843,064
      48838,843,112
      88838,847,880
      11221,624
      11219,552
      11219,392
      11222,288
      11218,256
      11223,776
      11218,160
      11218,816
      11219,784
      11219,208
      11219,200
      11220,592
      11221,304
      1121,032
      9655,184
      • Total: 15 entries
      Accumulated Objects
      Class NameShallow HeapRetained HeapPercentage
      88838,847,88088.33%
      48838,843,11288.33%
      2,097,168838,843,06488.33%
      2451,409,1285.41%
      2412,173,1921.28%
      2410,685,7601.13%
      2410,587,2081.11%
      2410,443,6641.10%
      249,886,2801.04%
      249,723,2161.02%
      249,254,2880.97%
      246,494,4880.68%
      246,178,3200.65%
      244,008,0480.42%
      243,542,5520.37%
      243,503,0720.37%
      242,908,1200.31%
      242,197,0560.23%
      242,050,5280.22%
      242,034,2240.21%
      241,846,8080.19%
      241,846,7840.19%
      241,846,7840.19%
      • Total: 20 entries
      		480162,619,5200.171
      Accumulated Objects by Class
      LabelNumber of ObjectsUsed Heap SizeRetained Heap Size
      java.util.HashMap$Entry
      First 10 of 190,465 objects      		190,4654,571,160836,711,024
      java.lang.String
      First 10 of 253 objects      		2536,07226,408
      java.lang.reflect.Method
      First 10 of 57 objects      		574,5604,560
      coldfusion.runtime.ClientScope
      All 1 objects      		1562,632
      java.lang.Class[]
      First 10 of 73 objects      		731,2481,248
      coldfusion.runtime.ClientScopeKey
      All 1 objects      		12424
      java.lang.Class
      All 2 objects      		200
      • Total: 7 entries
      		190,8524,583,120836,745,896
        • 1. Re: Memory leak after upgrading to ColdFusion 10
          Sharma Nimit Adobe Employee
          Sharma Nimit Sep 24, 2014 11:22 PM (in response to DFresh4130)

          Hi,

           

          Please share the following details:

           

          1. Settings summary

          2. jvm.config

          3. How frequently you are facing this issue?

          4. How much time it take to throw error after restarting the ColdFusion server?

           

          Regards,

          Nimit

          • 2. Re: Memory leak after upgrading to ColdFusion 10
            DFresh4130 Community Member
            DFresh4130 Sep 25, 2014 9:31 AM (in response to Sharma Nimit)

            Hi Sharma,

             

            Here's the settings summary from the console.  The OOM issue is now happening about 1-2 times weekly after increasing the heap size.  Before that it was almost daily.  It seems we can go about 75-85 hours before we restart due to either OOM or blocked threads.

             

            Settings Summary

             

            Server Settings > Settings Summary

             

            Report generated on Sep 25, 2014 12:08 PM

             

            This report shows the status of all ColdFusion configuration settings. To display the area of the ColdFusion Administrator where you can edit the group settings, click any of the groups in the report.

             

            System Information

            Server Details

            Server Product ColdFusion

            Version ColdFusion 10,286680

            Edition Enterprise 

            Operating System Windows 2003 

            OS Version 5.2 

            Update Level /D:/ColdFusion10/cfusion/lib/updates/chf10000012.jar 

            Adobe Driver Version 4.1 (Build 0001) 

             

            JVM Details

            Java Version 1.7.0_15 

            Java Vendor Oracle Corporation 

            Java Vendor URL http://java.oracle.com/ 

            Java Home D:\ColdFusion10\jre 

            Java File Encoding Cp1252 

            Java Default Locale en_US 

            File Separator \ 

            Path Separator ; 

            Line Separator Chr(13) 

            User Name SYSTEM 

            User Home C:\Documents and Settings\Default User 

            User Dir D:\ColdFusion10\cfusion\bin 

            Java VM Specification Version 1.7 

            Java VM Specification Vendor Oracle Corporation 

            Java VM Specification Name Java Virtual Machine Specification 

            Java VM Version 23.7-b01 

            Java VM Vendor Oracle Corporation 

            Java VM Name Java HotSpot(TM) Server VM 

            Java Specification Version 1.7 

            Java Specification Vendor Oracle Corporation 

            Java Specification Name Java Platform API Specification 

            Java Class Version 51.0 

            CF Server Java Class Path ;D:/ColdFusion10/cfusion/lib/updates/chf10000012.jar;  D:/ColdFusion10/cfusion/lib/ant-launcher.jar;  D:/ColdFusion10/cfusion/lib/ant.jar;  D:/ColdFusion10/cfusion/lib/antlr-2.7.6.jar;  D:/ColdFusion10/cfusion/lib/apache-solr-core.jar;  D:/ColdFusion10/cfusion/lib/apache-solr-solrj.jar;  D:/ColdFusion10/cfusion/lib/asm-all-3.1.jar;  D:/ColdFusion10/cfusion/lib/asn1.jar;  D:/ColdFusion10/cfusion/lib/axis.jar;  D:/ColdFusion10/cfusion/lib/backport-util-concurrent.jar;  D:/ColdFusion10/cfusion/lib/bcel-5.1-jnbridge.jar;  D:/ColdFusion10/cfusion/lib/bcel.jar;  D:/ColdFusion10/cfusion/lib/bcmail-jdk14-139.jar;  D:/ColdFusion10/cfusion/lib/bcprov-jdk14-139.jar;  D:/ColdFusion10/cfusion/lib/cdo.jar;  D:/ColdFusion10/cfusion/lib/cdohost.jar;  D:/ColdFusion10/cfusion/lib/certj.jar;  D:/ColdFusion10/cfusion/lib/cf-acrobat.jar;  D:/ColdFusion10/cfusion/lib/cf-assembler.jar;  D:/ColdFusion10/cfusion/lib/cf-logging.jar;  D:/ColdFusion10/cfusion/lib/cf4was.jar;  D:/ColdFusion10/cfusion/lib/cf4was_ae.jar;  D:/ColdFusion10/cfusion/lib/cfusion-req.jar;  D:/ColdFusion10/cfusion/lib/cfusion.jar;  D:/ColdFusion10/cfusion/lib/chart.jar;  D:/ColdFusion10/cfusion/lib/clibwrapper_jiio.jar;  D:/ColdFusion10/cfusion/lib/commons-beanutils-1.8.0.jar;  D:/ColdFusion10/cfusion/lib/commons-codec-1.3.jar;  D:/ColdFusion10/cfusion/lib/commons-collections-3.2.1.jar;  D:/ColdFusion10/cfusion/lib/commons-compress-1.0.jar;  D:/ColdFusion10/cfusion/lib/commons-digester-2.0.jar;  D:/ColdFusion10/cfusion/lib/commons-discovery-0.4.jar;  D:/ColdFusion10/cfusion/lib/commons-httpclient-3.1.jar;  D:/ColdFusion10/cfusion/lib/commons-lang-2.4.jar;  D:/ColdFusion10/cfusion/lib/commons-logging-1.1.1.jar;  D:/ColdFusion10/cfusion/lib/commons-logging-api-1.1.1.jar;  D:/ColdFusion10/cfusion/lib/commons-net-3.0.1.jar;  D:/ColdFusion10/cfusion/lib/commons-vfs2-2.0.jar;  D:/ColdFusion10/cfusion/lib/crystal.jar;  D:/ColdFusion10/cfusion/lib/derby.jar;  D:/ColdFusion10/cfusion/lib/derbyclient.jar;  D:/ColdFusion10/cfusion/lib/derbynet.jar;  D:/ColdFusion10/cfusion/lib/derbyrun.jar;  D:/ColdFusion10/cfusion/lib/derbytools.jar;  D:/ColdFusion10/cfusion/lib/dom4j-1.6.1.jar;  D:/ColdFusion10/cfusion/lib/dpHibernate.jar;  D:/ColdFusion10/cfusion/lib/ehcache-core-2.5.1.jar;  D:/ColdFusion10/cfusion/lib/ehcache-web-2.0.4.jar;  D:/ColdFusion10/cfusion/lib/esapi-2.0.1.jar;  D:/ColdFusion10/cfusion/lib/EWSAPI-1.1.5.jar;  D:/ColdFusion10/cfusion/lib/FCSj.jar;  D:/ColdFusion10/cfusion/lib/flashgateway.jar;  D:/ColdFusion10/cfusion/lib/flex-messaging-common.jar;  D:/ColdFusion10/cfusion/lib/flex-messaging-core.jar;  D:/ColdFusion10/cfusion/lib/flex-messaging-opt.jar;  D:/ColdFusion10/cfusion/lib/flex-messaging-proxy.jar;  D:/ColdFusion10/cfusion/lib/flex-messaging-remoting.jar;  D:/ColdFusion10/cfusion/lib/flex-rds-server.jar;  D:/ColdFusion10/cfusion/lib/geronimo-stax-api_1.0_spec-1.0.1.jar;  D:/ColdFusion10/cfusion/lib/hibernate3.jar;  D:/ColdFusion10/cfusion/lib/httpclient-4.1.1.jar;  D:/ColdFusion10/cfusion/lib/httpclient-cache-4.1.1.jar;  D:/ColdFusion10/cfusion/lib/httpclient.jar;  D:/ColdFusion10/cfusion/lib/httpcore_4.1.2.jar;  D:/ColdFusion10/cfusion/lib/httpmime-4.1.1.jar;  D:/ColdFusion10/cfusion/lib/ib6addonpatch.jar;  D:/ColdFusion10/cfusion/lib/ib6core.jar;  D:/ColdFusion10/cfusion/lib/ib6http.jar;  D:/ColdFusion10/cfusion/lib/ib6swing.jar;  D:/ColdFusion10/cfusion/lib/ib6util.jar;  D:/ColdFusion10/cfusion/lib/im.jar;  D:/ColdFusion10/cfusion/lib/iText.jar;  D:/ColdFusion10/cfusion/lib/iTextAsian.jar;  D:/ColdFusion10/cfusion/lib/izmado.jar;  D:/ColdFusion10/cfusion/lib/jai_codec.jar;  D:/ColdFusion10/cfusion/lib/jai_core.jar;  D:/ColdFusion10/cfusion/lib/jai_imageio.jar;  D:/ColdFusion10/cfusion/lib/jakarta-oro-2.0.6.jar;  D:/ColdFusion10/cfusion/lib/jakarta-slide-webdavlib-2.1.jar;  D:/ColdFusion10/cfusion/lib/java-xmlbuilder-0.4.jar;  D:/ColdFusion10/cfusion/lib/javasysmon-0.3.3.jar;  D:/ColdFusion10/cfusion/lib/jax-qname.jar;  D:/ColdFusion10/cfusion/lib/jaxb-api.jar;  D:/ColdFusion10/cfusion/lib/jaxb-impl.jar;  D:/ColdFusion10/cfusion/lib/jaxb-libs.jar;  D:/ColdFusion10/cfusion/lib/jaxb-xjc.jar;  D:/ColdFusion10/cfusion/lib/jaxrpc.jar;  D:/ColdFusion10/cfusion/lib/jcifs-1.3.15.jar;  D:/ColdFusion10/cfusion/lib/jdom.jar;  D:/ColdFusion10/cfusion/lib/jeb.jar;  D:/ColdFusion10/cfusion/lib/jersey-core.jar;  D:/ColdFusion10/cfusion/lib/jersey-server.jar;  D:/ColdFusion10/cfusion/lib/jersey-servlet.jar;  D:/ColdFusion10/cfusion/lib/jets3t-0.8.1.jar;  D:/ColdFusion10/cfusion/lib/jetty-continuation-7.0.0.v20091005.jar;  D:/ColdFusion10/cfusion/lib/jetty-http-7.0.0.v20091005.jar;  D:/ColdFusion10/cfusion/lib/jetty-io-7.0.0.v20091005.jar;  D:/ColdFusion10/cfusion/lib/jetty-security-7.0.0.v20091005.jar;  D:/ColdFusion10/cfusion/lib/jetty-server-7.0.0.v20091005.jar;  D:/ColdFusion10/cfusion/lib/jetty-servlet-7.0.0.v20091005.jar;  D:/ColdFusion10/cfusion/lib/jetty-servlets-7.0.0.v20091005.jar;  D:/ColdFusion10/cfusion/lib/jetty-util-7.0.0.v20091005.jar;  D:/ColdFusion10/cfusion/lib/jetty-xml-7.0.0.v20091005.jar;  D:/ColdFusion10/cfusion/lib/jintegra.jar;  D:/ColdFusion10/cfusion/lib/jnbcore.jar;  D:/ColdFusion10/cfusion/lib/jpedal.jar;  D:/ColdFusion10/cfusion/lib/js.jar;  D:/ColdFusion10/cfusion/lib/jsch-0.1.44m.jar;  D:/ColdFusion10/cfusion/lib/jsr107cache.jar;  D:/ColdFusion10/cfusion/lib/jsr311-api-1.1.1.jar;  D:/ColdFusion10/cfusion/lib/jta.jar;  D:/ColdFusion10/cfusion/lib/jutf7-0.9.0.jar;  D:/ColdFusion10/cfusion/lib/ldap.jar;  D:/ColdFusion10/cfusion/lib/ldapbp.jar;  D:/ColdFusion10/cfusion/lib/log4j-1.2.15.jar;  D:/ColdFusion10/cfusion/lib/lucene-analyzers-3.4.0.jar;  D:/ColdFusion10/cfusion/lib/lucene-core-3.4.0.jar;  D:/ColdFusion10/cfusion/lib/lucene-highlighter-3.4.0.jar;  D:/ColdFusion10/cfusion/lib/lucene-memory-3.4.0.jar;  D:/ColdFusion10/cfusion/lib/lucenedemo.jar;  D:/ColdFusion10/cfusion/lib/macromedia_drivers.jar;  D:/ColdFusion10/cfusion/lib/mail.jar;  D:/ColdFusion10/cfusion/lib/metadata-extractor-2.4.0-beta-1.jar;  D:/ColdFusion10/cfusion/lib/mlibwrapper_jai.jar;  D:/ColdFusion10/cfusion/lib/msapps.jar;  D:/ColdFusion10/cfusion/lib/mysql-connector-java-commercial-5.1.17-bin.jar;  D:/ColdFusion10/cfusion/lib/namespace.jar;  D:/ColdFusion10/cfusion/lib/nekohtml.jar;  D:/ColdFusion10/cfusion/lib/netty-3.2.5.Final.jar;  D:/ColdFusion10/cfusion/lib/ooxml-schemas.jar;  D:/ColdFusion10/cfusion/lib/pdfencryption.jar;  D:/ColdFusion10/cfusion/lib/poi-contrib.jar;  D:/ColdFusion10/cfusion/lib/poi-ooxml-schemas.jar;  D:/ColdFusion10/cfusion/lib/poi-ooxml.jar;  D:/ColdFusion10/cfusion/lib/poi-scratchpad.jar;  D:/ColdFusion10/cfusion/lib/poi.jar;  D:/ColdFusion10/cfusion/lib/portlet_20.jar;  D:/ColdFusion10/cfusion/lib/postgresql-8.3-604.jdbc3.jar;  D:/ColdFusion10/cfusion/lib/quartz.jar;  D:/ColdFusion10/cfusion/lib/relaxngDatatype.jar;  D:/ColdFusion10/cfusion/lib/ri_generic.jar;  D:/ColdFusion10/cfusion/lib/rome-cf.jar;  D:/ColdFusion10/cfusion/lib/saaj.jar;  D:/ColdFusion10/cfusion/lib/saxon9he.jar;  D:/ColdFusion10/cfusion/lib/serializer.jar;  D:/ColdFusion10/cfusion/lib/slf4j-api-1.5.6.jar;  D:/ColdFusion10/cfusion/lib/slf4j-log4j12-1.5.6.jar;  D:/ColdFusion10/cfusion/lib/smack.jar;  D:/ColdFusion10/cfusion/lib/smpp.jar;  D:/ColdFusion10/cfusion/lib/STComm.jar;  D:/ColdFusion10/cfusion/lib/tagsoup-1.2.jar;  D:/ColdFusion10/cfusion/lib/tika-core-0.6.jar;  D:/ColdFusion10/cfusion/lib/tika-parsers-0.6.jar;  D:/ColdFusion10/cfusion/lib/tools.jar;  D:/ColdFusion10/cfusion/lib/tt-bytecode.jar;  D:/ColdFusion10/cfusion/lib/wc50.jar;  D:/ColdFusion10/cfusion/lib/webchartsJava2D.jar;  D:/ColdFusion10/cfusion/lib/wsdl4j-1.6.2.jar;  D:/ColdFusion10/cfusion/lib/wsrp4j-commons-0.5-SNAPSHOT.jar;  D:/ColdFusion10/cfusion/lib/wsrp4j-producer.jar;  D:/ColdFusion10/cfusion/lib/xalan.jar;  D:/ColdFusion10/cfusion/lib/xercesImpl.jar;  D:/ColdFusion10/cfusion/lib/xml-apis.jar;  D:/ColdFusion10/cfusion/lib/xmlbeans-2.3.0.jar;  D:/ColdFusion10/cfusion/lib/xmpcore.jar;  D:/ColdFusion10/cfusion/lib/xsdlib.jar;  D:/ColdFusion10/cfusion/lib/;  D:/ColdFusion10/cfusion/lib/axis2/axiom-api-1.2.13.jar;  D:/ColdFusion10/cfusion/lib/axis2/axiom-dom-1.2.13.jar;  D:/ColdFusion10/cfusion/lib/axis2/axiom-impl-1.2.13.jar;  D:/ColdFusion10/cfusion/lib/axis2/axis2-adb-1.7.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/axis2-adb-codegen-1.7.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/axis2-codegen-1.7.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/axis2-jaxws-1.7.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/axis2-kernel-1.7.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/axis2-transport-http-1.7.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/axis2-transport-local-1.7.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/commons-fileupload-1.2.jar;  D:/ColdFusion10/cfusion/lib/axis2/commons-io-1.4.jar;  D:/ColdFusion10/cfusion/lib/axis2/geronimo-ws-metadata_2.0_spec-1.1.2.jar;  D:/ColdFusion10/cfusion/lib/axis2/httpcore-4.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/neethi-3.0.2.jar;  D:/ColdFusion10/cfusion/lib/axis2/woden-api-1.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/woden-impl-commons-1.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/woden-impl-dom-1.0.jar;  D:/ColdFusion10/cfusion/lib/axis2/wsdl4j-1.6.2.jar;  D:/ColdFusion10/cfusion/lib/axis2/wstx-asl-3.2.9.jar;  D:/ColdFusion10/cfusion/lib/axis2/XmlSchema-1.4.8.jar;  D:/ColdFusion10/cfusion/lib/axis2/;  D:/ColdFusion10/cfusion/gateway/lib/examples.jar;  D:/ColdFusion10/cfusion/gateway/lib/;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/flex/jars/cfgatewayadapter.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/flex/jars/concurrent.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/flex/jars/;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-awt-util.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-css.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-ext.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-transcoder.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-util.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/commons-discovery.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/commons-logging.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/concurrent.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/flex.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/jakarta-oro-2.0.7.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/jcert.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/jnet.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/jsse.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/oscache.jar;  D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/; 

            Java Class Path D:\\ColdFusion10\\cfusion\lib\oosdk\lib;

            D:\\ColdFusion10\\cfusion\lib\oosdk\classes;

            D:\ColdFusion10\cfusion\bin\..\runtime\bin\tomcat-juli.jar;

            D:\ColdFusion10\cfusion\bin\cf-bootstrap.jar 

            Java Ext Dirs D:\ColdFusion10\jre\lib\ext;C:\WINDOWS\Sun\Java\lib\ext 

            Printer Details

            Default Printer

            Printers

             

            Server Information

            General Settings

            Timeout requests Yes 

            Enable Per App Settings Yes 

            Request Time Limit 120 seconds

            Use UUID for CFToken Yes 

            Disable Service Factory No 

            Protect serialized JSON No 

            Protect Serialized JSON Prefix // 

            Missing Template Handler /404b.cfm 

            Site-wide Error Handler /errorCatch.cfm 

            Enable HTTP status codes Yes 

            Enable Global Script Protection Yes 

            ORMSearch Index Directory 

            Default CFForm ScriptSrc Directory /CFIDE/scripts/ 

            Google Map Key 

            Maximum size of post data 100 MB

            Request Throttle Threshold 4 MB

            Request Throttle Memory 200 MB

            Request Tuning

            Simultaneous request limit 20 

            Flash Remoting request limit 5 

            Web Service request limit 20 

            CFC request limit 20 

            CFThread Pool Size 10 

            Maximum number of report threads 8 

            Request Queue Timeout 120 seconds

            Request Queue Timeout Page 

            Caching

            Template cache size 150 templates

            Enable trusted cache No 

            Cached query limit 500

            Save Class Files Yes 

            Client Variable Settings

            Default client variable store ecom 

            Purge Interval 1 hours 7 minutes 

            Client Stores

            Registry 

            Type REGISTRY 

            Description System registry. 

            Purge data after time limit Yes 

            Time limit 90 days

            Disable global updates No 

            ecom 

            Type JDBC 

            Description 

            Purge data after time limit No 

            Time limit 2 days

            Disable global updates Yes 

            Cookie 

            Type COOKIE 

            Description Client based text file. 

            Purge data after time limit Yes 

            Time limit 10 days

            Disable global updates No 

            Memory Variables

            J2EE Sessions No 

            Application Variables

            Enable Application Variables Yes 

            Default Timeout 2,0,0,0 

            Maximum Timeout 2,0,0,0 

            Session Variables

            Enable session variables Yes 

            Default Timeout 0,0,20,0 

            Maximum Timeout 2,0,0,0 

            ColdFusion Mappings

            /gateway   D:\ColdFusion10\cfusion\gateway\cfc 

            /CFIDE   D:\ColdFusion10\cfusion\wwwroot\CFIDE 

            Mail Connection Settings

            Default Server Port 25 

            Connection Timeout 60 seconds

            Spool Interval 30 seconds

            Mail Delivery Threads 10 

            Maintain Connection to Mail Server Yes 

            Spool Messages To disk

            Max Messages Spooled to Memory 50000 

            Default CFMail Charset UTF-8 

            Use SSL Connection No 

            Use TLS No 

            Default Mail Server

            smtp.example.com

            Server smtp.example.com

            Port 25 

            Username 

            Password 

            Backup Mail Servers

            smtp-backup.example.com

            Server smtp-backup.example.com

            Port 25 

            Username 

            Password 

            Mail Logging Settings

            Log Severity warning 

            Log all E-mail messages sent by ColdFusion Yes 

            Charting

            Cache Type disk images

            Maximum number of images in cache 50 images

            Maximum number of charting threads 4 

            Disk cache location D:\ColdFusion10\cfusion\charting\cache 

             

            Data & Services

            Database Data Sources

            GES 

            CF data source name GES 

            Description 

            Driver MSSQLServer 

            JDBC URL jdbc:macromedia:sqlserver://MSSQL:1433;databaseName=GES;SelectMethod=direct;sendStringPar ametersAsUnicode=false;querytimeout=0;MaxPooledStatements=100

             

            Username sa 

            Login timeout 30 seconds

            Long text buffer size 64000 

            Timeout 1200 seconds 

            Maintain connections Yes 

            Interval 420 seconds

            Restricted SQL operations 

            Disable connections No 

            cfcodeexplorer 

            CF data source name cfcodeexplorer 

            Description 

            Driver Apache Derby Embedded 

            JDBC URL jdbc:derby:D:\ColdFusion10\cfusion\db\cfcodeexplorer;create=false 

            Username 

            Login timeout 30 seconds 

            Long text buffer size 64000 

            Timeout 1200 seconds

            Maintain connections Yes 

            Interval 420 seconds 

            Restricted SQL operations 

            Disable connections No 

            ecom 

            CF data source name ecom 

            Description 

            Driver 

            JDBC URL jdbc:macromedia:oracle://10.10.120.13:1521;serviceName=ecom;alternateServers=(10.10.120.1 4:1521);loadBalancing=true

             

            Username user 

            Login timeout 30 seconds

            Long text buffer size 64000 

            Timeout 1200 seconds 

            Maintain connections Yes 

            Interval 420 seconds

            Restricted SQL operations 

            Disable connections No 

            cfbookclub 

            CF data source name cfbookclub 

            Description 

            Driver Apache Derby Embedded 

            JDBC URL jdbc:derby:D:\ColdFusion10\cfusion\db\bookclub;create=false 

            Username 

            Login timeout 30 seconds 

            Long text buffer size 64000 

            Timeout 1200 seconds

            Maintain connections Yes 

            Interval 420 seconds 

            Restricted SQL operations 

            Disable connections No 

            ecomd 

            CF data source name ecomd 

            Description 

            Driver Oracle 

            JDBC URL jdbc:macromedia:oracle://ORACLE:1521;SID=ecomd;AuthenticationMethod=userIDPassword;sendSt ringParametersAsUnicode=false;querytimeout=0;MaxPooledStatements=100

             

            Username ges 

            Login timeout 30 seconds

            Long text buffer size 64000 

            Timeout 1200 seconds 

            Maintain connections Yes 

            Interval 420 seconds

            Restricted SQL operations 

            Disable connections No 

            GES_New 

            CF data source name GES 

            Description 

            Driver MSSQLServer 

            JDBC URL jdbc:macromedia:sqlserver://MSSQL\MSSQL:1433;databaseName=GES;SelectMethod=direct;sendStr ingParametersAsUnicode=false;querytimeout=0;MaxPooledStatements=100

             

            Username user 

            Login timeout 30 seconds 

            Long text buffer size 64000 

            Timeout 1200 seconds

            Maintain connections Yes 

            Interval 420 seconds 

            Restricted SQL operations 

            Disable connections No 

            E1_Oracle 

            CF data source name E1 

            Description 

            Driver Oracle 

            JDBC URL jdbc:macromedia:oracle://ORACLE:1516;SID=E1;AuthenticationMethod=userIDPassword;sendStrin gParametersAsUnicode=false;querytimeout=0;MaxPooledStatements=100

             

            Username user

            Login timeout 30 seconds

            Long text buffer size 64000 

            Timeout 1200 seconds 

            Maintain connections Yes 

            Interval 420 seconds

            Restricted SQL operations 

            Disable connections No 

            cfdocexamples 

            CF data source name cfdocexamples 

            Description 

            Driver Apache Derby Embedded 

            JDBC URL jdbc:derby:D:\ColdFusion10\cfusion\db\cfdocexamples;create=false 

            Username 

            Login timeout 30 seconds 

            Long text buffer size 64000 

            Timeout 1200 seconds

            Maintain connections Yes 

            Interval 420 seconds 

            Restricted SQL operations 

            Disable connections No 

            cfartgallery 

            CF data source name cfartgallery 

            Description 

            Driver Apache Derby Embedded 

            JDBC URL jdbc:derby:D:\ColdFusion10\cfusion\db\artgallery;create=false 

            Username 

            Login timeout 30 seconds

            Long text buffer size 64000 

            Timeout 1200 seconds 

            Maintain connections Yes 

            Interval 420 seconds

            Restricted SQL operations 

            Disable connections No 

            Web Services

             

            Debugging & Logging

            Debugging Settings

            Enable debugging No 

            Enable Robust Exception Information No 

            Display format classic.cfm 

            Execution times Yes 

            Execution time format summary 

            Execution time highlight threshold 250 ms

            Database activity Yes 

            Exception information Yes 

            Tracing information Yes 

            Timer Information No 

            Variables Yes 

            Variables

            Application Yes 

            CGI Yes 

            Client Yes 

            Cookie Yes 

            Form Yes 

            Request Yes 

            Server No 

            Session No 

            URL Yes 

            Debugging IP Addresses

            Debugging IP Address Restrictions 

            Line Debugger Settings

            Allow Line Debugging NO 

            Debugger Port 5005 

            Max Simultaneous Debugging Sessions 5 

            Logging Settings

            Log directory D:\ColdFusion10\cfusion\logs 

            Maximum file size 5000 KB

            Maximum number of archives 10 

            Log slow pages Yes 

            Slow page time limit 60 seconds

            Log CORBA calls No 

            Log scheduled tasks No 

             

            Schedule Tasks & Probes

            Scheduled Tasks

            Auto Replenish Notify S9 Order Min Levels Reached 

            Start Date 9/2/2014 

            End Date 

            Interval Daily 

            Start Time 2:05:00 AM 

            URL http://server1.example.com/index.cfm?fuseaction=rfid.SendReplenishEmails 

            Username 

            Timeout 

            Proxy Server 

            Proxy Port 80 

            Save output to file No 

            Publish path 

            Publish filename 

            Resolve URLs No 

            Auto Replenish Notify Customer Badge Replenishment 

            Start Date 9/2/2014 

            End Date 

            Interval Daily 

            Start Time 2:00:00 AM 

            URL http://server1.example.com/uri1/act_notifyUserBadge.cfm 

            Username 

            Timeout 

            Proxy Server 

            Proxy Port 80 

            Save output to file No 

            Publish path 

            Publish filename 

            Resolve URLs No 

            Auto Replenish 

            Start Date 9/2/2014 

            End Date 

            Interval Daily 

            Start Time 2:10:00 AM 

            URL http://server1.example.com/uri1/uri2/uri3/Replenish2.cfm 

            Username 

            Timeout 

            Proxy Server 

            Proxy Port 80 

            Save output to file No 

            Publish path 

            Publish filename 

            Resolve URLs No 

            Contact Updates and Registrations 

            Start Date 9/2/2014 

            End Date 

            Interval 65 

            Start Time 12:05:00 AM 

            URL http://server1.example.com/index.cfm?fuseaction=intCRM.sendCRMObjects 

            Username 

            Timeout 

            Proxy Server 

            Proxy Port 80 

            Save output to file No 

            Publish path 

            Publish filename 

            Resolve URLs No 

            System Probes

             

            Extensions

            Java Applets

            CFX Tags

            cfx_xml2cfx 

            Tag name cfx_xml2cfx 

            Description

            Type CPP 

            Server library c:\cfusion8\cfx\xml2cfx\cfx_xml2cfx.dll 

            Keep library loaded true 

            Procedure ProcessTagRequest 

            cfx_encrypt_aes 

            Tag name cfx_encrypt_aes 

            Description DB encryption library

            Type CPP 

            Server library C:\cfusion8\cfx\aes\cfx_encrypt_aes.dll 

            Keep library loaded true 

            Procedure ProcessTagRequest 

            cfx_exec 

            Tag name cfx_exec 

            Description

            Type CPP 

            Server library c:\CFusion8\cfx\exec\cfx_exec.dll 

            Keep library loaded true 

            Procedure ProcessTagRequest 

            Custom Tag Paths

            D:\web\app1\app2\app3\activedit3 

            D:\ColdFusion10\cfusion/CustomTags 

            D:\ColdFusion8\CustomTags 

            CORBA

            Selected connector [ none] 

            Connectors

             

            Event Gateways

            Settings

            Enable Event Gateway Yes 

            Thread Pool Size 10 

            Max Queue Size 25000 

            Gateway Types

            SMS 

            Description Handles SMS text messaging 

            Class coldfusion.eventgateway.sms.SMSGateway 

            Timeout 30 seconds 

            Kill On Timeout Yes 

            XMPP 

            Description Handles XMPP instant messaging 

            Class coldfusion.eventgateway.im.XMPPGateway 

            Timeout 30 seconds

            Kill On Timeout Yes 

            SAMETIME 

            Description Handles Lotus SAMETIME instant messaging 

            Class coldfusion.eventgateway.im.SAMETIMEGateway 

            Timeout 30 seconds 

            Kill On Timeout Yes 

            DirectoryWatcher 

            Description Watches a directory for file changes 

            Class examples.watcher.DirectoryWatcherGateway 

            Timeout 30 seconds

            Kill On Timeout Yes 

            Socket 

            Description Listens on a socket 

            Class examples.socket.SocketGateway 

            Timeout 30 seconds 

            Kill On Timeout Yes 

            CFML 

            Description Handles asynchronous events through CFCs 

            Class coldfusion.eventgateway.cfml.CfmlGateway 

            Timeout 30 seconds

            Kill On Timeout Yes 

            JMS 

            Description Handles Java Messaging Service messages 

            Class examples.JMS.JMSGateway 

            Timeout 30 seconds 

            Kill On Timeout Yes 

            ActiveMQ 

            Description Handles Apache ActiveMQ JMS messages 

            Class examples.ActiveMQ.JMSGateway 

            Timeout 30 seconds

            Kill On Timeout Yes 

            DataServicesMessaging 

            Description Handles Data Services Messaging messages 

            Class coldfusion.eventgateway.flex.FlexMessagingGateway 

            Timeout 30 seconds 

            Kill On Timeout No 

            FMS 

            Description Handles Flash Media Server shared objects 

            Class coldfusion.eventgateway.fms.FMSGateway 

            Timeout 30 seconds

            Kill On Timeout Yes 

            DataManagement 

            Description Notifies Data Management Services of data changes 

            Class coldfusion.eventgateway.flex.FlexDataManagementGateway 

            Timeout 30 seconds 

            Kill On Timeout Yes 

             

            Security

            CF Admin Authentication

            Enable authentication for the ColdFusion Administrator Yes 

            Allow access to ColdFusion Administrator with a Single password No 

            RDS Authentication

            Enable authentication for RDS access Yes 

            Allow access through RDS with Single password Yes 

            Security Sandboxes

            Enable ColdFusion Security No 

             

             

            jvm.config

            #

            # VM configuration

            #

            # Where to find JVM, if {java.home}/jre exists then that JVM is used

            # if not then it must be the path to the JRE itself

             

            java.home=D:\\ColdFusion10\\jre

            application.home=D:\\ColdFusion10\\cfusion

             

            #

            # If no java.home is specified a VM is located by looking in these places in this

            # order:

            #

            #  1) ../runtime/jre

            #  2) registry (windows only)

            #  3) JAVA_HOME env var plus jre (ie $JAVA_HOME/jre)

            #  4) java.exe in path

            #

             

            # Arguments to VM

            java.args=-server  -Xms768m -Xmx1280m -XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dges.config.serverEnvironment=prod2 -Dges.config.webServerHostName=server1.example.com -Dcom.sun.management.jmxremote.port=8880 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application .home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/j ars,{application.home}/wwwroot/WEB-INF/cfform/jars

             

            # Comma separated list of shared library path

            java.library.path={application.home}/lib,{application.home}/jintegra/bin,{application.home }/jintegra/bin/international,{application.home}/lib/oosdk/classes/win

             

            java.class.path={application.home}/lib/oosdk/lib,{application.home}/lib/oosdk/classes

            • 3. Re: Memory leak after upgrading to ColdFusion 10
              PhilBro Community Member
              PhilBro Sep 26, 2014 7:31 AM (in response to DFresh4130)

              How many active sessions are there when you start to have the OOM and blocked threads issues? Your app may be holding on to sessions longer than it really needs to. If you get a lot of traffic and bots, store a lot of info in the session scope and they don't time out fast enough, you will have OOM and blocked threads issues.

               

              I had this happen earlier this year and it was a nightmare to figure out that this was the problem.

              • 4. Re: Memory leak after upgrading to ColdFusion 10
                DFresh4130 Community Member
                DFresh4130 Oct 1, 2014 3:47 PM (in response to PhilBro)

                When I go to the server monitor and view active sessions it's showing anywhere from 4-5K active sessions when the blocked threads are not present.  I'll have to check keep checking regularly and see if I can find a trend relating to the blocked threads.  Note that in the thread dump we've found two ajp threads blocked on each other so it's a complete deadlock which it appears it'll never recover from.

                • 5. Re: Re: Memory leak after upgrading to ColdFusion 10
                  PhilBro Community Member
                  PhilBro Oct 2, 2014 10:31 AM (in response to DFresh4130)

                  4-5K active sessions isn't a lot but I'd like to see if that number keeps growing or levels out.

                   

                  Are you running just the one cfusion instance?

                  What is the sessionTimeout set to in each applcation.cfc?

                  What are you putting in your client and session scopes?

                  Are you seeing high CPU usage when you start seeing the OOM errors?

                   

                  It looks like your using a database to store your client variables but "Purge data" isn't checked. If I understand correctly, you should have it set to purge data. Otherwise the client tables in the database will grow out of control and cause some real performance issues. I've never used client vars with a database so I could be very wrong.

                  • 6. Re: Re: Re: Memory leak after upgrading to ColdFusion 10
                    DFresh4130 Community Member
                    DFresh4130 Oct 3, 2014 2:14 PM (in response to PhilBro)

                    After running for about 48 hours the number of active sessions doesn't grow much (about 500).  We're running 2 CF 10 instances on separate VMs.  Traffic goes through 2 apache servers with each apache pointed directly at one CF instance, no load balancing at the app level.  It looks like the session timeout is set to 45 minutes, but I'm not sure.  Seeing the below line in the code.

                     

                    THIS.SessionTimeout = "#CreateTimeSpan(0,0,45,0)#";

                     

                    I'm not a developer so I don't know the answer to what's in the client and session scopes.  I'll have to see if I can find a dev who can answer that.  We do see CPU usage gradually climb due to the amount of time being spent in GC as the heap gets full.  When we finally get an OOM error we sometimes see very high CPU or just above normal.  I also verified the Purge Data option is checked in the console and it's set to 90 days.

                     

                    *EDIT* Looking at the client variables section in more detail I see our default option is a database.  It currently does not purge the data and the box for "" is checked.

                    • 7. Re: Re: Re: Memory leak after upgrading to ColdFusion 10
                      PhilBro Community Member
                      PhilBro Oct 6, 2014 9:32 AM (in response to DFresh4130)

                      Definitely expiring sessions in 45 min so that sounds fine.

                       

                      In the CF Admin Server Monitor you can can go to the "Statistics" tab and open "Memory Usage" then click "Memory Usage /summary". How much memory is being used by sessions, application, and server? Can you post a screenshot of this page so I can see what the garbage collection graph and memory usage graph look like?

                       

                      Its common to see CPU spikes when GC runs. Usually with memory issues it will get to the point where GC is running longer than it should and the CPU will eventually max out.

                       

                      At this point I don't think its anything to do with sessions but I want to see what is going on to be on the safe side. It sounds more like an issue with the way client vars are being used and I don't have a lot of experience with them. It is good to know you don't have "" checked. For the most part it adds overhead to each page that isn't needed.

                       

                      Purge Data is on so that is good but 90 days is a long time to store client vars. You timeout sessions after 45 min. Why keep client vars around for 90 days? Unless there is a good reason to keep client vars that long, I would reduce the Purge Data to something like 2-3 days. My reasoning. Every time someone comes to your site you are creating a new session to store in server memory and new set of client vars to store in a database. The sessions are removed from memory after they time out. Right now your client vars are being stored and saved for ever in your database in tables that most likely are not indexed. This will cause performance issues as the tables fill up. According to the docs, "To improve performance, create indexes when you create these tables. For the CDATA table, index these cfid and app columns. For the CGLOBAL table, index the cfid column." I would also use a dedicated database for client vars.

                       

                      I looked over your JVM arguments and have some suggestions.

                      Its recommended to have Xms and Xmx and -XX:PermSize and -XX:MaxPermSize set to the same value. I would consider changing your settings to -Xms1280m -Xmx1280m -XX:PermSize=192m -XX:MaxPermSize=192m. If you have enough server memory you may want to allocate more to Xms and Xmx.