10 Replies Latest reply on Sep 2, 2015 8:15 AM by brentil

    Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?

    brentil Level 1

      We have a development copy of CF11 that we installed as a RTM build that has since gone through CU1 through CU5.  We just recently installed CF11 on our production server which was done from the latest deployment package and was then upgraded to CU4 then CU5.  We've noticed instabilities with the production copy that we haven't seen once on the development copy the years it has been running.

       

      The most common are "Error Executing Database Query. Timed out trying to establish connection" errors at random while other applications using the same DB server have no issues at all.  If you reload the page the DB call goes through the 2nd time.  Only CF applications on this web server have these issues and not .NET or Java based ones.

       

      So wondering if I had configuration wrong I merge compared our development versus production server and found a number of core parts of the CF11 that started as RTM to be much older.  Various jar files, the Jetty runtime is v6 instead of v9, tomcat files, etc.  It's obvious at some point the CF11 deployment package has been updated but typically in the past we'd see a CF 9.0.1 type of release for major deployment changes but both of my servers report as being Version 11,0,05,293506.

        • 1. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
          carl type3 Level 4

          Hi Brent,

           

          The CF11 release installer laid down Java 7. The latter CF11 installer uses Java 8. If one went release then U1 to U5 that would not alter the underlying Java version.

           

          I have seen issues where database connections are broken due to a poorly performing Java heap or non heap issue. You could do well to check how well the heap or non heap IE metaspace values are performing and if one or other need adjustment. Not to be forgotten with Java memory is CodeCache and CompressedClass to know either of those are consuming memory in good boundaries. Use traditional Java logging to a file or enable JMX then use JDK tools like jconsole or jmc to check the memory consumption is stressed or not.

           

          HTH, Carl.

          • 2. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
            brentil Level 1

            I noticed our development server still had -XX:MaxPermSize=192m where the new production server had -XX:MaxMetaspaceSize=192m set instead.  Which reading up on that setting not having it defined would have given an unlimited access size.  I've changed the development server to limit to 192m to see if it causes issues there.  Are there any good articles written on metaspace tuning for CF like the connector one?

             

            We don't run the internal Java but instead external JRE copies so both machines are running JRE 1.8.0_45 (working on upgrading to _51 soon).

            • 3. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
              Sandip_halder Adobe Employee

               

              Regarding the Performance issue, here are some resources, that will help you.

               

              * Performance tuning for ColdFusion applications (http://www.adobe.com/devnet/coldfusion/articles/coldfusion_performance.html),

              *         JVM Tuning (http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html)

               

              These articles are for JRE 7, however, the logic remains the same for JRE 8 as well.

               

              • 4. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
                brentil Level 1

                After setting my development system to have the same -XX:MaxMetaspaceSize=192m I started getting the same errors just browsing around a development website.

                 

                Error Executing Database Query. 

                  Timed out trying to establish connection 

                ...

                java.sql.SQLException: Timed out trying to establish connection

                    at coldfusion.server.j2ee.sql.pool.JDBCPool.requestConnection(JDBCPool.java:843)

                    at coldfusion.server.j2ee.sql.pool.JDBCManager.requestConnection(JDBCManager.java:125)

                    at coldfusion.server.j2ee.sql.JRunDataSource.getConnection(JRunDataSource.java:137)

                    at coldfusion.sql.CFDataSource.getConnection(CFDataSource.java:44)

                    at coldfusion.sql.DataSrcImpl.getCachedConnection(DataSrcImpl.java:156)

                    at coldfusion.sql.DataSrcImpl.getConnection(DataSrcImpl.java:110)

                    at coldfusion.sql.SqlImpl.execute(SqlImpl.java:363)

                    ...

                • 5. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
                  carl type3 Level 4

                  Seems to me you are creeping up on the answer since setting dev to -XX:MaxMetaspaceSize=192m 

                  you get the database connectivity problem. So you are consuming  greater than 192m of MetaSpace which

                  is not unusual on 64bit.

                   

                  With Java 8 it is able to manage MetaSpace automatically if you do not define a value for MaxMetaspaceSize= however it may not make good decisions and will be inclined to re-size down to it's lower value. All this resizing invokes a Full garbage collection which is a performance loss.

                   

                  You can do well to define an initial and maximum setting for MetaSpace. I find this works well in most cases:

                   

                  -XX:MetaspaceSize=312m -XX:MaxMetaspaceSize=512m

                   

                  Having said that some Java logging or JMX monitoring might reveal better values to apply given the

                  type of workload.

                   

                  HTH again, Carl.

                  • 6. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
                    brentil Level 1

                    I enabled JMX and installed VisualVM so I've been able to watch the resource usage.  Oddly the DB timeout happened when only 80mb of metaspace was in use when limited to 192.  We were doing similar min/max settings with the previous Java 7 settings, thank you for providing me the terms for the new settings.

                     

                    I'm also looking into tomcat connector tuning as well but I don't think the DB timeouts are related to it (at least I don't think they are).

                    • 7. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
                      carl type3 Level 4

                      It is a good idea to apply tomcat tuning tho like you say adjusting tomcat pools, threads and timeout is probably not related to the database timeout problem.

                       

                      I guess you have adjusted MetaSpace. Does Jvisualvm show it consuming more memory now?

                       

                      While you have JMX enabled you might like use jconsole or jmc to check CodeCache and CompressClass memory consumption to know those values are well.

                       

                      Regards, Carl.

                      • 8. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
                        brentil Level 1

                        Unfortunately the increased memory space and Tomcat tuning did not resolve the issue on our production servers.  I was hoping they might happen less frequently but they still seem to happen at about the same rate. 

                        • 9. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
                          brentil Level 1

                          I've created a Adobe Bug for the timeouts as we've been unable to make them stop on our production environment and can reproduce them on development as well.

                           

                          https://bugbase.adobe.com/index.cfm?event=bug&id=4038865

                          • 10. Re: Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?
                            brentil Level 1

                            After countless attempts to stop this from happening the solution we finally found was to run the Microsoft JDBC drivers v4.2 inside of ColdFusion11. Once we stopped using all Adobe JDBC drivers to connect to our MS-SQL 2014 server the server stopped being unstable. If even one active DB is using the built in Adobe MS-SQL driver then the system in both our development and production enviroment will eventually behave as documented previously.

                             

                            https://www.microsoft.com/en-us/download/details.aspx?id=11774

                             

                             

                            1. CF11 CFIDE -> Data Sources -> Type name as usual but choose “other” as driver
                            2. JDBC URL is where you define the connection string including the server to connect to, the DB, and encryption (if you use encryption make sure to use the FQDN
                              1. jdbc:sqlserver://SERVERNAME:PORT;databaseName=DATABASETOUSE;encrypt=TRUE;
                            3. Driver Class must contain the name of the JDBC driver
                              1. com.microsoft.sqlserver.jdbc.SQLServerDriver
                            4. Driver Name must contain the correct name
                              1. SQLServerDriver
                            5. Fill out the User Name and Password as usual
                            6. The Advanced area contains all of the same settings except for the “Max Pooled Statements”

                             

                            I have a set of benchmark scripts that run an exhaustive set of DB queries I use to evaluate performance changes over time.  On my development test server these take 630 seconds to run on average and moving to the MS JDBC driver the on average time dropped by 30 seconds to complete for unencrypted connections.  For encrypted connections it took on average the same amount of time to complete.

                             

                            One issue we’ve discovered is that the driver enforces some MS coding practices by disallowing the running of certain queries. For example a select identity inside an insert query will error using the MS JDBC driver but run with the Adobe driver.