39 Replies Latest reply: Dec 31, 2014 7:10 AM by cpb07 RSS

    CFThread GC Overhead Limit Exceeded

    cpb07 Community Member

      I have an application that trades virtual items and have a single page which gets all my accounts and for each one creates a thread that firstly logs the account in and then searches and buys for items for as long as the session is active.  I should point out at this point that this is my first experience of using cfthread.

       

      I'm having issues with it. Every 30 minutes (if not less) my ColdFusion server comes to a standstill and I have to restart the service. Upon restarting the service I check the logs and there are errors that say "GC Overhead Limit Exceeded".

       

      I have looked extensively online but as much as cfthread is new to me, so is the JVM and how it operates. I'm running on CF10 Enterprise Edition and have loaded up the server monitor and surely enough I can see the JVM memory usage grow and grow until the limit has been reached (just now I have it set as 2gb as when I had it set higher the memory seemed to fill up quicker). Even when I select the Run GC option in the monitor it does not reduce the memory usage very much, if at all.

       

      Is this more than likely something to do with my code? At the moment I have just under 50 threads being created but as I add more accounts to the application then the more threads that will be required.

       

      Here is the code from the page...

       

      <script>

        /* RELOAD PAGE EVERY 65 MINUTES */

        setTimeout(function(){

          window.location.reload(1);

        }, 3900000);

      </script>

       

      <!--- GET ACTIVE ACCOUNTS --->

      <cfquery name="getLogins" datasource="myDB">

      SELECT * FROM Logins WHERE active = 1

      </cfquery>

       

      <!--- LOOP THROUGH ACCOUNT --->

      <cfloop query="getLogins">

       

        <!--- HAVE A SLEEP SO IP DOESN'T GET FLAGGED FOR SENDING TOO MANY REQUESTS AT ONCE --->

        <cfset Sleep(30000) />

       

        <!--- CREATE THREAD FOR ACCOUNT --->

        <cfthread

        name="#getLogins.accountName#"

        action="run"

        accountName="#Trim(getLogins.accountName)#"

        email="#Trim(getLogins.email)#"

        password="#Trim(getLogins.password)#"

        resourceId="#Trim(getLogins.resourceID)#">

       

        <!--- DEFAULT SESSION VARIABLES --->

        <cfset SESSION["#attributes.accountName#LoggedIn"] = 0 />

        <cfset SESSION["#attributes.accountName#LoginAttempts"] = 0 />

       

        <!--- WHILE ACCOUNT NOT LOGGED IN AND LESS THAN 8 LOGIN ATTEMPTS MADE --->

        <cfscript>

        while (SESSION['#attributes.accountName#LoggedIn'] EQ 0 AND SESSION['#attributes.accountName#LoginAttempts'] LT 8) {

       

        // ATTEMPT LOGIN

        THREAD.logInAccount = Application.cfcs.Login.logInAccount(attributes.email,attributes.password);

       

        // IF LOGIN ATTEMPT UNSUCCESSFUL

        if (THREAD.logInAccount EQ 0) {

        // INCREASE ATTEMPT COUNT

        SESSION['#attributes.accountName#LoginAttempts'] = SESSION['#attributes.accountName#LoginAttempts'] + 1;

        }

        // ELSE IF RETURNED VALUE IS 481 THEN ACCOUNT IS LOCKED

        else if (THREAD.logInAccount EQ 481) {

        // SET LOGIN ATTEMPT COUNT TO STOP LOOP

        SESSION['#attributes.accountName#LoginAttempts'] = 8;

        // UPDATE ACCOUNT TO MARK AS LOCKED

        THREAD.updLogin = Application.cfcs.Login.updLogin(attributes.email);

        }

        }

        </cfscript>

       

        <!--- IF ACCOUNT LOGGED IN --->

        <cfif SESSION['#attributes.accountName#LoggedIn'] EQ 1>

       

        <!--- SET ID FOR SEARCHING --->

        <cfset THREAD.definitionID = attributes.resourceID - 1610612736 />

       

        <!--- WHILE ACCOUNT LOGGED IN --->

        <cfloop condition="SESSION['#attributes.accountName#LoggedIn'] EQUALS 1">

       

        <!--- GET LATEST LOWEST BUY NOW PRICE --->

        <cfquery name="THREAD.getMinBIN" datasource="FUT" cachedWithin="#CreateTimeSpan(0,0,1,0)#">

        SELECT TOP 1 * FROM v_FUT14BINPrices WHERE resourceID = #attributes.resourceId# ORDER BY lastUpdated DESC

        </cfquery>

       

        <!--- INCLUDE FILE THAT CALCULATES BUYING AND SELLING PRICES --->

        <cfinclude template="sellingPrices.cfm" />

       

        <!--- IF BIDDING PRICE HAS BEEN SET --->

        <cfif StructKeyExists(THREAD,"biddingPrice")>

       

        <!--- MAKE SEARCH REQUEST, TIMING THE REQUEST --->

        <cfset THREAD.requestStart = GetTickCount() />

        <cfset THREAD.search = Application.cfcs.Search.dosearchOld(attributes.resourceId,THREAD.biddingPrice,0) />

        <cfset THREAD.requestDuration = GetTickCount() - THREAD.requestStart />

       

        <!--- IF SEARCH CONTAINS FILE CONTENT  --->

        <cfif StructKeyExists(THREAD.search,"FileContent")>

       

        <!--- DECLARE NUMBER OF RESULTS VARIABLE --->

        <cfset THREAD.numResults = 0 />

       

        <!--- IF JSON RETURNED --->

        <cfif IsJSON(THREAD.search.FileContent)>

       

        <!--- DESERIALIZE JSON --->

        <cfset THREAD.searchResults = DeserializeJSON(THREAD.search.FileContent) />

       

        <!---  IF PLAYER SEARCH RETURNS AUCTIONINFO STRUCT --->

        <cfif StructKeyExists(THREAD.searchResults,"auctionInfo")>

       

        <!--- SET NUMBER OF CARDS RETURNED FROM SEARCH --->

        <cfset THREAD.numResults = ArrayLen(THREAD.searchResults.auctionInfo) />

        <cfset THREAD.statusCode = "Successful" />

        <cfif THREAD.numResults EQ 0>

        <cfset THREAD.statusCode = "Successful - No Results" />

        </cfif>

       

        <!--- ELSE IF ERROR CODE RETURNED --->

        <cfelseif StructKeyExists(THREAD.searchResults,"code")>

       

        <cfset THREAD.statusCode = THREAD.searchResults.code />

       

        <!--- IF CODE 401 THEN SESSION HAS EXPIRED --->

        <cfif THREAD.statusCode EQ 401>

        <!--- SET SESSION AS LOGGED OUT AND ATTEMPT SESSION REFRESH --->

        <cfset SESSION['#attributes.accountName#LoggedIn'] = 0 />

        <cfset THREAD.logInAccount = Application.cfcs.Login.logInAccount(attributes.email,attributes.password) />

        </cfif>

       

        <!--- ELSE SOMETHING ELSE HAS HAPPENED --->

        <cfelse>

        <cfset THREAD.statusCode = "Something Else - " & THREAD.searchResults.code />

        </cfif>

       

        <!--- IF RESULTS RETURNED --->

        <cfif THREAD.numResults GT 0>

       

        <!--- LOOP ROUND RESULTS AND CHECK IF MATCH BUYING CRITERIA --->

        <cfloop index="i" from="1" to="#THREAD.numResults#">

       

        <!--- ***SAFETY CHECK*** - ENSURE ID OF CURRENT CARD IS SAME AS ONE SEARCHING FOR --->

        <cfif THREAD.searchResults.auctionInfo[i].itemData.resourceID EQ attributes.resourceId AND THREAD.getMinBIN.resourceID EQ attributes.resourceId>

       

        <!--- ENSURE BIN PRICE SET AND IS LESS THAN SET BUYING PRICE --->

        <cfif THREAD.searchResults.auctionInfo[i].buyNowPrice GT 0 AND THREAD.searchResults.auctionInfo[i].buyNowPrice LTE THREAD.biddingPrice>

       

        <!--- SET AUCTION END TIME --->

        <cfset THREAD.timeLeft = THREAD.searchResults.auctionInfo[i].expires />

        <cfset THREAD.auctionEnds = DateAdd("s",THREAD.timeLeft,Now()) />

        <!--- BUY CARD --->

        <cfset THREAD.buyCard = Application.cfcs.Bid.doBIN(THREAD.searchResults.auctionInfo[i].tradeID,THREAD.searchResul ts.auctionInfo[i].buyNowPrice,THREAD.searchResults.auctionInfo[i].startingBid,THREAD.searc hResults.auctionInfo[i].itemData.ID,THREAD.searchResults.auctionInfo[i].itemData.resourceI D,THREAD.startPrice,THREAD.binPrice,THREAD.lowestBIN,THREAD.searchResults.auctionInfo[i].i temData.discardValue,THREAD.auctionEnds,THREAD.requestStart,THREAD.requestDuration) />

       

      </cfif>

       

        </cfif>

       

        </cfloop>

       

        </cfif>

       

        <cfelse>

        <cfset THREAD.statusCode = THREAD.search.FileContent />

        </cfif>

       

        <cfset THREAD.sleepDuration = 1000 - THREAD.requestDuration />

        <cfif THREAD.sleepDuration GT 0><cfset Sleep(THREAD.sleepDuration) /></cfif>

       

        </cfif>

       

        <!--- INSERT SEARCH RECORD --->

        <cfset THREAD.insSearchRecord = Application.cfcs.Search.insSearchRecord(THREAD.definitionID,THREAD.statusCode,THREAD.requ estDuration,THREAD.numResults,THREAD.biddingPrice) />

       

        </cfif>

       

        </cfloop>

       

        </cfif>

       

        </cfthread>

       

      </cfloop>

       

       

      I would have thought that the memory would have stayed around the same usage as each loop is performing the same set of actions so once the loop has went back to the start then I thought the previous loop would have been removed from memory (freeing up space) and then the same actions would be performed so the same memory total would then be used up but it seems almost as if each loop is being kept in memory and that is why it is growing.

       

      Could someone please help me out and offer some guidance on how I could remedy this issue? If you need any more info then just let me know

       

      Thanks in advance

        • 1. Re: CFThread GC Overhead Limit Exceeded
          BKBK MVP

          You seem to be generating too many threads in a loop, each of them holding live objects. There is a high chance the heap of live objects that the loop generates keeps growing. The error message is telling you that the garbage collector has been unsuccessful in clearing up (because the objects are live).

           

          For a start, I see no motivation for the use of cfthread and for the calls to sleep(). The page implicitly runs in a thread, the main Coldfusion execution thread. To monitor the number of requests per minute, use a counter in onRequestStart instead of sleep().

           

          You should strip your code down to its bare bones. Something like this

           

          <script>

            /* RELOAD PAGE EVERY 65 MINUTES */

            setTimeout(function(){

              window.location.reload(1);

            }, 3900000);

          </script>

           

          <!--- GET ACTIVE ACCOUNTS --->

          <cfquery name="getLogins" datasource="myDB">

          SELECT * FROM Logins WHERE active = 1

          </cfquery>

           

          <!--- LOOP THROUGH ACCOUNT --->

          <cfloop query="getLogins">

              <cfset accountName=getLogins.accountName>

              <cfset email=getLogins.email>

              <cfset password=getLogins.password>

              <cfset resourceId=getLogins.resourceID>

              <!--- DEFAULT SESSION VARIABLES --->

              <cfset SESSION["#accountName#LoggedIn"] = 0 />

              <cfset SESSION["#accountName#LoginAttempts"] = 0 />

           

              <!--- WHILE ACCOUNT NOT LOGGED IN AND LESS THAN 8 LOGIN ATTEMPTS MADE --->

              <cfscript>

              while (SESSION['#accountName#LoggedIn'] EQ 0 AND SESSION['#accountName#LoginAttempts'] LT 8) {

           

                  // ATTEMPT LOGIN

                  logInAccount = Application.cfcs.Login.logInAccount(email,password);

               

                  // IF LOGIN ATTEMPT UNSUCCESSFUL

                  if (logInAccount EQ 0) {

                      // INCREASE ATTEMPT COUNT

                      SESSION['#accountName#LoginAttempts'] = SESSION['#accountName#LoginAttempts'] + 1;

                  }

                  // ELSE IF RETURNED VALUE IS 481 THEN ACCOUNT IS LOCKED

                  else if (logInAccount EQ 481) {

                      // SET LOGIN ATTEMPT COUNT TO STOP LOOP

                      SESSION['#accountName#LoginAttempts'] = 8;

                      // UPDATE ACCOUNT TO MARK AS LOCKED

                      updLogin = Application.cfcs.Login.updLogin(email);

                  }

              }

              </cfscript>

           

              <!--- IF ACCOUNT LOGGED IN --->

              <cfif SESSION['#accountName#LoggedIn'] EQ 1>

                  <!--- SET ID FOR SEARCHING --->

                  <cfset definitionID = resourceID - 1610612736 />

               

                  <!--- WHILE ACCOUNT LOGGED IN --->

                  <cfloop condition="SESSION['#accountName#LoggedIn'] EQUALS 1">

                      <!--- GET LATEST LOWEST BUY NOW PRICE --->

                      <cfquery name="getMinBIN" datasource="FUT" cachedWithin="#CreateTimeSpan(0,0,1,0)#">

                      SELECT TOP 1 * FROM v_FUT14BINPrices WHERE resourceID = #resourceId# ORDER BY lastUpdated DESC

                      </cfquery>

               

                      <!--- INCLUDE FILE THAT CALCULATES BUYING AND SELLING PRICES --->

                      <cfinclude template="sellingPrices.cfm" />

               

                      <!--- IF BIDDING PRICE HAS BEEN SET --->

                      <cfif StructKeyExists(SESSION,"biddingPrice")>

                          <!--- MAKE SEARCH REQUEST, TIMING THE REQUEST --->

                          <cfset requestStart = GetTickCount() />

                          <cfset search = Application.cfcs.Search.dosearchOld(resourceId,biddingPrice,0) />

                          <cfset requestDuration = GetTickCount() - requestStart />

                   

                          <!--- IF SEARCH CONTAINS FILE CONTENT  --->

                          <cfif StructKeyExists(search,"FileContent")> 

                              <!--- DECLARE NUMBER OF RESULTS VARIABLE --->

                              <cfset numResults = 0 />

                       

                              <!--- IF JSON RETURNED --->

                              <cfif IsJSON(search.FileContent)>

                                  <!--- DESERIALIZE JSON --->

                                  <cfset searchResults = DeserializeJSON(search.FileContent) />

                           

                                  <!---  IF PLAYER SEARCH RETURNS AUCTIONINFO STRUCT --->

                                  <cfif StructKeyExists(searchResults,"auctionInfo")>

                                      <!--- SET NUMBER OF CARDS RETURNED FROM SEARCH --->

                                      <cfset numResults = ArrayLen(searchResults.auctionInfo) />

                                      <cfset statusCode = "Successful" />

                                      <cfif numResults EQ 0>

                                      <cfset statusCode = "Successful - No Results" />

                                      </cfif>

                                  <!--- ELSE IF ERROR CODE RETURNED --->

                                  <cfelseif StructKeyExists(searchResults,"code")>

                                      <cfset statusCode = searchResults.code />

                                      <!--- IF CODE 401 THEN SESSION HAS EXPIRED --->

                                      <cfif statusCode EQ 401>

                                          <!--- SET SESSION AS LOGGED OUT AND ATTEMPT SESSION REFRESH --->

                                          <cfset SESSION['#accountName#LoggedIn'] = 0 />

                                          <cfset logInAccount = Application.cfcs.Login.logInAccount(email,password) />

                                      </cfif>

                                  <!--- ELSE SOMETHING ELSE HAS HAPPENED --->

                                  <cfelse>

                                      <cfset statusCode = "Something Else - " & searchResults.code />

                                  </cfif>

                                  <!--- IF RESULTS RETURNED --->

                                  <cfif numResults GT 0>

                                  <!--- LOOP ROUND RESULTS AND CHECK IF MATCH BUYING CRITERIA --->

                                      <cfloop index="i" from="1" to="#numResults#">

                                          <!--- ***SAFETY CHECK*** - ENSURE ID OF CURRENT CARD IS SAME AS ONE SEARCHING FOR --->

                                          <cfif searchResults.auctionInfo[i].itemData.resourceID EQ resourceId AND getMinBIN.resourceID EQ resourceId>

                                              <!--- ENSURE BIN PRICE SET AND IS LESS THAN SET BUYING PRICE --->

                                              <cfif searchResults.auctionInfo[i].buyNowPrice GT 0 AND searchResults.auctionInfo[i].buyNowPrice LTE biddingPrice>

                                              <!--- SET AUCTION END TIME --->

                                              <cfset timeLeft = searchResults.auctionInfo[i].expires />

                                              <cfset auctionEnds = DateAdd("s",timeLeft,Now()) />

                                              <!--- BUY CARD --->

                                              <cfset buyCard = Application.cfcs.Bid.doBIN(searchResults.auctionInfo[i].tradeID,searchResul ts.auctionInfo[i].buyNowPrice,searchResults.auctionInfo[i].startingBid,searc hResults.auctionInfo[i].itemData.ID,searchResults.auctionInfo[i].itemData.resourceI D,startPrice,binPrice,lowestBIN,searchResults.auctionInfo[i].i temData.discardValue,auctionEnds,requestStart,requestDuration) />

                                              </cfif>

                                          </cfif>

                                      </cfloop>

                                  </cfif>

                              <cfelse>

                                  <cfset statusCode = search.FileContent />

                              </cfif>

                          </cfif>

                          <!--- INSERT SEARCH RECORD --->

                          <cfset insSearchRecord = Application.cfcs.Search.insSearchRecord(definitionID,statusCode,requestDuration,numResult s,biddingPrice) />

                      </cfif>

                  </cfloop>

              </cfif>

          </cfloop>

          • 2. Re: CFThread GC Overhead Limit Exceeded
            cpb07 Community Member

            Having held a conversation with BKBK and given them a deeper insight in to how my application is structured and works, the issue of the GC Overhead Limit being exceeded is still requiring tackled.

             

            The functionality of the app works very well whilst there is memory available and the cfthreads are required to allow the accounts to search synchronously.

             

            The question at hand here is how can I go about implementing a solution that ensures the JVM does not run out of memory?

            • 3. Re: CFThread GC Overhead Limit Exceeded
              BKBK MVP

              Yes, in our private mail discussions we weighed up alternative options. For example, using event gateways for asynchronous calls instead of cfthread, and websockets to dispatch notifications to users.

               

              This would mean redesigning and rewriting the code, with no guarantee that that would solve the memory problem. We agreed that a better solution would be to find ways to optimize the current code.

              • 4. Re: CFThread GC Overhead Limit Exceeded
                BKBK MVP

                So, here we go with suggestions for optimizing the code.

                 

                1) It would help to 'var' local function variables in all the functions in all the component classes, i.e. in Application.cfcs.Login, Application.cfcs.Search and Application.cfcs.Bids.

                 

                2) Disable server monitoring and profiling in the Coldfusion Administrator.

                • 5. Re: CFThread GC Overhead Limit Exceeded
                  cpb07 Community Member

                  All local function variables are set as LOCAL scope variables, which is the same difference as var-ing the variables if I'm not mistaken?

                   

                  Also, when server monitor is not running the JVM issue still occurs although I'll try and start capturing up-time so there's basis for comparison.

                  • 6. Re: CFThread GC Overhead Limit Exceeded
                    BKBK MVP

                    <cffunction>

                    <cfset var x1=0>

                    </cffunction>

                     

                    is what you should have for all local function variables. That is different from

                     

                    <cffunction>

                    <cfset local.x2=0>

                    </cffunction>

                     

                    I meant that you should disable server monitoring altogether.

                    • 7. Re: CFThread GC Overhead Limit Exceeded
                      cpb07 Community Member

                      So.....

                       

                      I've been monitoring how the app has been running and there's no real consistency with it. At first I thought BKBK had cracked it as I changed my locals to vars and disabled the monitoring and things seemed to be running smoothly but it seems to have went down around 4am this morning (running for around 12 hours).

                       

                      I restarted the service when I got in at 9 and it ran for about an hour before I got the java heap space error and had to restart the service. I then started the application again and it was only running for 15 minutes before the java heap space error appeared and the service came to a standstill and most recently the application was running for the last hour before it ran out of memory.

                       

                      Is there any other approaches I can try? I'm so exasperated with this as the application accomplishes its task so well whilst it's running!

                      • 8. Re: CFThread GC Overhead Limit Exceeded
                        BKBK MVP

                        <!--- DEFAULT SESSION VARIABLES: this line writes the session variable within the thread --->

                        <cfset SESSION["#attributes.accountName#LoggedIn"] = 0 />

                         

                        <!--- IF ACCOUNT LOGGED IN: this line is the first test of the value of the session variable within the thread --->

                        <cfif SESSION['#attributes.accountName#LoggedIn'] EQ 1>

                         

                        Yet, nowhere is the value of the session variable updated.

                        • 9. Re: CFThread GC Overhead Limit Exceeded
                          cpb07 Community Member

                          <!--- IF CODE 401 THEN SESSION HAS EXPIRED --->

                          <cfif statusCode EQ 401>

                                        <!--- SET SESSION AS LOGGED OUT AND ATTEMPT SESSION REFRESH --->

                                        <cfset SESSION['#accountName#LoggedIn'] = 0 />

                                        <cfset logInAccount = Application.cfcs.Login.logInAccount(email,password) />

                          </cfif>

                           

                          Sure it does. The response of each search is checked and the session variable maintained accordingly.

                          • 10. Re: CFThread GC Overhead Limit Exceeded
                            BKBK MVP

                            Not quite. The session key "#attributes.accountName#LoggedIn" is an attribute of the thread of a particular account name, and so is directly coupled to it. Whereas, '#accountName#LoggedIn' is ambiguous, and may even stand for '#getLogins.accountName#LoggedIn'.

                             

                            Therefore we cannot assume them to be the same. The query runs in the main execution thread. Whereas each accountname thread runs asynchronously to the main thread.

                            • 11. Re: CFThread GC Overhead Limit Exceeded
                              BKBK MVP

                              cpb07 wrote:

                               

                              <!--- IF CODE 401 THEN SESSION HAS EXPIRED --->

                              <cfif statusCode EQ 401>

                                            <!--- SET SESSION AS LOGGED OUT AND ATTEMPT SESSION REFRESH --->

                                            <cfset SESSION['#accountName#LoggedIn'] = 0 />

                                            <cfset logInAccount = Application.cfcs.Login.logInAccount(email,password) />

                              </cfif>

                               

                              Sure it does. The response of each search is checked and the session variable maintained accordingly.

                              In any case, that still fails to show us how the value of SESSION["#attributes.accountName#LoggedIn"] changes from 0 to 1.

                              • 12. Re: CFThread GC Overhead Limit Exceeded
                                cpb07 Community Member

                                Apologies, I'd actually made a change to the initial code posted in this thread as follows

                                 

                                <cfif THREAD.statusCode EQ 401>

                                     <!--- SET SESSION AS LOGGED OUT AND ATTEMPT SESSION REFRESH --->

                                     <cfset SESSION['#attributes.accountName#LoggedIn'] = 0 />

                                     <cfset THREAD.loginRequired = 1 />

                                </cfif>

                                 

                                In terms of the value being changed, the call at the start of the thread to Application.cfcs.Login.logInAccount makes all the http requests and at the very end of that function, if the account has been logged in successfully, it sets the logged in session variable to true.

                                • 13. Re: CFThread GC Overhead Limit Exceeded
                                  BKBK MVP

                                  That clarifies matters somewhat, but the point I raised still has to be addressed. Here it is again:

                                   

                                  <!--- WHILE ACCOUNT NOT LOGGED IN AND LESS THAN 8 LOGIN ATTEMPTS MADE --->

                                    <cfscript>

                                    while (SESSION['#attributes.accountName#LoggedIn'] EQ 0 AND SESSION['#attributes.accountName#LoginAttempts'] LT 8) {

                                  /* The value of SESSION['#attributes.accountName#LoggedIn'] remains 0 in this code block */

                                    }

                                    </cfscript>

                                   

                                     <!--- IF ACCOUNT LOGGED IN: How so? Nowhere does this value change from 0 to 1 --->

                                    <cfif SESSION['#attributes.accountName#LoggedIn'] EQ 1>

                                  • 14. Re: CFThread GC Overhead Limit Exceeded
                                    cpb07 Community Member

                                    It gets set to 1 in the logInAccount function in the Login CFC that gets called within the while (SESSION['#attributes.accountName#LoggedIn'] EQ 0 AND SESSION['#attributes.accountName#LoginAttempts'] LT 8) { } block

                                    • 15. Re: CFThread GC Overhead Limit Exceeded
                                      BKBK MVP

                                      So SESSION['#attributes.accountName#LoggedIn'] is updated away from view, in the function loginAccount. That's OK. But it raises an analysis issue.

                                       

                                      The function knows nothing about the attributes scope (as this belongs to the context of the thread). As such, you should have passed attributes.accountName explicitly to the function. You currently pass it implicitly through attributes.email and attributes.password. In any case, such indirect relationships raise the complexity of the code. More on the use of the attributes scope in a moment.

                                       

                                      This brings me to 3 questions, which are interrelated.

                                       

                                      1) Suppose a logged-in user were to arrive at this CFM page. Then the following code would practically log him out:

                                       

                                      <!--- DEFAULT SESSION VARIABLES --->

                                      <cfset SESSION["#attributes.accountName#LoggedIn"] = 0 />

                                        <cfset SESSION["#attributes.accountName#LoginAttempts"] = 0 />

                                       

                                      Was it perhaps your intention to use this instead:

                                       

                                      <cfparam name="SESSION[attributes.accountName & 'LoggedIn']" default="0">

                                      <cfparam name="SESSION[attributes.accountName & 'LoginAttempts']" default="0">

                                       

                                      2) Using both the attributes and getLogins scopes seems like duplication. I can see no difference in purpose between thread variables attributes.X and the respective query variables getLogins.X. Why not just use the following version of the cfthread tag, and delete every occurrence of the attributes scope?  At least, the code will then be much simpler.

                                       

                                        <cfthread  name="#getLogins.accountName#"  action="run">

                                       

                                      3) Your last remark apparently contains a serious design issue. Here it is again:

                                      It gets set to 1 in the logInAccount function in the Login CFC that gets called within the while (SESSION['#attributes.accountName#LoggedIn'] EQ 0 AND SESSION['#attributes.accountName#LoginAttempts'] LT 8) { } block

                                      This is bound to have a serious impact on CPU resources. In fact, I wonder if it would run at all for any modest number of accounts.

                                       

                                      Suppose that there are just 10 accounts and that they are all unauthenticated at the start. Then Coldfusion will immediately create 10 threads. They will possibly run simultaneously, indefinitely.

                                       

                                      The while-block suggests that each such thread will only stop after the account owner logs in. That is, after human interaction.

                                       

                                      However, by the time you raise your hand to scratch your head, Coldfusion will have run each while-block ten million times, say. By the time the first of the 10 users successfully logs in, Coldfusion will have run while-loops in the hundreds, or even thousands, of millions. That is a lot of processing power, most of it spent doing nothing.

                                       

                                      The following test will show you what I mean. You will need a tool to display memory usage(for example, Sysstat tool or Top command for Linux, Process Explorer or Task Manager for Windows).

                                       

                                      Run a CFM page containing the code,

                                       

                                      <cfset x=0>

                                      <cfset y=500000000>

                                       

                                      <cfthread name="th_x" action="run">

                                      <cfscript>

                                        while (x LT 500000000) {

                                      x=x+1;

                                        }

                                        </cfscript>

                                      </cfthread>

                                       

                                      <cfthread name="th_y" action="run">

                                      <cfscript>

                                        while (y GT 0) {

                                      y=y-1;

                                        }

                                        </cfscript>

                                      </cfthread>

                                       

                                      <cfthread action="join" name="th_x,th_y" />

                                       

                                      <cfscript>

                                      writeoutput('x=' & x & " ; y=" & y);

                                      </cfscript>

                                       

                                      View the memory consumption while the code runs. It runs on my system (Win 7, 64-Bit, 4GB RAM, CF 11) for about 5 minutes, during which Coldfusion's CPU usage stays at around 90%.

                                      • 16. Re: CFThread GC Overhead Limit Exceeded
                                        cpb07 Community Member

                                        Your first point is absolutely spot on, I should default the session variables as a cfparam rather than a cfset, although this page theoretically only requires to ever be called once because once the page has been loaded then each account will spawn its own thread and the criteria within the while condition should ensure that the account is always active (even if the session expires then there is a call to log the account back in).

                                         

                                        Point 2 probably stems from my inexperience with having cfthread in a cfloop, I'll update the code accordingly.

                                         

                                        As for point 3, there is no human interaction with this application. It is fully automated. Once I load this page once I should then not have any more interaction with it. I leave it to get on with the task in hand. The majority of the time the account will log in during the first attempt so the while (SESSION['#attributes.accountName#LoggedIn'] EQ 0 AND SESSION['#attributes.accountName#LoginAttempts'] LT 8) { } block will only loop once, for each account. And after all, the loop waits on the logInAccount function to complete before completing so it will not run any more than this?

                                         

                                        Running the code you posted, the page completes in 48 seconds each time (3 attempts) and the current percent of CPU consumption by the Adobe ColdFusion Launcher Application (coldfusion.exe) never surpasses 50%. That's running CF10 on a Windows 7 x64 machine with an Intel core i3-2100 processor and 8GB RAM.

                                        • 17. Re: CFThread GC Overhead Limit Exceeded
                                          BKBK MVP

                                          cpb07 wrote:

                                           

                                          As for point 3, there is no human interaction with this application. It is fully automated.

                                          Ah, that is a surprise!

                                           

                                          The majority of the time the account will log in during the first attempt so the while (SESSION['#attributes.accountName#LoggedIn'] EQ 0 AND SESSION['#attributes.accountName#LoginAttempts'] LT 8) { } block will only loop once

                                          Certainly makes for a workable design, assuming that there are no hold-ups in the logInAccount function. Under which conditions will 2 or more attempts be required?

                                           

                                          Running the code you posted, the page completes in 48 seconds each time (3 attempts) and the current percent of CPU consumption by the Adobe ColdFusion Launcher Application (coldfusion.exe) never surpasses 50%. That's running CF10 on a Windows 7 x64 machine with an Intel core i3-2100 processor and 8GB RAM.

                                          Judging by your times, you might have run the version with 100 000 000 loops that I posted earlier. In a later, more representative test, I used 500 000 000, and corrected my post accordingly.

                                          • 18. Re: CFThread GC Overhead Limit Exceeded
                                            cpb07 Community Member

                                            Sometimes the host server returns a message saying that it has reached its max connections so it sometimes takes several attempts until a successful login is achieved, although this isn't particularly common. The only time that an account won't log in at all is if the host server is down or if the account has been banned (usually for continually exceeding the request threshold). Having the login attempt set at 8 is probably a bit overkill.

                                             

                                            Yeah it was the version with the 100,000,000 loops that I had tried, I've just ran the one with 500,000,000 loops and again the ColdFusion CPU consumption stays around the 50% mark with the overall CPU usage sitting around 59%.

                                            • 19. Re: CFThread GC Overhead Limit Exceeded
                                              BKBK MVP

                                              Thanks for the explanation. Your result,

                                              I've just ran the one with 500,000,000 loops and again the ColdFusion CPU consumption stays around the 50% mark with the overall CPU usage sitting around 59%.

                                              seems more like it. It shows us the astonishingly high memory consumption of two loops that do practically nothing besides looping. If each was to create objects, the garbage collector might not have the opportunity of cleaning up before the overhead limit is reached.

                                               

                                              The moral is clear. We should avoid stagnation wherever possible. A function that fails to return, for example, might hold objects that cannot then be garbage-collected.

                                              Sometimes the host server returns a message saying that it has reached its max connections so it sometimes takes several attempts until a successful login is achieved, although this isn't particularly common. The only time that an account won't log in at all is if the host server is down or if the account has been banned (usually for continually exceeding the request threshold). Having the login attempt set at 8 is probably a bit overkill.

                                              That sounds good. It is crucial for the loops to keep going, without obstruction, and to terminate quickly in case of login failure.

                                               

                                              By the way, how many accounts are there in your database? I ask because I have just noticed parts of the code that may cause concurrency issues.

                                                <!--- INCLUDE FILE THAT CALCULATES BUYING AND SELLING PRICES --->

                                                <cfinclude template="sellingPrices.cfm" />

                                              I doubt that a thread can do an include like that into the current page. If it can, it will lead to a shambles. Since the threads are generated asynchronously, they run simultaneously, and in no particular order. If the include were possible, the different accounts will each include sellingPrices.cfm, in unpredictable order.

                                               

                                              The solution is to copy the content of sellingPrices.cfm to this particular point in the code.

                                              • 20. Re: CFThread GC Overhead Limit Exceeded
                                                cpb07 Community Member

                                                At the moment there are about 70 (although this is likely to increase dramatically over time). In the included sellingPrices.cfm I have set all variables in the thread scope and this seems to work fine (as I keep a record of each item request and all the prices match up the way they should). There is not a great deal amount of code in that page so I shall move it in to the main thread code.

                                                 

                                                Is it better to avoid focusing on settings in the JVM at the moment as we're trying to eradicate the issue that is causing the overhead limit issue, rather than postponing the inevitable?

                                                • 21. Re: CFThread GC Overhead Limit Exceeded
                                                  BKBK MVP

                                                  The focus is not on JVM settings. It is on detecting where the application might be creating and accumulating live objects.

                                                  • 22. Re: CFThread GC Overhead Limit Exceeded
                                                    cpb07 Community Member

                                                    I didn't mean that we were focusing on that at the present, I was just inquiring whether I should ignore exploring different JVM settings.

                                                    • 23. Re: CFThread GC Overhead Limit Exceeded
                                                      BKBK MVP

                                                      You actually brought up a good topic. We should have explored your JVM settings, if only to rule them out.

                                                       

                                                      So, what are your JVM settings - particularly maximum heap (Xmx), minimum heap (Xms) and XX:MaxPermSize?

                                                      • 24. Re: CFThread GC Overhead Limit Exceeded
                                                        cpb07 Community Member

                                                        I had tinkered with them previously but at present they are as follows:

                                                         

                                                        -server -XX:MaxPermSize=192m -XX:PermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfjvmGC.log

                                                         

                                                        Minimum and maximum JVM Heap Sizes are both set to 2048mb.

                                                        • 25. Re: CFThread GC Overhead Limit Exceeded
                                                          BKBK MVP

                                                          Ah, thanks. I would experiment with

                                                           

                                                          Minimum heap size: 1024

                                                          Maximum heap size: 2048

                                                          -XX:MaxPermSize=1024m

                                                           

                                                          I suspect that -XX:MaxPermSize:192m is too small for your purposes.

                                                          • 26. Re: CFThread GC Overhead Limit Exceeded
                                                            cpb07 Community Member

                                                            Okay, thanks. Just restarting the service so the changes take effect and will keep you updated.

                                                            • 27. Re: CFThread GC Overhead Limit Exceeded
                                                              cpb07 Community Member

                                                              Okay...so that's ColdFusion just grind to a halt after 4 hours (which has been an improvement on recent up-time) and from keeping an eye on the resource monitor in task manager, the ColdFusion CPU consumption was sitting below 20% but it is currently at 85%+ (as I've not stopped the service yet) so I'm wondering if it is just a gradual filling up of the memory (although I'd have expected the CPU consumption to gradually increase) or if there is something going wrong and demanding all the memory suddenly until such point as there is no more free memory.

                                                               

                                                              I'll restart the service and keep an eye on the resource monitor again and see if there is sudden spike in CPU consumption.

                                                              • 28. Re: CFThread GC Overhead Limit Exceeded
                                                                carl type3 Community Member

                                                                Hello,

                                                                 

                                                                I see thread has moved to discussing JVM arguments so thought I would join in as I refrain from commenting on CFM syntax.

                                                                What Java version is CF10 using? Early CF10 installer applied Java 6 (which is Oracle EOL) and latter installer used Java 7.

                                                                 

                                                                I see Java logging is enabled so perhaps the details in your last cfjvmGC.log would be interesting. Can you ZIP and host that somewhere so I can download and read log?

                                                                 

                                                                Regards, Carl.

                                                                • 29. Re: CFThread GC Overhead Limit Exceeded
                                                                  cpb07 Community Member

                                                                  Progress report...

                                                                   

                                                                  The service has ran for 3-4 hours on three occasions, the other occasion I ran it the GC overhead limit reached message was encountered after only the 10th account logged in. I'm starting to think that rather than the memory filling up gradually (although increasing the MaxPermSize seems to have helped greatly) that eventually a request is getting stuck and this is causing all the subsequent requests to throttle the memory. Sound plausable?

                                                                   

                                                                  Carl, I've uploaded my last cfjvmGC.log which can be found at http://www.filedropper.com/cfjvmgc

                                                                   

                                                                  EDIT: The service just ran for 20 minutes before the GC overhead limit exceed message appeared so that would suggest that a request is getting stuck (for lack of a better term).

                                                                   

                                                                  Now, with the inclusion of more accounts I have a cfsetting variable at the very start of the file that sets the request timeout to 600 seconds (10 minutes - at present it takes around 6-8 minutes to log all accounts in but the more accounts added the longer this will take obviously).

                                                                   

                                                                  Could it be that because this overwrites the default CFAdmin timeout setting of 60 seconds that for some reason, quite randomly, a cfhttp or cfquery request is not returning a response in a timely manner and this is causing any subsequent requests to have to wait for this request to finish and this is throttling the memory?

                                                                  • 30. Re: CFThread GC Overhead Limit Exceeded
                                                                    BKBK MVP

                                                                    cpb07 wrote:

                                                                     

                                                                    I'm starting to think that rather than the memory filling up gradually ... that eventually a request is getting stuck and this is causing all the subsequent requests to throttle the memory. Sound plausable?

                                                                    My instincts exactly. There is filling up all right, and not gradually. Should there be talk of 'subsequent requests', in plural? From what I understand, there is just the one request, albeit one that spawns 70+ threads.

                                                                     

                                                                    EDIT: The service just ran for 20 minutes before the GC overhead limit exceed message appeared so that would suggest that a request is getting stuck (for lack of a better term).

                                                                    When you consider the 20 minutes together with the line <cfset Sleep(30000) />, it tells you the code never made it past 40 threads. [20 minutes = 1200 seconds. Coldfusion sleeps for 30000 ms, that is, 30 seconds during creation of subsequent threads. 1200 / 30 = 40 ]

                                                                     

                                                                    Now, I'd mention to BKBK in a discussion we had that there is a scheduled task that runs every 10 minutes that basically gets a list of all the items I'm trading and gets the current lowest buy now price for each. This can often take anywhere between 4-8 minutes to complete so I have a cfsetting variable in the .cfm file that sets the request timeout to 600 seconds (10 minutes - so that even if the task has not completed, it will start again as the lowest buy now price data will now be out of date).

                                                                     

                                                                    Could it be that because this overwrites the default CFAdmin timeout setting of 60 seconds that for some reason, quite randomly, a cfhttp or cfquery request is not returning a response in a timely manner and this is causing any subsequent requests to have to wait for this request to finish and this is throttling the memory?

                                                                    To answer these questions, it might help to have a closer look into the threads. Suppose you wish to know how they stand after, say, 30 minutes.

                                                                     

                                                                    You could then include an extra thread, named sleeper30min, say, that does nothing but wait for 30 minutes. Next, join the main page execution thread to sleeper30min.

                                                                    At this point, dump all the threads.

                                                                     

                                                                    It goes something like this:

                                                                     

                                                                    <cfloop query="getLogins">

                                                                       <!--- HAVE A SLEEP SO IP DOESN'T GET FLAGGED FOR SENDING TOO MANY REQUESTS AT ONCE --->

                                                                      <cfset Sleep(30000) />

                                                                      <!--- CREATE THREAD FOR ACCOUNT --->

                                                                      <cfthread ...>

                                                                    etc.

                                                                    </cfthread>

                                                                     

                                                                    <!--- Join page thread to thread sleeper30min --->

                                                                    <cfthread action="join" name="sleeper30min" timeout="1800000" />

                                                                     

                                                                    <cfif sleeper30min.STATUS is "TERMINATED">

                                                                    <!--- Get out of loop --->

                                                                    <cfbreak>

                                                                    </cfif>

                                                                     

                                                                    </cfloop>

                                                                     

                                                                    <cfif sleeper30min.STATUS is "TERMINATED">

                                                                    <cfloop query="getLogins">

                                                                      <cfset thread_name = getLogins.accountName>

                                                                       <cfdump label="Thread #thread_name#" var="#evaluate(thread_name)#">

                                                                    </cfloop>

                                                                    </cfif>

                                                                    • 31. Re: CFThread GC Overhead Limit Exceeded
                                                                      carl type3 Community Member

                                                                      You will need to make a copy of cfjvmGC.log since when CF starts a new log is created and existing log with the memory details and garbage collection information lost. The log you attached has 1 full garbage collection at 29 seconds for 0.03
                                                                      seconds duration with just over 2 minutes of up or log time.


                                                                      Be sure to save your next log when fail occurs and host that.

                                                                       

                                                                      HTH, Carl.

                                                                      • 32. Re: CFThread GC Overhead Limit Exceeded
                                                                        carl type3 Community Member

                                                                        As for finding out what Java version is in use, in CFadmin select System Information i button. What are the details EG:

                                                                         

                                                                        JVM Details 

                                                                        Java Version  1.7.0_51   

                                                                        Java Vendor  Oracle Corporation   

                                                                        Java Vendor URL  http://java.oracle.com/ 

                                                                        Java Home  D:\Program Files\Java\jdk1.7.0_51\jre   

                                                                        Java VM Specification Version  1.7   

                                                                        Java VM Version  24.51-b03   

                                                                        Java VM Name  Java HotSpot(TM) 64-Bit Server VM

                                                                         

                                                                        Regards, Carl.

                                                                        • 33. Re: CFThread GC Overhead Limit Exceeded
                                                                          cpb07 Community Member

                                                                          Carl, had a fail this morning and the log can be found at http://www.filedropper.com/cfjvmgc_1. I'd forgot you'd asked about Java version, details are as follows:

                                                                           

                                                                          Java Version1.7.0_15 
                                                                          Java VendorOracle Corporation 
                                                                          Java Vendor URLhttp://java.oracle.com/ 
                                                                          Java HomeC:\Services\web\jre 
                                                                          Java VM Specification Version1.7 
                                                                          Java VM Version23.7-b01 
                                                                          Java VM NameJava HotSpot(TM) 64-Bit Server VM 

                                                                           

                                                                          BKBK, I'd reduced the sleep time to 5 seconds so all accounts log in okay. I'll implement the suggestion you made and let you know what I find.

                                                                          • 34. Re: CFThread GC Overhead Limit Exceeded
                                                                            cpb07 Community Member

                                                                            Not even getting 30 minutes up-time anymore, memory threshold seems to be getting exceeded after 10-15 minutes for whatever reason...grrrrrrr! Will report back further.

                                                                            • 35. Re: CFThread GC Overhead Limit Exceeded
                                                                              carl type3 Community Member

                                                                              Capture.JPG

                                                                               

                                                                              JVM looks like this thanks to gcviewer:


                                                                              The diagram does not show PermGen details. Tail of log says:

                                                                               

                                                                              PSPermGen  total 1048576K, used 80172K object space 1048576K, 7% used.

                                                                               

                                                                              While you may have something to resolve with the CFM code, for now to keep the system up perhaps you can do better to make some JVM adjustments. Keep in mind this is probably not a fix for the overall problem more try this to see if the system stays up while you continue to work on other CFM matters.

                                                                               

                                                                              I expect JVM details are like this currently:


                                                                              Minimum heap size: 1024
                                                                              Maximum heap size: 2048
                                                                              -XX:MaxPermSize=1024m

                                                                               

                                                                              The overall system is:

                                                                               

                                                                              CF10 on a Windows 7 x64 machine with an Intel core i3-2100 processor and 8GB RAM.


                                                                              Seeing not many objects are maintained in PermGen you can make that smaller. Double heap sizes and set a value for the new part of heap. EG:

                                                                               

                                                                              Minimum heap size: 2048
                                                                              Maximum heap size: 4096
                                                                              -XX:MaxPermSize=324m
                                                                              -Xmn256m

                                                                               

                                                                              Or this way if you prefer in CFadmin:

                                                                               

                                                                              -server -Xmn256m -XX:MaxPermSize=324m -XX:PermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfjvmGC.log

                                                                               

                                                                              Or this way by editing JVM.CONFIG:

                                                                               

                                                                              -server -Xms2048m -Xmx4096m -Xmn256m -XX:MaxPermSize=324m -XX:PermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfjvmGC.log

                                                                               


                                                                              For now it might be advisable to set JVM to perform full garbage collection every 10 minutes but as I post I am undecided so will post again more thoughts on that if it seems more like a good bandaid idea to me. I think stay with UseParallelGC because it tends to keep the heap evacuated rather than alter garbage collector to something that tends to maintain objects in memory.


                                                                              Java is 7 so not EOL 6 tho fair to say 7u15 is old with 7u67 current. Java 8 is also release however no support statement from Adobe with running  CF10 on Java 8. Since you are not using a newer garbage collection algorithim like G1GC (and for now I do not recommend you do) I think stay with 7u15 for now.

                                                                               

                                                                              HTH, Carl.

                                                                              • 36. Re: CFThread GC Overhead Limit Exceeded
                                                                                cpb07 Community Member

                                                                                I added the sleeper thread that you suggested and the information it gives me on the thread is elapsed time, name, output (which is empty), priority, start time and status. Were you expecting there to be more information?

                                                                                 

                                                                                I check the error logs whenever ColdFusion is grinding to a halt and cannot see any other errors than the GC overhead limit exceeded one, but cfhttp requests are taking 30-40 seconds to return rather than a few hundred milliseconds.

                                                                                 

                                                                                Would anyone be willing to cast their eye over the application on their own development environment to see if they A) encounter the same issue and B) see if they can find what the cause is?

                                                                                • 37. Re: CFThread GC Overhead Limit Exceeded
                                                                                  cpb07 Community Member

                                                                                  Carl

                                                                                   

                                                                                  I implemented the suggestions you made but unfortunately this does not seem to have had any impact on the availability of the application, for the last couple days I've never had longer than 25-30 minutes up-time

                                                                                  • 38. Re: CFThread GC Overhead Limit Exceeded
                                                                                    BKBK MVP

                                                                                    At this point, the key values we wish to know are those of status. You want to know which threads have run completely (status=COMPLETED), which were prevented from running (status=TERMINATED) and which have yet to run (status=NOT_STARTED).

                                                                                    • 39. Re: CFThread GC Overhead Limit Exceeded
                                                                                      cpb07 Community Member

                                                                                      I assume these are found in a thread dump?