Skip navigation
ddelbiondo
Currently Being Moderated

Client Storage Purge

Mar 30, 2012 7:34 AM

Hello,

 

We are having an issue with CPU loading on our server.  It has to do with Client Storage Purge. I see where this issue has been around for a while but could not find a solid solution.  Our sever is running Windows Server 2003 Standard and ColdFusion 8.  There is only one ecommerence website utilizing CF and the database is MSSQL.

 

The server average CPU usage is between 5% and 10%.  Then this appears in the log; 03/30 05:56:34 Information [scheduler-3] - Run Client Storage Purge.  The CPU usage jumps to 50% to 80% for about 4 to 5 hours.  Then after it settles back down, with in the next few hours it starts over again.

 

Settings,

ColdFusion Administrator > Server Settings > Client Variables

Select Default Storage Mechanism for Client Sessions is set to Registry

Purge Interval is set to 1 hour and 7 minutes.

 

Now, my working knowledge of ColdFusion is not at an expert level.  I really do not know what Client Storage Purge is doing, what client variables are being stored and why.

 

I hope I have included enough information to start the 21 question routine.  Any help with this issue would be appreciated.

 

Regards,

David

 
Replies
  • Currently Being Moderated
    Mar 30, 2012 10:42 AM   in reply to ddelbiondo

    David, there’s no need for 21 questions. But conversely there is also not as simple an answer as some may purport.

     

    Bottom line, yes, the CPU use is likely due to the purge that’s running (by default every 67 minutes). And because your default client storage is “registry”, it means that it’s purging the registry, and that can take a lot of CPU. But the answer is NOT to just change the default location to cookie, as some will assert.  If you are really using client variables, and you change the location like that, then all the client vars will be “lost” for your users, which you may not want. But I appreciate the concern over performance, and the temptation to “do whatever it takes” to stop the madness of high CPU use.

     

     

    But let me propose something else: it may not be that you need to be having CF write the client variables at all.  If you don’t really use client variables in your code (setting values for variables like client.somename), note that you will STILL be writing to that client store on EVERY page request where you have enabled clientmanagement (in a cfapplication tag, typically in application.cfm, or with this.clientmanagement in application.cfc).

     

    If you click on that “registry” link (or any database listed) on that client storage Admin page, you will see a setting for “disable global client variable updates”. That is NOT checked by default, which means that CF then tracks its own client variable (hitcount and lastvisit) for EVERY page that enables clientmanagement, even if it never “uses” client variables.

     

    Now, should you turn that off? Likely so. The only caveat is that if you have any code in your app that USES these special variables (client.lastvisit and client.hitcount). If you do not, then you don’t need that feature enabled. Disabling it will stop CF writing those to that location (the registry, or a database selected to be a client storage mechanism), at least for requests to pages that don’t otherwise really use client variables. Many have found that doing that alone has helped a lot, in that each page request no longer writes to the client store. (And it’s worse for visits from spiders, bots and other automated requests, as they would CREATE A NEW client repository entry on EVERY page.

     

    But doing that change alone WILL NOT STOP the hourly purge. That data is still there, and CF will still try to purge it. Now, some are tempted to say “well just change the purge interval to 24 hours or something”. And that will indeed stop it happening hourly, but it means that it will now happen at least daily, and the clock starts when CF comes up, so it may happen in the middle of the day. Not good.

     

    The real solution is to remove the client variables yourself. But even that is not trivial.

     

    First, if it’s a database, you may be tempted to just delete all the records in the cglobal table, but beware: if there are records in the cdata table (real client variables you’ve written from your code), then you don’t want to delete the records in cglobal that are related to those.

     

    Second, it’s even harder when they’re in the registry. First: DO NOT go into regedit and click to expand the keys in the location where they are written (HKEY_LOCAL_MACHINE\SOFTWARE\Macromedia\ColdFusion\CurrentVersion\Cli ents). Doing that could bring your server down itself, if there are millions of keys (client variable entries) that have been written (which again is NOT that unusual).

     

    Instead, you would want to delete them from a command line tool (like REG). But even that is not as straightforward as it seems. You would want to delete all those that are not really holding “real” client variables (if you use them at all). That’s not easy to figure out. But if you’re scrambling and willing to sacrifice any “real” client variables in purging the evil demons, you can do it from the command line with:

     

    REG DELETE HKLM\SOFTWARE\Macromedia\ColdFusion\CurrentVersion\Clients

     

    Just beware that may take a long time to run. You may want to do it during times when users would not be as affected.  Then again, most servers are hit more by spiders and bots than by real people, so there’s never a “good time” when they may not be affected, but you may not care as much about them. That said, don’t ignore the significance of them (spiders, bots, and other automated requests) to be the real root cause of all this client variable bloat. In my day-to-day CF troubleshooting consulting practice, I find this to be a problem for someone pretty much every week.

     

    Indeed, I have written about it and you can find more in the following entry(on the impact of both client variables and session variables, and by association memory, which makes people think CF has a “leak”):

     

    http://www.carehart.org/blog/client/index.cfm/2006/10/4/bots_and_spide rs_and_poor_CF_performance

     

    You’ll see there also a link to a recorded presentation I also did on the topic.

     

    So, no, not 21 questions. But perhaps 21 points to ponder. If this problem was easy (to understand and solve) it wouldn’t be such a burden. I do what I can, with the blog and presentation, but obviously many (most) will never see it. I didn’t just point you to it since your question was more specifically just about the client vars in the registry. I’ve drawn out the most salient points here.

     

    Let us know if that helps.

     

     

     

    /charlie arehart

    charlie@carehart.org

     

    Providing fast, remote, on-demand troubleshooting services for CF (and CFBuilder)

    More at http://www.carehart.org/consulting

    See also http://www.cf911.com for more on CF troubleshooting resources

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 30, 2012 4:54 PM   in reply to ddelbiondo

    David, just to be clear, it would in fact be the jrun.exe process which would show the high CPU during the registry purge.

     

    BTW, I meant to add also (and do say so in the blog entry) that if one is on Linux/Unix/OS X, things are even worse. The option for “registry” is still there in the CF Admin (and still the default), but whereas those *nix-based don’t have a real registry, CF instead simulated it in a TEXT FILE! It’s typically in /opt/coldfusion/registry/cf.registry.

     

    I’m curious: when you say there are a few options you “might be able to deploy”, are you suggesting something other than the things I proposed? I’d be curious what else you may consider. That said, I did make all that recommendation on your specific identification of CPU related to the client var purge and your use of the registry. All the signs point to what I shared. I’ll be curious if you’re not thinking it’s something else, and if so, what has led you to think that.

     

    /charlie

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 30, 2012 6:39 PM   in reply to ddelbiondo

    Good to hear. Thanks for the update. Hope it works out.

     

     

     

    /charlie

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 10, 2012 11:12 PM   in reply to ddelbiondo

    David, these references say - it is most efficient to store client variables in a database.

     

    http://livedocs.adobe.com/coldfusion/8/htmldocs/help.html?content=shar edVars_08.html

     

    http://livedocs.adobe.com/coldfusion/8/htmldocs/help.html?content=basi config_08.html

     

    Seems CF9 has similar references.

     

    Perhaps you so well to change from cookie to MS SQL (or other?) database for client variables.

     

    Could some other problem not related to client variables be causing the CPU to bottle-neck on Jrun.exe?

     

    HTH, Carl.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 11, 2012 6:37 PM   in reply to ddelbiondo

    David, changing those admin defaults does NOT then cause CF to somehow clear out the registry faster, no. They merely set what the default storage should be if code does not specify its own using clientstorage (in cfapplication or the this scope in application.cfc). CF will continue to purge (and you can’t stop it from that Admin interface). If you click the “registry” link in the client variable page, what is the number of days they can last? The default is 90, so that it would take 90 days for the registry to be purged.

     

    You could try changing that to 0, but then you’ll wait for CF to clear the registry (which could also hang up CF for some time while it does, if you have hundreds or thousands or millions of records). Instead, you could clear it yourself using a command-line REG tool. I document that, and much more on this whole subject of dealing with client vars (and the implication on them and session vars from spiders, bots and other automated requests), in a blog entry here:

     

    http://www.carehart.org/blog/client/index.cfm/2006/10/4/bots_and_spide rs_and_poor_CF_performance

     

    Just be careful: too many people offer one-sentence or one-paragraph answers. If it was that simple, it wouldn’t be a challenge deciding how to remediate the problem. See the entry for more (where even I myself have still more I’d like to add in an updated version of it some day).

     

     

     

    /charlie

     
    |
    Mark as:
  • Currently Being Moderated
    May 20, 2012 8:08 AM   in reply to ddelbiondo

    Hi David,

    I hope this information is helpful to you.

    Symptom: CPU spins at 100% - and high memory usage - about every hour after startup (large memory use too).

    Here's a common one that some fall into going from development to production. The default for storing client variables is "REGISTRY". Once the number of records in the registry is large, the query to get all records and delete expired client records can take 100% CPU for minutes at a time.

    Registry client storage should never be used for production systems but often developers fall into this by accident by not explicitly specifying a data source (and the ColdFusion admin defaulting to "REGISTRY"). Since the registry isn't a real database, ColdFusion has to retrieve the entire registry client tree (high memory usage) and compare the date/time one at a time to decide whether to purge a record. This is a CPU-intensive operation with an hourly purge that may only delete a few records.

    Regards,

    Nimit

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points