We are having an issue with CPU loading on our server. It has to do with Client Storage Purge. I see where this issue has been around for a while but could not find a solid solution. Our sever is running Windows Server 2003 Standard and ColdFusion 8. There is only one ecommerence website utilizing CF and the database is MSSQL.
The server average CPU usage is between 5% and 10%. Then this appears in the log; 03/30 05:56:34 Information [scheduler-3] - Run Client Storage Purge. The CPU usage jumps to 50% to 80% for about 4 to 5 hours. Then after it settles back down, with in the next few hours it starts over again.
ColdFusion Administrator > Server Settings > Client Variables
Select Default Storage Mechanism for Client Sessions is set to Registry
Purge Interval is set to 1 hour and 7 minutes.
Now, my working knowledge of ColdFusion is not at an expert level. I really do not know what Client Storage Purge is doing, what client variables are being stored and why.
I hope I have included enough information to start the 21 question routine. Any help with this issue would be appreciated.
David, there’s no need for 21 questions. But conversely there is also not as simple an answer as some may purport.
Bottom line, yes, the CPU use is likely due to the purge that’s running (by default every 67 minutes). And because your default client storage is “registry”, it means that it’s purging the registry, and that can take a lot of CPU. But the answer is NOT to just change the default location to cookie, as some will assert. If you are really using client variables, and you change the location like that, then all the client vars will be “lost” for your users, which you may not want. But I appreciate the concern over performance, and the temptation to “do whatever it takes” to stop the madness of high CPU use.
But let me propose something else: it may not be that you need to be having CF write the client variables at all. If you don’t really use client variables in your code (setting values for variables like client.somename), note that you will STILL be writing to that client store on EVERY page request where you have enabled clientmanagement (in a cfapplication tag, typically in application.cfm, or with this.clientmanagement in application.cfc).
If you click on that “registry” link (or any database listed) on that client storage Admin page, you will see a setting for “disable global client variable updates”. That is NOT checked by default, which means that CF then tracks its own client variable (hitcount and lastvisit) for EVERY page that enables clientmanagement, even if it never “uses” client variables.
Now, should you turn that off? Likely so. The only caveat is that if you have any code in your app that USES these special variables (client.lastvisit and client.hitcount). If you do not, then you don’t need that feature enabled. Disabling it will stop CF writing those to that location (the registry, or a database selected to be a client storage mechanism), at least for requests to pages that don’t otherwise really use client variables. Many have found that doing that alone has helped a lot, in that each page request no longer writes to the client store. (And it’s worse for visits from spiders, bots and other automated requests, as they would CREATE A NEW client repository entry on EVERY page.
But doing that change alone WILL NOT STOP the hourly purge. That data is still there, and CF will still try to purge it. Now, some are tempted to say “well just change the purge interval to 24 hours or something”. And that will indeed stop it happening hourly, but it means that it will now happen at least daily, and the clock starts when CF comes up, so it may happen in the middle of the day. Not good.
The real solution is to remove the client variables yourself. But even that is not trivial.
First, if it’s a database, you may be tempted to just delete all the records in the cglobal table, but beware: if there are records in the cdata table (real client variables you’ve written from your code), then you don’t want to delete the records in cglobal that are related to those.
Second, it’s even harder when they’re in the registry. First: DO NOT go into regedit and click to expand the keys in the location where they are written (HKEY_LOCAL_MACHINE\SOFTWARE\Macromedia\ColdFusion\CurrentVersion\Cli ents). Doing that could bring your server down itself, if there are millions of keys (client variable entries) that have been written (which again is NOT that unusual).
Instead, you would want to delete them from a command line tool (like REG). But even that is not as straightforward as it seems. You would want to delete all those that are not really holding “real” client variables (if you use them at all). That’s not easy to figure out. But if you’re scrambling and willing to sacrifice any “real” client variables in purging the evil demons, you can do it from the command line with:
REG DELETE HKLM\SOFTWARE\Macromedia\ColdFusion\CurrentVersion\Clients
Just beware that may take a long time to run. You may want to do it during times when users would not be as affected. Then again, most servers are hit more by spiders and bots than by real people, so there’s never a “good time” when they may not be affected, but you may not care as much about them. That said, don’t ignore the significance of them (spiders, bots, and other automated requests) to be the real root cause of all this client variable bloat. In my day-to-day CF troubleshooting consulting practice, I find this to be a problem for someone pretty much every week.
Indeed, I have written about it and you can find more in the following entry(on the impact of both client variables and session variables, and by association memory, which makes people think CF has a “leak”):
You’ll see there also a link to a recorded presentation I also did on the topic.
So, no, not 21 questions. But perhaps 21 points to ponder. If this problem was easy (to understand and solve) it wouldn’t be such a burden. I do what I can, with the blog and presentation, but obviously many (most) will never see it. I didn’t just point you to it since your question was more specifically just about the client vars in the registry. I’ve drawn out the most salient points here.
Let us know if that helps.
Providing fast, remote, on-demand troubleshooting services for CF (and CFBuilder)
See also http://www.cf911.com for more on CF troubleshooting resources
I appreciate the time you put into that reply and yes it helps. I forgot to include in my original post that it is the jrun.exe process using the CPU. There are few options that I might be able to deploy to help with this issue. First off, I am going to contact the software vendor and find out how client variable intensive their software is. Also, maybe they have run into this problem in the past and have a possible solution.
UPDATE: I contacted the software vendor and below is their reply.
You want to store them as cookies … we don’t use them specifically so moving them to cookies will be a good thing for you,.
I am going to go into ColdFusion Administrator and change the storage session as Cookie. Now I am not sure if this plays a part in it or not, but I do have several Cookie errors stating "error Cannot create cookie" all on the domain that is or ecommerence site. I wonder if this change will fix that issue.
David, just to be clear, it would in fact be the jrun.exe process which would show the high CPU during the registry purge.
BTW, I meant to add also (and do say so in the blog entry) that if one is on Linux/Unix/OS X, things are even worse. The option for “registry” is still there in the CF Admin (and still the default), but whereas those *nix-based don’t have a real registry, CF instead simulated it in a TEXT FILE! It’s typically in /opt/coldfusion/registry/cf.registry.
I’m curious: when you say there are a few options you “might be able to deploy”, are you suggesting something other than the things I proposed? I’d be curious what else you may consider. That said, I did make all that recommendation on your specific identification of CPU related to the client var purge and your use of the registry. All the signs point to what I shared. I’ll be curious if you’re not thinking it’s something else, and if so, what has led you to think that.
The options I was referring to were the ones you suggested. I did contact the software vendor a few weeks ago regarding this issue, but their response was vague at best. But today, due to the information in your post, I was able to ask a much more clear and directed question.
I am going to monitor the server and in the next few days hope to see the jrun.exe process not require so much CPU resources. I believe the registry will eventually be cleared out since no new data is being posted. I will post back in a few days to update the progress.
I do administer 2 linux servers, but do not have ColdFusion installed. Instead they have PHP and MySQL. I did read in a forum where someone posted that the file /opt/coldfusion/registry/cf.registry had, what they believe, over a million lines in it. I can see where that would cause problems.
I appreciate the you help and better understanding of this issue. I also did read your blog entry, which also helped.
I need to resurrect this thread. In the ColdFusion Administrator, Server Settings, Client Variables I changed the client sessions to Cookie from Registry. I thought over a few days that when Client Storage Purge would clear out the Registry and eventually jrun.exe would not be using so much CPU resources. Nothing has changed, Client Storage Purge still runs for about 4 hours and then in a few hours starts all over again. Is there a way to stop this. I thought once I stopped using the Registry for client sessions, the Registry would clear itself out.
David, these references say - it is most efficient to store client variables in a database.
Seems CF9 has similar references.
Perhaps you so well to change from cookie to MS SQL (or other?) database for client variables.
Could some other problem not related to client variables be causing the CPU to bottle-neck on Jrun.exe?
David, changing those admin defaults does NOT then cause CF to somehow clear out the registry faster, no. They merely set what the default storage should be if code does not specify its own using clientstorage (in cfapplication or the this scope in application.cfc). CF will continue to purge (and you can’t stop it from that Admin interface). If you click the “registry” link in the client variable page, what is the number of days they can last? The default is 90, so that it would take 90 days for the registry to be purged.
You could try changing that to 0, but then you’ll wait for CF to clear the registry (which could also hang up CF for some time while it does, if you have hundreds or thousands or millions of records). Instead, you could clear it yourself using a command-line REG tool. I document that, and much more on this whole subject of dealing with client vars (and the implication on them and session vars from spiders, bots and other automated requests), in a blog entry here:
Just be careful: too many people offer one-sentence or one-paragraph answers. If it was that simple, it wouldn’t be a challenge deciding how to remediate the problem. See the entry for more (where even I myself have still more I’d like to add in an updated version of it some day).
I hope this information is helpful to you.
Symptom: CPU spins at 100% - and high memory usage - about every hour after startup (large memory use too).
Here's a common one that some fall into going from development to production. The default for storing client variables is "REGISTRY". Once the number of records in the registry is large, the query to get all records and delete expired client records can take 100% CPU for minutes at a time.
Registry client storage should never be used for production systems but often developers fall into this by accident by not explicitly specifying a data source (and the ColdFusion admin defaulting to "REGISTRY"). Since the registry isn't a real database, ColdFusion has to retrieve the entire registry client tree (high memory usage) and compare the date/time one at a time to decide whether to purge a record. This is a CPU-intensive operation with an hourly purge that may only delete a few records.