Copy link to clipboard
Copied
Greetings,
I have a fresh install of CF11 standard on Windows 8 server 64x
16GB of ram
JVM settings:
-server -Xmn1024m -Xss1024k -XX:PermSize=256m -XX:MaxPermSize=2048m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random
Min Heap: 2048mb, Max Heap 6144mb
VFS settings:
Global memory limit for VFS: 1000mb
Memory Limit per Application for VFS: 400mb
Currently CF is using 1.7GB of ram
I have a 6 applications on this server. 1 of which makes use of the VFS.
The others are simple cfm applications that have NOTHING to do with the VFS system.
I have a scheduled task that calls simple cfm pages to update tables in our database. When these tasks run, i get a global error email (ones i have setup if a cf error happens) and the gist of the error is
Browser | CFSCHEDULE |
DateTime | {ts '2014-08-16 15:07:00'} |
Diagnostics | The Global Memory Limit 1000 MB for In-Memory Virtual FileSystem exceeded . <br>Increase the VFS memory limit in ColdFusion Administrator. <br>The error occurred on line 17. |
GeneratedContent | [empty string] |
HTTPReferer | [empty string] |
Mailto | [empty string] |
Message | The Global Memory Limit 1000 MB for In-Memory Virtual FileSystem exceeded . |
QueryString | [empty string] |
RemoteAddress | xxx.xxx.xxx.xxx |
I have a utility i can run to view all the files in the VFS. and there is currently a total of 8 cfm files there that make up 4064KB of space within the VFS.
The 1 application that actually uses the VFS has several functions. One of which basically writes out CFM files and then includes the cfm as a content variable and returns the variable.
Under CF9 we were also deleting the ram drive CFM files after including them. But under CF11 this fails with an error saying the modified date could not be found. I have since removed the delete file part and the system works but we get this out of memory error with the VFS.
Can someone shed some light on this for me?
It makes no since.
Thanks
I am sorry to say to anyone else who my find this issue, i could not resolve it.
Thank you to BK for all your great suggestions and help.
We just switched the entire system over to hard drive. This will end up hurting us in the end but at this point we dont have an option, our system has been down since the day we upgraded it to CF11. We did a lot of load testing prior to the migration and from our testing it worked. But there were a lot of issues with IIS connectors and we also had new memory ins
...Copy link to clipboard
Copied
-Xmn1024m -Xss1024k -XX:PermSize=256m -XX:MaxPermSize=2048m
Min Heap: 2048mb, Max Heap 6144mb
Why muck about? What about 3 simple settings such as
-Xmx2048m -Xms1024k -XX:MaxPermSize=256m
Copy link to clipboard
Copied
BKBK,
so i should update the server JVM settings to what you are suggesting?
Copy link to clipboard
Copied
Yes, the JVM settings in the Coldfusion Administrator. Then restart Coldfusion. Test and see what happens.
Copy link to clipboard
Copied
Service wont start with those settings.
I reverted it back to what i had before.
Thanks anyway
Copy link to clipboard
Copied
chewymole wrote:
Service wont start with those settings.
Beg your pardon. There was a typo: it should have read "-Xmx2048m -Xms1024m -XX:MaxPermSize=256m"
Also, did you by any chance duplicate the flags -Xmx and -Xms in the field "JVM Arguments"? That would have been an error.
I later realized it would have been less confusing to represent -Xmx and -Xms in exactly the way this is done in the Administrator, namely:
Minimum JVM Heap Size (in MB):1024 | Maximum JVM Heap Size (in MB):2048 |
Then you only had to modify the flag -XX:MaxPermSize in the field "JVM Arguments".
Copy link to clipboard
Copied
I am actually trying to set the max heap to 6GB and min to 4GB
Can you update the response to reflect that?
I am not sure now because i just copied and pasted into the jvm config.
When it did not start, I just reverted it back to previous version and saved.
Thanks again
Tim
Copy link to clipboard
Copied
chewymole wrote:
I am not sure now because i just copied and pasted into the jvm config.
That's what I suspected. In so doing, you quite likely entered the respective values for minimum heap size and maximum heap size twice. No wonder the server refused to start.
My suggestion was that you change the JVM settings in the Coldfusion Administrator. As I said earlier, the following fields (in the Administrator)
Minimum JVM Heap Size (in MB): |
Maximum JVM Heap Size (in MB):
are, respectively, equivalent to the flags -Xms and -Xmx. Hence there is no need to include the flags in the field "JVM Arguments".
I am actually trying to set the max heap to 6GB and min to 4GB
Can you update the response to reflect that?
Why such high values? Increasing memory is not always the answer. It can in fact be dangerous. It is analogous to stopping to worry about the gas or electricity leaks in your apartment, once you've won the lottery, and just writing out the energy cheques.
The 8 CFM files in VFS weigh in at just 4MB, so you don't have any reason to use such high memory values. At least, not yet.
Coldfusion is at 1.7 GB, so my suggestion is to start with comparable values for the JVM memory. Hence, JVM settings like
Minimum JVM Heap Size (in MB):1024 |
Maximum JVM Heap Size (in MB):1200
Then -XX:MaxPermSize=256m for the flag in "JVM Arguments".
The VFS settings remain at:
Global memory limit for VFS: 1000mb
Memory Limit per Application for VFS: 400mb
There is then one main issue to look into:
I have a utility i can run to view all the files in the VFS. and there is currently a total of 8 cfm files there that make up 4064KB of space within the VFS....The Global Memory Limit 1000 MB for In-Memory Virtual FileSystem exceeded.
As 4MB is far less than 1000MB, it means that some other process - other than the CFM pages - is taking up more than 1 GB of memory. Something to do with the database job perhaps? That is what you have to look into.
Copy link to clipboard
Copied
Bk
thanks for the correction
Ok so here is my JVM FROM CFADMIN
-server -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib
Min Heap: 1024, Max Heap 2048
Ill be running some test to see if this corrected the error from VFS.
Regardless of what is consuming the CF Heap, why is CF saying the VFS limit is exceeded?
Seems like a bug to me, this did not happen in CF9. its the same code base.
Will report any changes soon
Thanks again
Tim
Copy link to clipboard
Copied
So today i found a ton of errors saying the Global VFS limit had been reached.
The errors came from all of the site, and a few from pages that actually touch the VFS.
Again there are very files maybe 20 for a grand total of 699.74 KB.
This is getting ridiculous. Is there any way i can track down what is causing this?
My application and customers are depending on this.
At this point my options are to revert back to CF9 or switch from VFS to physical disks.
Anybody know how i can track this issue down?
Copy link to clipboard
Copied
Leave your JVM settings at modest values at a moment, and assume Coldfusion is reporting correctly. There might be an infinite loop somewhere in the sequence of calls, perhaps in the Application file.
If you wish, I could have a look for you. Send me a private mail, attaching the 20 files you mention and your Application file.
Copy link to clipboard
Copied
I am sorry to say to anyone else who my find this issue, i could not resolve it.
Thank you to BK for all your great suggestions and help.
We just switched the entire system over to hard drive. This will end up hurting us in the end but at this point we dont have an option, our system has been down since the day we upgraded it to CF11. We did a lot of load testing prior to the migration and from our testing it worked. But there were a lot of issues with IIS connectors and we also had new memory installed. There has been a lot going on and it is hard to track down the exact cause.
None the less, the error should still be reported as a bug. It misrepresents what is actually the issue and it will lead someone else in the future down a ghost rabbit hole that does not exist.
Thanks
Tim
Copy link to clipboard
Copied
Hi Tim,
Can you share your current settings in jvm.config?
Regards,
Anit Kumar
Copy link to clipboard
Copied
min heap: 6144, max heap: 6144
jvm config:
-server -Xss1024k -XX:MaxPermSize=512m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -javaagent:C:/FusionReactor/instance/cfusion.CF11.1.1/fusionreactor.jar=name=cfusion.CF11.1.1,address=8088
Windows 2012 R2 Standard
16GB ram
VFS global limit is now 1024MB
Per app limit 1024MB
Tim
Copy link to clipboard
Copied
Anit Kumar Panda wrote:
Hi Tim,
Can you share your current settings in jvm.config?
This might also help:
<cfdump var="#getVFSMetaData('ram')#">
Copy link to clipboard
Copied
First of all, BKBK I appreciate your responses and involvement.
Apart from all the suggestions so far, my experience says that, only tuning Xmx or Xms is not sufficient. Tuning the permgen memory is equally important. You may run, some third party tools to analyse the memory breakthrough.
You can download Java (JDK) : http://java.com/en/download/index.jsp. Install JDK
If the non-heap memory is increasing then, the perm gen value needs to be recalculated. Whereas, if the heap memory is increasing then the Xmx needs to changed.
Regards,
Anit Kumar
Edit: corrected the version typo
Copy link to clipboard
Copied
Hi Anit Kumar,
Your contribution, too, is much appreciated. It is a stubborn issue, and the new direction you point us to is certainly worth exploring.
P.S.: By the way, Tim is on Coldfusion 11.
Copy link to clipboard
Copied
Just corrected it. Let us see, if this helps us in some direction.
Regards,
Anit Kumar
Copy link to clipboard
Copied
Hi Anit,
I have not done as you asked, since we were in a pinch in production and had to something now to stop the bleeding.
we just switched to a physical disk path to write out our files instead of the ram:/// drive.
Now we are attempting to setup a new server on another OS, windows server 2008 R2 standard, 16GB ram and 64bit OS as well.
We are still testing it but we are planning on using the ram drive. its CF11 update 4.
During all the testing we have done, i have not seen this issue yet except for a few very strange times.
It appears we are being scanned by a spider, www.baidu.com. for what ever reason they are hitting a page that produces a 404 on our CMS.
our CMS 404 page emails us the CGI, whats so strange is the CGI.SERVER_NAME and CGI.HTTP_HOST variable values are "www.baidu.com" AND NOT our server IP/Host name.
Since our CMS is using this value to determine what site to render it causes an error and 404. The error is you guessed it
"The Global Memory Limit 2048 MB for In-Memory Virtual FileSystem exceeded . " |
I have done some searching and cant figure out if CF will allow this CGI var to be overwritten or not. regardless it seems like this is another bug and potential security vulnerability.
Not only that but i have disabled ALL of the Cache settings in CFAdmin and CF is still caching the templates and refuses to update them unless i log into CFAdmin and click the clear template cache buttons.
And the CFSchedule is all jacked up to, there are settings you can disable if your on standard edition and the scheduler runs each task 2 sometime 3 times.
After all this we are to the point that we want our money back for the CF11 upgrade and want to go back to CF9, or at least until all these MAJOR bugs are fixed with CF11
Completely dissatisfied with Adobe and CF11!
Copy link to clipboard
Copied
baidu.com needs to be firewalled out. I have had to insert hard code at the top of application.cfm to block this spider.
Copy link to clipboard
Copied
chewymole wrote:
We just switched the entire system over to hard drive.
By this you mean you went back to Coldfusion 9, where the application works, without the error message? Just to confirm.
Copy link to clipboard
Copied
We see the same thing sometimes on CF11 update 3 on Linux. Stack traces point at either CreateObject() or "new Query()" but all the files involved are on normal disks, no RAM drive involved.
I think CF is leaking somewhere and filling up some internal invisible RAM disk. Like you, getVFSMetaData('ram') indicates lots of space, and our per-application limits is 20meg, and doesn't appear to have been involved.
Copy link to clipboard
Copied
We are seeing the same thing! The VFS seems to fill up so quickly, but without actually being used! And setting it to a lower setting seems to work better.
You also mentioned compiling and including CFML. Did you find this way slower on CF11 vs CF9? We're seeing a huge performance hit and slow down in compile times. Were you ever able to move to CF11? Kinda wish we were back on CF9 now...
Any pointers appreciated! Thanks in advance.
Brook