CF11 VFS error: The Global Memory Limit 1000 MB for In-Memory Virtual FileSystem exceeded .

[chewymole]

Greetings,

I have a fresh install of CF11 standard on Windows 8 server 64x

16GB of ram

JVM settings:

-server -Xmn1024m -Xss1024k -XX:PermSize=256m -XX:MaxPermSize=2048m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random

Min Heap: 2048mb, Max Heap 6144mb

VFS settings:

Global memory limit for VFS: 1000mb

Memory Limit per Application for VFS: 400mb

Currently CF is using 1.7GB of ram

I have a 6 applications on this server. 1 of which makes use of the VFS.

The others are simple cfm applications that have NOTHING to do with the VFS system.

I have a scheduled task that calls simple cfm pages to update tables in our database. When these tasks run, i get a global error email (ones i have setup if a cf error happens) and the gist of the error is

Browser	CFSCHEDULE
DateTime	{ts '2014-08-16 15:07:00'}
Diagnostics	The Global Memory Limit 1000 MB for In-Memory Virtual FileSystem exceeded . <br>Increase the VFS memory limit in ColdFusion Administrator. <br>The error occurred on line 17.
GeneratedContent	[empty string]
HTTPReferer	[empty string]
Mailto	[empty string]
Message	The Global Memory Limit 1000 MB for In-Memory Virtual FileSystem exceeded .
QueryString	[empty string]
RemoteAddress	xxx.xxx.xxx.xxx

I have a utility i can run to view all the files in the VFS. and there is currently a total of 8 cfm files there that make up 4064KB of space within the VFS.

The 1 application that actually uses the VFS has several functions. One of which basically writes out CFM files and then includes the cfm as a content variable and returns the variable.

Under CF9 we were also deleting the ram drive CFM files after including them. But under CF11 this fails with an error saying the modified date could not be found. I have since removed the delete file part and the system works but we get this out of memory error with the VFS.

Can someone shed some light on this for me?

It makes no since.

Thanks

[BKBK]

-Xmn1024m -Xss1024k -XX:PermSize=256m -XX:MaxPermSize=2048m

Min Heap: 2048mb, Max Heap 6144mb

Why muck about? What about 3 simple settings such as

-Xmx2048m -Xms1024k -XX:MaxPermSize=256m

[chewymole]

BKBK,

so i should update the server JVM settings to what you are suggesting?

[BKBK]

Yes, the JVM settings in the Coldfusion Administrator. Then restart Coldfusion. Test and see what happens.

[chewymole]

Service wont start with those settings.

I reverted it back to what i had before.

Thanks anyway

[BKBK]

chewymole wrote:

Service wont start with those settings.

Beg your pardon. There was a typo: it should have read "-Xmx2048m -Xms1024m -XX:MaxPermSize=256m"

Also, did you by any chance duplicate the flags -Xmx and -Xms in the field "JVM Arguments"? That would have been an error.

I later realized it would have been less confusing to represent -Xmx and -Xms in exactly the way this is done in the Administrator, namely:

Minimum JVM Heap Size (in MB):1024

Maximum JVM Heap Size (in MB):2048

Then you only had to modify the flag -XX:MaxPermSize in the field "JVM Arguments".

[chewymole]

I am actually trying to set the max heap to 6GB and min to 4GB

Can you update the response to reflect that?

I am not sure now because i just copied and pasted into the jvm config.

When it did not start, I just reverted it back to previous version and saved.

Thanks again

Tim

[BKBK]

chewymole wrote:

I am not sure now because i just copied and pasted into the jvm config.

That's what I suspected. In so doing, you quite likely entered the respective values for minimum heap size and maximum heap size twice. No wonder the server refused to start.

My suggestion was that you change the JVM settings in the Coldfusion Administrator. As I said earlier, the following fields (in the Administrator)

Minimum JVM Heap Size (in MB):

Maximum JVM Heap Size (in MB):

are, respectively, equivalent to the flags -Xms and -Xmx. Hence there is no need to include the flags in the field "JVM Arguments".

I am actually trying to set the max heap to 6GB and min to 4GB

Can you update the response to reflect that?

Why such high values? Increasing memory is not always the answer. It can in fact be dangerous. It is analogous to stopping to worry about the gas or electricity leaks in your apartment, once you've won the lottery, and just writing out the energy cheques.

The 8 CFM files in VFS weigh in at just 4MB, so you don't have any reason to use such high memory values. At least, not yet.

Coldfusion is at 1.7 GB, so my suggestion is to start with comparable values for the JVM memory. Hence, JVM settings like

Minimum JVM Heap Size (in MB):1024

Maximum JVM Heap Size (in MB):1200

Then -XX:MaxPermSize=256m for the flag in "JVM Arguments".

The VFS settings remain at:

Global memory limit for VFS: 1000mb

Memory Limit per Application for VFS: 400mb

There is then one main issue to look into:

I have a utility i can run to view all the files in the VFS. and there is currently a total of 8 cfm files there that make up 4064KB of space within the VFS....The Global Memory Limit 1000 MB for In-Memory Virtual FileSystem exceeded.

As 4MB is far less than 1000MB, it means that some other process - other than the CFM pages - is taking up more than 1 GB of memory. Something to do with the database job perhaps? That is what you have to look into.

[chewymole]

Bk

thanks for the correction

Ok so here is my JVM FROM CFADMIN

-server -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib

Min Heap: 1024, Max Heap 2048

Ill be running some test to see if this corrected the error from VFS.

Regardless of what is consuming the CF Heap, why is CF saying the VFS limit is exceeded?

Seems like a bug to me, this did not happen in CF9. its the same code base.

Will report any changes soon

Thanks again

Tim

[chewymole]

So today i found a ton of errors saying the Global VFS limit had been reached.

The errors came from all of the site, and a few from pages that actually touch the VFS.

Again there are very files maybe 20 for a grand total of 699.74 KB.

This is getting ridiculous. Is there any way i can track down what is causing this?

My application and customers are depending on this.

At this point my options are to revert back to CF9 or switch from VFS to physical disks.

Anybody know how i can track this issue down?

[BKBK]

Leave your JVM settings at modest values at a moment, and assume Coldfusion is reporting correctly. There might be an infinite loop somewhere in the sequence of calls, perhaps in the Application file.

If you wish, I could have a look for you. Send me a private mail, attaching the 20 files you mention and your Application file.

[chewymole · Accepted solution]

I am sorry to say to anyone else who my find this issue, i could not resolve it.

Thank you to BK for all your great suggestions and help.

We just switched the entire system over to hard drive. This will end up hurting us in the end but at this point we dont have an option, our system has been down since the day we upgraded it to CF11. We did a lot of load testing prior to the migration and from our testing it worked. But there were a lot of issues with IIS connectors and we also had new memory installed. There has been a lot going on and it is hard to track down the exact cause.

None the less, the error should still be reported as a bug. It misrepresents what is actually the issue and it will lead someone else in the future down a ghost rabbit hole that does not exist.

Thanks

Tim

[Anit_Kumar]

Hi Tim,

Can you share your current settings in jvm.config?

Regards,

Anit Kumar

[chewymole]

min heap: 6144, max heap: 6144

jvm config:

-server -Xss1024k -XX:MaxPermSize=512m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -javaagent:C:/FusionReactor/instance/cfusion.CF11.1.1/fusionreactor.jar=name=cfusion.CF11.1.1,address=8088

Windows 2012 R2 Standard

16GB ram

VFS global limit is now 1024MB

Per app limit 1024MB

Tim

[BKBK]

Anit Kumar Panda wrote:

Hi Tim,

Can you share your current settings in jvm.config?

This might also help:

<cfdump var="#getVFSMetaData('ram')#">

[Anit_Kumar]

First of all, BKBK I appreciate your responses and involvement.

Apart from all the suggestions so far, my experience says that, only tuning Xmx or Xms is not sufficient. Tuning the permgen memory is equally important. You may run, some third party tools to analyse the memory breakthrough.

You can download Java (JDK)  : http://java.com/en/download/index.jsp. Install JDK

1. Stop ColdFusion service
2. Launch Command prompt as Administrator
3. Browse to cf_root\cfusion\bin and run the following command coldfusion.exe -start -console.
4. Try to access the CF admin, once the services are started.
5. Once ColdFusion 11 is started from Command prompt, Launch jvisualvm.exe present in C:\ProgramFiles\Java\jdk1.7.0_25\bin

If the non-heap memory is increasing then, the perm gen value needs to be recalculated. Whereas, if the heap memory is increasing then the Xmx needs to changed.

Regards,

Anit Kumar

Edit: corrected the version typo

[BKBK]

Hi Anit Kumar,

Your contribution, too, is much appreciated. It is a stubborn issue, and the new direction you point us to is certainly worth exploring.

P.S.: By the way, Tim is on Coldfusion 11.

[Anit_Kumar]

Just corrected it. Let us see, if this helps us in some direction.

Regards,

Anit Kumar

[chewymole]

Hi Anit,

I have not done as you asked, since we were in a pinch in production and had to something now to stop the bleeding.

we just switched to a physical disk path to write out our files instead of the ram:/// drive.

Now we are attempting to setup a new server on another OS, windows server 2008 R2 standard, 16GB ram and 64bit OS as well.

We are still testing it but we are planning on using the ram drive. its CF11 update 4.

During all the testing we have done, i have not seen this issue yet except for a few very strange times.

It appears we are being scanned by a spider, www.baidu.com. for what ever reason they are hitting a page that produces a 404 on our CMS.

our CMS 404 page emails us the CGI, whats so strange is the CGI.SERVER_NAME and CGI.HTTP_HOST variable values are "www.baidu.com" AND NOT our server IP/Host name.

Since our CMS is using this value to determine what site to render it causes an error and 404. The error is you guessed it 

"The Global Memory Limit 2048 MB for In-Memory Virtual FileSystem exceeded . "

I have done some searching and cant figure out if CF will allow this CGI var to be overwritten or not. regardless it seems like this is another bug and potential security vulnerability.

Not only that but i have disabled ALL of the Cache settings in CFAdmin and CF is still caching the templates and refuses to update them unless i log into CFAdmin and click the clear template cache buttons.

And the CFSchedule is all jacked up to, there are settings you can disable if your on standard edition and the scheduler runs each task 2 sometime 3 times.

After all this we are to the point that we want our money back for the CF11 upgrade and want to go back to CF9, or at least until all these MAJOR bugs are fixed with CF11

Completely dissatisfied with Adobe and CF11!

[mtnhomes]

baidu.com needs to be firewalled out. I have had to insert hard code at the top of application.cfm to block this spider.

[BKBK]

chewymole wrote:

We just switched the entire system over to hard drive.

By this you mean you went back to Coldfusion 9, where the application works, without the error message? Just to confirm.

[ChivertonT]

We see the same thing sometimes on CF11 update 3 on Linux. Stack traces point at either CreateObject() or "new Query()" but all the files involved are on normal disks, no RAM drive involved.

I think CF is leaking somewhere and filling up some internal invisible RAM disk. Like you, getVFSMetaData('ram') indicates lots of space, and our per-application limits is 20meg, and doesn't appear to have been involved.

[brookd]

We are seeing the same thing! The VFS seems to fill up so quickly, but without actually being used! And setting it to a lower setting seems to work better.

You also mentioned compiling and including CFML. Did you find this way slower on CF11 vs CF9? We're seeing a huge performance hit and slow down in compile times. Were you ever able to move to CF11? Kinda wish we were back on CF9 now...

Any pointers appreciated! Thanks in advance.

Brook