Freeing up Memory During CFFILE

Report · Jan 12, 2007

I have a process that generates flat files using CFFILE. The process uses a database reference table that feeds the query parameters. I have several loops in this process which I thought would be more efficient and free up memory after each CFFILE "action append" but before execution of the next query. (execute query then write all the files, loop to query then write all files, etc....)

Problem is that it is not freeing up memory and I am getting java.lang.OutOfMemoryError errors after about 15 minutes of run time.

Current solution is to run this process in batches. This unfortuantly requires someone baby sit the process and change a parameter and submit.

Is there anyway after the one of the loops is complete (query has completed and first group of files have been written) and before the next execution of the query that I can free memory?

I need to trick the process in thinking that the page is complete then immediately initiate the next query to process the files.

We have tuned the box as best we can, applied all patches (MX7) . Process runs great in batches, but, just can't handle the volume. Millions of rows are being processed.

Thanks
Bill

Report · Jan 12, 2007

Don't know if this will work, but it's worth a shot.
1) Finish a "batch" in your longRunningPage and do a CFLOCATION to foo.cfm.
2) In the <HEAD> section of foo.cfm, insert the code below. The notation ../../.. is the path from your webroot to the page. a, b and c are parareters sent to foo.cfm by the CFLOCATION. When you sent them back to foo.cfm, use them to let longRunningPage know where to start on the second batch and other control information. Of course, you could use session variables for this purpose.

Report · Jan 12, 2007

Current solution is to run this process in batches. This unfortuantly
requires someone baby sit the process and change a parameter
and submit.

java.lang.OutOfMemoryError means that there are too many objects in memory at one time. However, it is difficult to remove objects in one continuous process while it is still running.

The word batches carries a hint. In the circumstances, I would break the process up into smaller jobs and use cfschedule for each. Each job could store the data that subsequent jobs need in a file or in a database table specifically created for that purpose. Use cached queries for static data. Coldfusion will then read the data from memory, rather than from a file or database, sparing you the creation of yet more objects.

Adobe Community

Freeing up Memory During CFFILE