I appologise for the long post in advance...
Ok... so I have this script that, using cfdirectory, will
check a directory for any files that may have been uploaded, if
there are files, it loops through the results and reads the files
one at a time, line by line, using the FileReader.cfc (Uses the
Java FileInputStream, InputStreamReader, and BufferedReader to
provide a way to incrementally read large files). The files are
just pipe "|" delimited data, each line represents a record for a
db table.
Now as it's reading each line, it will perform some basic
string parsing to clean up the file line to make sure the data is
valid, blah blah blah and then it will write that "cleaned" line to
another file using FileWriter.cfc (Java component once again). Once
it's completely done reading the original file, it will close it
and it will open the new "cleaned" version of the file, read it
(FileReader.cfc), create an INSERT statement and then update the
database table.
Now... this all works GREAT... until it has to loop through
more than a few files... 3 - 4 files are NO problem! works like a
charm, but throw 6 - 8 files at it and it dies, not a timeout mind
you but an actual "java.lang.OutOfMemoryError" (now, I've tried
making all the files exactly the same (just changed the name) and
the weird thing is, it takes longer and longer to process each as
it goes through the loop... I have the script write some stats as
it's looping:
FILE 1 STATS
------------
Name: COA0607_Intranet1.DAT
Status: Import Successfull
Line Count: 32,971
Processing Time: 74,237ms
FILE 2 STATS
------------
Name: COA0607_Intranet2.DAT
Status: Import Successfull
Line Count: 32,971
Processing Time: 82,018ms
FILE 3 STATS
------------
Name: COA0607_Intranet3.DAT
Status: Import Successfull
Line Count: 32,971
Processing Time: 94,476ms
FILE 4 STATS
------------
Name: COA0607_Intranet4.DAT
Status: Import Successfull
Line Count: 32,971
Processing Time: 145,108ms
I know what you guy are probably thinking; "Woah man... CF
isn't really meant to do that kind of processing...", I know, trust
me I know... however, I really neeeeeed it too lol.
Ok, so as the script is running, I watch the Virtual Memory
use of jrun.exe, processing say 3 - 4 of these files brings up the
usage to approx 300,000k which yes, is a LOT but that's fine...
this process is meant to run at night via a Scheduled Task...
When I run more than 4 files, things start to get ugly, keep
in mind that these are EXACTLY the same files just re-named
differently. The script will start lagging BIG time and on the last
file (usually the last file) I'll see the memory usage spike from
350,000K all the way up to 600,000K and that's when it throws the
"java.lang.OutOfMemoryError" and dies... I've tried commenting out
the part of the script that updates the db, but still get the same
problem...
So... what gives? How come CF Server does this??? I mean, it
runs fine for the first few files... and then WAM, it dies... sorry
for the long post... any insight here is VERY much appreciated...
it would be AWESOME if the wonderful folks at Adobe could shed some
light on this for me : )
CFMX 6.1 version: 6,1,0,83762
Windows XP Pro SP2
Intel P4 2.8Ghz
1Gb of Ram
3
Replies
AdChoices