3 Replies Latest reply on Jul 13, 2006 7:07 AM by Al69

    CFMX 6.1's Virtual Memory Use problem!!

    Al69
      I appologise for the long post in advance...

      Ok... so I have this script that, using cfdirectory, will check a directory for any files that may have been uploaded, if there are files, it loops through the results and reads the files one at a time, line by line, using the FileReader.cfc (Uses the Java FileInputStream, InputStreamReader, and BufferedReader to provide a way to incrementally read large files). The files are just pipe "|" delimited data, each line represents a record for a db table.

      Now as it's reading each line, it will perform some basic string parsing to clean up the file line to make sure the data is valid, blah blah blah and then it will write that "cleaned" line to another file using FileWriter.cfc (Java component once again). Once it's completely done reading the original file, it will close it and it will open the new "cleaned" version of the file, read it (FileReader.cfc), create an INSERT statement and then update the database table.

      Now... this all works GREAT... until it has to loop through more than a few files... 3 - 4 files are NO problem! works like a charm, but throw 6 - 8 files at it and it dies, not a timeout mind you but an actual "java.lang.OutOfMemoryError" (now, I've tried making all the files exactly the same (just changed the name) and the weird thing is, it takes longer and longer to process each as it goes through the loop... I have the script write some stats as it's looping:

      FILE 1 STATS
      ------------
      Name: COA0607_Intranet1.DAT
      Status: Import Successfull
      Line Count: 32,971
      Processing Time: 74,237ms

      FILE 2 STATS
      ------------
      Name: COA0607_Intranet2.DAT
      Status: Import Successfull
      Line Count: 32,971
      Processing Time: 82,018ms

      FILE 3 STATS
      ------------
      Name: COA0607_Intranet3.DAT
      Status: Import Successfull
      Line Count: 32,971
      Processing Time: 94,476ms

      FILE 4 STATS
      ------------
      Name: COA0607_Intranet4.DAT
      Status: Import Successfull
      Line Count: 32,971
      Processing Time: 145,108ms

      I know what you guy are probably thinking; "Woah man... CF isn't really meant to do that kind of processing...", I know, trust me I know... however, I really neeeeeed it too lol.

      Ok, so as the script is running, I watch the Virtual Memory use of jrun.exe, processing say 3 - 4 of these files brings up the usage to approx 300,000k which yes, is a LOT but that's fine... this process is meant to run at night via a Scheduled Task...

      When I run more than 4 files, things start to get ugly, keep in mind that these are EXACTLY the same files just re-named differently. The script will start lagging BIG time and on the last file (usually the last file) I'll see the memory usage spike from 350,000K all the way up to 600,000K and that's when it throws the "java.lang.OutOfMemoryError" and dies... I've tried commenting out the part of the script that updates the db, but still get the same problem...

      So... what gives? How come CF Server does this??? I mean, it runs fine for the first few files... and then WAM, it dies... sorry for the long post... any insight here is VERY much appreciated... it would be AWESOME if the wonderful folks at Adobe could shed some light on this for me : )

      CFMX 6.1 version: 6,1,0,83762
      Windows XP Pro SP2
      Intel P4 2.8Ghz
      1Gb of Ram