5 Replies Latest reply: Jul 10, 2009 8:41 AM by Texas_Ranger RSS

    CF8 DirectoryWatcher Gateway Doesn't Run When I Overwrite a File

    molaro Community Member
    molaro May 15, 2009 6:54 AM

      I've set up a Coldfusion Directory Event Gateway on my site (Win2K3 server, CF8 Ent).  The DW monitors an FTP directory where my users upload a batch of 5 PDF files each week.  The files are named file1.pdf, file2.pdf and so on.  When my DW gateway sees an addition or a file change, it copies the file to a new location, reads the PDF's meta data and uses that to rename the file and then add an entry to a database table for that file (the table entry gets used to populate a PDF archive list on a different page). In general this all works with a few exceptions.

       

      ISSUE 1: If I add a new file, the gateway sees it and calls my onAdd function (which actually just calls my onChange function). If I rename a file in that directory, the onChange event fires and my code runs.  However, if I upload a new file1.pdf file which overwrites the old file1.pdf, nothing happens. The new file is truly different and has a new modification date.  I read through the Java class and mod date is how the gateway determines a change.  Now, so far am not actually FTPing a new file in, but rather just coping it in from my local machine to the server directory via a mapping.

       

      I saw a solution for a similar issue which talked about setting up the onDelete event b/c the FTP will actually delete then add the file. Does this sound like a solution?

       

      ISSUE 2: I had a lot of CFM files that needed to be put into the archive, so I converted those to PDFs.  I then put all of them in my watch directory so my script would process them.  To my surprise, the script would only load 11 files at a time.  So after some tests to confirm the issue and the correct number of files, I then loaded my batch, 11 files at a time.  Any ideas why this would happen?

       

      My Specs again:

      • Windows 2003 Server
      • Coldfusion 8 Enterprise
      • In my config file:
        • checking my directory every minute (60000 milliseconds)
        • only checking for .pdf extension
        • Set an onAdd and onChange function call, but no onDelete
        • my onAdd function simply calls my onChange function, passing
        • in the CFEvent argument that it received
      • 2653 Views
      • Tags:
        • 1. Re: CF8 DirectoryWatcher Gateway Doesn't Run When I Overwrite a File
          molaro Community Member
          molaro May 15, 2009 7:40 AM (in response to molaro)

          Ok, I now know what's happening in ISSUE 2, where only 11 files uploaded at a time.  In the CFAdmin I can set the nu,ber of threads to dedicate to the Event Gateway, and then the max number of requests to queue.  The defaults ar 1 thread and 10 queue requests (my 11 files).  So I will set it higher.  Is there a danger limite with the threads or the request queue I should avoid?  If I set up 5 threads and 200 requests does that sound safe in general (I'm sure it depends on my server specs).

          • 2. Re: CF8 DirectoryWatcher Gateway Doesn't Run When I Overwrite a File
            BKBK Community Member
            BKBK May 24, 2009 10:25 AM (in response to molaro)

            I would in any case code all 3 methods, onAdd, onChange and onDelete, and include the cflog tag in each to identify the action.

            • 3. Re: CF8 DirectoryWatcher Gateway Doesn't Run When I Overwrite a File
              BKBK Community Member
              BKBK May 24, 2009 10:47 AM (in response to molaro)
              If I set up 5 threads and 200 requests does that sound safe in general (I'm sure it depends on my server specs).

              There is one way to find out: just do it, and see. The default values in the Administrator always good order-of-magnitude estimates. Incidentally, my default is 10 gateway processing threads.

              • 4. Re: CF8 DirectoryWatcher Gateway Doesn't Run When I Overwrite a File
                molaro Community Member
                molaro May 25, 2009 8:10 AM (in response to molaro)

                I still haven;t determined why the Directory Watcher Gateway doesn't seem to run when I simply overwrite a file.  This is disappointing because as of now I have taken my function code and updated it to be a scheduled task.  This isn't as elegant because I have to add code to decide if the files in the directory are new or not (which is somewhat fuzzy) and run based on that determination.  Does anybody have any ideas, tricks, workarounds as to why the DW gateway event doesn't fire when a file in the directory is overwritten??

                • 5. Re: CF8 DirectoryWatcher Gateway Doesn't Run When I Overwrite a File
                  Texas_Ranger Community Member
                  Texas_Ranger Jul 10, 2009 8:41 AM (in response to molaro)

                  We had the same problem here, and I agree that it is disappointing that the DW doesn't consider that a new file.  As far as a workaround, what I did was change the way our process dealt with incoming files entirely.  I used to handle the files as they came in, relying on Dave Ferguson's tricks for getting around files that were still being written to or otherwise locked (thanks, Dave!).  Now, instead of that, I only do one thing when a file is received: I rename it with a datestamp and write this to our db as the received filename so that the additional processes can be run on it.  That way, when that same filename is used by our client again, which is every time, the DW picks it up again as a new file.  I don't know if that helps or not, because it still means you lose the functionality that used to be inside the DW instead of a scheduled task, but at least you know when an actual new file is received, regardless of its name.  You may even be able to put your functions inside the onChange or onDelete if you keep the reference to the original filename, then have those do your work after the file is renamed and/or moved.  In our case, the DW also kept hanging every couple of days so we were missing very important files.  I ended up just restarting the gateway through code as the last piece of the scheduled task that runs every 30 mins to process the files.