Directory watcher gateway stops seeing files added after working fine for awhile

[raelehman]

We have a directory watcher gateway that has worked quite well in the past, but over the past couple days has started having problems. What seems to happen is that the gateway cfc stops firing when a file is added to a directory. Stopping and restarting the gateway doesn't fix the problem. However we are running 4 instances on the server, so I removed the gateway from one instance and added it to another. Things started working again. Then the next day, the new gateway stopped working. Moved it back to the original instance, and it's working again.

We also have a scheduled task that cleans up files that the gateway misses, it runs every 3 minutes. The gateway does log some "file does not exist" errors but it seems to only happen when the scheduled task runs - that is, it seems like the gateway doesn't notice the files until another process tries to do something with them!

Incidentally we have been doing a pretty high volume of traffic lately and the server has had some other issues (another process has brought it down multiple times in the past week) - so the server has actually been restarted a few times lately (it's in a cluster of 2). I'm not sure how these things would really affect the gateway's operaion though.

Does anyone know what might be going on here? Thanks in advance!

[bw_bloodletter]

Same exact issue.

Running CF9 Hotfix 1 on Windows Server 2008 R2.

DirectoryWatcher Event Gateway runs fine, then suddenly gives up.

When a file is added to the target directory, the event counter in increments as expected, but the appropriate function (onAdd) in the .cfc is never called.

There is zero information in any of the logs to indicate the directory watcher gateway instance has done anything.

The only indication that the gateway is alive is the event counter in the cfide administration page.

Restarting the gateway does not fix it.

Restarting jrun does fix it.  But since this is a production server, we can't just be restarting willy-nilly.

Does anyone have any suggestions?

[bw_bloodletter]

Poking around in process monitor, I found that the following error was being thrown incessantly.

Date	<Timestamp>
Thread	<Thread ID>
Class	File System
Operation	CreateFile
Result	NAME NOT FOUND
Path	path\to\file.ext
Duration	<duration>

This error was being thrown incessantly even though all the gateway instances were stopped.

(I even stopped them using the global disable on the gateway settings page.)

So I went ahead and dumped a blank file with the name the error specified into the path the error specified.

Process monitor reported a success, and the errors stopped flooding process monitor.

I reenabled the gateway instances and tried a test file.  Same problem.  Counter gets incremented, nothing happens, nothing in logs, process monitor shows the same error over and over for the new file.

[bw_bloodletter]

Some progress...

Stop all the directory watchers instances

Add a blank file with the corresponding path and name that Prosess Monitor shows jrun is trying to create

Repeat until process monitor stops complaining about NAME NOT FOUND for files in your watched directories

Delete all remaining blank files you added (they may have been processed by the event gateway even while it was stopped, so kill any files your cfc might create in watched directories, too)

Restart the gateways

Test

Success!

Without having to remove gateway instances and without having to restart jrun.

Now it's a matter of figuring out why it got into that state in the first place, and how to prevent it...

Don't know if this would work if you use an onDelete function.  It seems as if the function calls get queued up, and can only be cleared by stopping the gateway (to prevent more function calls from queueing up), and making jrun happy (seeing the file that it expects to exist).