I have a user who reports my plugin consumes memory until Lr/System is no longer operable, when doing large batches of photos.
I have this type of problem too from time to time, but not always, and in the most recent case, *not* for the same operation my client is complaining about.
Begs the question: is there a way to control whether excessive memory is used, or force it to be released, when doing an operation upon large batches of photos.
Note: the operation is already concluding catalog transactions every 1000 photos (exits with-write function, and re-enters). My client reports Lr/System slowdown at about 4000 photos. He is running Lr3, Windows OS - system details not yet known.
the operation is already concluding catalog transactions every 1000 photos
I wonder if reducing that to 100 might have a significant impact. I recall a previous thread here that discussed long-running operations and the memory impact on the LR internal undo stack (not SQLite's undo logs). Perhaps in some circumstances LR is expanding some memory buffer for the undo stack and not releasing it for some reason.
Have you already tried John R. Ellis idea of reducing the transaction size? I remember from another project we had to limit the transaction size on a SQLite based database due to memory problems.
Maybe you are facing a different problem - not sure how efficient LUAs garbage collection is and your code causes some kind of memory leaks somewhere.
Only one user was reporting a problem, and I think I had just about maxed out his "good will" in testing my plugin. I can't reproduce the problem he was having on my machine. So, I left it at 1000. He got past the problem by dividing operation into chunks. Next week he'll have new machine with upgraded Lr, so... - sorry for verbose answer.......
It could be dozens of other users have same problem, but don't report it. If they report it, I'll fix it! - otherwise...