the operation is already concluding catalog transactions every 1000 photos
I wonder if reducing that to 100 might have a significant impact. I recall a previous thread here that discussed long-running operations and the memory impact on the LR internal undo stack (not SQLite's undo logs). Perhaps in some circumstances LR is expanding some memory buffer for the undo stack and not releasing it for some reason.
Have you already tried John R. Ellis idea of reducing the transaction size? I remember from another project we had to limit the transaction size on a SQLite based database due to memory problems.
Maybe you are facing a different problem - not sure how efficient LUAs garbage collection is and your code causes some kind of memory leaks somewhere.
Only one user was reporting a problem, and I think I had just about maxed out his "good will" in testing my plugin. I can't reproduce the problem he was having on my machine. So, I left it at 1000. He got past the problem by dividing operation into chunks. Next week he'll have new machine with upgraded Lr, so... - sorry for verbose answer.......
It could be dozens of other users have same problem, but don't report it. If they report it, I'll fix it! - otherwise...