The way a video file is encoded the majority of frames are not full image frames. Full frames are every now and then and when scenes change cuts happen. Most video file are not a sequence of images. Frames are more like changes to the current image. It cuts down on files size and data rate required for video. Actions works layers however however Photoshop filters and adjustments do not work on video layers. Photoshop only has basic video support cuts splice transitions etc. To really do video editing you need to use Video editing software like Adobe After effects and Adobe Premier
I am trying to touch up pixels, not re-cut video. I have used After Effects for special effects before, but it can't currently spot-heal over time. Photoshop will do this task if I do it by hand, so why can't it do the action as batch?
*** Example: If I retouch a video frame and advanced it by hand 1000 times it works fine. So why can't photoshop advance the frames 1000 times for me.
If your video less then around 5 minutes you can create frame layers for you for your video layer frames Photoshop support up to 8,000 layers. You would loose the audio and your machine would most likely be taxed manipulating thousands of layers. Spot healing may require days.