I think what you might want to try is content aware fill where your filling the newly added transparent border. If that is possible. Don't know how well it would work. For it sound like you don't want to scale the frame. That your actually trying to protect them with an alpha selection. I also think that content aware scaling each frame would produce a very jittery video.
However I think the best way would be to just resize video the fill the aspect ratio area size you need, Crop to the required aspect ratio then resize. You will loose a borders on the top and bottom or the sides when you change aspect ratios. Something like you see on TV this film has been modified to fit your screen. Or you could distort the video resize the video to the new aspect ratio size without constrain.
The entire point of using content aware filling/scaling is not having to resize and cope and loose details/resolution around the subject. Also I think with the right consistent alpha channel content aware scalling will be much more consistent than content aware fill because you're kind of telling PS what content to stretch as opposed to letting it guess how it should generate it. In either case I'll see if the fill actually works...didn't think to try it
The entire point of using content aware filling/scaling is not having to resize and cope and loose details/resolution around the subject.
Filling and Scaling are separate thing. Filling an area does not resize. Scaling a document to some new pixel size is a resize. When the resize is done with the content aware option is all about changing aspect ratio and controlling distortion somewhat. Objects recognized by Photoshop and those masks by the users will not be distorted they will be resized in a constrained way and repositioned for the new aspect ratio and size. The remaining areas will be distorted to fit and fill the new aspect ratio and size.
What you seem to be trying to do is mask the whole original Image/frames which is more or less asking Photoshop to resize everything is a constrained way, That is like resizing an image so it just fits onto a known paper size so it just fill the width or height which ever comes first. That leave white borders on the top and bottom or on the sides. That is why I suggested suggested trying to use content aware fill. What you seem to want to do is have Photoshop fill the two empty borders in a content aware mode.
No that's not what I was trying to do, I mask the central subject of my frame and as I said it was surrounded by a fairly uniform background(main subject is a rock formation in the middle of the ocean) I want to change the aspect ratio of that frame, so I created an alpha mask to cover the main subject, the surrounding background was not covered in this alpha channel mask. This way when I scale using the content aware feature all that happens is the background(the water) is streteched evenly with lesser distorion(than normal scaling) while the masked main subject is "avoided" with completely no distrotion. I think the key point you missed in my description is that I am in fact creating an alpha matte for only my subject not the entire frame and using that in the "protect" option of content aware scaling.
The result is essentially a frame aspect ratio change that does not sacrifice the resolution around the main subject(because the main subject remains of the same size and PS doesn't have to interpolate any pixels...this is obviously is not the same as just resizng and cropping where the subject and background are streched evenly to fill an essentially larger area of the screen causing "softness"). The result using this method is also much more predictable from frame to frame than content aware filling because it's taking information from the current frame and using is to fill the empty background resulting from the aspect ratio change. I know this works because I tried it on individual still frames from the video then compiled them back in on AfterEffects...thats not my problem. My problem is when you try to natively try to do this inside PS on a video layer(rather than a still frame) it doesn't work and I'm not sure if this is a bug or if video content aware streching is just not supported.I have CS6 Production Premium.
I beleive content aware scale is more about not distorting know objects during resize. If your changing the oveall canvas pixel sizes as well as the aspect ratio everyting need to be interpolated even the protected area you mask. Also if your subject is moving in the video how do you mask each frame. How do you mask a moving object?.
If your just changing the canvas width or height to achieve the new aspect ratio. The interpolation of the protected areas will not be much.
The method I describe works 100% on still frames extracted from the video, changing the canvas/aspect ratio is not the issue here because I do the same thing on a still frame and it works...just does not work on a video type layer. I appreciate you taking the time to respond but you're anwering the wrong question
As for masking a moving object, you create a track matte using standard rotoscoping techniques, it's very simple....in fact Mocha for AE or Pro generate by default BW mattes for tracked objects without any transfer mode/track matte tricks that you would have to do using masks in After Effects to get a BW or alpha matte.
That explains it I do not have the AE add on for Photoshop. AE cost to much for the amount of use I would have for it. I also do not do video. I think if you want to do video you would be better off using Adobe product Premiere Pro. I would think what your trying to do in Photoshop would be more a standard feature of Premiere