Warp stabilizer & slow-motion footage

[David Coiffier]

Hi everyone,

I have a serious load of ultra slow-motion footage to stabilize, mostly follow pans. Given it's slomo stuff, warp stab happens to be way out of limits I would need.

I understand stab is highly related to movement over time, and slomo gives analyzer to think everything is smooth, while it is only because of the slomo rate.

We're talking here of 1 or 2k fps footage, meaning a 1/40 or 1/80 of a standard frame rate.

I think the main issue I have is limit of the smoothness parameter. It can't go over 1000%, where I could easily need 5 or 10k %.

This setting limit, when divided by 40 (for 1000fps) is only 25%, or 12.5% for 2000fps, which is of course way out of useful range.

I've also used the crop less/smooth more variable, but it has no visible effect in this case.

May I ask Adobe team any consideration for next update on this module ?

Meanwhile, if anyone has a brilliant idea to help here, please be very welcome!

Unrelated to previous issue, but still about this warp stab module, I think it would be a very good idea to make analysis & correction data accessible to user, for further editing. I'm thinking here of data we can get with regular tracker, such as anchor, pos, rot, scale… It probably makes no sense when using the warper mode, but would have in all the other modes. As a matter of fact, for now we don't have a cloud tracker that leads to anchor/pos/rot/scale regular editable data...

Thanks for reading

David

[Szalam]

Could you please describe what you are trying to do in more detail? If it's so smooth that the Warp Stabilizer isn't doing anything, I don't understand what you are trying to do.

Can you use the legacy stabilizer to do what you're asking for?

[Dave_LaRonde]

I concur with Szalam.  I have no idea what you're ultimately trying to accomplish.  At 2000 fps conformed to... oh, say 60 fps, EVERYTHING should be smooth!

[David Coiffier]

The idea here is to stabilize 1000fps footage, so that when speeded up to regular speed, it's still clean.

Actually, it's not, because shots were made with extreme long focal (560mm on a 2cm sensor, that is equivalent to 1000mm on a full frame camera). Follows are quite ok on the x axis, but there is serious wiggle on Y axis.

I agree that this wiggle is hardly noticeable at nominal speed, but I will need to do some speed effects in editing, switching from 1/40th to realtime speed, therefore my issue here, trying to stabilize 1/40th speed footage

[David Coiffier]

And shots are actually planes at take off & landing. I follow the airplanes, so shot are never static, making it hard to track with regular tracker techniques...

[Rick Gerard]

Interpret your 1000 fps footage as 1000 fps, drop it into a comp that is 29.97 or 23.976 or 25 (any standard frame rate), pre-compose, warp stabilize the pre-comp. Done....

It does not matter what the frame rate of the footage is, if it is interpreted correctly, when you drop it into a standard frame rate comp it will play back in real time. That is why 24fps footage plays back in real time in a 30 fps comp. Always has, always will. At least in the latest release of AE you are no longer limited to interpreting footage to 99 fps.

If the only thing in the shot is an airplane then it's going to be hard to get a good warp stabilize because there is so little fixed geometry in the shot. You might be better off living with a little cropping and motion stabilizing the footage so the plane doesn't move. If you have a bunch of fixed geometry, like the ground and horizon line you'll have a better chance of using Warp Stabilizer.

[David Coiffier]

Hi Rick, thanks for your help!

Actually warp stab does work well because I have enough foreground under plane to provide useful details for analysis.

My 1000fps footage is actually very regular 25fps that will play at 25fps, providing 1/40th regular speed material.

Realtime here would be achieved with a 4000% speed, or in AE with a 2.5% stretch.

The thing is I don't wanna stab only realtime version, but wanna be able to stabilize the slomo footage so that  I can later play with varispeed fx while editing. To be clear, I'm really in need to stab the 25fps slomo material, for moving to a further step with stabbed material. Your solution, unless I didn't fully get it, seems ok but only on a realtime version. I've also thought about this way, speeding up the playback, run analysis, and then remove the speed up, but I didn't find a way to keep all my frames without concatenating (warp stab won't run on concatenate precomp), so it looks like a dead end to me at this point...

[Rick Gerard]

If the original footage was 25 fps then it is not slow motion. If you are trying to slow it down so that it acts like it was originally shot at 1000 fps then you are asking the software to make up 39 in-between frames and somehow calculate where the pixels would move. That's a difficult task and you will need Twixtor and some motion vectors to get good results. AE is not going to do a very good job figuring out where the pixels are moving in those 39 new frames.

I'm now really confused:

My 1000fps footage is actually very regular 25fps that will play at 25fps, providing 1/40th regular speed material.

Realtime here would be achieved with a 4000% speed, or in AE with a 2.5% stretch.

This makes no sense to me at all. Are you telling me that you shot at 25 fps then you slowed it down? If so, then my first paragraph is accurate, you are going to have a very difficult creating all those extra frames and making them look good.

If you have slowed down the 25fps footage to simulate shooting at 1000fps and then want to speed up that footage again and you are using time stretch, you are using the wrong tools. Speeding up and slowing down the same shot in AE is best handled with Time Remapping. If you are going for anything over creating more than 4 or 5 new frames from the original footage, which would mean 1 second in real time would take 5 seconds to playback, then you should be looking at Twixtor. 

If the original 25fps footage is smooth then slowing it down should not make it less smooth. Again, I'm really confused. I don't think you have a very good understanding of how frame rates and time work.

[David Coiffier]

OK, let's try to be clear : This IS 1000fps material, but I use regular field recorder to record 25fps material. At 1000fps, only inboard camera RAM can record. So I replay my camera RAM at 25fps then record this 25fps playback. This is my actual original footage. It is technically 25fps footage file, but where action is replaying at 1/40th of realtime. If you prefer, 1s of realtime will have 40s duration on this 25fps footage file...

[David Coiffier]

And Rick, I'm really sorry you think I don't have a good understanding of how frame rates and time work, as this has been my main professional activity for at least 10 years...

[Rick Gerard]

Ok, so the footage was recorded at 1000 fps so when you play it back at 25 fps one second of real time takes 40 seconds. Now that makes sense.

If that footage has too much movement when you time remap or otherwise speed up the action so that one second of real time plays back in one second then I would suggest that you try pre-composing the speeded up material and Warp Stabilize the pre-comp. If there is too much movement between frames try speeding it up so 40 seconds takes 10 seconds and Warp Stabilize that pre-comp.

Without actually having a sample of your footage to work with it is going to be nearly impossible to give you a formula that will work well for a specific shot because the exact method for optimal results depends entirely on the shot. Post a typical shot that is about 20 seconds long and I will take a look. I am beginning to think that Warp Stabilizer - Which makes up new frames from the original ones - may not be your best option.

[David Coiffier]

Rick, thanks again for your time and expertise here,

I'm actually encoding a typical shot so you can see what the issue is. It will  be available in some minutes.

Seems to me that your solution can't work because if I speed  footage up, I will need to "unspeed" it later, right ? So if I wanna keep all my original frames heres, I will have to concatenate, otherwise I'll loose some frames. Or is there something I didn't get here ?

What I'm looking for is of course preserving all my frames AND stabbing them hard enough so that when I later speed 'em up by 40, result won't show any vertical wiggle, that could have earlier been "masked" by the very slow rate.

Again, warp stab does an excellent job, but won't judge bad moves on a sufficient time range, so it won't allow me to smooth enough...

In 5 minutes now, you should get here one shot, that is 400MB : Dropbox - CRUCIAL_S001_S001_T009 h264.mov

As previously stated, it will look good at nominal speed, but when played back 40x faster, simulating realtime, it will show very unpleasant vertical wiggle...

[Rick Gerard]

Thanks for the footage. Downloading now. I'll take a look right after church.

From looking at the first frame I don't think I would warp stabilize, I think I would motion stabilize then put the camera move back in later to correct the framing. I'll let you know. If you wanted to show the approach in real time then slow down the touchdown I would probably use Time Remapping. It would probably give better results.

Edit: Here is a project file that took about 10 minutes using your footage. I didn't accurately speed the footage up at the head but give the comp a look see using your footage and see if this is an approach that would work.  A couple of minutes with a calculator would have told me where to put the second time remapping keyframe.

Stabilized motion, decided I wanted slow motion just before the wheels touched down so deleted all stabilizition after this time, pre-composed, collapsed transformations, applied Time Remapping and set a keyframe just before the wheels touched down then grabbed the last 2 Time Remapping keyframes and moved them to the left to speed up the first part of the footage. Set a position keyframe at the head of the shot, then adjusted the position when the wheels touched down. Set a couple of position keyframes to center up the footage in the Comp panel and added 2 scale keyframes to hide edges until the motion stabilize ended. Here's the comp: Dropbox - TimeRemappingSlowMo.aep  After looking at the ram preview a couple of times I think I would have been better off to track something on the ground. Another 15 or 20 minutes messing with the clip should result in a smooth approach to a slow motion landing.

NOTE: your browser may add a .txt extension to the project. Just delete it and use your footage.

[Roei Tzoref · Accepted solution]

I would do it this way with Mocha Ae:

1. right click on on the footage -> track in mocha Ae

2. select a decent area in on the foreground. select only translation in the motion panel

3. unlink the mocha shape from the tracker

4. track

5. if you did not start on the first frame, then on the first frame nudge the planar surface so it will record it as the full frame

6. export tracking data - transform, click invert

7. paste in Ae. now you have a stabilized shot for both X and Y. you will also get scale and rotation and can remove those

8. separate dimensions of X and Y and click on the stopwatch to remove the X movement

now you have stabilized the shot on the Y axis, all you have to do is precomp (for order sake...) make sure you collapse the precomp to avoid a render hit. make any speed changes you need. scale and position it a bit to make up for the missing part of the frame.

[David Coiffier]

Thank you very much to both of you!

I'll try both methods tomorrow, and will give you feedback!

Have a nice evening,

David

[Rick Gerard]

Roei's technique is a really good one for stabilizing motion. Same basic idea. Warp stabilizer is just the wrong tool for the job.

I spent another 10 minutes with your footage, motion stabilized the cockpit window and came up with this. Pretty good. Could have also used Mocha. There is quite a bit of Y movement as well as X in the shot so you will probably need to do both.

Dropbox - Plane Stabilize.aep (note: your browser may add a .txt extension to the .aep file. Just delete it and then use your sample footage)

[David Coiffier]

Hello Rick, hello Roei,

Thanks again for your help here. As a conclusion, and as Rick stated, warp stab is probably not the best tool here, because of range limits I've found. Without those, it could have been an excellent choice on most shots, since it always focused on foreground and background rather than airplane.

Using AE classic tracker is indeed a good option, but many other shots I have won't easily fit, because being quite closer than this example, image will pan from one plane part to another, hence my idea to use warp stab to track ground rather than plan itself. For no obvious reason, my mind totally forgot that mocha has been part of AE for years...

And of course, it totally did the job. Tracking is not quite as fast as warp stab analysis, but it works great anyway.

So, thanks a last time to both of you, I really appreciate the time & expertise you've freely spent on this!

Best,

David

[Roei Tzoref]

you can try other software to stabilize like mocha or syntheyes. If we could see the shot we could run some tests.