I am afraid your choice of codec will not improve quality (and keying) once the footage is already shot.
Converting HDV to either ProRes or Cineform can give you better performance and a more responsive feel, because you won't be dealing with interframe compression.
But the MPEG-2 compression (used in HDV) and the reduced color sampling already ocurred when you recorded the material. The 4:2:0 color samplig scheme in HDV is really bad for keying, since it throws away a lot of color information that is needed for pulling a clean matte.
There are some third party keyers (like DV Matte Pro), which specialize in disguising the flaws you get when you key low color sampled material like DV or HDV.
Some HDV cameras have an uncompressed HD-SDI output that bypasses HDV compression/color sampling completely. In this case, yes, recording (with a capture card) to ProRes or Cineform would make a world of difference compared to HDV.
What codec was it originally shot in? HDV can be pretty painful to composite. Also, there are a lot of different flavors of pro-res, but it can totally work for greenscreen. Maybe get your cameraman to render a tiff sequence with an alpha channel for you if he's already got it working perfectly.
As pointed out, if it is HDV, then you already have lost a lot of information. I would not put too much face value on what your camera operator showed you on his system. The thing is, since Premiere is YUV native and operates on HDV native, it has a different way of dealing with such issues, where on the other hand everything that comes into AE is blown up back to full RGB. Furthermore, the way how AE deals with fields may give a wrong impression, which is anotehr thing that works better/ differntly in Premiere. Also you should not over-simplify the process. Rarely ever does simply throwing on a keyer produce good results from the getgo. For compressed footage, you should make it a point to remove the block artifacts first. This can be done with tools like Magic Bullet Frames, SmoothKit or a bunch of manual techniques. Likewise, noise may need to be smoothed out or fields removed. All depends. Of course do not forget, that nobody really cares how a matte is generated - duplicate the layer, ramp up contrast or whatever and key that. You can always recombine the result by using matte modes and such. Specific to Keylight: reduce the matte range by adjusting the black and white points, then use the bias and rollback settings. If you force the effect to only operate on specific ranges, the result is usually good, but if you allow it to stay at the full range, it tries to be a bit too smart and actually messes up...
I have used all kinds of techniques for smoothing out compressed footage that has to be keyed. The first think you need to do is to fix problems caused by the reduced color sampling used in HDV, AVCHD or DV footage. The problem is that there is no one click solution. I can give you a basic idea that seems to work fairly well for some situations working strictly inside AE without any third party plug-ins.
The luminance value (black and white information) is what gives you the apparent sharpness and you end up with a unique value for each pixel in the image. The color values are blocks of color that are 1/2 or even 1/4 the resolution of the luminance. A solution is to blur the color using box or gaussian blur then add back in the sharpness of the luminance. The simplest way do to this, which helps immensely, is to duplicate your layer, blur the layer, change the blend mode to color, then pre-compose the pair of layers and pull your key from the pre-comp.
When you have higher rez material, like your HDV footage you can also scale it down from 1080 to 720, to resample the color info, then render the footage, then bring add a copy of your original footage and the scaled down footage to a 1080 comp, place the 720 footage on top, scale it to fit horizontally in the frame, then change the blend mode of the rendered and scaled version to color.
Another technique is to use the blur and blend mode in Premiere with the original footage, then render out to a suitable codec for keying in AE.
The third party plug-in solutions mentioned also work quite well. The rub is that the best technique and most effective results are different for each shot. The only way to get you a first rate solution is to have a sample of about 10 or 20 frames of the footage where there's a lot o motion blur or transparency (like the actor holding up a wine glass) or a shallow depth of field problem you can't seem to resolve, or a color bleed problem like a blond with curly hair shot against green instead of blue. See what I mean, there's at least three things that have to be fixed with every keying project I've ever encountered. Show me some frames and tell me what tools you have available and I'll give you a suggestion as to how I would handle the problem. The codec only becomes a problem when compression artifacts are worse than color artifacts and the rest of the problems combined.
Thanks Mylenium & Rick, lots of great advice there.
I take your point about it depends on the footage. So I have uploaded 1 second of my footage to my web site, encoded from a 720p m2t file into a QuickTime Animation based codec, which I pressume will have lost none of the orginal informatin?
Download the clip here www.federatedfilms.com/GreenScreenTest.mov
You may like to know it was shot using a JVC GYHD201 camera using a Datavideo CKL-100 Chromakey system and shot progressive.
Software wise, I have Adobe Production Studio CS4 so Premiere Pro and After Effects as well as Final Cut Studio 2
Thanks for your help :-)
The blockiness is caused by the color sampling artifacts of HDV video. There are a couple of ways to fix this in AE.
The first is to pre-blur the matte in Keylight. A value of 1.2 to 1.8 is probably sufficient. The second is demonstrated in this image. The footage is duplicated, top layer blurred using either box blur or gaussian blur, and then the top layer's blend mode is changed to color. The image below is viewed at 1600%. You can clearly see the 4 pixel blocks of color that are repaired using the blur/blend technique.
A little tweaking in Keylight and you should have a decent key. Here's a sample CS4 project showing both techniques.
Thanks, that is very kind of you Rick to go to all of that work.
There is still some noise around the top of the hair which is where I was strugling before but I will try playing with Mylenium and your suggestions and see if I can get it perfect!
also, besides the excellent advice given, you can deal with problematic areas like the hair with a completly serperate key, (duplicate layer and mask).