I just went back to work on my PPro timeline, and discovered what makes the gray halo shy - realtime preview of unrendered footage! So here are the before and after snapshots I was trying to get for you. For some reason realtime preview was being stubborn on my pink rose, but this orange one is actually the one I plan to really use with this background, so here it is:
So, the halo only shows up when the frame is actually rendered. Weird, huh! Is that related to the type of alpha, or something else? If it's something else, I guess I'll need to ask over int the PPro forum, but I posted here because I though ti was something I did wrong in AE.
NM, it looks like you can't specify the type of alpha interpretation in Premiere.
Any thoughts on this? I need to send this project off to the replicator, but I don't want to do it with gray lines around my roses... Should I be trying in the PPro forum? I don't want to get in trouble for cross-posting, but I don't really know which program is causing the problem.
I doubt this is relevent, but my original "footage" is actually photo sequences (snapped once a minute for a couple days for each rose) - I first used VirtualDub with a deflicker plugin to bring the sequences in, remove the camera aperture flicker, and output as AVIs with Lagarith. Then I brought them into AE for Keylight. But none of that should matter, as there is no alpha before AE - it's how AE renders it and PPro interprets it that matter.
Try to render RGB and Alpha separately, then use Premiere's Alpha key effect to recombine them. When you do, consider dilating the RGB data by one pixel using the Minimax effect.
Thanks for the suggestion - it basically worked. PPro CS3 doesn't appear to have an effect called "Alpha Key", so I used "Track Matte Key", specifying the track where I put the alpha clip and selecting Luma Key for the setting (setting it to Alpha Key didn't do anything, and it was clear that the "alpha" AVI was not an alpha channel per se, but just a regular video clip with a B&W representation of the transparency). I also don't see any effect called "Minimax". Anyway, I followed your suggestion as best I could with the tools I have - you must be using a newer version of PPro. I can indeed see about a pixel or so that would be better if it was more transparent - I suspect your Minimax suggestion would have taken care of that. But this is definitely usable.
Key point for those reading this later: You have to choose "Straight" for the Color setting when you render by this method, even though you get some strong warnings that dissauded me the first time I tried it (I thought they were saying I had to include an alpha channel in the stream, not the method I was about to use).
One more question: When rendering the separate alpha clip (essentially just a grayscale video of the transparency) to be used as the luma matte for the Track Matte Key, does it matter whether the Color setting is Straight or Premultiplied? I can't imagine that it would matter for that clip, but if there is a subtle difference, let me know.
I have no idea why I can't get it to work with an embedded alpha channel, though - surely hundreds or thousands of people use these two programs to handle RGB+Alpha video files. For this time, I'll re-render my roses (twice for each rose to get the two streams - this will give my CPU a workout!) and use this workaround, but if someone has any insight about the more direct method, please enlighten us all.