You could try duplicating the sequence and applying a Levels effect to the layer; set the channel to only target the Alpha, then adjust the sliders as necessary to solidify the semitransparent section.
Then, use this as an alpha matte as usual.
Thanks for the reply, Zack.
I have tried that, but unfortunately, it still gives the same result.
With alpha track matte on the landscape sequence, the background just disappears. With inverted alpha, the landscape sequence disappears and leaves the background matted in the shape of the landscape area, as opposed to the "horizon" area (the portion that is fully transparent).
It appears to recognize that the fully transparent portion is alpha, but when I try to use it as the alpha matte, it just isn't working as expected. Which is strange, because as I said, when I use inverted track matte, it does exactly what I would expect it to do... it reveals the background matted in the shape of the landscape animation.
The easiest way to fix this would be to get the footage right in the first place... i.e. getting the transparency issues sorted out in the 3D application.
The footage itself is fine, Dave. The 3d application is rendering exactly what it is supposed to render; a semi-transparent landscape on a transparent background, rendered out to 32 bit PNGs.
When I view the frames in Photoshop, they are exactly what I wanted. In the final composition, there are other image elements and pre-rendered objects provided by the client that need to be visible beneath the "ground" and "water", which is why I rendered the landscape as semi-transparent. Normally, I would render with a color background for compositing, but using a color background in this case was tinting the semi-transparent landscape.
I think my track mattes are failing because the images are made up of varying degrees of alpha.
Going back to the 3d application, the only option there would be to re-render a pure alpha pass of the animation. I can test the render time per frame, but I was hoping that there would be an easier solution... the initial landscape animation took 70+ hours to render, using 32 cores and 24 gb of RAM. The pure alpha render shouldn't take anywhere near that long, BUT, if it did, that would not be feasible on my current schedule.
I did find another possible solution by creating a recorded script to batch convert luma images from the PNGs. If the alpha pass proves unworkable, I can always fall back to batching the luma images and using stencil luma.
Thanks for helping me further think this out!
Perhaps I phrased it incorrectly: can't you use the 3D app to generate a simple black-and-white matte -- or mattes -- to separate the elements correctly, then just composite in AE?
That is exactly what I ended up doing... I didn't want to, because I was afraid that the 3d application would try to interpret the black and white as something other than the matte it was set for... and with such lengthy render times for the product, I didn't have that kind of time to wait to get my AE composition together and get some product out.
Fortunately, the 3d application took the matte literally and rendered the entire thing in about two and a half hours.
Glad it worked.