I wish your description was a little more detailed or you could show us a screenshot of all the layers with their modified properties revealed (Press the u key twice). Your description isn't detailed enough to troubleshoot the problem. Given the correct kind of footage and the correct placement of the effect Set Matte always works.
Here is a little more clarification, along with some screenshots.
1. The image below shows the video footage with some black tree outlines (png images) that I have keyframed to match some existing trees in the video for the purpose of subtracting that area from the 3d animated layer.
2. This image shows the animated layer on top of the video footage and the png images. Trees from the video should appear in front of the animation layer.
3. This image shows the animation layer with the png trees turned off, so you can see the current state of my composition. The existing trees are being cut off by the road.
I have applied the set matte effect to the animation layer, and selected the png trees as the alpha source, but it is not subtracting anything from the layer. Does that make sense?
Pre-compose the masked and scaled layers. AE has a specific rendering order and as far as the program is concerned, it cannot see your changes using your current method.
Thank you! I was unfamiliar with pre-composing. This made it especially easy since there were multiple trees. Once I precomped those together, setting them as a matte to my animation layer worked perfectly.