No way. Your structure is simply a mess and you can't defeat how AE works. At best you may be able to bypass some of the matte issues by using the Set Matte effect, Channel Combiner or similar, but otherwise only greater restructuring operations can save your bacon, i.e. the mattes would have to be generated in a duplicate comp where stuff is linked to the main comp with pickwhip expressions and then the ultimate results combined strictly in 2D without CR/ CT.
I found that even using layer masks instead of track mattes does not change anything and your answer predicts this.
So that structure is a mess? It would seem like a simple hierarchical containing relationship for shared precompositions to me... and so I expect I have much to learn.
FWIW, turning on CT for the C comps inside the main comp and turning on CT for all the B comps in C properly rasterizes everything except for the objects in comp A. So I have that going for me.
Thanks for your help, Mylenium!
Here's what you are probably missing. Let's set up something very simple. You have a Main Comp and a PreComp. The Main comp contains the PreComp and a Camera. The PreComp only contains single layer - TestShape that is a 20 X 20 pixel red solid and it has a circular mask applied so it looks like a dot. When you add a mask to a solid you have essentially turned that layer into a vector layer.
Let's go through a couple of things in the PreComp. Fist, scale TestShape 600% and look at the edges. They are a mess. Turn on Continuously Rasterize and they are perfect.
Look at the Main comp and scale up the PreComp layer 600% and the edges look terrible. Turn on Collapse Transformations and the edge is sharp.
Notice that I used two different terms here for the same button. Collapse Transformations only applies to layers that are compositions. It means that the items inside that composition are treated as if they were in the main comp. In other words when Collapse Transformations is applied we are scaling the TestShape layer 600% and then scaling that again 600% effectively making the layer 3600% bigger. Did you follow that? It's important.
Now let's go back to the PreComp and add a second 20 X 20 blue solid with a circular mask. Call the Blue Solid TestShape 2 and make it 3D and turn on CR. Add a camera to the comp. Now adjust the z value of the TestShape 2 layer so that the layer is about the same size as the 2D TestLayer. If the TestShape layer is scaled to 600% then the Z position of TestLayer 2 will be about 80% of the Zoom value of the camera. Now move the camera's X position just a little to the right so you have a Blue circle overlapping a Red circle in your comp. Turn CR on and off for the TestShape 2 layer to see how that works. Moving the TestShape 2 layer closer to the camera is exactly like scaling the layer.
Now return to your Main Comp. With CT turned on you should see only the blue circle. This is because CT is ignoring the camera in the nested comp and using only the Active Camera in the MainComp. Turn off CT and you'll see what you see in the Pre-comp but the edges will be soft. The only way to get the same view in both comps is to copy the camera in the PreComp and paste it in the Nested Comp.
Here's one more interesting thing that happens. If you make the PreComp a 3D layer it will multiply the position value of any 3D layers in the PreComp making camera moves an interesting but not impossible thing to deal with. In this example if you were to make the PreComp 3D it would look like the blue layer disappeared, but just move the camera back and it will show up again.
Now let's add a track matte by creating a new rectangular solid that is 1000 pixels wide and 200 pixels tall and add an elliptical mask. Scale this up 600% use it as a track matte and turn on and off CR. The results should be predictable. If you make a track matte layer 3D moving the Z position around will not effect what is matted out in the nested comp. If you had 2 3D layers in the nested comp you could not move the track matte between the two 3D layers and only mask one of them.
I hope this helps. It's a good exercise to help you understand how CT and CR work. Start throwing in effects and you'll learn more.
This was a terrific exercise!
I now understand much more about how After Effects works (i.e., where/when layers are rasterize and how the various transforms are collected). I also now see what Mylenium was talking about when he indicated that the structure I chose is a mess. You just can't do that in AE (for some good reasons I now understand better).
I think AE should ship with this tutorial. I hope that it will help others as well, here on the forum.
Thanks again to the forum community for such fast and comprehensive explanations!