Are you scaling up the precomp past 100%? You'll have that problem if you do. What kind of animated text is this? Is it in an AE comp? Is it a movie?
Here's the basic workflow when you want to use 3-D objects in a pre-comp. Make the layers in the pre-comp all 3-D. Leave the pre-comp 2-D in the main composition and turn on collapse transformations. Now the camera will interact with the 3-D layers in the pre-comp.
Masking should work but if it doesn't that is telling me you are using Ray-traced rendering for some reason. If that is not the case and you still cannot get the masking to work try using a shape layer as a track matte.
If you are using Ray-traced rendering then I need to know exactly what you're trying to accomplish to walk you through using a matte. We need full resolution screenshots of both comps with the modified properties of your layers revealed.
I was using Ray-traced render for...reasons!
Switched to classic 3D, switched object within precomp to 3D, collapsed and it all works!
Ray-traced rendering never worked well even with a supported card. It's much easier and faster to create your 3D text in C4D Lite that comes with AE. Adobe is no longer developing Ray-traced rendering.
If you really must use Ray-traced rendering then you have yo do a bunch of work arounds and only use Ray-traced rendering in the nested comp. If I knew exactly what you were trying to do then I could help you with the work around to get masking to work.