So here's the details: I have a precomposed layer that I need to apply a mask to because I only want certain areas of the layer to be visible. Inside of this precompossed layer is some solids that I have changed the blending modes of to emulate a light effect. There is also another precomposed layer inside of this precomposed layer that consists of some more solids with masks. I have toggled collapse transformations to make the effects visible in another composition. Anyways, I need to add a mask to the final precomposed layer, but when I do the layer seems to lose its attributes and as result all of the blending modes are out of order. For example: normally when I apply a mask and change the mode to subtract, the area inside of the mask will not be shown. This is what I want, but instead what I'm getting around the mask is not the original composed layer's look. Instead the composed layer looks like all the blending modes turned off. I hope this makes sense .
You can't use CT with masks. The behavior is perfectly normal. That's just how it works. If you want masks, you have to add one more level of nesting and do it in the topmost comp or forego CT. The rest we can't know without seeing screenshots. Reorganizing the layers and using different effects might easily avoid the issue of even having to use CT.
As Mylenium said it's a workflow problem. You can use CR and CT, you just have to do your masking in a different way. I would add a solid or use a shape layer above the nested comp you want to mask, change the blend mode to something like screen so you can see through the layer to draw your mask, then draw the mask and use it as a track matte. That should work just fine unless I'm completely misunderstanding your project.