I apologize for what must be easy to fix, but I am new to AE, and just learning my way around it. I can make a nice matte (using an inverse luma matte), where the opaque areas are dark and crisp, and the background shows through the transparent parts perfectly...so long as I don't touch anything. If I try to move the matte, or scale it to a smaller size, the opacity increases, fading the dark areas of the matte, and turning the transparent parts cloudy. A way to avoid this must exist, but I can't figure it out. Levels just changes opacities and colors; I still don't get a clear matte with scaling.
Pre-compose the matte stuff and scale the resulting composition-based layer. You simply misunderstand how mattes work. A matte merely acts the Alpha channel. The behavior is perfectly normal - even in Photoshop things behave that way if you scale only scale a layer mask. The resulting blend of all channel components is different than before.
since you are new to Ae make sure you watch this video first to get a sense of how this is supposed to work:
if you are still struggling with this, please show us your setup: your matte layer, your background layer and the composite in both situations so we can see your workflow and what exactly is wrong.