I am presuming that your image is a flat picture, set as a 3D layer.
If your text is a 2D layer, it won't work because it won't respect/move with the camera view. If your text is a 3D layer and oriented in any way other than straight on, and in any Z position other than that of your wall image, it won't work because of the nature of 3D parallax.
So the ideal way to fix this problem is to get the text in the correct position, then precompose the text and image. In your subsequent MAIN comp, set the precomp layer to 3D and do the camera move. The text and image should remain in the same position.
Of course, you could just add the text in Photoshop, then do the move in AE, if you wanted to.
As Andrew said, you're just not understanding how AE works and not using the right workflow. I would position and distort the text layer on the photo using corner pin or one of the other distortion effects to that the perspective matches, then pre-compose those elements and make the Pre-comp the 3D element. The other option would be to make the text layer 3D and position it just in front of the photo layer by one one pixel, then parent the two layers and you will be good to go. If the perspective of the text layer doesn't match the photo then you'll need to distort it rather than rotate the text layer. Depending on your still image and what you are trying to accomplish in the composite, you may also need to match the angle of view and camera position of the comp camera and the camera used to take the photo. Without details or a screenshot of exactly what you are trying to accomplish it is pretty hard to suggest the most efficient workflow.