It looks "off" because you forgot that the structure inside the stone would look different than on the outside - the strata would have folded differently, there would be inclusions, microfractures and all that. Create the letter on a duplicate of the layer which you have slightly shifted/ skewed around. also frizzing up the edges a bit might help. Depending on what look you want, the carved region would also look darker because it hasn't bleached as much in the sun or in reverse, look a lot brighter depending on teh composition of the material and dust from the "carving" having covered its surface... Also, naturally, with that strong a shadow implying a deeper region, you may need to simulate the side walls...
Do a web search for "Photoshop displacement maps". You'l find lots of tutorials about this.
It's not a nice rock picture as well. The area you are using is very over exposed and taken in very harsh lighting conditions. Plus the rock is not flat, so the logo/text would wrap around it to some extent.
The usual way to do this is to add a Bevel & Emboss layer style, and take the layer Fill down to zero %. Is that what you did? I like Mylenium's idea of clipping a different rock texture inside the logo/text to show what might lay beneath. Or how about using the same texture, but shifting it just enough to show a sort of parallax effect. Probably the wrong word, but you are looking at it a wee bit deeper, so it would be just a little bit misaligned with the surface.
But main thing is to use a decent rock image.
If only I could change the rock, but I am stuck with it.