I don't understand exactly what you are doing. 3D camera tracking will give you a place holder in 3D space where you can basically stick things to the ground.
Here's what I think you have going on. You shot yourself with a green screen behind you using a moving camera. You shot a background plate to replace the green screen. The background plate was also shot with a moving camera. In order for the background plate to match you have to match the cameras and you can't do that with Camera Tracking, that takes motion control. Even when you have motion control your background plate must match the perspective of the first shot. Let's look at a simplified example.
You have your friend shoot you sitting at a desk with a green screen behind you. The camera moves from the left side of the desk to the right side of the desk. Now you go to a fancy office building and shoot a room with no desk. You want to composite the green screen shot of you at the desk with the fancy room so it looks like you are in the office. If the camera was 6' from the desk on the green screen set then the camera in the office needs to be 6' from where the desk would be if it were in the room. The camera filming the room would need to move around the room exactly the same as the camera on the green screen stage or the perspective will not match up. You can get close by using dolly tracks or by carefully measuring and rehearsing, but it will never be a perfect fit without a motion control camera because the perspective will not match.
There is a solution though, It's called projection mapping. You would do it with a program like Syntheses, where you can generate a 3D model from a filmed scene. You would film your fancy room in the office with the camera approximately in the same place relative to the desk and with approximately the same camera move. You would then track in Syntheyes and create a 3D mesh from the planes in the shot. This would turn into a 3D model of the room that that you could place in your Camera Tracked scene.
If I did not guess correctly then please provide some screenshots or sample videos and a complete description of what you are trying to do. Camera tracking software has made compositing a lot easier, but you still have to know what you are doing if you are combining two shots from different cameras.