I do all the moving and zooming of the clip in the time line, I then nest the clip and add a garbage matte or linear wipes to the nested sequence to make the still the right size. You may have to then correct some of the movements to keep it within the new size.
Try keyframing the anchor point instead of position.
I simply start by adding a Keyframe (one for Scale and one for Position), at the first Frame, and the last Frame, to set things to zero adjustment setting. Then, I go to where I want the Pan, and/or Zoom to begin, and add another Keyframe at the zero adjustment settings, for each Effect. Next, I add another, where I want the Pan, and/or Zoom to end, retuning the view to the zero addjustment settings, and add a Keyframe for where the Pan, and/or Zoom should stop, and a Keyframe where I want the Effect to begin going back to zero adjustment.
Actually, the "anchor Keyframes" at first and last Frames are not necessary, unless the animation will begin at that point, but placing them is easy, and is just part of my Keyframing workflow. They also insure that zero adjustments will be locked down, on both ends of the Clip (if I wish to return to zero adjustments at the end).