In such a case one would do a full replacement - mask out the ground and the actors, track them, apply the inverse tracking motion e.g. to a photo, recombine everything.
I've done this a bunch of times in the last 20 years and the technique is always the same. Grab some frames from the video to create a clean plate, roto your subject to cleanly separate them from the background, track the camera move (not to be confused with Camera Tracking) apply tracking to the clean plate, use a feathered mask to hide the seams, then put your subject in front of the clean plate and compensate for motion blur.
Here are the details. I'm assuming that you intentionally blurred the shot, or is that motion blur? If it's motion blur then I would find a frame of your video where the people in the background are on the right side of the frame that has the least amount of motion bur, export frame to Photoshop, find another frame where the people in the background are on the right side of the screen with the least amount of motion blur, export that frame to Photoshop, then use both exported frames to create a background plate that is wide enough to cover the pan by matching up details, then clone or paint out the people and the actor.
Back in AE, import the background "clean plate" you just made and put it above a copy of your original footage on the bottom layer. Your Rotobrushed actor should be on the top layer. Turn off the top layer, find the spot where you can match up the clean plate you created in Photoshop with the shot then create a feathered mask to hide the edges. Set some markers on the clean plate and the original video.
Now track the original video using position and rotation, add a null to the timeline and apply the tracking info to the null, move to the markers you set and parent the clean plate to the null. Enable motion blur on the clean plate and adjust the comp settings under the advanced tab to match the motion blur of the original video. You may need to enable CC force motion Blur on the clean plate and the original plate to match the both shots so that the result is a perfect clean plate with no actor and no people in the background.
Now just turn on your actor layer and make sure that the motion blur on the actor matches the rest of the shot.
A little time consuming, but the easiest way I know to do what your asking and have it work perfectly.
You can also use Mocha or Mocha pro to create your clean plate. (or SynthEyes.. or a bunch of other solutions) but if there is significant detail in your shot that you can track and you can composite a clean plate in Photoshop then AE will do the trick just fine.
If your shot is hand held and you are moving toward the actor then there are other problems creating a clean plate because of perspective changes so the procedure is more difficult and may require a lot more work to fix the parallax changes of a moving camera, but the technique is basically the same, Create a clean plate with actors and distractions removed, then add back in only the actors.
Thanks for the answer Mylenium. Makes sense to me now. Thanks Rick for the really detailed answer.