There is only one way. Warp stabilize your footage to get it where you like it and then render a digital intermediate using a suitable production format. That means no MP4's. I use image sequences. That we import your rendered DI and camera track that.
As a sidenote, I always render any work stabilized footage to a suitable DI before use in any other part of the project.
It's hard to give you a great answer to your question without seeing how good/bad your footage is. If you are starting with .mp4 or h.264 .mov files, it is likely you have bad pixels, incomplete color information, rolling shutter, poor contrast or weak focus, or any combination of these. Even after you've stabilized your footage, you will still have bad pixels during any moment of extreme camera shake, which leaves loose targets for your tracking points. Just because the motion appears smooth, doesn't mean the trackers have enough clear info to effectively track. Add to this, that using the warp stabilizer in "warp" mode will produce not only a less-sharp image, but one with geometry out of sync with the natural physics of your scene. Whenever i stabilize a shot before match-moving, I stick to using the "old-fashioned" motion tracking system so I have complete control over the subject being stabilized and the method of correction. Then, I will use an adjustment layer or precomp the footage to re-fill edges that go in and out of frame and apply an unsharp mask to bring back some detail. In extreme cases, I will place my own high contrast targets throughout the scene via tracking and nulls to aid in the camera tracking. Sometimes it works, others not so much. If you have stabilized a shot linearly - ie. you smoothed out a pan or dolly shot, you can delete all the camera tracking keyframes between your first and last keyframe, since you should now be working with footage that moves at a perfectly constant rate. Other than that, sometimes you can apply a smoothing expression to the position and/or orientation of the tracker camera to smooth it out. Something like: smooth(.2,2). You can also manually adjust the tracker camera keys for any that go way out of range between one frame and the next. This is not so easy to do with the orientation keys though, since they often straddle the 360 and 0 degree mark, and can vary by less than a single degree throughout the whole scene. In some cases I will set up a 3D null object and copy the orientation of the camera to the rotation of the null and then pickwhip them back, so I can adjust them individually and with better accuracy. It is also easier to manually adjust the position keys if you separate the dimensions so you can see each graph individually and adjust only the dimensions that you know are wrong.
But again, starting with high quality, preferably 4:2:2 or 4:4:4 footage with as little compression as possible will yield the best results in the least amount of time.
Along with what Rick said, I prefer working with ProRes 422 HQ footage, and will pre-render any stabilization and correction out to a ProRes clip to have a render friendly base for my camera tracking.