These kind of tracking problems are common when the tracking target is too small and the cameras have rolling shutter and shock vibration problems that foul up the source video. Electronic image stabilization can also introduce weird motion artifacts. Fortunately, if the errors are not too bad the problems can be fairly easily corrected.
The first thing you need to do is repair the rolling shutter problems with your video and stabilize the shot. The Warp Stabilizer should do a good job. I might also add CC Force Motion Blur to the project. Then you would render an image sequence or a production format movie of your stabilized and repaired footage. Then you would track that making sure that you picked a big enough tracking target to get a solid track. Then you carefully verify the track inside AE or Mocha, or how ever you track the shot and you then redo your project using the repaired footage. BTW I would track nearly this entire area to include the keyhole shaped area instead of just a corner. You can put your attach point anywhere, but track more of the image.
It's my standard practice to repair any footage that was shot with electronic stabilization and/or with a camera that has a rolling shutter before I do any tracking. If possible I also correct for lens distortion. Then I work with a DI (digital intermediate) that is either an image sequence or production format video that is at least 10 bit color and 4:4:4 chroma sampling. You would be amazed at how much difference that makes.