Without seeing your shots nobody can tell you much. Simple things like additional recognizable patterns, be that just a bunch of stones on the floor, a traffic sign or some withered bush can make all the difference. A shot may look perfectly "logical" to the human eye, but can be just terrible and inconsistent in terms of the underlying math used by tracking algorithms.
I see a lot of planar surfaces. I'd try using Mocha instead.
At the time I was working on this the AE version with mocha was having a pasting problem and I sort of gave up with the whole thing...
My plan was to export/ paste tracking data from mocha into AE, is that what you're suggesting?