You need to study up on
a) how AE 3D works on a general level
b) how the 3D tracker works
c) why fisheye lenses are a terrible idea for 3D tracking
Since AE doesn't have a way of calibrating its 3D tracks, this is pretty much to be expected - it simply calculated an arbitrary tracking solution that kinda works, but yields tiny values. You will have to scale the camera keyframes and adjust the zoom level accordingly to even come close to something workable. Otherwise shooting with a flatter lens in the first place or using a fisheye removal plug-in is definitely advisable.
The first thing you should do is set an origin and ground plane, then check to see if things are working by inserting a solid and scrubbing through the shot. I suspect that your camera track isn't that good and that things fall apart quickly as the placed 3D layers move to the edge of the frame.
AE's camera has no distortion. If you shot that footage with a GoPro I would highly recommend using the GoPro Studio to import the footage, convert it to a production codec and remove the distortion. If you shot with some other camera then I would suggest that you shoot a straight on shot of something that is geometrically square like some big graph paper or even a big square picture frame making sure that you are perpendicular and centered on the grid, then use the Lens Correction tools in AE to remove the distortion before tracking. You can then either render an undistorted DI of your footage or you can pre-compose the fixed layer and Camera Track the pre-comp. This is the ONLY way you can get camera tracking to work, especially around the edges of the frame, when you have lens distortion. If you really want the fisheye look when you complete your composite nest the master comp in another comp and use the lens correction tools to put the fisheye back in the shot. That is how all compositing that uses camera tracking has to be done if it is going to be accurate because virtual cameras in 3D and compositing apps do not have any lens distortion unless you add it with an effect.