1. It could be that your PSD / VP grids are incorrectly drawn (wall/floor/wall)
Remember that you need to have the grids drawn properly in order to allow AE to interpret the camera data.
2. Your grid is skewed
The VPE data is dependent on those grids so that the camera placement can be determined. If you skew the grid, the camera might be 10000 px away
3. Multi grids
You'll get multiple cameras, which SHOULD break AE.
4. Camera Rotation
Sometimes the camera is just looking in the wrong place.
NOTE When you import the VPE, USE the VPE project that is created - don't just pull in the elements. It's camera mapping, so you'll need the correct layers and correct (crazy) camera to see the image properly.
In addition to Ko's suggestions, make sure the images you use are halfway decent in size. It seems that the behavior is better with bigger files, as obviously the reconstructed camera has larger values for position which may improve precision.