When you animate path vertices in space, they will find the shortest route (a straight line) from one location to the next. the most explicit example is when you use rotation so if this is the case, you should use transform rotation in order to perform rotation for this type of animation and not the free-transform of the mask.
from the help files: Managing and animating shape paths and masks in After Effects
"When you animate rotation using Free Transform Points, the vertices of the mask are interpolated in a straight line from keyframe to keyframe. For this reason, the results may be different from what you expect."
it makes sense because you are animating vertices, not a whole shape. so the transforming is per vertex and not the whole mask in it's entirety.
Here's a demonstration. I have set numbers for each vertex 1,2,3,4 notice both shapes start and finish in the same place. only one is rotating as a whole, and the other only shifts its vertices to a new location
the principle is the same for all vertices interpolation so It doesn't matter if you used free-transform to rotate a path or manually dragged the vertices, you will still get this behavior i.e shortest line from keyframe to keyframe.
Now that you know how animated vertices are interpolated, you probably have better understanding on how to perform what you set out to do. there is much more information in the help files link I provided that could help you get smoother shape morphing animation (first vertex, (Smart) mask interpolation panel, and more...). if you still have questions after all of this, you should probably share a screenshots or video capture so we can see exactly what your are dealing with and offer solutions.
In some cases you can improve the transformation by playing around with Window>Mask Interpolation...
If you are rotating a square you'll need to set a bunch of intermediate keyframes and experiment with interpolation methods.
Unfortunately mask interpolation is not available for Shapes. you could use a workaround and copy the shape paths to a mask path, do the interpolation, then copy back. not ideal. race you to the feature request page?
here is some more information for Op and everybody who is interested in demistifying vertices mysteries:
No, but you can apply it to a mask, copy the keyframes, start a shape layer with a single click using the pen tool and then paste your interpreted masks paths to a shape layer.