There are four vertices on your square. There are four vertices on your circle... but the circle's vertices are 90 degrees away from the square's vertices. Rotate the circle vertices 90 degrees and life gets better.
When you animate a path the vertices (points) always try and take the shortest straight path to the new position. It's as simple as that.
If you want to morph a square smoothly in a circle you need to add a vertex at the center of each side, then add a vertex at 45º, 135º, 225º, and 315º and then set the first vertex to the same position, like top center for each path.
There is a tool in AE to help with this and it works sometimes on some shapes. You can find it in the window menu. It's called Mask Interpolation.
To do more complex shapes the only real solution is to decide how long you want the transition to take and then use the blend and expand and release to layers feature in Illustrator to create an animation there. You have complete control over the artwork and can do things that are just not possible trying to animate a path in After Effects. Take a quick look at these mini tutorials that I did on the technique. They may give you some ideas:
Thank you both for your quick answers!
@Dave The anchor point rotation does work for these shapes but I was also wondering how to do this to more complex shapes in other videos.
@Rick Thank you for those tutorials! I'm unable to try them right now but I think you have answered my question.