It's simply the relative tangent length and direction of the underlying Bezier formula in realtion to the distance between points/ the length of the side.
In practical terms if your polygon has 3 sides a percentage of about 120% will give you something pretty close to a circle. If the polygon has 4 sides it takes about 142%. It's pretty easy to see what is going on if you set up a circle that's twice the size of the polygon radius and then use the difference mode to look at how changing the percentage or the number of sides effect the roundness.
thank you very much, Rick and Mylenium, for your answers and the nice animation.
The hint to Bezier leads into the right direction.
I spent a lot of hours to find out a formula which would calculate the maximum distance from a polygon side to the resulting Bezier curve at a given roundness percentage. No success! It seems that the rounded pentagon works well, assuming that 100% roundness equals to positioning the control points of the Bezier to the intersection of the tangent and the rectangle above a side (height of this rectangle = outer radius * ( 1 - sin (54)). Unfortunately, there are differences when applying this approach to other polygons, compared to the results of After Effects.
Maybe a developer at Adobe can tell more...
Naturally the actual percentages to form a perfect circle will vary for every type of n-gon, but can still be derived from the Bezier formula in relation to the distance between two points/ edge length. It's really just math.