You should be using shape layers, not masked solid layers for vector graphics like this. Shape layers are always continuously rasterized, and you have fine control over such things as strokes.
ya, i started out with shapes, but then i had a problem to scale the rounded corner rectangle vertically, it squeezes the rounded corners and i didn't see a way to animate the vertices of a shape.
So i went over to masks, but that now pixelate when scaled ...
Is there another trick to achieve the goal of scaling a rounded corner rectangle vertically without squeezing the corners ? One that is being used as a pre-comp where i need continuous rasterization all the way in the goal-comp when i scale the pre-comp again. (I needed to pre-comp, because there is no grouping for the bunch of layers in AE i figured.)
Adjust the scale in the major transform properties rather than in the shape transform properties.
do shapes have an Anchor ? When animating the size, my goal is to enlarge it downwards only. I checked it with pre-comp and then use the anchor of the pre-comp ... but that squeezes the corners again.
Just reveal the transform properties for the rectangle. Adjust the anchor point there and you should be good to go. You just didn't dig deep enough.
Sorry, I didn't complete my post. I was replying on my phone. After I said you just didn't dig deep enough I meant to give you the solution, or actually two solutions.
The first is to add an expreession to the anchor point that points to the size of the rectangle. If you divide the size value by 2 and subtract that from the current value of the anchor point the anchor point will animate as you change the size and the rectangle's left and top edge will maintain their position.
Here's the expression:
x = value  - content("Rectangle 1").content("Rectangle Path 1").size / 2;
y = value  - content("Rectangle 1").content("Rectangle Path 1").size / 2;
Change the minus sign just before content to a plus and the bottom or right edge will maintain position.
After creating the rectangle and adding the expression you'll have to reposition the rectangle in your design. I could add a bunch of other language to the expression to maintain the original position but frankly, it's a waste of time.
The other option is to hold down the Alt/Option key as you create your rounded corner (or any other geometric shape layer) so that a Path instead of a Rounded Rectangle is created. When you have a path you can just animate the path selecting as many of the points as you wish.
Here's a project file (created in CC but saved as CS6) with both options so you can see how this works....
a) works great
b) is the key tip on how to use paths for any shape creation in shape layers :-)