Think of it: Masks are relative to the layer, not the comp. Applying the mask directly to a stabilized layer therefore naturally also moves the mask, which is what happens - the pixels seem to be stationary, but in fact the layer moves all the time. Simply add a solid on top and use it as a matte, then do all your masking work on the solid. Or alternatively, pre-compose the stabilized shot without the masks, then mask the pre-comp layer.
I did as you suggested, I precomped the stabilized layer, then applied the mask to it. I still get the same results, the mask keeps bouncing up and down while the video is smooth. Is there something that I;m missing on the precompositing stage?
Oooops, I see what I did wrong now. I forgot to make the precomposotion with the stabilization included. Note to self, read the options that you get when saving a precomp file. Thanks for your help, it works now.