The effect derives its contours from the image and at lower resolutions of course the result may change. Your image is small to begin with, so any furthe scaling may simply degrade the Alpha too much for the effect to work. Either set the resolution pop-up to Full or indeed always review your settings at 100%. It's the only reliable way to predict the outcome, anyway.
I can't get Vegas to fail when adjusting the zoom factor. The Zoom factor shouldn't have anything to do with an effect. Add vegas to a solid with a mask and check it out. There must be something else going on. Is your width 1? Select the layer that's giving you problems and press the U key twice to reveal all modified properties and if that does not help you figure it out let us know. I can't even get vegas to fail if I scale the layer way down.
thx for your fast support! hmm, so i will test this again – because this "error" happen on windows and osx machine with this file. i will try this with another file and come back to you guy tomorrow!
thx a lot!