When you apply it to a clip, you'll be able to shape the "effects area" by maniuplating the corner handles.
Then, whatever effect(s) you apply to the clip will only be applied to the video within the effects area.
I give you step by step instructions for using it in my book if you're interested.
In PRE11 , You can right click the clip in Expert mode and Choose >Effect mask >Apply . You will see a box on the monitor , you can resize this box .
Go to Effects in action bar , choose any effect and drag and drop on monitor in the mask box .
You will now see the effect applied on only the portion( mask box) of entire clip .
You can also Edit or remove the Effect mask using clip>right click >Effect mask >Edit/remove