I was trying to make a vector mask by using the pen tool to just draw a simple rectangle so that the layer below would show up. What I went to use the pen tool to make the first anchor point the whole top layer disappeared and showed my bottom layer that I wanted to show through the rectangle. Instead of the bottom layer showing through the rectangle it showed the other layer cut out and placed in the rectangle shape over the bottom layer. Version CC 2017 has quite a few differences in how to create the results you want vs the version of CS6 that the online course I am learning with uses. Such as when I was trying to do a layer mask and completing the step to have the beach scene below show up behind the layer with the people I had to click reveal all/ hide all instead of the instruction they told me which was click the create mask icon on the layers panel.
Here is an example:
I wanted the beach to show up in the square behind the layer with the people What did I do wrong?
You explanation was not 100% clear however it sounds like you need to look at the pen setting for Combine or Subtract shape
1. If you have vector mask and want the top layer in the square
Then set you pen tool as shown to set the mask to show the top layer in a square and the bottom layer around it
2. If you have a vector mask but want the top layer outside the square
Set the pen tool as shown
I hope that helps you
That must have been what I had it set to. I was not sure what those settings meant