Choose anything else from the View menu than "Fit Page in Window" or "Fit Spread in Window".
I guess Command+1 Actual Size, is closer to what I expect but I'd like the behavior of the paste board to be like illustrator where the paste board remains stationary when I resize the window.
The image sliding all around while moving a window corner is bothersome. Such as when I am trying to open 2 document windows side by side on the screen to compare. But it is less bohersome than sliding around AND resizing.