I solved it.
When the main AE window slightly goes into the second monitor it prevents video preview.
So just get the AE window a bit to the side. Unfortunately the green button on the top left to put the window to the actual monitor size does not work.
Yes, this is a known issue with Mercury Transmit, in both Premiere Pro and After Effects. Mercury Transmit can not use a monitor on which any part of the application frame appears. The limitation is noted here:
Glad you got it figured out.
Thanks for getting back to me about this issue.
Actually this is a minor problem, or none at all.
What is a bit frustrating that Blackmagic won't do any scaling. Aja does a pretty niche job.
This is mentioned in the AE_video_preview document as well.
The point is that it is possible because Davinci Resolve does it with the same card.
Understood about Blackmagic. We've been working with them about implementing the automatic scaling functionality that the Mercury Transmit API allows, and which AJA and Matrox use. Blackmagic will need to release a new driver to support this, so I can't comment on when that might happen.