This is a known issue. Might get fixed in one of those updates one day...
This already has been fixed, in After Effects CC (12.1). What version are you using?
Thanks for your replies.
I'm using version 188.8.131.52 on Win 7 professionnal, on both my desktop computer and my laptop
(I have this issue on both computers)
Just to let you know, this still isn't fixed in my current version of After Effects (184.108.40.206 on win7, and win8.1) and it can be reeeeeaaaaally annoying to have to always add a close/cancel button on every windows...
It's still not working in CC2015 (220.127.116.117)