I've noticed the same. I am using Firefox 4 since yesterday on MacOS 10.5.8 (Leopard). When clicking on interactive elements within the Flash interface a dotted line (or better rectangle) appears around the entire flash window or maybe just around parts of it. I just discovered that and have to research it more to know what precisely happens.
My guess is it has to do with accessibility issues. It looks somehow similar like when you click on linked images within a web page. There is a CSS property: outline:none; which helps to avoid those dotted lines around images embedded in <a> anchors.
There are even more problem with Flash in Firefox 4. Using the property:
stage.displayState = StageDisplayState.FULL_SCREEN;
in Actionscript in conjunction with:
allowFullScreen = true;
in the web page code doesn't seem to work anymore in Firefox 4. When fullscreen is activated the only thing I get is a full blank screen.
There are dodgy things going on with Firefox 4 and Flash Player. Anybody knows already more?
Setting outline to none on the flash object seems to be working: