In Flash Builder 4, if you are building an actionscript project with "Generate HTML wrapper" off, Flash Builder opens the default application associated with .SWF file.
If you manually associate the flash player debugger with the .SWF file, then you are good to go. (Do an open with on a SWF file, say always use this application)
P.S: This has changed in the upcoming version of Flash Builder: you can simply point to any flashplayerdebugger.exe in Window -> Preferences -> Flash Builder -> Debug. See labs.adobe.com for a preview release.
I am working with "Generate HTML wrapper" turned off. And manually associating worked nicely! (For any Mac people who read this, you do a command+i and change the Open with drop-down menu: to the "Flash Player Debugger" and then click the "Change All..." button. Now all .SWFs will open with the Debugger version of the player)
I'm really liking Flash Builder, just gotta move away from the procedural style I've been using for years. Thanks again!