Unfortunately this doesn't have anything to do with the FlexUnit framework, it is Flash Builder and Eclipse.
Personally though, I rarely use the Execute FlexUnit Tests. Here is my workflow and, it if works for you, I find it much faster. I make one top level suite that includes others. This means I can continue adding to it over time, but I am always running the same suite. I launch the FlexUnit tests once for that suite. They will run successfully and report their results in the FlexUnit panel.
Then, instead of closing that browser, I keep it open in the background. I never again launch the tests from FlashBuilder, instead i just compile it through FlashBuilder, switch to the browser and refresh it. The results will still be sent to FlashBuilder, etc. but I find it to be significantly faster.
BTW, the compile time has less to do with there being one test and instead the project/workspace in general. To run your tests, it has to make sure your project is compiled so anything you can do to make the project compile faster will make this part run faster too