I think you've answered your own question here: http://forums.adobe.com/message/5776745. I don't know enough about the RTMP implementation to offer any helpful suggestions, but if you want to file a bug report, you can do so here: https://bugbase.adobe.com/.
From your FLM, I can make a couple of general comments:
1) Framerate and frames only matter if you're actually using them - since in your case, you don't seem to have any timers listening for ENTER_FRAME, nor are you doing any rendering, it's meaningless to complain about the "waiting for next frame" time. I suspect (although I'm not entirely sure) that Flash Player has an optimization to throttle the frame ticker, if nothing is listening to it. The first delay in connecting to your server could be an artifact of this, since you say that it works fine if you increase the size of the swf to have non-zero dimensions.
2) Make sure you use a release SWF for performance testing - debug SWFs (as in your case) have significantly worse performance, because of all the extra debug information and processing that happens.
Thanks for your reply, Michael.
Regarding debug SWF, in my testing I can reproduce both the worst-case performance and "regular" performance with the same debug SWF so I don't think that is playing a factor here. So right now I'm treating it as just a bug.
I've filed a bug report so we'll see where that leads.