Sorry but this is one of the things you have to expect with rich media annotations (RMAs) in PDFs. The media content is decompressed into memory when the RMA is instructed to activate, and a copy of Flash Player is created and loaded onto the screen canvas. Depending on the structure of the media file, and the specifications of the computer, that can take a noticeable time. When the RMA is deactivated, the Flash Player instance has to be cleaned from the canvas, again on some machines this can take a while to happen.
I would typically expect a 1-5 second delay for activation and cleanup on an average-spec computer. 30 seconds is very long, I've never seen a file take that amount of time. Without knowing your hardware setup and the structure of the file, I can't suggest how to reduce it.
Machine specs don't matter once you clear the base requirements for Acrobat or Reader - both are 32-bit applications so can't make use of extended ram, hyperthreading, etc. and there's no CUDA acceleration in the Acrobat Family. It's more about the complexity of the PDF file and what else Acrobat is trying to handle at the time (how complex is the page, etc.)
If you're embedding a SWF it will need to have a stop keyframe on the timeline, but if it has a scripted interface that controls looping you can pass it flashVars using the advanced settings on the RMA dialog.