few steps will help
-. increase the buffer size on flash player client swf to 2-5 seconds.
if doesn't help then it might be client side machine(on which video is getting played back) processing power issue
--> reduce the video size/bitrate.
--> try switching off video smoothing in swf if it is on.
--> monitor per core cpu usage on client machine, flash player taking a complete cpu core or all of the cpus to >70-80% usage is a sign on cpu limitations.