Captivate can handle quite large video files. However, the course may take a long time to load if the video is too big.
I am no expert, but a video with a final size in the 6-8 MB range doesn't seem to give me or my users any problems.
The most common video formats are supported:
From Adobe's page on Captivate 9 (probably works on 8 as well):
Record system audio along with narration or import video in a wide variety of formats (AVI, MOV, FLV, MPEG).
Hope this helps,