Sorry, but we need just a bit more information here.
Do you have a Captivate project that will be used to create HTML 5 and you want to add a video to that project?
Or do you want to simply export the Captivate project out as a video that might be played within an HTML 5 page?
Unfortunately, when it comes to video, it's quite an "alphabet soup" or "word salad" of different things. There is no standard "format". What we have are a collection of "containers" with each container being capable of storing different kinds of video content inside. So you could have three different MP4 files and two may fail to play while one plays beautifully. It's all in the internal encoding of the video. Life with video is bizarre.
I want to convert a video file to be played in adobe Muse. I created one in the past but I forgot how I did it.
My sincere apologies. Email messaging got mixed up and I thought I was replying to a post in the Captivate forums. Hence mentioning Captivate!
You might see if you have Adobe Media Encoder installed. I believe that's what Adobe prefers to use to tanscode video.