You may want to email me offline about more questions here as it can get a little specific but I'll hit the general problems here.
MP4 is a fine format, and I won't speak ill of it, but it does have weaknesses. In FMS implementation those weaknesses tend to manifest around the combination of recording and very large files, so some of these things are a known issue.
The problem is that MP4 recording is achieved through what's called MP4 fragmentation. It's a part of the MP4 spec that not every vendor supports, but has a very particular purpose, namely the ability to continually grow an MP4 style file efficiently. Without fragments one has the problem that a large file must be constantly rewritten as a whole for updating the MOOV box (index of files) - fragments allow simple appending. In other words it's tricky to make mp4 recording scalable (like for a server ) and still have the basic MP4 format - so fragments.
There's a tradeoff to this however, in that the index of the file is broken up over the whole file. Also likely these large files are tucked away on a NAS for you or something similar. Normal as you likely can't store all of them locally. However that has the bad combo of needing to index the file (touching parts of the whole thing) and doing network reads to do it. This is likely the cause of the long delay you're facing - here are some things you can do to help.
1. Post process the F4V/MP4 files into non fragmented format - this may help significantly in load time, though it could still be considered slow it should increase in speed. Cheap to try it out on a few files. (F4V and MP4 are the same thing for this purpose - so don't worry about the tool naming)
2. Alternatively this is why we created the raw: format. For long recording mp4 is just unideal and raw format solves many of the problems involved in doing this kind of recording. Check it out
3. You may also want to check out FMS HTTP Dynamic Streaming - it also solves this problem, along with others like content protection and DVR and it's our most recent offering in tech, so it has a lot of strengths the other areas don't.
Hope that helps,