I'm not on Mac but on PC, but I often do recordings of Captivate 5 with another instance of CP5. Some captions will not be correct (sometimes it shows as 'Click on the top-level window' and will have to be edited, but otherwise this works great. Since I'm on Win I can also capture CP5 with CP4 but prefer working with CP5 and accept the little bit of extra editing.
Can you document the process you are using to record Captivate 5 with 5? I have Captivate 4 on P.C.s and I have done it many times. I've got the eLearning suite2 (with 5) on laptop and desktop Mac only. I really need to use the Mac if it is at all possible, but it doesn't work the way that it has in the past so I do need some help.