The dialog box was in Captivate 4. When you insert a slidelet in Captivate 5 you configure it using the Properties tab on the right side of your Captivate screen.
Hello and welcome to the forum,
Did you find those instructions in the Help of Captivate 5? If yes, please add your comment there if you can find the time, that way it could be corrected.
When you insert a Rollover Slidelet you see both the Slidelet and the Rollover Slidelet as a square on the stage. To set the Properties for the slidelet, select it and you will be in the Slidelet Timeline automatically whereas the Properties panel will be the one for the slidelet. To set the Properties for the rollover slidelet, do the same, you'll be in the main timeline and the Properties panel will be the one for the rollover area.