You are correct, the editing points (yellow ones) are not behaving well at all. I have checked it out. You should log a bug.
Do you heed the smart shape as a container for text? If not, you could use one of its brothers and rotate it, but if you need text, that will be rotated as well.
The other workaround is to convert it to a freeform shape and edit the points then. More work, I know....
This is a bug in Captivate 6. Here is the solution to make it work:
1) Download the xml present in Acrobat.com Link --> https://acrobat.com/#d=T0o7dfXsRuBnJD1dsTLfOA
2) Go to AutoShape folder present under Gallery inside Captivate install location.
For my Win7 64 bit --> "C:\Program Files\Adobe\Adobe Captivate 6 x64\Gallery\AutoShape"
3) Take the backup of S2_12_UpArrowCallout.xml present in AutoShape folder.
4) Replace S2_12_UpArrowCallout.xml with the one downloaded from Acrobat.com.(Make sure the name is same)
5) Re-Launch Captivate6.
6) Insert Up Arrow call out smart shape and handles should work fine.
For any querries you can contact me at firstname.lastname@example.org