Every LMS is different. Even SCORM-compliant LMSs do some things differently.
So the fact that the client's old LMS had no issues with the particular settings that these courses used, does not mean their next LMS must of necessity do likewise. The reason there are so many options and settings in the SCORM area of Captivate is due to the fact that you usually have to experiment to find the combination of settings that work with a particular LMS. In effect, you have to 'tune' the content for the LMS.
So, like it or not, changing the LMS will definitely require some adjustments, and there are limitations to what the LMS vendor can do.
I would suggest to your client that they try plugging their courses into SCORM Cloud to see what the diagnostics there show up. If their content works perfectly on SCORM Cloud without issue, they'll have a better case to present to the LMS vendor than saying their "home grown" LMS handled it OK.
Rod, that you for your input. Answer was a little more general than I was looking for. Do you have any suggestions for what I might look for regarding the specific problems I mentioned (i.e. bookmarking, auto-close not working, x in playbar not working, button count showing up in the score area of XML?
Two of those issues are likely caused by the same thing. Auto-close and the Exit button in the playbar. More information on that is offered at the link below.
My own guess on those issues is that the window presenting the course is now being opened in a different manner than it was in the other LMS.
Helpful and Handy Links