Sounds like you might be in the same situation as this thread - Getting message, "Cannot checkout all the files that needed to upgrade this project." - over in the Source Control forum. Since they didn't get any response, I suggest you contact Adobe Support & then report back what worked, so that others can benefit from the experience.
A few things to try.
When upgrading a project in source control, I always used to manually check out the entire project before opening it, then manually check it in afterwards. I found I had fewer problems this way.
Also check to make sure you haven't got any files in the "Pending Add" or similar state - you might need to look in a different place/report thing (sorry for the vagueness; I haven't used source control for a number of years, so I forget the exact details.) rather than just looking at the status next to each file in your source control tool.
Oh, and double-check you do not have the .cpd file checked in to source control - this can cause all sorts of weirdness. (I think the full list that shouldn't be in source control is .cpd, .ldb, .pss ? )
I finally figured it out. The project had a couple files that were messed up due to broken bookmarks. Have you ever had a bookmark break and RH creates a file with the #[bookmark] appended to it? For example, Test.html#bookmark. I believe this happens when you have a hyperlink to a bookmark in the project that no longer exists, so RH tries to "create" the bookmark by adding a new file. Well, source control cannot check out that file because of the wonky extension. Basically, I had to open the project in the older version (RH10 in my case), removed the project from source control (which allowed me to delete the wonky file), readd the project to source control, and then upgrade. Sigh.
Thanks for your input Jeff and Amebr!