Try opening the problem files using a text editor or file viewer to see what the first few bytes contain. All valid FM binary files for FM 11 will contain <MakerFile 11.0> in the first bytes of the file.
When updating books, it's sometimes better to just to create a new book file and add the files to that.
When renaming files in a book, changes at the system level will break any links/cross-references between files, so it's always best to use the Rename option in the Book file to change FM file names. This will maintain the correct linkages.