This error message usually means that an open, save, place, save as dialog window is opened somewhere and is waiting for you to click on a button.
Can you confirm which version of InDesign you are using and which operating system you run this?
Can you confirm also that you have 'synchronized your book' to unify your list of styles?
I am using CC2017 and it is up to date. I have a MacBook Pro (Retina, 15-inch, Late 2013) running Yosemite 10.10.5. I am also using Radish plug in for versioning and the WordsFlow plug in to data merge excel charts.
There is no other dialog window open. I've made sure to "synchronize book", "Update all numbers", save all documents and save the book file BEFORE selecting Layout > Update Table of Contents.
I have 8 documents composing a 300 page book. The TOC items for the first document and the last 5 documents show up correctly. The items in the 2nd and 3rd book do not show up correctly--some of the TOC style links are correct, but the L2 items don't show up at all. I've tried copying a subhead that SHOWS up in Book 4 for instance and put it into book 2 content. The TOC item still does not show up for Book 2 reference.
Just figured out that, even though it is following style from Document 1, some of the style sheets got duplicated and were not nested the same way in the Paragraph styles. Once I deleted these and reassigned them the nested style that was referenced in the TOC styles, problems were solved.
Hopefully this is all. I keep getting overset text error ( Synchronization will cause some text in Document3 to beforeme overset), but I hope I can ignore this. I also keep bombing AFTER I save and QUIT InDesign which is weird...