You post here saved me a lot of time tracking down the problem. I often just delete all of the cross-reference links after I copy the document. (Wouldn't it be nice if they updated to the new path when you package the book document...)
I have another experience. I've noticed that usin GREP styles slows InDesign a lot. I have delays even when while selecting text. I removed all GREP rules, and problem was gone. I mean - was replaced with a problem of not using GREP style rules.