One thing that usually works is to save the file as MIF, then use a plaintext editor to search the MIF by color name. If the name is too similar to common MIF content, you may need to rename the color to something unique before saving as MIF.
The first hit will usually be the color def in the Color Catalog.
You'll then need to learn enough MIF to decipher what the associated object is on the next hits.
If the color is coming in with an imported object, the name may not appear at all.
One thing that could work is to use the Color Views (say #6) to set a View with the Black turned off and leave the colours that are reported in use to be visible. You will still have to page through your document, but you will then only see objects/text using those colours, i.e. no black text or graphics. Simply use the esv v 1 shortcut to toggle back to your default view showing everything.
You can import the Colorr Views into other documents via the File > Import > Formats to check other documents. The shortcut toggle also works at the book level to toggle the views in all files in the book.
Slightly tedious, but you'll see exactly what has those colours. Also, don't forget to check the Master and Reference Pages.
This is a task I find myself facing more often than I would like, and I agree that the safest way is to roll up your sleeves and poke around in the .mif file.
If the unused colours are related to a change in styles, it might help to start by high-handedly deleting all paragraph and character styles from [a copy of …] a sample document. Then you can use File > Utilities > Create and apply formats before creating the .mif – that gets rid of any style definitions that aren't being applied.
Once you have one file set up with styles and colour definitions doing what you want, then I reckon you can go ahead ruthlessly: delete all styles and colour definitions from the next suspect file, then import them from your new master source. This of course assumes that your set of styles and colours is identical and complete in all the files you need to clean ;-}