Well you might not really want to delete a swatch or style that appears in only one file in a book if it's actually in use. Someone over in scripting might have a script to step through all the files in a book and do the "Select All Unused" commands and delete in all the various panels that would apply.

OK, I was dumb enough to volunteer, but it turns out this is a bit harder than it looks.

There's not actually a command in scripting-land to "Select All Unused," so the trick becomes figuring out "what's an unused paragraph style?

What's an unused swatch?"

So, cribbing heavily from some previous threads: Delete Unused Paragraph Styles, [CS3][VBS] Delete Unused Swatches ALSO clears Gradient Swatches!, and Deleting unused swatches change the gradient, here's a solution that might work. But it might also cause problems for your gradients, if you have any. And it doesn't bother with Character Styles, because the method I've chosen (not necessarily the best) for finding paragraph styles looks at every paragraph. And looking at every character can be too slow (but you can uncomment that section if you like).

If you have a fixed list of swatches you want to remove (you could generate one by, for instance, selecting all unused swatches, exporting them to an ASE file, loading the ASE file in a new document, and then either using a quick script to print out the names of all the current swatches, or exporting the document to IDML and pulling them out of Resources/Graphic.xml or whatever), and the current implementation doesn't work, it'd be easy enough

to modify it to handle that.

Anyhow, this script opens each document of the book in turn, removes paragraph styles and swatches, pops up an alert telling you how many it removed, and moves on to the next document. It doesn't save anything, deliberately so.

As always with scripts, paste it into a text file (such as cleanBook.jsx), and then follow the instructions at How to Install InDesign Scripts.

// Return an object whose keys are all the unique values of
// property prop among the elements of array arr.
function uniqProp(arr, prop) {
    var
      i, o = {};
      
    for (i=arr.length-1; i>=0; i--) {
        o[arr[i][prop]] = 1;
    }
    return o;
}
 
// Call the .remove() method on all objects in array all
// such that their property prop does not exist as a key
// in used.
function removeUnused(used, all, prop) {
    var i;
    
    for (i=all.length-1; i>0; i--) {
        if (!used.hasOwnProperty(all[i][prop])) {
            all[i].remove();
        }    
    }
}
 
// Remove unused paragraph styles and swatches from doc
function cleanDoc(doc) {
    var
        pstyles = uniqProp(app.activeDocument.stories.everyItem().
            paragraphs.everyItem().
            appliedParagraphStyle, "name"),
        // cstyles = uniqProp(app.activeDocument.stories.everyItem().
        //    characters.everyItem().
        //    appliedCharacterStyle, "name"),
        i, p1, p2, s1, s2;
        
    p1 = doc.allParagraphStyles.length;
    removeUnused(pstyles, doc.allParagraphStyles, "name");
    p2 = doc.allParagraphStyles.length;
    
    // removeUnused(cstyles, doc.allCharacterStyles, "name");
 
    s1 = doc.swatches.length;
    for (i=doc.unusedSwatches.length-1; i>=0; i--) {
        doc.unusedSwatches[i].remove();
    }
    s2 = doc.swatches.length;
    
    alert("Removed "+(p1-p2)+" paragraph styles, "+
        (s1-s2)+" swatches from doc "+doc.name);
  
}
 
// Call cleanDoc on all documents in the book.
function main() {
    var
        docs = app.activeBook.bookContents.everyItem().fullName,
        i;
        
    for (i=0; i<docs.length; i++) {
        cleanDoc(app.open(docs[i]));
    }
}
 
main();

No warranty :-)