Show only colors already used in the document?

Hi!

Is it possiblé to see somewhere which colors I have previously used while working with a document? I think it would be nice to have the option for example for the swatches panel to only show user selected colors so that I can go back and forth in my 6-10 color palette really fast. It would be kind of a "color history". Is this possible to achieve somehow?