I believe you have to sign in to your color.adobe.com account, select My Themes, hover over the color swatches and you'll see a trash can that says Delete. That should then be reflected in the InDesign panel.
Or, from Actions on My Themes, choose view online and you'll see the option to delete it.
Great Thanks! Both answers seem to work, but it's way easier to just choose view online.