You can rearrange, delete groups, etc. in the preset manager.
And you can already load/replace/append new files of swatches.
The only thing you requested that is missing in Photoshop is grouping swatches visibly in the palette (swatch sets in the UI).
Yes, well - if you read between the lines, the real thing I'm looking for here is consistency between applications. It's frustrating to a user to have a "Creative Suite" but each application handles swatches, and many other tools differently. In several cases VERY differently. Maybe you can manage stuff in Photoshop from the preset manager (I would consider myself an advanced Photoshop user - 8+ years - and I don't even know what the preset manager is), but InDesign and Illustrator each have different ways of editing and grouping swatches. In my opinion, Illustrator is the most friendly at handling swatches for my stated purposes overall.
And don't get that condescending tone with me boy!
There won't be complete consistency, becuase the applications work in very different ways. Photoshop's swatches are just a way to pick colors, not live objects connected to the artwork as in Illustrator and InDesign.
Also, each app has considerable history, and major changes to functionality aren't undertaken lightly.
And remember that each app addresses many different kinds of user: what may be appropriate to your workflow may hinder someone else's. (and we can't offer an infinite number of UI variants)
But that said, we have several requests to make Photoshop's swatches (and other presets) easier to manage. We have some ideas, but haven't had time to work on them yet.