For example most of my pictures are clustering in areas and avoiding small but powerful splashes of color. In full disclosure, we are making a color puzzle and attempting to auto select 4 colors from a photo but finding the various solutions a bit of a rabbit hole. Love everything about Adobe Capture App.
Within "Colors", you should be able to manually select which areas of the image to pull from. If the color pickers are moving too quickly, tap the screen and that should freeze the image. From there, drag the color pickers to the area you'd like to pull color from. Are you, perhaps, using the "Looks" tool? If so, go back to the main menu and choose "Colors" from the toolbar.
As for the algorithm, I'm not sure I can share that information
Let me know if this helps!