This is an art/design question, not really a Photoshop question. "Subtle" is not one of the physical parameters of color. It is subjective and situational. Basically, any color is defined by 3 physical parameters: Hue, Value, and Saturation. Hue refers to its position on the color wheel. Value refers to its position on a black to white scale, or equivalent brightness. Saturation refers to purity/intensity vs a neutral gray of the same value, in other words, how much red is in this red? How "watered down" or "greyed out" is it? These 3 factors are theoretically capable of defining any color, but while absolutely necessary to know about, they will not help you to achieve subtlety. To approach "Subtle," we need to understand that what makes one color stand out or "pop" in relation to another is Contrast, and there are several forms of contrast: for example, warm/cool, light/dark, weak/intense are forms of contrast. To achieve subtlety, lessen contrast and use simple, pleasing color schemes. Good luck, and just know that the above items require years of study to understand, even by talented artists and designers.
