Beyond setting up a multichannel document there is no magic. And it is still merely an approximation because some colors can neither be simulated on-screen nor actually printed that way. That ultimately is the whole point - PS can't simulate anything that is impossibole in the "real world" and expressed as a color profile. Anyway, I have the feeling that you have a wrong understanding what spot colors actually are/ are meant for. They simply won't respond like the basic CMYK channels because their actual inks don't behave that way. Your ink percentages will never represent anything equivalent to a mathematical mixing formula as when using RGB or CMYK, instead they are the inks color multiplied by the opacity. In case of an on-screen simulation/ proof they would not even necessarily include effects that happen due to spectral absorption when light travels through the inks or the inks mixing chemically/ physically at the pigment level. It really doesn't exactly make sense for what you have possibly in mind due to the aforementioned specifics and limitations of the simulation. One could even go so far as to craft everything in red on screen and then swap the color for green during the printing process, assuming only 100% opacities are used. That is to say spot colors are often more considered like masks that constrain the actual ink to certian areas and when you think of it that way, you may begin to understand how and why your grand plan may fall apart. Of course we don't know what your artwork looks like, so it may work, after all, but I realyl think you are heading in the wrong direction, dreaming up something that doesn't exist.
You may try
Image > Mode Grayscale
Image > Mode > Duotone > Quadtone
Here you can mix four spot colors visually.
For first tests one shouldn't use spot colors which are out-of gamut
for sRGB. This and the result can be checked by
Image > Mode > Lab
View > Proof Setup > Custom > sRGB
View > Proof Colors
View > Gamut Warning
Best regards --Gernot Hoffmann
I like exploring the realm of impossibility! But, I do understand what you're getting at. I'm actually looking for a way to slighty alter the colour of the CMYK channels to more accurately represent the CMYK that I am achieving with fabric dyes.
Synthetic dyes come in 4 main colors [turquoise, fushia, lemon, black, or their version of CMYK] and all other colors are mixed from those. Now, most of the time, colors are just thrown together in rough estimations of what percentage of each would be necessary to achieve a desired color. What I've done is dyed 4 pieces of fabric, each one of the base colors. I've matched the subsequent colors to 4 pantone colors, that are slightly different than the CYMK process colors.
Pantone 306 C for Cyan
Pantone 227 C for Magenta
Pantone 101 C for Yellow
Pantone 433 C for Black
So, really what I'd like to do is just adjust the CMYK process colors to match the colors above so that I can [theoretically] "mix" dyes on screen to show me what color will come of it. Doing this would also provide me with the percentages of each so that I could calculate the percentage of each dye required.
The only thing I can think to do is to use four solid color adjustment layers, each with your pantone color, and all set to multiply blend mode.
Selecte the layer masks for each one and use apply image to apply the actual CYMK values from the base (normal) layer. Once you get all the channels assigned to the layer mask, you can tweak the tones of the masks with the normal tools: levels, curves, etc. Not totally sure about measuring the percentage; however, I would imagine that you could read the mask black values to get your percentages.
P.S. I just tried this, and you need to invert the layer mask. I also agree with Mylenium about this being a total hit and miss - mostly miss process.
How about using old-fashioned, pre-ICC-profile colour management settings?
Go into your Color Settings and select Custom CMYK as your CMYK working space. Under Ink Colors, select Custom, set the four colours by eye, and check Estimate Overprints. It won't be very accurate, and would definitely be better done by using colour profiles, but I guess you're not expecting too much accuracy. And remember to change your settings back to something more appropriate for litho print work, if you do any.
Awesome! Thank you @DannyWhitehead. I used a Panton to RGB/CMYK coversion tool to get the closest approximations of the colors, plugged in the HTML color codes and had it approximate the overprints.
Seems to be working as I would expect something like this to work. I had, at one point, gone into the color settings and tried to do it that way but didn't realize that I could switch the the "Ink Colors - Swop" section to "Customize".
Thanks everyone for their responses!