13 Replies Latest reply on Aug 17, 2012 7:23 PM by Autumnmantis

# Luminance Formulae and Red Distortion

I'm working on a chart that displays and names a couple hundred color swatches.  I have all of their color data in a table, and I want to arrange them by hue and lightness, so I applied the Rec.709 coefficients to my data.  This arranges my colors into a natural gradient--except for the reds, which have some glaring missplacement.  Any idea why .709 coefficients do this, or what coefficients I should use instead?

• ###### 1. Re: Luminance Formulae and Red Distortion

Your observation is correct, like here in chapter 7:

http://www.fho-emden.de/~hoffmann/gray10012001.pdf

It's the Helmholtz-Kohlrausch effect/phenomenon.

The effect is taken into account in Mark Fairchild's

Color Appearance Models.

Using Lab (CIELab) or even Munsell wouldn't improve the

situation, in my opinion. For me, saturated red/orange

appears brighter than gray or blue in planes of constant

Lightness (Lab) or Value (Munsell).

http://www.fho-emden.de/~hoffmann/cielab03022003.pdf

http://www.fho-emden.de/~hoffmann/munsell15052009.pdf

Best regards --Gernot Hoffmann

• ###### 2. Re: Luminance Formulae and Red Distortion

Okay, so it'll take me a while to digest all of that, but thanks in advance just for knowing what I'm talking about!  I've asked this question to a half dozen people and only made it to a shoulder shrug.

• ###### 3. Re: Luminance Formulae and Red Distortion

So if I want to generate values for lightness in a table the way Photoshop calculates its greys, I should use the .907 primaries but with a Gamma of 2.2?

Y = 0.2126*R^2.2 + 0.7152*G^2.2 + 0.0724*B^2.2

I'm still missing something, because I can't make sense of my resulting values.

• ###### 4. Re: Luminance Formulae and Red Distortion

Thanks to Prof Hoffman as usual. I am glad he is again contributing to the forum.

• ###### 5. Re: Luminance Formulae and Red Distortion

I think I've cracked it now.  Most of the formulae you see online are missing the gamma.  The formula should be:

Y = (0.2126*R^2.2 + 0.7152*G^2.2 + 0.0722*B^2.2) / 772.4106

• ###### 6. Re: Luminance Formulae and Red Distortion

Autumnmantis and Lundberg,

thanks for the feedback. In my opinion there are two mistakes:

a) wrong exponent (G instead of 1/G)

b) wrong concept - the ordering sequence does not depend on

gamma encoding.

This formula delivers the CIE(1931) luminance Y in linear coordinates,

specifically for Rec-709 primaries and D65 white:

r=0.2126

g=0.7152

b=0.0722

Y = rR + gG + bB

Gamma encoded values are characterized by an apostroph, but we

have to apply an inverse exponent k=1/G = 1/2.2, because the monitor

applies a transform with exponent G to data (rule of thumb).

R'=R^k

G'=G^k

B'=B^k

Y' = rR' + gG' + bB'

Now let's assume, we have sorted some colors by their luminances:

Y1<Y2<Y3...

How would the gamma encoded numbers be sorted?

Probably exactly  in the same order (I thought I had proved this

mathematically, but who knows):

Y1'<Y2'<Y3'

Therefore no gamma encoding can cure the Helmholtz-Kohlrausch-

"problem" - sorting by perceived lightness (by brightness).

There are other problems - the perception  of a color depends

much on the environment.

Autumnmantis, what exactly is your application?

Best regards --Gernot Hoffmann

• ###### 7. Re: Luminance Formulae and Red Distortion

Professor Hoffmann,

Thanks again for your taking the time.

Changing the gammas absolutely changes the order of the values.  The distortion I was originally experiencing is due to the fact that the linear coefficients give an intense value (say 255,0,0) a lower Y than a moderate shade of the same hue (say 175,30,30).  I don't fully understand the science behind gamma correction, but mathematically, if we want to boost the values of intense hues relative to others, the exponent must be greater than one; using 1/2.2 makes the problem worse.

For example, I have three values in my table: red (229, 0, 0); cardinal (178.6, 28.8, 52.8); and oxblood (122.1, 32.1, 41.5).  Red is clearly the lightest, oxblood is clearly the darkest, and desaturating programs across different software and monitors agree.

Ranked light to dark by the Average method, we get Cardinal>Red>Oxblood

With the linear .709 coefficients, we get Cardinal>Oxblood>Red

With a gamma of 1/2.2, we get Cardinal>Oxblood>Red

With a gamma of 2.2, we get Red>Cardinal>Oxblood.

The last formula works emperically for my data, but I suspect it should actually be a little smaller because my resulting data seems skewed to the extremes.

My application is really pretty trivial--I'm just making a table of color names.  But now I'm caught up in the intellectual exercise.  You can take a look at my old draft (using linear .907 coefficients) here.

• ###### 8. Re: Luminance Formulae and Red Distortion

Things you might want to read:

http://www.poynton.com/notes/colour_and_gamma/GammaFAQ.html

http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html  Particularly section 9.

You have gamma encoded RGB values, you need to convert those to linear values, then apply the weights to calculate Y.

• ###### 9. Re: Luminance Formulae and Red Distortion

Autumnmantis,

you're right, my assumption that sorting by linear luminance Y

or - alternatively - gamma encoded luminance Y' should deliver

the same sequence, is probably wrong. Thanks for insisting.

Anyway, there is - in the sense of CIE(1931) colorimetry - only one

correct approach, here for rec.709 and D65:

Convert the gamma encoded file data R', G', B' into data R,G,B

in the linear space X,Y,Z and calculate Y, the true CIE luminance:

Y = r R'^e + g G'^e + b B'^e  with exponent e=gamma=2.2.

The Helmholtz-Kohlrausch effect isn't  part of the CIE colorimetry,

therefore this sorting doesn't deliver a visually correct sequence.

According to Poynton (quoted below):

http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html#RTFToC11

in video technology a linear combination of R',G',B', called

Luma,  is used as a substitute for luminance Y:

The coefficients that correspond to the "NTSC" red, green

and blue  CRT phosphors of 1953 are standardized in ITU-R

Recommendation BT.  601-2 (formerly CCIR  Rec.  601-2).

I call it Rec.  601.

To compute nonlinear video luma from nonlinear red, green

and blue:

This well known formula is often applied to sRGB though it is based

on NTSC primaries and illuminant C whitepoint.

You may try sorting by Luma...

Here's another attempt how to sort colors for special purposes:

http://www.fho-emden.de/~hoffmann/palette30082003.pdf

Best regards --Gernot Hoffmann

• ###### 10. Re: Luminance Formulae and Red Distortion

Sorry, Professor, I didn't mean to be ingracious.  Thanks again for your help.

• ###### 11. Re: Luminance Formulae and Red Distortion

So my formula should be:

Y = 0.2126*R^2.2 + 0.7152*G^2.2 + 0.0724*B^2.2

and I am not supposed to use the inverse of the gamma (1/2.2) after all?

This seems to put my names in the right order, but it skews my data very dark (about one third of my data points wind up with less than 20% lightness).

Is it safe to assume that my RGB values are gamma-encoded? My values are based on component averages of responses to digital color swatches, so they haven't been converted into RGB from somewhere else.

At this point I'm tempted to just desaturate the image in PhotoShop and manually eyedrop each color into my data.  None of the formulas we've discussed in theory match the function in practice-- I don't mean to be a pain about it, I'm just surprised that it's such a mystery.

• ###### 12. Re: Luminance Formulae and Red Distortion

Yes 2.2.

Think about 50% gray -- you have a gamma encoded value of 127/255, that should map to a linear value near 18% (reflectance matching 50% perceived lightness).

127/255 ^ 2.2 = 0.216

• ###### 13. Re: Luminance Formulae and Red Distortion

Okay, that helps.  So then I can add all of the components and apply an exponent of 1/2.2 to get back to percentages (50%, in this case).

My final formula is then:

Y = (0.2126*R^2.2 + 0.7152*G^2.2 + 0.0722*B^2.2)^(1/2.2)

This will give me a scale to 255, obviously you can divide by 255 to get %.