3 Replies Latest reply on Apr 1, 2009 5:21 PM by jbfiedler

    Color Conversion using FP10 vs PBToolkit

    jbfiedler Level 1

      I am working on a simple pair of shaders to convert an image from RGB to HSL and HSL to RGB the math of which can be found here. Although the filters convert perfectly in PBT(Version: 1.1.364221, Build 101) there is a large difference when used with FP10 (WIN 10,0,22,87 debug).

      The main test case: original image => rgb2HSL => hsl2RGB => original image, works in PBT but FP10 produces noticeable color shifts. I have PBT setup with all the FP settings on (warnings, errors, cpu, pre/unmultiply by alpha) and I believe that I have complied to the limitations of FP (no functions, no loops, etc). The only thing I can think of is that FP loads images 8-bit per channel rather than PBT's 32. Any help would be greatly appreciated. Thank you in advance.
        • 1. Re: Color Conversion using FP10 vs PBToolkit
          Kevin Goldsmith Level 3
          I'll see if I can get a chance to check this out in the player. The math in the player is done at 32 bpc, but you are right, it is expanded/compressed to 8bpc on the way in/out. I'd have to see how serious the color shifts were to see if that might be the culprit, it certainly wouldn't be at the same fidelity as in the PBT. There was also a bug in how we export min/max in PBJ that might be partly to blame. We'll be releasing a new toolkit with a fix for that issue very soon and you could try re-exporting the PBJ and seeing if the issue is fixed.
          • 2. Re: Color Conversion using FP10 vs PBToolkit
            AIF Bob Level 3
            Can you try replacing the min and max functions with their equivalents using if statements. So instead of

            r = min( a, b );

            you'd have

            r = a;
            if( b < a )
            r = b;

            and the similar construction for max.

            I am also a little worried that you're doing exact comparisons with floats - e.g.

            delta_Max == 0.0

            depending on the exact way in which delta_Max is caluclated (and that exact way is different betwen the flash runtime and the toolkit runtime) you might end up with a result that is exactly 0.0 in one case, and just a fraction off in the other case. The usual fix for this is to use something like:

            abs( delta_Max ) < epsilon

            where epsilon is set to whatever small value you think is appropriate for your application
            • 3. Re: Color Conversion using FP10 vs PBToolkit
              jbfiedler Level 1
              Thank you both for your posts. The problem was in the min and max functions. Once I replaced them with Bobs suggestion the shaders worked wonderfully.