Skip navigation
Currently Being Moderated

VertexShader - Parameter / Matrix Issues

Aug 23, 2011 5:34 AM

I'm trying to implement a simple mask-shader for my 2D engine and I'm having multiple issues getting it to work.


Basically the shader takes two images and two matrices and tries to map back the vertex position of the sprites vertex back to the local mask space position and then calculate the local uv-coordinates for the mask, but I'm stuck with some basic calculations here.


Bug 1: Divisions in the material-vertex kernel don't work at all. That's why I have to put invertedMaskSize (which is 1.0 / 128.0) into the shader (but we already know that )

Bug 2: interpolated vars have to be float4, float2 is not possible (That's a pretty old bug as well)


I tried the following changes in the shader. The shader posted here, just tries to display the resulting uv-coordinates. No textures are used.


Case 1:

interpolatedMaskUV = float4((vertexPos.x + halfMaskSize.x) * invertedMaskSize.x, (vertexPoss.y + halfMaskSize.y) * invertedMaskSize.y, 0.0, 0.0);

The output is this: Just like you expect! Perfect, let's proceed.


Case 2:

Change the halfMaskSize and invertedMaskSize to float2 and set set the parameters as two vectors of length two of course in AS. The output:


Case 3:

Masking Test, matrix multiplication. First calculating the world space position of the vertex:


float4 worldSpacePos = float4(vertexPos.x, vertexPos.y, 0.0, 1.0) * objectToClipSpaceTransform;


Then mapping it back to the local space of the mask:

float4 localMaskSpacePos = worldSpacePos * maskObjectToClipSpaceTransform;


And calculating the uv-coords:

interpolatedMaskUV = float4((localMaskSpacePos.x + halfMaskSize.x) * invertedMaskSize.x, (localMaskSpacePos.y + halfMaskSize.y) * invertedMaskSize.y, 0.0, 0.0);


For testing, I set the maskObjectToClipSpaceTransform to the inverse of the objectToClipSpaceTransform. In theory and on paper, this should work.

But, I think, something gets out of order and maybe the maskObjectToClipSpaceTransform is screwed up in the shader, just like when I set the halfMaskSize and invertedMaskSize to float2. The result is this: and I have no idea how to fix this...


<languageVersion : 1.0;>
material kernel texture
    namespace : "ND2D_Shader";
    vendor : "nulldesign";
    version : 1;
    input vertex float2 uvCoord
        id : "PB3D_UV";


    input vertex float2 vertexPos
        id : "PB3D_POSITION";


    parameter float2 uvOffset;
    parameter float4x4 objectToClipSpaceTransform;
    parameter float4x4 maskObjectToClipSpaceTransform;

    // if set to float2, strange things happen

    parameter float4 halfMaskSize;
    parameter float4 invertedMaskSize;


    interpolated float4 interpolatedUV;
    interpolated float4 interpolatedMaskUV;


    void evaluateVertex()
        // not used in the current test ...

        interpolatedUV = float4(uvCoord.x + uvOffset.x, uvCoord.y + uvOffset.y, 0.0, 0.0);


        float4 worldSpacePos = float4(vertexPos.x, vertexPos.y, 0.0, 1.0) * objectToClipSpaceTransform;

        // doesn't work as expected

        float4 localMaskSpacePos = worldSpacePos * maskObjectToClipSpaceTransform;


        interpolatedMaskUV = float4((localMaskSpacePos.x + halfMaskSize.x) * invertedMaskSize.x,
                                    (localMaskSpacePos.y + halfMaskSize.y) * invertedMaskSize.y,
                                     0.0, 0.0);


    input image4 textureImage;
    input image4 textureMaskImage;
    parameter float4 color;


    output float4 result;


    void evaluateFragment()
          // just visualize the uv-coords

          result = float4(interpolatedMaskUV.x, interpolatedMaskUV.y, 0.0, 1.0);


        float4 texel = sample(textureImage, float2(interpolatedUV.x, interpolatedUV.y), PB3D_2D | PB3D_MIPNEAREST | PB3D_CLAMP);
        float4 texel2 = sample(textureMaskImage, float2(interpolatedMaskUV.x, interpolatedMaskUV.y), PB3D_2D | PB3D_MIPNEAREST | PB3D_CLAMP);


        result = float4(texel.r * color.r,
                        texel.g * color.g,
                        texel.b * color.b,
                        texel.a * color.a * texel2.a);


I know that we're working with a four month old version of pb3d and I hope that a new version will be out soon and maybe all these bugs I encountered are already solved, but if not.... here's another shader to fix

  • Currently Being Moderated
    Aug 24, 2011 11:09 AM   in reply to .lars

    First of all, thank you for posting such an informative description of your problem. Having this much detail helps us to track down what's going on. We have fixes for some of the bugs you've encountered, and we're working on the other bugs. We'll make sure this gets added to our test suite.


    We're working on a new release which we plan to have out soon (weeks rather than months).



    Mark as:
  • Currently Being Moderated
    Aug 25, 2011 10:42 AM   in reply to .lars

    One of the problems we have is that PB3D is built on top of Molehill which has severe hardware restrictions, particularly around the number of registers that are available for use. Once you start using float4x4, particularly in conjunction with arrays you run out of registers almost immediately (there are only 8 temporary registers available - since a float4x4 uses of 4 of them you can see that we're squeezed really tight).


    This means we haven't been focussing on array handling, so you're probably stuck with a clunky method of getting values into arrays for the next release.



    Mark as:
  • Currently Being Moderated
    Sep 1, 2011 10:25 AM   in reply to .lars

    I just discovered one more issue when I was working through your example. If you have a parameter that's used by the vertex kernel, and a parameter that's used by the evaluateVertex function of the material kernel they must have different names, even if they are supposed to be the same value. The particular parameter that gave me trouble is objectToClipSpaceTransform. I'm using it in my standard vertex kernel, but it's also used in your example as an input to the evaluateVertex function.


    We haven't yet worked out what the right way to deal with this situation is, for the time being, the work around is to make sure that the parameters have two different names.



    Mark as:
  • Currently Being Moderated
    Aug 24, 2013 7:22 PM   in reply to .lars

    This may not be the answer to .lars' issue, and may be totally obvious to some, but for readers who, like me, were stuck wondering why offset UVs weren't working when the offset was applied within the evaluateVertex() function of the material kernel ( as opposed to the evaluateVertex() of the vertex kernel ):


    You need to pass the UV offset parameter ( aka. constant register ) to the VERTEX shader, NOT to the FRAGMENT shader, even though you are targeting the evaluateVertex() of the material kernel.


    In other words, use:


    com.adobe.pixelBender3D.utils.ProgramConstantsHelper::setNumberParamet erByName(  Context3DProgramType.VERTEX, "myUVOffset", Vector.<Number>( [ value1, value2 ] ) );



    P.S. My question to Adobe: did we really need to split the vertex shader in two: on portion in the vertex kernel and the other in the material kernel? ..  It seems like a really odd choice, making things unnecessarily more complicated.

    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points