Skip navigation
Royi A
Royi A
Currently Being Moderated

Pixel Bender Notes

Sep 8, 2010 1:45 AM

Hello.

I've been using Pixel Bender for over a month now.

Implementing some Image Processing Algorithms into it (Advanced Sharpening, Blurring, NR etc...).

 

I have some notes regarding it to my special usage - Implementing Algorithms for Photoshop / After Effects.

I'm not a programmer so I thought this would be the right tool for me, just taking care of the Algorithm.

I don't have a deep knowledge of what is possible using GPU's and what's not, All written here is in the spirit of the limitations you made clear in the PDF's of PB. Anyhow, Looking on the supported GPU's by you it seems you support pretty old GPU's. Since it's an experimental project for now and the big changes GPU had in the last 2 years you might want to reconsider dropping support for anything below DX10 to give PB all the features and flexibility it should have.

 

The Notes:

1. Hidden Codes - I know it is possible (Oil Painting Plug In). This should be a first priority. If I got is write, PB Kernels for Flash are compiled into some kind of a binary form. Let's use that, Let the Photoshop / AE Plug In read those files.

 

2. CPU Precomputing - There are some computations which suit the CPU much better than the GPU. Could the "Calculate Dependencies" concept be expanded? Let's say compute some (Small) data which later be used by Pixel Bender. A simple case would be Face Recognition. Let's say I could create a Gray Mask (Faces would be brighter grays the rest black...) on the CPU and then this mask would be passed as a source image to pixel bender. The CPU procedure could be created using AS3 or something. I know it is easy to do in Flash, yet what about Photoshop / After Effects?

 

3. Static Node / Loop of Nodes- Let's say I convert the image into LAB Colorspace. Now I apply an Algorithm, Convert it back into RGB and that's it. Let's say the user have 3 sliders controlling some parameters of the Algorithm applied on the LAB image. Now each time the user moves a slider the whole process is being calculated. The solution should be letting us define "Static Nodes", Which means they are calculated only once. Their result is saved for later use. They won't be regenerated. In the example above, The node which Convert the image into LAB would calculated once and that's it. This is a simple example, but there are decompositions of images which are very demanding and needed to be calculated only once. A generalization of this would be Looping over nodes. Which means do this node till something happens. This would make iterative algorithms possible in PB.

 

4. Changing Graphs Connections on The Fly - In a later note I'll ask for an UI improvements - Radio Buttons. Let's say I built a Graph with 2 different Local Contrast Algorithms. I allow the user to chose one of them (Radio Buttons). I want to change the graph connections according to his choice. The way things now I have to calculate both and use a boolean variable to chose one of them.

 

5. Performance Measure - We need some way to measure if the optimizations we do really improves performances. I know there's an option to see the FPS. I don't how accurate this is. Could be other way?

 

6. Image Data - I'm pretty sure it would be easy to pass from Photoshop / AE into PB some basic information about the Image. Dimensions, Min Value, Max Value, Mean Value, etc...

 

7. Built In Functions - Colorspace Conversions (LAB / HSL).

 

8. Radio Buttons.

 

9. Vector Casting - Casting from Bool / Int / Float Vectors into Bool / Int / Float in one line. Sometimes it is needed to cast each item in the vector individually.

 

10. Defining DOD() - Sometimes Auxiliary images are needed. Let us create a DOD(). Define an array of RGBA in the size we want. This would make Resizing algorithms possible in PB. This could be even some kind of Auxiliary matrix for various calculations. Moreover, it's needed for many other uses (Building Matrices for other calculations such as Least Squares methods etc...).

 

11. Sample Nearest Mode B - In the current method ("Sample Nearest Mode A") if the Kernel access out of the DOD() pixel it gets (0, 0, 0, 0) as a result. In Mode B I would like it to actually return the value of the nearest pixel (Euclidean Distance), Basically "Padding" the image with its border. This would result in a much better Convolutions results.

 

 

That's it for now.

Would love to hear your opinions.

2119 Views   9 Replies   Latest reply: Royi A, Sep 23, 2011 3:47 AM
 
Replies
  • Royi A
    Currently Being Moderated
    1. Royi A,
    Sep 14, 2010 12:22 PM   in reply to Royi A
    Report

    Kevin / Bob, I would really be happy to hear your opinions.

     

    Anyhow, got one more thing to add:

     

    12. Advanced Math Tools - Many advanced algorithms requires advanced Math Tools. Could you create the flexibility needed to run them. Today there are many ways to run highly parrarlized algorithms of Matrix Algebra and Optimizations problems on the GPU:

         http://http.developer.nvidia.com/GPUGems2/gpugems2_chapter44.html

         http://docs.google.com/viewer?url=www.cs.columbia.edu/cg/pdfs/28_GPUSi m.pdf

     

    Just give us the basic flexibility to run those kind of algorithms.

     
    |
    Mark as:
  • Royi A
    Currently Being Moderated
    2. Royi A,
    Oct 4, 2010 8:26 AM   in reply to Royi A
    Report

    Got another one - Lightroom Support.

     
    |
    Mark as:
  • Kevin Goldsmith
    Currently Being Moderated
    3. Kevin Goldsmith,
    Oct 5, 2010 12:15 PM   in reply to Royi A
    Report

    That one you should ask the lightroom team for. We can tell them they need Pixel Bender, but if they hear it from their customers, it will carry a lot more weight...

     
    |
    Mark as:
  • Kevin Goldsmith
    Currently Being Moderated
    4. Kevin Goldsmith,
    Oct 5, 2010 12:51 PM   in reply to Royi A
    Report
    1. Do you mean protecting your Pixel Bender filters? We know that this is a priority for developers and are working on some ideas that would allow you to do this. The difficult part is getting something that works across multiple Adobe applications in a simple way. A binary representation of Pixel Bender files wouldn't be sufficient because it could get decompiled simply. We're still thinking through this.
    2. This is a good suggestion. We have been thinking about ways of combining more general purpose languages with Pixel Bender that would still be high-performance.
    3. The static node case is already handled within the Pixel Bender run-time. We will cache values at a node and will not update that node unless one of its parameters changes or one of the nodes connected to its inputs change. Looping is difficult to handle with region reasoning logic, but we are definitely aware that it would be useful to support.
    4. Good suggestion. This has come up before as a request. Again, it is a region reasoning issue, but it is certainly do-able.
    5. Testing GPU performance is particularly problematic as it is an external processor. Our GPU FPS counts are currently approximations. Our CPU performance measurements should be completely accurate however. I've thought in the past about specific performance testing options that would be similar to how my team does performance measurements of filters, but it seemed like an obscure feature for most users. Glad to hear that you would be interested.
    6. There is the ability in Pixel Bender now to tag a float2 parameter as corresponding to the input size of the original image. The toolkit supports it, but AE and PS do not currently. We're working with them to support that. Something to remember though is that the original image size may not correspond to the image that you are receiving at your inputs because of region reasoning. Min/Max/Mean values are good ideas. There is the notion of the original image versus the image at the inputs of the filter however. Need to think about this a bit more.
    7. Yup. Agreed.
    8. We currently have the enum parameter semantic hinting metadata. In the toolkit I show that as a drop-down, but it could be shown as radio buttons.
    9. Good suggestion.
    10. Interesting suggestion.
    11. Padding or clamping? GPUs support clamping (sampling off the image gives the color of the row or edge pixel). That would be more possible than actually computing distance. That could be a big performance hit.
     
    |
    Mark as:
  • Kevin Goldsmith
    Currently Being Moderated
    5. Kevin Goldsmith,
    Oct 5, 2010 12:52 PM   in reply to Royi A
    Report

    we're definitely looking at that.


     
    |
    Mark as:
  • Royi A
    Currently Being Moderated
    6. Royi A,
    Oct 6, 2010 8:33 AM   in reply to Kevin Goldsmith
    Report

    Hello Kevin.

    Thank for the detailed response.

     

    1. For now, even a Binary Format is better than nothing. If it is doable at the near future (Month or so) It would be great till you get the "Optimal Solution".

     

    3. Caching or "Calculate Only Once" flag would be great. Looping would be awesome. I don't have any knowledge in GPU's but since I can do it manually (Create few nodes which refer to the same kernel) couldn't it be done naively - I write the number of loops and the program which translates my code into the GPU code just create "On The Fly" nodes of the same kernel. I guess I'm missing something.

     

    5. If people wanna do "Heavy Filtering" Performance Measurement" is a must. A tool for that would be great. If it makes it easier it doesn't have to be in the IDE.

     

    6. The Image size is important for any "Generalized Convolution" Kernel. Just to know when you pick something outside. There's a Bypass, a Symmetric Continuation of the Image is easily done. As long as the number is consistent with the XY plane you created in PB that's great (Means if I have a Width of 2048 I know that (2047.5, y) is the right most pixel). Any other info on the image is great. (Min Max for every Channel, Mean etc...).

     

    10. There are many Algorithms for Calculations on the GPU (Least Squares, Conjugate Gradients etc). I saw some Cg code which creates a Texture or something like that (I guess that's just an Image) of arbitrary size. Then you use it as it was a Matrix and calculate whatever you want. Since dod() is the native way to store data in PB (and only thing which could be changed in the EvaluatePixel phase) it would be great to be able to create arbitrary one for auxiliary computations. Unless there are other ways you may hint of.

    http://http.developer.nvidia.com/GPUGems2/gpugems2_chapter44.html

    This is what I meant.

    They create arbitrary sized Textures (Each is a dod() or something, isn't it?).

    They loop over it.

    They mention DirectX 9.0C compliance. You support older hardware I think. Might be that the reason?

     

    11. I'm not familiar with Clamping. I meant Padding as in Matlab notation. Basically, Let's say we have image of 2048 x 2048. I use a Convolution mask of 9 x 9. Let's say the current Pixel is (0.5, 0.5) . I try to access (-3.5, 0.5). In regular "sampleNearest" I get (0, 0, 0, 0). What I would like to have back is the value of pixel (0.5, 0.5). Think of it as padding the image with the values of the pixels at the borders of the image. Look at this:

    http://www.mathworks.com/help/toolbox/images/ref/padarray.html

    Something like "Replicate" mode.

    I don't know if it's possible, yet it is useful.

     

    I posted a request on the Lightroom forum about PB filter.

     

    Thank You!!!

     
    |
    Mark as:
  • Royi A
    Currently Being Moderated
    7. Royi A,
    Oct 29, 2010 11:29 AM   in reply to Royi A
    Report

    13. Random Number Generator (Uniform and Gaussian).

     

    Kevin, Did you get the previous post of mine?

    Thanks.

     
    |
    Mark as:
  • lewiscarlton1
    Currently Being Moderated
    8. lewiscarlton1,
    Jun 26, 2011 7:09 AM   in reply to Royi A
    Report

    Those are some very helpful tips for Pixel Bender, thanks Rovi!

     

     

    Regards,

    Lewis Carlton

    SEO Myths

     
    |
    Mark as:
  • Royi A
    Currently Being Moderated
    9. Royi A,
    Sep 23, 2011 3:47 AM   in reply to lewiscarlton1
    Report

    Thanks.

    Yet it seems all those suggestions are left untouched.

     

    PB could be great.

    I wish they start working on it again.

     
    |
    Mark as:
Actions

More Like This

  • Retrieving data ...

Incoming Links

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points