Skip navigation
Currently Being Moderated

Pixel Bender Notes

Sep 8, 2010 1:45 AM

Hello.

I've been using Pixel Bender for over a month now.

Implementing some Image Processing Algorithms into it (Advanced Sharpening, Blurring, NR etc...).

 

I have some notes regarding it to my special usage - Implementing Algorithms for Photoshop / After Effects.

I'm not a programmer so I thought this would be the right tool for me, just taking care of the Algorithm.

I don't have a deep knowledge of what is possible using GPU's and what's not, All written here is in the spirit of the limitations you made clear in the PDF's of PB. Anyhow, Looking on the supported GPU's by you it seems you support pretty old GPU's. Since it's an experimental project for now and the big changes GPU had in the last 2 years you might want to reconsider dropping support for anything below DX10 to give PB all the features and flexibility it should have.

 

The Notes:

1. Hidden Codes - I know it is possible (Oil Painting Plug In). This should be a first priority. If I got is write, PB Kernels for Flash are compiled into some kind of a binary form. Let's use that, Let the Photoshop / AE Plug In read those files.

 

2. CPU Precomputing - There are some computations which suit the CPU much better than the GPU. Could the "Calculate Dependencies" concept be expanded? Let's say compute some (Small) data which later be used by Pixel Bender. A simple case would be Face Recognition. Let's say I could create a Gray Mask (Faces would be brighter grays the rest black...) on the CPU and then this mask would be passed as a source image to pixel bender. The CPU procedure could be created using AS3 or something. I know it is easy to do in Flash, yet what about Photoshop / After Effects?

 

3. Static Node / Loop of Nodes- Let's say I convert the image into LAB Colorspace. Now I apply an Algorithm, Convert it back into RGB and that's it. Let's say the user have 3 sliders controlling some parameters of the Algorithm applied on the LAB image. Now each time the user moves a slider the whole process is being calculated. The solution should be letting us define "Static Nodes", Which means they are calculated only once. Their result is saved for later use. They won't be regenerated. In the example above, The node which Convert the image into LAB would calculated once and that's it. This is a simple example, but there are decompositions of images which are very demanding and needed to be calculated only once. A generalization of this would be Looping over nodes. Which means do this node till something happens. This would make iterative algorithms possible in PB.

 

4. Changing Graphs Connections on The Fly - In a later note I'll ask for an UI improvements - Radio Buttons. Let's say I built a Graph with 2 different Local Contrast Algorithms. I allow the user to chose one of them (Radio Buttons). I want to change the graph connections according to his choice. The way things now I have to calculate both and use a boolean variable to chose one of them.

 

5. Performance Measure - We need some way to measure if the optimizations we do really improves performances. I know there's an option to see the FPS. I don't how accurate this is. Could be other way?

 

6. Image Data - I'm pretty sure it would be easy to pass from Photoshop / AE into PB some basic information about the Image. Dimensions, Min Value, Max Value, Mean Value, etc...

 

7. Built In Functions - Colorspace Conversions (LAB / HSL).

 

8. Radio Buttons.

 

9. Vector Casting - Casting from Bool / Int / Float Vectors into Bool / Int / Float in one line. Sometimes it is needed to cast each item in the vector individually.

 

10. Defining DOD() - Sometimes Auxiliary images are needed. Let us create a DOD(). Define an array of RGBA in the size we want. This would make Resizing algorithms possible in PB. This could be even some kind of Auxiliary matrix for various calculations. Moreover, it's needed for many other uses (Building Matrices for other calculations such as Least Squares methods etc...).

 

11. Sample Nearest Mode B - In the current method ("Sample Nearest Mode A") if the Kernel access out of the DOD() pixel it gets (0, 0, 0, 0) as a result. In Mode B I would like it to actually return the value of the nearest pixel (Euclidean Distance), Basically "Padding" the image with its border. This would result in a much better Convolutions results.

 

 

That's it for now.

Would love to hear your opinions.

 
Replies
  • Currently Being Moderated
    Oct 5, 2010 12:15 PM   in reply to Royi A

    That one you should ask the lightroom team for. We can tell them they need Pixel Bender, but if they hear it from their customers, it will carry a lot more weight...

     
    |
    Mark as:
  • Currently Being Moderated
    Oct 5, 2010 12:51 PM   in reply to Royi A
    1. Do you mean protecting your Pixel Bender filters? We know that this is a priority for developers and are working on some ideas that would allow you to do this. The difficult part is getting something that works across multiple Adobe applications in a simple way. A binary representation of Pixel Bender files wouldn't be sufficient because it could get decompiled simply. We're still thinking through this.
    2. This is a good suggestion. We have been thinking about ways of combining more general purpose languages with Pixel Bender that would still be high-performance.
    3. The static node case is already handled within the Pixel Bender run-time. We will cache values at a node and will not update that node unless one of its parameters changes or one of the nodes connected to its inputs change. Looping is difficult to handle with region reasoning logic, but we are definitely aware that it would be useful to support.
    4. Good suggestion. This has come up before as a request. Again, it is a region reasoning issue, but it is certainly do-able.
    5. Testing GPU performance is particularly problematic as it is an external processor. Our GPU FPS counts are currently approximations. Our CPU performance measurements should be completely accurate however. I've thought in the past about specific performance testing options that would be similar to how my team does performance measurements of filters, but it seemed like an obscure feature for most users. Glad to hear that you would be interested.
    6. There is the ability in Pixel Bender now to tag a float2 parameter as corresponding to the input size of the original image. The toolkit supports it, but AE and PS do not currently. We're working with them to support that. Something to remember though is that the original image size may not correspond to the image that you are receiving at your inputs because of region reasoning. Min/Max/Mean values are good ideas. There is the notion of the original image versus the image at the inputs of the filter however. Need to think about this a bit more.
    7. Yup. Agreed.
    8. We currently have the enum parameter semantic hinting metadata. In the toolkit I show that as a drop-down, but it could be shown as radio buttons.
    9. Good suggestion.
    10. Interesting suggestion.
    11. Padding or clamping? GPUs support clamping (sampling off the image gives the color of the row or edge pixel). That would be more possible than actually computing distance. That could be a big performance hit.
     
    |
    Mark as:
  • Currently Being Moderated
    Oct 5, 2010 12:52 PM   in reply to Royi A

    we're definitely looking at that.


     
    |
    Mark as:
  • Currently Being Moderated
    Jun 26, 2011 7:09 AM   in reply to Royi A

    Those are some very helpful tips for Pixel Bender, thanks Rovi!

     

     

    Regards,

    Lewis Carlton

    SEO Myths

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points