19 Replies Latest reply on Aug 4, 2010 3:16 PM by AIF Bob

    Image Processing Algorithms - From Matlab to Pixel Bender

    Royi A Level 1

      Hello.

      Got a few Image Processing (Mainly Image Enhancement) Algorithms I created in Matlab.

       

      I would like to make them run on Photoshop, Create a Plug In out of it.

      Would Pixel Bender be the right way doing it?

       

      The Algorithms mainly use Convolutions and Fourier Domain operations.

       

      All I need is a simple Preview Window and few Sliders, Dropbox and Buttons.

      I'd appreciate your help.

        • 1. Re: Image Processing Algorithms - From Matlab to Pixel Bender
          Kevin Goldsmith Level 3

          the convolutions are no problem, but FFTs are problematic. There have been successful attempts to do them using multiple passes of gather-model shaders (like Pixel Bender), but if you are not experienced in this domain it will be very challenging.

           

          Pixel Bender would drastically reduce the complexity of getting the filter running in Photoshop (or After Effects), but I'm not sure it would be the best tool for the job given your algorithms.

           

          We are working on some stuff that would make this easier, but I can't go into detail about that at this point.

          • 2. Re: Image Processing Algorithms - From Matlab to Pixel Bender
            Royi A Level 1

            Well, Actually I have no Experience in what you mentioned. I guess I'll have to read about Pixel Bender a little more.

            I can't create an array in Pixel Bender?

            I assumed I could which is all I needed.

             

            Could you suggest about the time frame of the tools you mentioned.
            What other options exists to do so?

             

            By the way, May I integrate scripting within a Pixel Bender Plug In?

            To make it clear, let's say I want to apply Gaussian Filter, Instead of doing "Manually" in Pixel Bender, May I use "Scripting" instead (Photoshop's Scripting Tool)?

             

            Thank You.

            • 3. Re: Image Processing Algorithms - From Matlab to Pixel Bender
              Kevin Goldsmith Level 3

              Pixel Bender has limited array support, but the output of a pixel bender program is a single pixel. you cannot write to multiple targets because all pixels can (and often do) run in parallel. This makes it unfriendly to traditional implementations of FFT.

               

              Within Photoshop the Pixel Bender plug-in is a smart filter and can be scripted.

               

                   Kevin

              • 4. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                Royi A Level 1

                OK, Well I hope the new release would be much more flexible.

                Anyhow, Where can I read about setting simple UI (Preview image, Sliders, etc) in Pixel Bender and using a Script with Pixel Bender.

                 

                Moeover, I saw in the Exchange section that it's easy to see the Algorithm.

                Is there a way to create a file which is complied so the Algorithm is hidden (Including the Scrip if used)?

                 

                Thanks.

                • 5. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                  Kevin Goldsmith Level 3

                  see: http://www.adobe.com/devnet/pixelbender/

                  There is a spec and developer guide linked from there. They are also included in the Pixel Bender Toolkit installer.

                   

                  We are working on some ways to make it easier for Pixel Bender developers to protect their algorithms if they choose, but there is not anything in place right now.

                  • 6. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                    Royi A Level 1

                    Well,

                    I searched through the guides.

                    Couldn't find a way where I can access Photoshop Filters within Pixel Bender.

                    I guess what you meant is I can access Pixel Bender from a Script.

                     

                    The question is will I be able to define variables for a Pixel Bender filter from a script.

                    How about the Preview window?

                     

                    Thanks.

                    • 7. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                      DavideBarranca Level 1

                      Hi,

                      I don't know how much this one will answer to your question, but I asked in this forum about the possibility to embed a PB kernel within a CS SDK panel - in short, a Flash extension that runs within Creative Suite applications as a panel. This way you could mix scripting (via ActionScript or ExtendScript) and call your binary PB code - by the way, you could access also shaders like Blur that are already in the Flash libraries without the need to code them again.

                       

                      But I got no answer, so I'm not really sure (neither I've ever tried) it's possible or not to encapsulate a PBJ within CS SDK - let's keep an eye on this ;-)

                      Cheers,

                       

                      Davide

                      • 8. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                        Kevin Goldsmith Level 3

                        you cannot currently access photoshop  filters from within Pixel Bender. I don't know enough about the PS Smart Plug-in SDK to know for sure if you can access the parameters of the Pixel Bender filter from the PS scripting environment, but I think it is possible. I also don't know if you can get image data in Panels in PS. If you could run image data through them, you would be able to do Pixel Bender operations since it is using a 10.x version of the player.

                        • 9. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                          Royi A Level 1

                          Just for the saying, How can apply a Gaussian Blur in PB the same way it's implemented in Photoshop?

                          The matrix size is so limited so it won't be practical for large Radius.

                           

                          Moreover, Let's say I just want to create a Blending Mode.

                          Is there a way to access 2 layers of Photoshop? Or am I limited only to the active one?

                           

                          I see here:

                          http://tv.adobe.com/watch/max-2008-design/pixel-bender-unleashed-by-ryan-taylor/

                          Around min 45 he uses library which isn't documented anywhere.

                          He uses it to apply a Blending Mode (Which I don't get how he 2 sources).

                           

                          Could you explain it?

                           

                          Thank you for your efforts.

                          I really think PB has a lot of potential. It just needs to get rid of some limitations.

                          • 10. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                            Kevin Goldsmith Level 3

                            you would use for loops, not a matrix. Variable radii blur are fairly easily written in Pixel Bender (for PS and AE).

                             

                            After Effects allows you to use multiple layers as inputs to a single pixel bender filter (allowing you to create your own blending modes). I submitted a request to Photoshop to enable something like this functionality for the next release.

                             

                            thanks for the conversation Royi!

                            • 11. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                              Royi A Level 1

                              Thank you for being so patient and helpful.

                              Once I'll be able to export the filters from Matlab into Photoshop you'll be credited for that .

                               

                              Well, Loops you say...

                              I guess I'm spoiled by Matlab :-).

                              If you mean this:

                              http://www.adobe.com/devnet/flash/articles/pixel_bender_basics_05.html

                              You'd agree with me that for radius of 25 it would be nasty.

                              Moreover, how would you deal when you "sample" pixels outside the boundaries without knowing it.

                               

                              What about the Library I mentioned, Could you explain it (Min 45:20 in Movies)?

                              It seems he passes 2 kernels over the same image.

                              I thought this could only be done using "Graphs".

                              Maybe it's just like a Function or something?

                               

                              If I may add one request for future releases.

                              Let us use ActionScript as a back end.

                              There are some things (Heuristic things mainly) I would really like to be able in a Photoshop Plug In.

                               

                              I know it is possible using the SDK, yet I want to focus on the Algorithm not the logistics and those IDE's let me do that.

                              Just more flexibility, that's all needed :-).

                               

                              Again, thank you.

                              • 12. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                                Kevin Goldsmith Level 3

                                I'm not sure when I'll get a chance to rewatch the clip from the talk that you mentioned, but Pixel Bender does support graphs. PBG files are XML representations of graphs with multiple kernels. I don't remember if he showed that in his talk, but I think he might have.

                                 

                                As far as actionscript on the back end. That is a bit beyond the scope of our stuff. More of a photoshop request. We'd need to figure out how to make Actionscript run on the GPU, which would be pretty impressive

                                • 13. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                                  Royi A Level 1

                                  Well, you don't need to watch the whole video.

                                  Let me show you:

                                   

                                  01.png

                                  02.png

                                   

                                  as you can see he creates a library.

                                  Later on he calls it just as it was a function or something.

                                  Is that just a set of functions?

                                   

                                  Another thing, I created a Gaussian Blur filter for arbitrary radius.

                                  I have issues in the "Edges" of the image.

                                  I get the result of convolution as the image is surrounded by '0' which is what the documentation say about accessing out of the image pixels.

                                   

                                  If I write a filter for Photoshop, how could I overcome it and get info about the image dimensions or any other trick to have the convolutions results as in Photoshop - The image is enlarged at the edges (Replicating the pixels)?

                                   

                                  Regarding graphs.

                                  Let say I created a graph with 4 sub kernels.

                                  The preview image would show the result of the last kernel, right?

                                   

                                  Thank you.

                                  • 14. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                                    Kevin Goldsmith Level 3

                                    thanks for posting the screenshots! That saves me some time!

                                     

                                    yeah, I forgot that he did that in his presentation. Pixel Bender supports the notion of function libraries, but we didn't document it well (at least in the most recent docs). People don't use libraries now because we don't currently allow you to reference functions or kernels in external files. We should improve the documentation there, but it won't really be useful until we allow you to reference other files.

                                     

                                    You are correct that if you read off the edges of the image, we will return transparent black for those pixels. If you want your blur to be edge aware you currently need to pass the size of the image in as a parameter.

                                     

                                    If you create a graph, the order of execution will depend on the structure of the graph. The entire graph is evaluated before the preview is updated, so whatever node is connected to the output will be the last node executed before the preview.

                                    • 15. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                                      Royi A Level 1

                                      AIF Kevin wrote:

                                       

                                       

                                      You are correct that if you read off the edges of the image, we will return transparent black for those pixels. If you want your blur to be edge aware you currently need to pass the size of the image in as a parameter.

                                      Is there a way doing so in Photoshop?

                                      It something you might be able to solve with built in convolution function (Maybe with 3 modes - Replicating the image, Periodically enlargement and the current state - Zero padded).

                                       

                                      Thank you.

                                      • 16. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                                        Kevin Goldsmith Level 3

                                        there is a way of expressing it in Pixel Bender, but we ran out of time to get it into the PS plug-in. It should be in the next version.

                                        • 17. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                                          Royi A Level 1

                                          If you need a beta tester for the next one, let me know :-).

                                           

                                          Anyhow, I need 2 more clarifications:

                                           

                                          • pixel vs float - Couldn't figure out what exactly is the difference if there is at all. I assume Pixel always get clipped into [0 1] and float won't until it gets to be shown on the screen as an output?

                                           

                                          • Regions - Let me see if I get is straight. For the example assuming Gaussian Blur Kernel of Radius 5 (Not the STD, but the radius - a 11x11 Matrix). I should use "needed()" in order to define the support of each pixel output in the input image. I should do it to make sure no one changes those values before the output pixel is calculated.
                                            Now, In the documentation is goes needed(region outputRegion, imageRef inputIndex). Should I assume that at default the outputRegion is actually the sampled pixel in the input? Now I use outset(outputRegion, float2(x, y)) to enlarge the "Safe Zone". I don't get this float2 number. Let's say it's (4, 3) and the current pixel is (10, 10). Now the safe zone goes 4 pixel to the left, 4 to the right, 3 up and 3 down? I assume it actually creates a rectangle area, right? Back to our example I should set outset(outputRegion, float2(5.0, 5.0)) right?

                                           

                                           

                                          Thanks...

                                          • 18. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                                            Kevin Goldsmith Level 3

                                            pixel and float are the same thing. We don't clamp pixels (for HDR applications).

                                             

                                            On the regions, I'm not 100% sure that I understand your question, but I think you do understand. If you are doing a square convolution kernel of radius 5 on an image input, you would outset that region by (5,5)

                                             

                                                 Kevin

                                            • 19. Re: Image Processing Algorithms - From Matlab to Pixel Bender
                                              AIF Bob Level 3

                                               

                                              • pixel vs float - Couldn't figure out what exactly is the difference if there is at all. I assume Pixel always get clipped into [0 1] and float won't until it gets to be shown on the screen as an output?

                                               

                                              There is no difference between them. At one stage of development we had some ideas about the way the pixel type should work that would make it different to float, but the ideas never came to anything and by the time we realized that it was too late to change. It's #1 on my list of "mistakes we made when developing Pixel Bender".

                                               

                                              • Regions - Let me see if I get is straight. For the example assuming Gaussian Blur Kernel of Radius 5 (Not the STD, but the radius - a 11x11 Matrix). I should use "needed()" in order to define the support of each pixel output in the input image. I should do it to make sure no one changes those values before the output pixel is calculated.
                                                Now, In the documentation is goes needed(region outputRegion, imageRef inputIndex). Should I assume that at default the outputRegion is actually the sampled pixel in the input? Now I use outset(outputRegion, float2(x, y)) to enlarge the "Safe Zone". I don't get this float2 number. Let's say it's (4, 3) and the current pixel is (10, 10). Now the safe zone goes 4 pixel to the left, 4 to the right, 3 up and 3 down? I assume it actually creates a rectangle area, right? Back to our example I should set outset(outputRegion, float2(5.0, 5.0)) right?


                                              Needed is the function the system calls to answer the question "what area of the input do I need in order to calculate a particular area of the output?".

                                               

                                               

                                              • I should do it to make sure no one changes those values before the output pixel is calculated.

                                               

                                              No, you should do it to make sure the input pixel values needed to compute the output pixel values have been calculated and stored.

                                               

                                              • Should I assume that at default the outputRegion is actually the sampled pixel in the input?

                                               

                                              No. When "the system" (i.e. After Effects, PB toolkit or the Photoshop plugin) decides it wants to display a particular area of the output, it will call the needed function with that area in the outputRegion parameter. The job of the needed function is to take whatever output region it is given and work out what input area is required to compute it correctly.

                                              • Let's say it's (4, 3) and the current pixel is (10, 10).

                                               

                                              Don't think in terms of "current pixel" when you're looking at the needed function. The region functions are not called on a per-pixel basis, they are called once at the start of computing the frame, before we do the computation for each pixel.

                                               

                                              • Back to our example I should set outset(outputRegion, float2(5.0, 5.0)) right?


                                              Yes - you're correct. Whatever size the output region is, you require an input region that has an additional border of 5 pixels all round to calculate it correctly.