22 Replies Latest reply: Jul 25, 2012 9:15 PM by Noel Carboni RSS

    'Super Resolution' Upscaling Algorithm

    Trevor.Dennis MVP

      This was linked from a flickr group, and I am not sure how new the paper is, but it is very impressive. I didn't even try to read or follow the complicated bits, but it compares various up-scaling methods with what they are calling a Super Resolution method, and the difference is amazing.

       

      http://www.wisdom.weizmann.ac.il/~vision/SingleImageSR.html

       

      ISTR that it was a couple of Weizmann Institute guys that first demonstrated Content Aware Scale/Fill, so - if that is right - they probably have a relationship with Adobe.  Something for the next Adobe Max perhaps?

        • 1. Re: 'Super Resolution' Upscaling Algorithm
          c.pfaffenbichler Community Member

          Interesting.

          As that paper is from 2008 I suppose it must have received some consideration at Adobe already.

          • 2. Re: 'Super Resolution' Upscaling Algorithm
            conroy Community Member

            Wow!!!

             

            Thanks for posting!

            • 3. Re: 'Super Resolution' Upscaling Algorithm
              JJMack MVP

              c.pfaffenbichler wrote:

               

              Interesting.

              As that paper is from 2008 I suppose it must have received some consideration at Adobe already.

              Adobe may have see it however I just did a test using one of the example on the web page the eye chart. the origanal upper left corner on to one of Adobe biliner to the right Adobe Bicubic Sharper right of thatr CS6 Bicubic Automatic then their with a blue border.

              res.jpg

              Png next if it works

              res.png

              • 4. Re: 'Super Resolution' Upscaling Algorithm
                Noel Carboni Community Member

                Some nice marginal improvements, but I don't know if it's "wow" material...

                 

                Anyone who's pondered the mechanics of JPEG compression has probably thought of using a database of imagery to "make up" missing detail, and there are already good commercial upsampling tools that can make good 400% enlargements.  An OnOne Software plug-in called Perfect Resize 7 is based on technology that recognizes recurring fractal patterns in imagery, and has been a commercial product (in the prior name of Genuine Fractals) for quite a long time.

                 

                The results shown on the Weizmann page seem only marginally better in some cases and a bit worse (with more artifacts) in others compared to Perfect Resize results.

                 

                Example:  Super-Resolution result vs. the OnOne Software Perfect Resize plug-in + some USM:

                 

                http://www.wisdom.weizmann.ac.il/~vision/single_image_SR/kitchen/res.png

                 

                http://Noel.ProDigitalSoftware.com/ForumPosts/PR7.png

                 

                -Noel

                • 5. Re: 'Super Resolution' Upscaling Algorithm
                  conroy Community Member

                  Noel, your kitchen image is an interesting example of how the OnOne method produces an extremely synthetic result, similar to an image which has had some kind of "watercolour paint" or "paint daubs" or "palette knife" filter applied. The Super Resolution result is softer than the OnOne but looks more like a photograph and less like a deliberately filtered image, in my opinion.

                  • 6. Re: 'Super Resolution' Upscaling Algorithm
                    Trevor.Dennis MVP

                    I was looking at the detail in the knitted material where what was little more than a blur with the other methods, clearly resolved each stitch with the SR method.  Yes there artefacts, but if you needed to use such a tool, your final medium would probably be a large print that would typically be viewed from far enough away to minimise the affect of those artefacts. 

                    • 7. Re: 'Super Resolution' Upscaling Algorithm
                      conroy Community Member

                      I hope nobody interpreted my stating that SR looked softer than OO with the kitchen scene enlargement as my finding fault with SR. The SR results are far sharper and detailed than the other methods and any slight loss in perceived sharpness when compared to OO is more than compensated for by the much more natural appearance of SR.

                      • 8. Re: 'Super Resolution' Upscaling Algorithm
                        Noel Carboni Community Member

                        I worry, though, with software that looks things up in databases and "fills in" the details with essentially false (or "best guess") information, where will this go...  Imagine a courtroom 50 years from now with a blown up security image clearly showing what looks like a great match to the defendant's face, when in fact the original security capture just didn't have enough information to convict...

                         

                        You might pooh pooh that idea, but I have heard fractal enhancement is already used in courtrooms.

                         

                        The additional complexity certainly seems to add some realism to the task at first glance.  But let's compare an image made by Bicubic upsampling then deconvolved (in this case with Maximum Entropy Deconvolution courtesy Astra Image), and combined with some parts from a fractal-upsampled image to reduce jaggies in key areas.  Which of these seems less plastic / made-up to you?

                         

                        UpsamplingComparison2.jpg

                         

                        -Noel

                        • 9. Re: 'Super Resolution' Upscaling Algorithm
                          Chris Cox Adobe Employee

                          Welcome to the many reasons why this is still an active field of research.

                          None of the algorithms are great, but are improving.

                          • 10. Re: 'Super Resolution' Upscaling Algorithm
                            Lundberg02 Community Member

                            Since the kitchen scene is blown out, what difference does it make [message edited]?

                             

                            Message was edited by: TerriStoneCHL

                            • 11. Re: 'Super Resolution' Upscaling Algorithm
                              Noel Carboni Community Member

                              Argh, I spelled deconvolved wrong above.  [fixed now]

                               

                              Lundberg, you're certainly critical.  Feel free to put up what you consider a good image to play with.

                               

                              -Noel

                              • 12. Re: 'Super Resolution' Upscaling Algorithm
                                conroy Community Member

                                Sure, the Super Resolution looks softer with less etched wrinkles in the face when the images are scaled down on the forum Web page, but when viewed full size, I find the jagged large-pixel appearance of the fractal plus deconvolution to be extremely unappealing. And you said the jaggedness has been reduced in the image on the left! How bad was it?

                                 

                                UpsamplingComparison2-detail.png

                                • 13. Re: 'Super Resolution' Upscaling Algorithm
                                  Noel Carboni Community Member

                                  More texture in the skin, more jaggedness.  Less texture, smoother wrinkles.  For this subject, while both enlargements leave something to be desiired, I find the left image looks slightly more natural because it has more texture.

                                   

                                  I reduced jaggedness around the edges.  The fractal resampling is good at smoothing rough edges.

                                   

                                  This Super Resolution stuff has promise, no question, but it's not really groundbreaking if existing methods can get in the same ballpark.  It's a tool I'd love to have in my toolbox though. 

                                   

                                  -Noel

                                  • 14. Re: 'Super Resolution' Upscaling Algorithm
                                    conroy Community Member

                                    More texture in the skin, more jaggedness.

                                     

                                    The enhanced jaggedness in the left image isn't natural - it's regular like a pixel grid. That's the aspect which made may say that I find the image to be unappealing.

                                    • 15. Re: 'Super Resolution' Upscaling Algorithm
                                      Noel Carboni Community Member

                                      Clearly it's just over your distraction threshold, and just under mine. 

                                       

                                      -Noel

                                      • 16. Re: 'Super Resolution' Upscaling Algorithm
                                        conroy Community Member

                                        Here's what I'm talking about, although I'm surprised that the enhancement is necessary. Look at the orthogonal lines and huge stairsteps on the left.

                                         

                                         

                                        jaggies.png

                                        • 17. Re: 'Super Resolution' Upscaling Algorithm
                                          Noel Carboni Community Member

                                          I knew what you were talking about.  Like I said, it didn't bother me.  I could probably do it better to avert more of that by combining the deconvolved data with the fractal data differentlyi.

                                           

                                          -Noel

                                          • 18. Re: 'Super Resolution' Upscaling Algorithm
                                            Trevor.Dennis MVP

                                            Noel Carboni wrote:

                                             

                                            I worry, though, with software that looks things up in databases and "fills in" the details with essentially false (or "best guess") information, where will this go...  Imagine a courtroom 50 years from now with a blown up security image clearly showing what looks like a great match to the defendant's face, when in fact the original security capture just didn't have enough information to convict...

                                             

                                            -Noel

                                             

                                            50 years?  They do that now, and I know it's true because I have seen in TV progs like 24 and CSI. 

                                             

                                            But being serious, it was surely not using look up tables when smoothing out the jaggies in the marble statue. Where bi-cubic was creating the 'affect' of straight lines by bluring them with aliasing, the SR has clean, sharp, high contrast lines.  I agree it was not so clear cut with skin tones, and I imagine that's the rub of the matter.  It kind of takes you back to magical debur demonstration at Adobe Max, or even third party applications like Topaz inFocus.  You see them demonstrated on best case subject material, which is not the same as real world subject matter.  I could never get the Topaz inFocus plugin to come even close to the demonstration on the Topaz site.  

                                             

                                            As I have said before, I treat such things as black box tecnology and leave it up to clever folk like Chris to make it happen.  I am just happy to use them when they become reality.

                                             

                                            [EDIT]  Not that anyone cares or noticed, but I have not been posting much recently because of a couple of big video projects that had to be done by this weekend.  Then I am overseas for a month, trying to keep clear of the Olymics in the UK. 

                                            • 19. Re: 'Super Resolution' Upscaling Algorithm
                                              Lundberg02 Community Member

                                              Let's see, Noel is the guy who upsamples everything, apparently in the belief that it causes no problems. Did you upsample before doing your examples?  In regard to super resolution, what value would it have in practice except for recon photos or maybe astronomy? Certainly no portrait or publicity work.  What rescue function does it have that deconvolution doesn't do better?

                                              • 20. Re: 'Super Resolution' Upscaling Algorithm
                                                Noel Carboni Community Member

                                                LOL Lundberg, why does what you write sound like a taunt?  You write, apparently in the belief that you know more than your audience.

                                                 

                                                Oh, and just so you know, some people actually DO work on astroimages...  This is a small crop from the very latest I've been working on, from a study of carbon stars...

                                                 

                                                NGC6871_Mosaic_Small_Crop.jpg

                                                 

                                                -Noel

                                                • 21. Re: 'Super Resolution' Upscaling Algorithm
                                                  Lundberg02 Community Member

                                                  I know you do work on astronomical images. Why would you want to use any form of artifact creating software on them? I realize that cosmology is part of the entertainment industry ever since Carl Sagan showed us billions and billions of billions, but I think they already see things that aren't there.

                                                  • 22. Re: 'Super Resolution' Upscaling Algorithm
                                                    Noel Carboni Community Member

                                                    Actually I have no interest in using the software in question on astroimagery.  I do need better deconvolution software.  I think I'll have to write something myself.

                                                     

                                                    Making more results from less data is why people would want a sophisticated upsampling tool such as this Super Resolution thing.  And this one is impressive, if only incrementally so.

                                                     

                                                    -Noel