29 Replies Latest reply on Jan 18, 2011 8:13 AM by MadManChan2000

    Cross Hatch Patterns

    Hudechrome Level 2

      I recently began running grayscale conversions in ACR and found a disturbing condition. In order to dramatize an image, I decreased the value of the Blue slider significantly, and the result, in neutral areas was a crosshatch pattern. Now, in Photoshop using B&W conversion and moving the Blue slider the same amount, the image becomes noisy, but random, much more to be preferred.

       

      Here is the result in ACR, (at 200% to see clearly):

       

      Example 1.jpg

       

      CS5, Win 7 64, all upgrades and patched installed.

        • 1. Re: Cross Hatch Patterns
          Noel Carboni Level 7

          What comes to my mind is that your camera has a Bayer pattern of photosites filtered with blue that are in a pattern that could ultimately lead to what you're seeing.

           

          http://en.wikipedia.org/wiki/Bayer_filter

           

          Perhaps the combination of settings you have chosen, coupled with the ordering of Camera Raw operations under the covers, yields a pattern when you do just what you did because only the blue measurements are participating in the conversion.

           

          By contrast, allowing the algorithms to use all the colors originally measured to participate in the generation of luminance information, then processing the resultant image in Photoshop yields a smoother final result.

           

          That's just a guess.

           

          What's interesting is that I can't reproduce what you're seeing.  When I zoom in on a neutral gray object (the side of my RV in shadow, as an example), then radically adjust the Blue slider, I get just more or less a random pattern.

           

          BlueRandom.jpg

           

          Can you be more specific about exactly what we're looking at in your image, and what other parameters you're setting to achieve this?

           

          -Noel

          • 2. Re: Cross Hatch Patterns
            Hudechrome Level 2

            Short answer: I believe you are right.

             

            The long answer got too long, so a synopsis follows.

             

            This image ran through two photo editing programs, the sum of which revealed the pattern. Of interest is that it is DXO and ACR for grayscale that does it, while DXO and Photoshop only does not. If I use the DXO/ ACR grayscale output in Photoshop, the pattern persists. In order to get the final image I wanted, I ran two versions, one through ACR which set the overall mood, the other through PS, layered them then erased the areas showing the Bayers.

             

            A lot of work! It actuall took to sets of layering to get it exactly right.

             

            I'm going back to see what I can do to get what I want at the front end.

             

            I also tried an entirely different image and got the result you did, using both DXO and ACR and/or PS. Random noise in the areas likely to produce the grid pattern, but no grid.

             

            Edit:

             

            I lookead at your image up large. One major difference is my blue slider is set to about -39, yours at +100.

             

            Message was edited by: Hudechrome

            • 3. Re: Cross Hatch Patterns
              Noel Carboni Level 7

              I reposted a -100 right after re-reading your comment about turning down the blue.  I can do it again with -39 if you'd like.

               

              What does the full image exhibiting the pattern look like?  Is it possible there was a window screen or something out of focus in there?

               

              -Noel

              • 4. Re: Cross Hatch Patterns
                Hudechrome Level 2

                Not at 9800' on the top od Steens Mountain! No window screens.

                 

                Here's the shot. The mound of snow in which the copyright notice resides (lower right corner) has the problem. In the rest of the image, where ever I saw it I eliminated it except around the middle of the shot. See those thingys that look like eyes? there are residuals there as well.

                 

                It's plainly visible in a 10x15 print.

                 

                20101008_0158 _print_1 copy.jpg

                • 5. Re: Cross Hatch Patterns
                  Noel Carboni Level 7

                  Was that snow patch slightly out of focus?

                   

                  I've seen lens problems not unlike what you're showing manifest as patterns in OOF areas...  In one case what should have been a good quality glass filter (protecting my 100-400 zoom) was at fault.  When I accidentally broke it (it doing its job nicely) I noticed a major improvement in my image quality.

                   

                  -Noel

                  • 6. Re: Cross Hatch Patterns
                    Hudechrome Level 2

                    Probably slightly. But the snow patch at the right is roughly in the same plane and it doesn't show the effect. The higher contrast may mask it.

                     

                    It's worth setting up  a study of sorts. Snow that is lit softly would be perfect, and I know where there is plenty!

                     

                    The clouds definitely were not OOF. Metadata says subject at Infinity. The shot was at f13, 1/320 at iso200, F/L 18mm on a D90.

                     

                    I can also graze through my files to see if I have suitable samples, but it probably will be better to set it up and do an OOF test.

                     

                    Curious minds want to know!

                     

                    No glass on front of the lens, either.

                     

                    Message was edited by: Hudechrome

                    • 7. Re: Cross Hatch Patterns
                      MadManChan2000 Adobe Employee

                      Do you have a raw file that you could provide?

                      • 8. Re: Cross Hatch Patterns
                        Hudechrome Level 2

                        The current file? I haven't shot any new ones yet.

                        • 9. Re: Cross Hatch Patterns
                          Hudechrome Level 2

                          I did some experiments after the last post. Several things fell out:

                           

                          1) Both ACR and B&W conversion produces the patterns, if you drag the blue slider down significantly. The numbers don't seem to have a 1:1 relationship.

                           

                          2) This only happens with the file that has been through DXO Single HDR conversion first, appearing as a .dng.

                           

                          3) DXO or Photoshop alone will not produce the patterns.

                           

                          4) Runnung a DXO file that does not include their HDR does not produce the grid.

                           

                          Given those, if you want to see the files, I'm glad to submit one that really shows it up, but I think it's important to see both the .nef and the adjusted .dng from DXO.

                           

                          Up to you.

                           

                          I suspect Noel is correct...it's showing a pattern from the sensor, at reduced resolution.

                          • 10. Re: Cross Hatch Patterns
                            Vit Novak Level 3

                            I never understood what's the point of single shot HDR and why it is called HDR 

                            • 11. Re: Cross Hatch Patterns
                              Hudechrome Level 2

                              I know and agree but my greed for any novel tweak, to at least try, leads me to try it. Particularly it can give a awful color to the image, but that gives conversion teeth to chomp away at the tonal scale. Seems here it bites off more than it can chew!  (With apologies to Jeff for the pun!)

                               

                              B&W conversion has always had it's caveats. I have otherwise neat conversions that absolutely suck when it comes to noise. Controlling that mostly leads to posterization, which is evident in the image I posted, though not offensive. (one little trick I try is to run a noise filter before conversion, Dfine is particularly good, that will see and correct for noise that may not be obvious to the eye, but does allow more aggressive B&W conversions. I didn't try that here, and I'll go back and run it.).

                               

                              That's were it sits at the moment. With respect to the image I posted, the dramatic change happened to the cloud bank at the center. The original image was so diffuse that the form all but disappeared. I tried to emulate that without any help from DXO, no dice. I might get there using the Adjustment brush on the RAW, that may also be a path. But without DXO showing me a possibility, I would not even be close.

                               

                              That's me. If someone tells me an object is unbreakable, I'll see what it takes to break it! Helps to define "unbreakable". (Still thinking of you, Jeff, if you are reading!)

                              • 12. Re: Cross Hatch Patterns
                                Noel Carboni Level 7
                                function(){return A.apply(null,[this].concat($A(arguments)))}

                                Hudechrome wrote:


                                2) This only happens with the file that has been through DXO Single HDR conversion first, appearing as a .dng.

                                 

                                3) DXO or Photoshop alone will not produce the patterns.


                                I suspect that DXO alone IS in fact creating the patterns, but it's just that you may not be seeing them visually.  I'll bet they're there in the data.

                                 

                                Not all raw processing is of equal quality.  Getting the math right and keeping it fast is a challenge.

                                 

                                Would you be willing to share the original raw file?  I wonder if it's possible to think up other ways to achieve results similar to what you're trying to get with DXO, but excluding that software.

                                 

                                -Noel

                                • 13. Re: Cross Hatch Patterns
                                  Hudechrome Level 2

                                  That would be great!

                                   

                                  DXO may be generating the patterns, or at least, some sort of output which forces the issue when run in Adobe. Neither software company is under any obligation to take the other into consideration, and if one can get clean, useable output from either alone, fine. But I use both, consistently. For lens and sensor corrections.

                                   

                                  I don't know of any editing software that is likely to be used in tandem with another, and if it wasn't for the powerful corrections DXO provides for lens deficiencies, as well as noise, using the devices themselves as the source rather than fiddling by eye, I wouldn't use it.

                                   

                                  DXO claims that no raw processing is done if all you ask for is the lens corrections. Noise, however is a function of the sensor, not the lens, and applying that correction...well, does demosaicing become a necessary part of that process? I would hope not.  So the question is one of what is required to pass along noise correction info for the sensor? DXO is not very clear on the matter, only saying that their dng isn't the same as the dng conversion from Adobe.

                                   

                                  Messing with raw files isn't for the fait of heart, or mind, for that matter!

                                   

                                  Edit: What it would take is a single image HDR correction that is tied to ACR rather than PS. Shadow/highlight does a fine job, but if I want further corrections through ACR, it's back and forth.

                                   

                                  Message was edited by: Hudechrome

                                  • 14. Re: Cross Hatch Patterns
                                    Jeff Schewe Level 5

                                    Hudechrome wrote:

                                     

                                    DXO claims that no raw processing is done if all you ask for is the lens corrections. Noise, however is a function of the sensor, not the lens, and applying that correction...well, does demosaicing become a necessary part of that process?

                                     

                                    If DxO is outputting Linear DNGs then yes, DxO's demosiacing has been applied–end of story. You really can't correct for the litany of various lens defects on a raw file unless you process the raw file into demosiaced RGB pixels. As I understand DxO's DNG process, it applies a white balance and then demosiacs the raw file into linear RGB pixels that are saved as DNG. So, a DNG from DxO is "processed".

                                    • 15. Re: Cross Hatch Patterns
                                      Hudechrome Level 2

                                      Thanks, Jeff.

                                       

                                      I have an ongoing conversation with DXO support, actually, for over a year now. With one person. As a bit of an aside, this kind of support is what I am sure that others here who complain about support would love to have, but that's for a different conversation, which I would like to pursue. So I sent them an inquiry last night, to which he responded this morning.

                                       

                                      He concurred with your statement about being a linear dng. If you choose the option to do no adjustments whatsoever, you get a linear dng, as stated here:

                                       

                                      "The DNG that Optics Pro delivers is not a CFA (Color Filtered Array - RGGB image) file, but a linear RAW file (RGB image - the sensor output is already de-mosaiced)."

                                       

                                      Adding corrections for optics and noise is safe, to use his word.

                                       

                                      So, we are on the same page (finally I might add, but that isn't a criticism of either Adobe or DXO; I own a big part of it!)

                                       

                                      As for the anomaly, they called it interesting, but don't go there! (Smiley mine, not theirs).

                                       

                                      There exists another  anomaly which is; supposedly ACR should not be seeing their color or tonal corrections except for  vignetting (and I assume, noise). That is not the case, and like I told them, if it wasn't the case, we would not be having this conversation, nor would I have this image, which I find striking.

                                       

                                      A question: If I run an nef through Adobe dng conversion, is it also demosaiced? Does that make a difference to the capabilities of ACR to be supplied with a demosaiced image from the start? I realize this is true if you supply a tiff or jpeg, but in those instances,all the tools available to nef are not available or marginalized.

                                       

                                      So, that's it in a nutshell, I don't know how much more I can reveal about the conversation with DXO, but at least I feel my findings validated at the basic level, here and with DXO.

                                       

                                      Lawrence

                                       

                                      Edited for a stray typo.

                                       

                                      Message was edited by: Hudechrome

                                      • 16. Re: Cross Hatch Patterns
                                        Jeff Schewe Level 5

                                        Hudechrome wrote:

                                         

                                        A question: If I run an nef through Adobe dng conversion, is it also demosaiced? Does that make a difference to the capabilities of ACR to be supplied with a demosaiced image from the start? I realize this is true if you supply a tiff or jpeg, but in those instances,all the tools available to nef are not available or marginalized.

                                         

                                        Assuming you are using the current DNG Converter 6.3, you have a wide variety of options...you can choose lossless compression (a good thing), you can choose to demosiac the file (create a linear DNG but YOU really don't want to unless you have to for some 3rd party reasons). You can also choose to embed the original proprietary raw file inside the DNG. That makes for a big DNG...I really only do that for testing purposes never an a real image.

                                         

                                        BTW, the ability to convert to Linear DNG has been buried (Adobe only want to have you do that is you NEED to do that). You need to luanch DNG Converter, click on Change Preferences then in the Compatibility dropdown select Custom and you'll get Custom DNG Compatibility dialog that allows you to select no compressions (it's on by default) and select Linear...

                                         

                                        To be clear, there is no direct benefit when using Camera Raw or Lightroom to convert proprietary raw files to DNG prior to opening in ACR/LR (unless you need to do it for backwards compatibility for a new camera in an older ACR/LR). If you DO have a Linear DNG, there ARE limitations to certain color related processing in ACR/LR. While a Linear DNG is still linear and can take advantage of ACR's tone mapping pretty well, the demosiacing process bakes in the color of ProPhoto RGB with a linear gamma. That can have an impact on the way white point and HSL correction may be rendered.

                                         

                                        I haven't done a lot of testing but in general, Linear DNG is a bit less good overall than undemosiaced DNG. And Linear DNG is what you get if you convert a TIFF or JPEG into a DNG format. You really can't un-ring the demosiacing bell.

                                        • 17. Re: Cross Hatch Patterns
                                          Hudechrome Level 2

                                          That being the case, I can easily check for any degradation by running an nef through their process w/o any adjustments. According to them, I should also see no difference except for the optical corrections if that's all I do, and at least to my eye, the two do look the same. Having the image jump in shape due to distortion correction does play games with the mind as to what I am seeing, so I'll try a totally unadjusted file and do a differential check, as a first order look.

                                          • 18. Re: Cross Hatch Patterns
                                            Hudechrome Level 2

                                            Ok, tried it on one image...no differential output visually. Total black.One as nef, one as dng from DXO with no adjustments. Open in both ACR then in Photoshop with default settings in ACR.

                                             

                                            I understand that this sort of test has problems and a more sensitive test is possible. But FAPP, they are identical.

                                            • 19. Re: Cross Hatch Patterns
                                              Noel Carboni Level 7
                                              function(){return A.apply(null,[this].concat($A(arguments)))}

                                              Hudechrome wrote:

                                               

                                              FAPP, they are identical.

                                               

                                              A visual comparison isn't really good enough to make that statement, since you're doing extreme things to the data to get the pattern to show up.

                                               

                                              Put one over the other, then set the top layer to Difference mixing. 

                                              Then put a Curves adjustment layer over the top and jack the curve way up.

                                               

                                              -Noel

                                              • 20. Re: Cross Hatch Patterns
                                                Hudechrome Level 2

                                                I figured that one out after posting. I remembered someone referring to the inability to see subtle differences at the black level, so I did just that. I first used levels to get a gray then Curves to up the contrast. Could not see any difference, nor could I see any value shift from the gray values in Info.

                                                 

                                                So, I stand by my initial assessment.

                                                • 21. Re: Cross Hatch Patterns
                                                  Noel Carboni Level 7

                                                  Very interesting.  And this is the same image that you can bring crosshatching out in through black and white conversion?

                                                   

                                                  -Noel

                                                  • 22. Re: Cross Hatch Patterns
                                                    sandy_mc Level 3
                                                    To be clear, there is no direct benefit when using Camera Raw or Lightroom to convert proprietary raw files to DNG prior to opening in ACR/LR (unless you need to do it for backwards compatibility for a new camera in an older ACR/LR). If you DO have a Linear DNG, there ARE limitations to certain color related processing in ACR/LR. While a Linear DNG is still linear and can take advantage of ACR's tone mapping pretty well, the demosiacing process bakes in the color of ProPhoto RGB with a linear gamma. That can have an impact on the way white point and HSL correction may be rendered.

                                                     

                                                    Let's be even clearer : Linear DNG doesn't require a linear gamma ProPhoto space, it can be whatever you want it to be. It doesn't in fact need to be any recognized color space. So there's no in-principle down side to converting to a linear DNG. In practice, ACR/Camera Raw seem to default to a space with ProPhoto primaries. But practically, for ACR/Lightroom/the DNG converter, that's not a downside - anything wider than ProPhoto can't be represented in ACR/Lightroom's internal working space, so it would have been cut off anyway, as soon as it was initially loaded into ACR/Lightroom.

                                                     

                                                    So there is no issue around "baking in color" or any issues at all with color in linear DNG (IMHO, sorry Jeff). The point about baking in the demosaicing process is however correct; demosiacing algorithms get better over time, so you probably don't want to do that unless you have to.

                                                     

                                                    Sandy

                                                    • 23. Re: Cross Hatch Patterns
                                                      Jeff Schewe Level 5

                                                      sandy_mc wrote:

                                                       

                                                      So there is no issue around "baking in color" or any issues at all with color in linear DNG (IMHO, sorry Jeff).

                                                       

                                                      Actually, if you convert a TIFF or JPEG to a DNG, then the original color space is tagged (inside the DNG) and you are limited to whatever the original gamut of the color space is (but processed in PPRGB Linear)...so yes, with TIFFs & JPEGs, the gamut is "baked in" for a Linear DNG. For a raw file converted to Linear DNG, I'm less sure of the color space issue...something Eric Chan could answer...

                                                      • 24. Re: Cross Hatch Patterns
                                                        sandy_mc Level 3
                                                        Actually, if you convert a TIFF or JPEG to a DNG, then the original color space is tagged (inside the DNG) and you are limited to whatever the original gamut of the color space is (but processed in PPRGB Linear)...so yes, with TIFFs & JPEGs, the gamut is "baked in" for a Linear DNG.

                                                         

                                                        Jeff, yes, but you're not losing anything. If the original gamut was whatever, the DNG still has all the colors that the original had. And as soon as you pull the file into e.g., Lightroom, you're in Lightroom's working space, and you can edit away to your hearts content. So you never lose anything in terms of color by going to DNG.

                                                         

                                                        Sandy

                                                        • 25. Re: Cross Hatch Patterns
                                                          Jeff Schewe Level 5

                                                          sandy_mc wrote:

                                                           

                                                          Jeff, yes, but you're not losing anything. If the original gamut was whatever, the DNG still has all the colors that the original had.

                                                           

                                                          Well, in the case of a proprietary raw file converted to Linear DNG (as DxO might do) I'm not 100% sure of that...in the case of a TIFF or JPEG converted to DNG, yes, you are right. The original "baked in" color space would be the limiting factor and the additional color manipulations (in ACR/LR) would be done in PP RGB, D50 and gamma 1.0.

                                                          • 26. Re: Cross Hatch Patterns
                                                            Hudechrome Level 2

                                                            So long as the raw file remains as the "go to" image, I don't worry about "baking in". In fact, having a fixed resolved (at some level anyway) image from a file, like a master print from a great negative, has importance.

                                                             

                                                            I've seen a number of prints Ansel Adams did of the same negative, and it's remarkable how things changed. But, at least in my work with analog photography, I have a master print, or rather, a reference print. I still do that today.

                                                             

                                                            It would be a weird situation if, when making a print from a negative, somehow that negative changes so that you could no longer make as good a print as you first did.

                                                            • 27. Re: Cross Hatch Patterns
                                                              Hudechrome Level 2

                                                              No, I grabbed a raw image by chance and did the test. It's only one image, one test, but it looks encouraging.

                                                              • 28. Re: Cross Hatch Patterns
                                                                Hudechrome Level 2

                                                                I'm looking at a number of different images to see if I can bring cross hatching out as a general case, if only to define the limits to which I can push. For the most part, any kind of raggedness in an image due to push processing defines limits, but that a regular, geometric pattern shows is more interesting than plain noise. It's mathematical. Like hearing Bach in a windstorm.

                                                                • 29. Re: Cross Hatch Patterns
                                                                  MadManChan2000 Adobe Employee

                                                                  There is nothing inherently lossy about the linear DNG format itself. For example, an already-rendered TIFF file in a RGB color space can be converted to linear DNG, with no data loss (the transform is reversible).

                                                                   

                                                                  But depending on where you start from, what tools are used, and what data processing is done, conversion to a linear DNG image may imply some non-reversible behavior. For example, since a linear DNG is not in mosaic format, then if you start from a mosaic image, the linear DNG is affected by the choice of the demosaic method. As another example, early stage sensor processing usually done prior to the demosaic method would also get baked into the linear DNG pixel values.