21 Replies Latest reply on Mar 10, 2014 5:38 PM by station_two

    Does ACR include all information from X-Trans sensors?

    ark@acm.org

      There has been a lot of discussion about how best to convert information from X-Trans sensors. Early conversion algorithms have been criticized in various fora and the algorithms have been improved. Therefore, it's reasonable to assume that the algorithms will be improved again in the future.

       

      If I store a raw file and then convert it to another form, say, TIFF, using a future conversion algorithm, I will obviously be able to take advantage of that algorithm. What if I store it in DNG form today and then convert it to TIFF in the future? Will I be able to use the future conversion algorithm, or will key information have been lost in converting to DNG?

       

      Here's what I think the situation is:

       

      If I use DNG Converter 8.4 and say that I want conversion that is compatible with ACR 7.1, then the X-Trans information is preserved. Moreover, converting to DNG does lossless compression, so the DNG file that results is likely to be smaller than the original raw file.

       

      If I use DNG Converter 8.4 and say that I want conversion that is compatible with ACR 6.3 or earlier, then DNG converter will do the conversion to RGB pixels. This will result in a DNG file that will probably be larger than the original raw file, and will make it impossible to use future raw conversion algorithms. However, it will also yield a DNG file that can be opened by versions of Photoshop before CS6 and versions of Lightroom before LR5.

       

      I would really appreciate it if someone from Adobe can tell me whether my understanding is correct, and correct it if it is not.

        • 1. Re: Does ACR include all information from X-Trans sensors?
          ssprengel Adobe Community Professional & MVP

          Your understanding is generally correct.

           

          At some point in the ACR 7.x series, maybe 7.2 or 7.3, Adobe updated their X-Trans demosaicking/interpolation algorithms in the Camera Raw engine used in ACR plug-in, DNGC, and LR, to produce a higher quality result.

           

          The newer DNG Converters do not need to interpolate the data for newer ACR plug-in versions.

           

          The newer DNG Converters do interpolate the data for older ACR plug-ins and older LRs using the better algorithms so those older products have the better interpolation results to work with.

           

          Older DNG Converters do a bad job of interpolating the data for even older ACR plug-ins and LR versions and this poor quality cannot be improved even if a new ACR plug-in or LR version is used because the damage done by the poor interpolation algorithms is already baked in.

           

          Another thing you might see is if you convert the data using a newer DNG Converter and DNG 7.1 compatibility options, but mistakenly process this DNG with an older ACR plug-in or older LR version, then you’ll see holes in the data, because the older CR engine is expecting interpolated data and the newer DNC Conversion options aren’t interpolating, anymore.

          • 2. Re: Does ACR include all information from X-Trans sensors?
            ark@acm.org Level 1

            Gotcha. So the question is whether this "older DNG Converters do a bad job..." situation is capable of happening again in the future if further improvements are made in the algorithms, or does ACR 8.4 (unlike its predecessors) preserve all of the information in the X-Trans raw file?

            • 3. Re: Does ACR include all information from X-Trans sensors?
              ark@acm.org Level 1

              Putting it more bluntly: Is it now safe to convert my X-Trans raw files to DNG and throw the originals away? Or might I regret doing so at some time in the future? This question is interesting because the raw files are about 32 megabytes each and the DNG files are between 17 and 20 megabytes.

              • 4. Re: Does ACR include all information from X-Trans sensors?
                deejjjaaaa Level 2

                > Putting it more bluntly: Is it now safe to convert my X-Trans raw files to DNG and throw the originals away?

                 

                no, because you don't know if it is safe and nobody else knows... do you believe in bugless code ?

                 

                > Or might I regret doing so at some time in the future? This question is interesting because the raw files are about 32 megabytes each and the DNG files are between 17 and 20 megabytes.

                 

                storage is cheap, may I suggest to use http://www.code42.com/crashplan/ at least ?

                • 5. Re: Does ACR include all information from X-Trans sensors?
                  Level 5

                  ark@acm.org wrote:

                   

                  Putting it more bluntly: Is it now safe to convert my X-Trans raw files to DNG and throw the originals away? Or might I regret doing so at some time in the future? This question is interesting because the raw files are about 32 megabytes each and the DNG files are between 17 and 20 megabytes.

                   

                  If you were in the habit of burning or throwing out your film negatives back in the days before digital photography, then I suppose one could understand your desire to discard your original raw files.  Oy, Weh!

                   

                  Storage is dirt cheap now.  You must value your work very little to consider throwing your raw originals away. 

                  • 6. Re: Does ACR include all information from X-Trans sensors?
                    ark@acm.org Level 1

                    Let me rephrase the question.

                     

                    Suppose in a couple of years a new version of ACR comes out that does a better job of demosaicing (or whatever) than the current one. Is there any reason to believe that it would be necessary to recreate every DNG file from the corresponding raw file (as it was in going from ACR 6.3 (I think) to ACR 7.1) or is the DNG file from ACR 8.4 contain all of the information in the raw file?

                     

                    This is a reasonable question about workflow, and I think the derisive remarks from station_two are rude and inappropriate.

                    • 7. Re: Does ACR include all information from X-Trans sensors?
                      ssprengel Adobe Community Professional & MVP

                      What are you converting the DNG in the first place?  The original raws obviously contain all the information in the original raws.

                      • 8. Re: Does ACR include all information from X-Trans sensors?
                        ark@acm.org Level 1

                        Because Lightroom cannot currently handle the original raws. As a secondary matter, the DNG files are half the size of the raw files.

                        • 9. Re: Does ACR include all information from X-Trans sensors?
                          Level 5

                          Well, ark, I don't appreciate being called rude and inappropriate, but I'm not here to argue with you.

                           

                          You still don't get it:

                           

                           

                          ark@acm.org

                           

                          Let me rephrase the question.

                           

                          Suppose in a couple of years a new version of ACR comes out that does a better job of demosaicing (or whatever) than the current one. Is there any reason to believe that it would be necessary to recreate every DNG file from the corresponding raw file (as it was in going from ACR 6.3 (I think) to ACR 7.1) or is the DNG file from ACR 8.4 contain all of the information in the raw file?

                           

                          This is a reasonable question about workflow…

                           

                          No, it's a question about a hypothetical… and it lacks foundation.  Yes, we do foresee the probability of future versions of ACR doing a better job of the conversion, that is precisely the point.  But for ACR to get to that point, it would require a breakthrough in technology that we cannot now predict.  Not knowing what such an advance might entail, the question of whether the DNG wrapper could be affecting the raw data in ways that we cannot currently discern becomes impossible to answer.

                           

                          Given current technology, we can say today that the raw data is indeed not affected by the DNG conversion, but no one can guarantee you that future technology advances will not force us to revise that view.

                           

                          That's why it is so unwise not to keep the original raw file in perpetuity, especially when the only drawback is the cost of file storage, which at this point is so ridiculously cheap as to be trivial.

                           

                          If you want to perceive rudeness and inappropriateness in my point of view, or in my expressing such an opinion, go ahead.  There's nothing I can do about it.

                           

                          You had a question, and I answered it to the best of my ability, that's all.  In my considered opinion, your stance is unwise.

                          • 10. Re: Does ACR include all information from X-Trans sensors?
                            sandy_mc Level 3

                            There is no way to know what information a hypothetical better X-Trans demosaicing scheme might require, and so no way for anyone to know whether that information is contained in a converted DNG or not.

                            • 11. Re: Does ACR include all information from X-Trans sensors?
                              ark@acm.org Level 1

                              Sandy_mc and station_two, if the information in a DNG file is sufficient to reconstruct exactly the original raw file, then it doesn't matter what information a better X-Trans demosaicing scheme might require. And I believe that the whole point of the DNG format is to make it possible to reconstruct the original raw information exactly.

                               

                              The whole point of my original question was to seek information to confirm or deny this belief. The question of whether, and if so how, to archive raw files separately from the DNG files that (if indeed they do) contain the same information is a completely independent question that deserves its own discussion.

                              • 12. Re: Does ACR include all information from X-Trans sensors?
                                deejjjaaaa Level 2

                                > Sandy_mc and station_two, if the information in a DNG file is sufficient to reconstruct exactly the original raw file, then it doesn't matter what information a better X-Trans demosaicing scheme might require. And I believe that the whole point of the DNG format is to make it possible to reconstruct the original raw information exactly

                                 

                                then does it surprise you that Adobe DNG converter does not have an option to convert from  DNG back to original raw :-) ? no, there is an option to embded the original raw file in DNG, yes - but except that solution :-) ? believing is a nice thing, but don't be taken for a ride like Library of Congress... unless you have a very specific defition of "original raw information" that suits you.

                                • 13. Re: Does ACR include all information from X-Trans sensors?
                                  Level 5

                                  For the reasons I have explained and which deejjjaaaa has expanded on, you're still dead wrong.

                                   

                                   

                                  ark@acm.org wrote:

                                   

                                  Sandy_mc and station_two, if the information in a DNG file is sufficient to reconstruct exactly the original raw file…

                                   

                                  Where on Earth did you get that from?    You cannot "reconstruct" the exact original raw file from a DNG file.  The option is simply not offered.

                                  • 14. Re: Does ACR include all information from X-Trans sensors?
                                    ark@acm.org Level 1

                                    There are two different questions. The first is whether Adobe--or anyone--offers a way to reconstruct the exact original raw file. The second, which is the one I am asking, is whether a DNG file contains all of the image information in the original raw file.

                                     

                                    For example, I have tried the following experiment: Save a raw file from a Fuji X-100 (note that the X100 does not have an X-Trans sensor). Convert the raw file to DNG and save the DNG file.

                                     

                                    Now load the raw file into Lightroom and export a TIFF image. Then do the same thing with the DNG file. The result is two TIFF images that are bit-for-bit identical. I think it is reasonable to conclude from that experiment that Lightroom does not use any information from the raw file that is not also in the DNG file, and this conclusion is reasonable whether or not there is a way to recreate the entire raw file.

                                    • 15. Re: Does ACR include all information from X-Trans sensors?
                                      Level 5

                                      ark@acm.org wrote:

                                       

                                      Sandy_mc and station_two, if the information in a DNG file is sufficient to reconstruct exactly the original raw file, then it doesn't matter what information a better X-Trans demosaicing scheme might require. And I believe that the whole point of the DNG format is to make it possible to reconstruct the original raw information exactly…

                                       

                                      That is a fallacy.

                                       

                                       

                                       

                                      ark@acm.org wrote:

                                       

                                      …The whole point of my original question was to seek information to confirm or deny this belief…

                                       

                                       

                                      Belief, schmelief. 

                                       

                                      With today's ACR technology, Adobe tells you that the raw to non-linear, demosaic'ed conversion of a DNG image file is identical to the raw to non-linear, demosaic'ed conversion of the original raw file, not that you can "reconstruct" said original raw file as recorded by your camera.

                                       

                                      Whether future ACR technology can advance to the point that a conversion from the original raw file as recorded by your camera generates more information than the current, latest version of ACR is mere speculation, and it would remain to be proven whether such theoretical enhanced ACR could also extract the same enhanced conversion from a DNG converted from the original raw with an improved DNG Converter that would incorporate the same technical advances as the hypothetical enhanced ACR plug-in.

                                       

                                      Sorry, if you take offense again, but I'm unabashedly back to my analogy of burning your film negatives.

                                      • 16. Re: Does ACR include all information from X-Trans sensors?
                                        Level 5

                                        Your last post, #14, contains the same flawed assumptions as your previous posts, plus another fallacy.

                                         

                                        You describe an experiment that uses today's ACR technology and pretend to make it valid for a theoretical, future, enhanced ACR technology that does not yet exist.

                                         

                                        That experiment is worthless and utterly irrelevant to the underlying question you posed.

                                         

                                        See post #15 in this thread:

                                         

                                        Whether future ACR technology can advance to the point that a conversion from the original raw file as recorded by your camera generates more information than the current, latest version of ACR is mere speculation, and it would remain to be proven whether such theoretical enhanced ACR could also extract the same enhanced conversion from a DNG converted from the original raw with an improved DNG Converter that would incorporate the same technical advances as the hypothetical enhanced ACR plug-in.

                                        • 17. Re: Does ACR include all information from X-Trans sensors?
                                          ark@acm.org Level 1

                                          station_two, you say that "Adobe tells you that the raw to non-linear, demosaic'ed conversion of a DNG image file is identical to the raw to non-linear, demosaic'ed conversion of the original raw file, not that you can "reconstruct" said original raw file as recorded by your camera."

                                           

                                          I believe that Adobe is now making a stronger statement, namely that starting with ACR 8.4, the demosaicing/linearization is done *after* conversion to DNG by default (i.e. unless you request compatibility with ACR 7.1 or earlier), and that the DNG file records the original pixel information from the X-Trans sensor.

                                           

                                          If you believe that this claim is not true, I would appreciate some factual evidence that refutes it. Perhaps someone from Adobe would be willing to clarify the matter.

                                          • 18. Re: Does ACR include all information from X-Trans sensors?
                                            Level 5

                                            …this whole discussion is sooooooo pointless, given the already trivial cost of storage space.

                                            • 19. Re: Does ACR include all information from X-Trans sensors?
                                              Level 5

                                              ark@acm.org wrote:

                                               

                                              …I believe that Adobe is now making a stronger statement…

                                              If you believe that this claim is not true, I would appreciate some factual evidence that refutes it…

                                               

                                              Jeebus Chrysler, ark!  Now you want "factual evidence" of a hypothetical contingency the nature of which no one is in a position to foresee.  That is preposterous!

                                               

                                              Why don't you go post in the DNG Converter forum and see if any of the DNG team members cares to engage in this discussion with you?

                                               

                                              http://forums.adobe.com/community/dng?view=discussions

                                               

                                              I'm out of here—and out of any other thread in which you choose to participate.

                                              • 20. Re: Does ACR include all information from X-Trans sensors?
                                                ccastleb Adobe Employee

                                                Hi All,

                                                 

                                                While the current DNG setup preserves the ability to take advantage of future improvements to ACR interpolation of X-Trans sensors, it isn't guaranteed to be a 1:1 substitute for interpolating directly from the original RAF files. Your best bet is to hold on to the originals to be certain you will get the best results.

                                                 

                                                Regards,

                                                 

                                                - Chris

                                                • 21. Re: Does ACR include all information from X-Trans sensors?
                                                  Level 5

                                                  Thanks for posting, ccastleb.  Maybe the OP wiill reconsider now that Adobe staff has chimed in.