30 Replies Latest reply on May 9, 2011 3:34 PM by Noel Carboni

    Camera Raw Policy of Writing Back into Input Files

    Noel Carboni Level 7

      This came up in another thread, and I think it deserves its own discussion.  I'd hope that we can influence the Camera Raw team's future direction.  I welcome your input and opinions.

       

      Given:  Under some conditions Camera Raw writes data back into (overwrites) its input files.

       

      Assuming you use Camera Raw to open your out-of-camera original files as many of us do, Adobe seems to be all over the road on whether to keep its hands off them or overwrite them...

       

      • Camera Raw will not touch a proprietary raw file, such as a Canon .CR2 or Nikon .NEF.  There's a whole process for remembering settings in a separate database or sidecar XMP files.  So far so good.

       

      • If you open a JPEG, TIFF, or DNG through Camera Raw, data WILL automatically be written back into it to tell another run of Camera Raw in the future what settings you used - without the software ever having warned you it will do so.

       

      • Some functions EXPLICITLY rewrite input files.  You can ask the software to write new thumbnails back into DNG files, for example.   This is fine - the user has instructed the software to overwrite the file, and the user is in charge, after all.

       

      Overwriting/rewriting an input file without being instructed to do so is NON-INTUITIVE BEHAVIOR for an application. No one would expect an input file to be overwritten.

       

      We do see that it causes people confusion.  I'm sure there are people right now reading this in disbelief.  I recommend you go test it for yourself (on a copy of one of your original files).

       

      The original file being overwritten is one of the reasons why I don't configure Photoshop to open my out-of-camera JPEGs through Camera Raw.

       

      Adobe's [mis]handling of input files on the surface seems to be derived from the history of DNG - where no camera actually writes the DNG file directly but it has been generated as an intermediate format through the DNG Converter, and as such can be handled with less "care" than an original camera file.

       

      It seems to me that Camera Raw should NEVER write back into an INPUT file it is opening without a) letting the user know or b) being directed to do so.

       

      Adobe:

       

      Please give those of us who don't want our input files overwritten an option for using the database/XMP sidecar instead in EVERY case.

       

      Thanks.

       

      -Noel

        • 1. Re: Camera Raw Policy of Writing Back into Input Files
          Noel Carboni Level 7

          By the way, I can think of one scenario where the requested option would make Camera Raw work more predictably...

           

          If the user does not have permission to actually write back to a file being opened through Camera Raw, the file will open anyway but the internal attempt to write back to it will simply fail quietly.  Allowing those "lost" settings to be stored in the local database would avert a problem with the user having to re-make the same settings again should they reopen the file at a later date.

           

          You might wonder why a user would not have permission to write back to his own files, but it's not hard to imagine a user accessing a shared resource with read-only access.  Networking is not something Photoshop does well now, and this is one reason why.

           

          -Noel

          • 2. Re: Camera Raw Policy of Writing Back into Input Files
            Jeff Schewe Level 5

            Noel,

             

            I think you need to make a distinction between proprietary raw files which Adobe treats as Read Only because there's no safe way to actually write metadata into them due to the undocumented nature of the file formats and other fully documented file formats such as DNG, JPEG & TIFF. Note, I also consider PSD as a proprietary file format since while generally documented, Adobe doesn't publicly document the entire file format without jumping through some NDA hoops.

             

            The issue of forced sidecar usage is another question. DNG, JPEG and TIFF can have .xmp metadata safely written into the files because the placement of the .xmp data is known and documented. I don't see a huger grassroots effort for forcing sidecar files for DNG, JPEG & TIFF files. In point of fact I suspect users would not want this. Yes, at this point in time only Adobe apps can really read the .xmp image settings and render adjusted previews. But once users understand that, I don't see widespread confusion or complaints.

             

            In the very limited discussion of Camera Raw re-writting entire DNG file when a user asks ACR to update the DNG preview, I don't disagree that perhaps the writing of the new DNG should give the user a warning that this will overwrite the original DNG and I don't disagree that the preferences for the new DNG should be exposed and the user given the opportunity to change the defaults. But I really think this is an edge case...Within an Adobe workflow, there is zero need to push new DNG previews out to DNG files. Bridge, ACR and Lightroom can all read the updated internal .xmp image settings and render a preview accurately. The _ONLY_ reason a user would really need to push a new DNG preview is in the case of 3rd party software that supports DNG in a limited bases but must rely upon the embedded DNG preview to show the image preview (meaning the software can't actually render the .xmp image settings).

             

            The root problem that we saw in the other thread is that the user didn't understand what he was doing, the 3rd party software didn't completely support DNG and by updating the DNG preview the resulting DNG could no longer be read correctly in the camera maker's software. I don't think there's any debate that it's really the 3rd party's failure to full implement the DNG spec that was the root of the problem (that and a user who didn't understand his own workflow).

             

            So far I find this all a tempest in a teapot. Yes, I've heard several cases where users got the short end of the stick because they thought DNG meant DNG and didn't understand that loose implementation by 3rd parties could cause problems...and make no mistake whose fault that is...it's the 3rd parties.

             

            So, what do you want Adobe to do? Add a preference and/or warning in ACR or DNG Converter? In terms of the complete ACR/LR/DNG user base, what percent of that base needs this? Is it mission critical for you? What new feature or function in the next version of ACR would you be willing to give up in order to get this new preference and warning? There are X number of engineers working X number of hours time X number of days towards the development of the new version of ACR. Unless you can make a use case that a large number of people need this, I think it's doubtful it's gonna happen.

             

            I would much prefer that Adobe evangelize the use of DNG better and work with those camera companies and 3rd party software developers to avoid the problems users have when they, the 3rd parties, don't follow the DNG spec properly.

            • 3. Re: Camera Raw Policy of Writing Back into Input Files
              Noel Carboni Level 7

              I appreciate your opinion, and I suspected you might be in favor of things just as they are, Jeff.

               

              I'm curious whether others, when pondering the thought that Adobe writes back to input files, really feel the same.

               

              I'm having difficulty thinking of another modern GUI-based application that takes a file as input and does things to it implicitly.   Is this more common on the Mac side?

               

              You could point out stuff that throws back to Unix command lines, I suppose (e.g., gzip unzipping a file by default deletes the .gz file)...  Make no mistake, Unix filters can be hugely powerful, but with power came the responsibility to get the commands just right.

               

              And since you asked, perhaps I wasn't clear in my first post:  I want the ability to configure Camera Raw to not write back to an input file, except where such an operation has been explicitly requested.  Something like:

               

              [ ] Save settings back to input file if possible.

               

              -Noel

              • 4. Re: Camera Raw Policy of Writing Back into Input Files
                JimGoshorn Level 2

                I have a question on this issue: Is there any way Adobe could verify the validity of 3rd party DNG formats when the image is opened? Even if this was a utility that the user could run a 3rd party DNG image through so they would be alerted in advance to possible problems.

                 

                If a camera maker is going to advertise that they output DNG files from their camera(s) then they should adhere to the spec otherwise the file is just another bastardized proprietary format.

                • 5. Re: Camera Raw Policy of Writing Back into Input Files
                  Jeff Schewe Level 5

                  JimGoshorn wrote:

                   

                  I have a question on this issue: Is there any way Adobe could verify the validity of 3rd party DNG formats when the image is opened?

                   

                  The problem is less the validity of the DNG file as it comes out of a camera or 3rd party software–I'm pretty sure ACR and DNG Converter does error check to make sure the file is valid. The problem is after DNG Converter or ACR writes a DNG file. If the 3rd party software isn't up to spec, then then they won't be able to read the DNG file. Of course, only ACR & Lightroom (and Bridge which uses the ACR plug-in) can read and render the .xmp image settings metadata into an accurate preview...

                   

                  In the grand scheme of things, a workflow that starts with a DNG as a camera file is prolly designed to be run through ACR/LR. Leica, which produces DNG files from the camera ships Lightroom as their default camera software. Other cameras such as the Pentax which has an option to write out DNG files is a bit different. 3rd party software which claims to support DNG is really all over the map. Some like Bibble will only support DNGs for camera files it already supports as native raw files–which is really not a full DNG implementation, only a subset subject to existing camera file support. The other issue with 3rd party DNG support is updating to the most recent DNG spec version which is currently v 1.3. V 1.3 includes a lot of new potential functionality which v 1.2 and 1.1 don't support.

                  • 6. Re: Camera Raw Policy of Writing Back into Input Files
                    areohbee Level 5

                    +1 vote on both counts:

                     

                    - Policy of re-writing input files.

                    - xmp sidecar option for all file types.

                     

                    PS - I'd even toss in a warning (that can be averted in future) when re-writing output files, especially DNG - at least until more implementations are compatible: "uncompliant software may not be able to read this...".

                     

                    PPS - I have no objection to Adobe "coming down hard" on half-baked DNG implementations too ;-}

                     

                    Question: How is DNG versioning handled when re-writing a DNG of a different version?

                     

                    R

                    1 person found this helpful
                    • 7. Re: Camera Raw Policy of Writing Back into Input Files
                      Vit Novak Level 3

                      In case of Sinar dng file I mentioned yesterday, original file had dng version tag 1.0.0.0, while updated file had version 1.3.0.0

                      So it seems ACR is using current dng specification when (over)writing the file

                       

                      I also wote for --possibility-- of having xmp files with dng files, the same workflow as with other raw formats

                      1 person found this helpful
                      • 8. Re: Camera Raw Policy of Writing Back into Input Files
                        areohbee Level 5

                        Vit Novak wrote:

                         

                        In case of Sinar dng file I mentioned yesterday, original file had dng version tag 1.0.0.0, while updated file had version 1.3.0.0

                        So it seems ACR is using current dng specification when (over)writing the file

                         

                        This seem presumptious to me, and like asking for trouble...

                         

                        (A file format version change should illicit a warning)

                        • 9. Re: Camera Raw Policy of Writing Back into Input Files
                          Vit Novak Level 3

                          Well, some programs (MS Word for instance) will inform you that file is an old version and ask what to do (save in that version or in new version). Or that file is not compressed and that it will be compressed ...

                          • 10. Re: Camera Raw Policy of Writing Back into Input Files
                            areohbee Level 5

                            Vit Novak wrote:

                             

                            Well, some programs (MS Word for instance) will inform you that file is an old version and ask what to do (save in that version or in new version). Or that file is not compressed and that it will be compressed ...

                             

                            Nuthin' wrong with that at all, IMO.

                             

                            Another example from the top of my head: Adobe Flash.

                            • 11. Re: Camera Raw Policy of Writing Back into Input Files
                              areohbee Level 5

                              JimGoshorn wrote:

                               

                              Is there any way Adobe could verify the validity of 3rd party DNG formats when the image is opened? Even if this was a utility that the user could run a 3rd party DNG image through so they would be alerted in advance to possible problems.

                               

                              If a camera maker is going to advertise that they output DNG files from their camera(s) then they should adhere to the spec otherwise the file is just another bastardized proprietary format.

                               

                              I dont know whether this would be a good idea or bad, but Adobe could support interoperability (spec compliance) certification, no? In which case, no warning need be issued for saving like version of certified format (when its an output that is - input files should never be covertly modified, imo)

                               

                              (kinda like how Microsoft has done for Windows drivers)

                               

                              Rob

                              • 12. Re: Camera Raw Policy of Writing Back into Input Files
                                Noel Carboni Level 7

                                Thanks for all your input so far.

                                 

                                areohbee wrote:

                                 

                                +1 vote on both counts:

                                 

                                 

                                - Policy of re-writing input files.

                                - xmp sidecar option for all file types.


                                Just to be clear, the second option needs two parts:  XMP sidecar - or - central database.  I happen to like using the central database myself.  I don't even want files written into the folder containing my original images.

                                 

                                As far as what version of DNG to write when a write-back is warranted, that seems to me to be pretty straightforward:  It needs another option (probably near the above) that would allow the user to choose whether to write the file back in the original version or current version.  I haven't read the spec completely through...  Is any part of version compatibility covered in there?

                                 

                                -Noel

                                • 13. Re: Camera Raw Policy of Writing Back into Input Files
                                  Jeff Schewe Level 5

                                  Noel Carboni wrote:

                                   

                                  As far as what version of DNG to write when a write-back is warranted, that seems to me to be pretty straightforward:  It needs another option (probably near the above) that would allow the user to choose whether to write the file back in the original version or current version.  I haven't read the spec completely through...  Is any part of version compatibility covered in there?

                                   

                                  When ACR or DNG Converter writes a DNG it will always use the most current spec version...pretty silly to step backwards. But ACR and DNG Converter has preferences for DNG backwards compatibilities so a DNG file written by ACR 6.4 for example could be opened in ACR 2.4 hosted in Photoshop CS. However, when updating the DNG preview it defaults to the current version with backwards compatibility set to v 1.1 (I believe). In the case of the Sinar DNGs, version 1.0 is horribly out of date–which is the root cause for the issues their software is experiencing...

                                   

                                  You really need to look into the spec (and understand why it's a bad idea to write old spec in new files) and the DNG backward compatibility options in ACR and DNG Converter.

                                   

                                  And as far as I'm concerned, saving edits in the Camera Raw database is primitive. Makes it real hard to have the edits transportable and while you can back them up, I view the database as more fragile than sidecar files. Personally, I much prefer writing the .xmp inside the file whether it's a DNG, KPEG or TIFF. Even when using Lightroom, I write the .xmp back to files even though I've got backups of the LR database. Having the .xmp as either sidecar or embedded makes working between LR and ACR possible.

                                  • 14. Re: Camera Raw Policy of Writing Back into Input Files
                                    Noel Carboni Level 7

                                    Jeff Schewe wrote:

                                     

                                    And as far as I'm concerned, saving edits in the Camera Raw database is primitive.


                                    Fine.  To each his own workflow.  That's why it's good there's an option.

                                     

                                    Oh, and I didn't count too awful carefully, but it seems like a majority of folks so far (myself included) have liked my idea of being able to separate the settings and never write them back to the original file.  I hope to get more thoughts on the subject.

                                     

                                    Eric / other Adobe people, please take note here.  This feature seems attractive to more than just me.

                                     

                                    -Noel

                                    • 15. Re: Camera Raw Policy of Writing Back into Input Files
                                      Vit Novak Level 3

                                      I also don't like the idea about saving something into common database

                                      • 16. Re: Camera Raw Policy of Writing Back into Input Files
                                        Noel Carboni Level 7

                                        Okay, but I think if the process for saving into XMP or common database is already in place, that another option to avoid writing back into original files should use that process.

                                         

                                        -Noel

                                        • 17. Re: Camera Raw Policy of Writing Back into Input Files
                                          Fred K. Level 1

                                          Hi Noel,

                                           

                                          Recently I re-viewed some old JPG-images in ACR to find out what I could do better in future.

                                          Yes... I know ... JPG in ACR..??... that is not where it is intended for.

                                          But now the focus is for altering the original image data..... so:

                                           

                                          When I open the JPG image in PS-ACR, make some changes and than quit (in ACR)... the date seems to stay OK.

                                          But....

                                          When I send the same image (after some changes made in ACR) to PS and then quit (in PS) the image seems to be also OK (unchanged) but the modification date is changed ! As far I can see that is done by ACR. ..... Unchanged >?> that is what the windows-thumbnail (also FireFox and also IrfanView) is telling me!

                                          But....

                                          When I open the same image again in PS-ACR it is chanced like I did in ACR before!

                                           

                                          Despite I quit the image...... the step between ACR and PS is modifying the image but you can't see it in windows-thumbnail.

                                          In Bridge however I can see for a split second the original (unmodified) image and then Bridge change the image like I did in ACR.... Bridge has read the EXIF-info.

                                           

                                          SO.... indeed: What meens the QUIT command.... (in ACR and PS).... what meens (in ACR) OPEN IMAGE command.

                                          OPEN IMAGE in ACR means WRITE to the same file .

                                           

                                          I have been a computerprogrammer since about 1980.....it is really new for me that OPEN means WRITE.

                                           

                                          So... I agree with you. Do not write any data.... but ask.

                                           

                                          Regards Fred

                                          • 18. Re: Camera Raw Policy of Writing Back into Input Files
                                            Hudechrome Level 2

                                            Well, this is fine for those who have the skills, knowledge and willingness to wade into these variations on a theme, but for users like myself, I definitely want a seamless process when X is invoked, ostensibly supported by all users, end and originating. As I look at what is written here, I keep thinking "OK but what about apps that send their output as dng, but are not originating suppliers like Leica or Sinar? And what am I to make of the cautions DXO counsels concerning the use of any Adobe product to even upload the RAW from the camera if contemplating the user of DXO? I spent a fair amount of time trying to get to the bottom of this with frequent e-mails to DXO and plain old checking myself, and the consensus was so long as I do not intend to use the complete editing suite, that is, modify the color and value with DXO it will not be a problem to use Bridge to upload. That's not completely true, because if I do any conversion in DXO, even nothing, the resulting dng has an Embedded choice in the Camera Calibration menu, which does not match anything else, including DXO own tiff version. I do not use the tiff!

                                             

                                            So there I am swinging in the breeze, so to speak. The saving grace is that if I stick with the Nikon selection Nikon Vivid, Standard and such, the visible tonal/color differences between the nef and the dng vanish except for things like vignetting compensation.

                                             

                                            Now I see that there are other factors in play, and I have no idea of their impact, particularly in terms of portability of the dng.  (Interesting that DXO cannot open that file either!)

                                            • 19. Re: Camera Raw Policy of Writing Back into Input Files
                                              Jeff Schewe Level 5

                                              Fred K. wrote:

                                               

                                              SO.... indeed: What meens the QUIT command.... (in ACR and PS).... what meens (in ACR) OPEN IMAGE command.

                                              OPEN IMAGE in ACR means WRITE to the same file .

                                               

                                              There is no QUIT command in ACR...only Open which will indeed save .xmp metadata back to the original file (or a sidecar if it's a read only file), Done which will also and Cancel which doesn't write anything to the file. And yes, if you make changes in ACR and Open or click Done, the file will be modified and have a new mod date as you should expect...

                                              • 20. Re: Camera Raw Policy of Writing Back into Input Files
                                                Noel Carboni Level 7

                                                Lawrence, I thank you for your thoughts, but...  I don't understand how what you've written applies to my question...  I'm not proposing any change to the fundamental way the software works for the end user - open an image of any type through Camera Raw and the settings used last time are re-applied.

                                                 

                                                With Camera Raw 6.4 and earlier, that's accomplished by Camera Raw writing those settings back into the original file in some cases.  In other cases, it writes the settings into a sidecar file or database.  I'm just asking that this write-back be made into a configurable option, so that Camera Raw will work the same way and always store its settings data in the sidecar file or database for all types of input files.

                                                 

                                                Putting this in terms that may be more familiar to you:

                                                 

                                                1.  You opened a .NEF file yesterday and applied some Camera Raw settings.  Today when you open it again the same settings are applied, without the .NEF file ever having been changed.  The settings were stored in an XMP file or in your central database.

                                                 

                                                2.  You opened a .JPG file yesterday through Camera Raw and applied some Camera Raw settings.  Today when you open it again the same settings are applied, but in this case metadata was added to the original .JPG file itself to accomplish this.  No XMP file or central database is used in this case.

                                                 

                                                I just want an option to have case 2 work the same as case 1, storing the settings elsewhere for all types of input files.

                                                 

                                                -Noel

                                                • 21. Re: Camera Raw Policy of Writing Back into Input Files
                                                  Hudechrome Level 2

                                                  Yes, I get that, in general if not in detail. What I am trying to say is why stop there if a change of this nature is contemplated?

                                                  • 22. Re: Camera Raw Policy of Writing Back into Input Files
                                                    Noel Carboni Level 7

                                                    Stop where?  I apologize for being particularly dense today.

                                                     

                                                    Keep in mind I'm suggesting that the new option prevent the writing back into any type of file, not just JPEG.

                                                     

                                                    -Noel

                                                    • 23. Re: Camera Raw Policy of Writing Back into Input Files
                                                      Hudechrome Level 2

                                                      More later. Gotta go out for dinner.

                                                       

                                                      Actually, what I was aiming at was Adobe more than you.

                                                       

                                                      Later....

                                                      • 24. Re: Camera Raw Policy of Writing Back into Input Files
                                                        Noel Carboni Level 7

                                                        Hudechrome wrote:

                                                         

                                                        Actually, what I was aiming at was Adobe more than you.

                                                         

                                                        If your suggestion doesn't bear directly on "Camera Raw writing back to input files", perhaps you should start a new thread?  It sounds like your DXO issue(s) might not have to do with this, though I'll be honest - I just don't understand what you wrote.

                                                         

                                                        I'd like to think good ideas get considered here by people who matter, and it could just get lost if expressed deep in this thread.

                                                         

                                                        -Noel

                                                        • 26. Re: Camera Raw Policy of Writing Back into Input Files
                                                          Peter DL Level 1

                                                          Noel wrote: >>  If you open a JPEG, TIFF, or DNG through Camera Raw, data WILL automatically be written back into it to tell another run of Camera Raw in the future what settings you used - without the software ever having warned you it will do so.<<

                                                           

                                                          It depends.

                                                          As far as I can tell:

                                                           

                                                          When a JPG or TIF file is Locked,
                                                          Camera Raw > Done results in a Write Permission Error.

                                                           

                                                          When a DNG file is Locked,

                                                          Camera Raw > Done - writes a separate xmp sidecar file.

                                                           

                                                           

                                                          The JPG / TIF handling seems to me inconsistent. With a locked file I'd prefer to have a sidecar xmp created as well. Some of my JPG / TIF files are originals for me.

                                                           

                                                          Peter

                                                           

                                                          --

                                                          Windows Vista, CS4 w/ACR 5.7

                                                          • 27. Re: Camera Raw Policy of Writing Back into Input Files
                                                            sandy_mc Level 3

                                                            Peter DL wrote:

                                                             

                                                            When a JPG or TIF file is Locked,
                                                            Camera Raw > Done results in a Write Permission Error.

                                                             

                                                            When a DNG file is Locked,

                                                            Camera Raw > Done - writes a separate xmp sidecar file.

                                                             

                                                             

                                                            The JPG / TIF handling seems to me inconsistent. With a locked file I'd prefer to have a sidecar xmp created as well. Some of my JPG / TIF files are originals for me.

                                                             

                                                            Actually, it get worse - locked/read-only file behaviour depends on operating system - basically, ACR and Lightroom depend (or at least did, round about LR 3.0, when I last did this test) on the operating system to reject attempts to overwrite locked files. Some do (e.g., Vista and Win 7). Some however don't e.g., Win XP in default user-is-an-administrator mode. So on XP, LR would happily overwrite a read-only file. And no, I'm not kidding, I demonstrated this on DNG files, and yes, I did report it to Adobe.

                                                             

                                                            Sandy

                                                            • 28. Re: Camera Raw Policy of Writing Back into Input Files
                                                              Vit Novak Level 3

                                                              Tried read only dng files on my comp: Windows XP SP3, ACR 6.4, local user has administrative rigts

                                                               

                                                              It works: dng files intact, settings written into xmp files

                                                               

                                                              I like undocumented features of ACR 

                                                              • 29. Re: Camera Raw Policy of Writing Back into Input Files
                                                                areohbee Level 5

                                                                Vit Novak wrote:

                                                                 

                                                                ...dng files intact, settings written into xmp files...

                                                                 

                                                                This works when ACR used via Photoshop, but *not* when used in Lightroom.

                                                                (Lightroom will read xmp sidecars for DNG, but not write them).

                                                                 

                                                                I would prefer same behavior in Lightroom too (preferrably as an option for all file-types, not just DNG - virtual copies too...).

                                                                 

                                                                R

                                                                • 30. Re: Camera Raw Policy of Writing Back into Input Files
                                                                  Noel Carboni Level 7

                                                                  Thanks for all your input.

                                                                   

                                                                  And just think, just the one setting I'm proposing could make all this work consistently... 

                                                                   

                                                                  -Noel