1 2 Previous Next 57 Replies Latest reply: Jan 28, 2013 1:24 PM by areohbee RSS

    Would other people like the option to write data to a side car for DNGs?

    JPenSuisse Community Member

      Hello All

       

      I have been using the Lightroom since since LR1 (when it was a beta for Mac only).  I skipped the LR3 upgrade and went directly from LR2 to LR4.   The one thing missing for me in all these versions is the fact that LRx writes the meta data directly into my DNG files.  There are 2 potential issues here:

       

      1.)  For me, I have to have an extra backup for my pictures from M8 and my scanned RAW files from VueScan, because they generate native DNG files.  For my Nikon's I don't have this problem, becuase LRx never writes into the proprietary files *.NEF files.

       

      2.)  Other people who have converted their images to DNGs (even if for instance they were NEF files) will probably have the same problem. 

       

      I think it is strange, because the whole point of this application is non-destructive editing!!  What do people think about this?

       

      Kind regards,

       

      John

        • 1. Re: Would other people like the option to write data to a side car for DNGs?
          areohbee Community Member

          +1 vote. DNGs, RGBs... - toss virtual copies in there too and we'd really be having things...

           

          Just a heads up - you/we are probably going to get a lecture at some point about how we shouldn't want this, and sidecars are bad whereas embedded xmp is good, and we know what's best for you...

           

          (this is about the 9-gillionth time this subject has come up)

           

          PS - xEmP provides somewhatofa work-around.

           

          R

          • 2. Re: Would other people like the option to write data to a side car for DNGs?
            JPenSuisse Community Member

            Rob, thanks for the feedback. 

             

            I am glad to know that I am not the only one experiencing this.  In fact, it is the reason why for a long time I have been modifying my Leica M8 files in Bibble 5.

             

            Unfortunately, for the ViewScan files raw scanner files, Bibble is not compatible.

             

            Cheers,

             

            John

            • 3. Re: Would other people like the option to write data to a side car for DNGs?
              john beardsworth Community Member

              I'm sure you're not the only one who doesn't like the approach, but Adobe has long taken the other view - rightly. DNG is a publicly-documented format and the image data is not affected by metadata being written to the file, so unless you want to change the definition of the term, it is indeed non-destructive. It's obviously pointless to continually backup the  DNGs - simply back them up when they're new. Backup the catalogue continually too, and all your images and Lightroom work are safe.

              • 4. Re: Would other people like the option to write data to a side car for DNGs?
                JPenSuisse Community Member

                That might be true about the image data itself.  As a layman, I cannot tell.

                 

                But the fact that I can only see that change after files are updated and saved makes me very suspicious.  Some metadata information is actually changed, and that makes it difficult to organize my photos.

                 

                Thus, I am not really safe as you so confidently proclaim.  It's the context that makes the difference.

                • 5. Re: Would other people like the option to write data to a side car for DNGs?
                  dj_paige Community Member

                  I agree with John. You are as safe as he proclaims, if you follow his advice.

                   

                  Some metadata information is actually changed, and that makes it difficult to organize my photos.

                  ???

                   

                  I don't understand

                  • 6. Re: Would other people like the option to write data to a side car for DNGs?
                    D Fosse Community Member

                    Don't worry so much. Keep your backup in a safe place and you'll be fine.

                     

                    There are bigger dangers out there - disk failures, fires, floods...

                    • 7. Re: Would other people like the option to write data to a side car for DNGs?
                      john beardsworth Community Member

                      JPenSuisse wrote:

                       

                      That might be true about the image data itself.  As a layman, I cannot tell.

                       

                      But the fact that I can only see that change after files are updated and saved makes me very suspicious.  Some metadata information is actually changed, and that makes it difficult to organize my photos.

                       

                      Thus, I am not really safe as you so confidently proclaim.  It's the context that makes the difference.

                       

                      You're right to start off from a suspicious point of view. But backing up new DNGs in their virgin condition means your photos are secured. Then whatever metadata LR subsequently writes to the DNGs shown in Library has second-rate value as a backup because the XMP is incomplete anyway - history, virtual copies, stacks, flags, assignment to collections, and lots of other LR work is never written to XMP (sidecars or embedded). Back up the catalogue and the virgin DNGs and all your work is secured.

                      • 8. Re: Would other people like the option to write data to a side car for DNGs?
                        JPenSuisse Community Member

                        Hi 21

                         

                        Look, let's not comapre apples and oranges here.  I have a complex back-up system with disks in multiple locations.  "Disk failures, fires and floods" are imaginable, non-human caused risks. 

                         

                        Software making some  strange manipulations to my files rendering part of my work no longer usable or requiring significant amount of re-work is human-caused or even self-inflicted stupidty.  How can I imagine all the things LR does to my files?

                         

                        Disk failures, fires and floods are OT here and diverting interest from the point of my thread.

                         

                        Kind regards,

                         

                        John

                        • 9. Re: Would other people like the option to write data to a side car for DNGs?
                          JPenSuisse Community Member

                          Yes, John, thanks. 

                           

                          This is exactly what I'm doing.  If however LR had the ability to write the changes to a DNG file in an XMP file it would reduce my storage usage by a factor of more than 2!  (This is becuase DNG's from my scanner "saved and updated" by LR have a about only 2/3s the files size as the virgin files!)

                           

                          KR, John

                          • 10. Re: Would other people like the option to write data to a side car for DNGs?
                            john beardsworth Community Member

                            Equally, what would reduce the backup size is not backing up these files unnecessarily. For instance, after backing up the new DNGs I keep the working DNGs (those shown in LR) on a drive that isn't covered by my backup routine. Lightroom can then write as much metadata as it wants to these working files without my backup being needlessly swollen. If anything ever goes wrong, the virgin + catalogue backups give me 100% coverage.

                             

                            Whatever one's view, I doubt Adobe are going to change.

                             

                            John

                            • 11. Re: Would other people like the option to write data to a side car for DNGs?
                              Mark Alan Thomas Community Member

                              This would be a welcome option. I keep multiple backups, one of which is through Crash Plan (a cloud backup service). Because DNGs save their xmp data internally, small changes to the image mean that the entire DNG needs to be backed up again. If DNGs could write sidecars, only the sidecars would need to be backed up again. This would mean faster, more efficient backups.

                              • 13. Re: Would other people like the option to write data to a side car for DNGs?
                                areohbee Community Member

                                I'd rather have catalog backup plus fresh xmp, than catalog backup alone.

                                 

                                Reminder: edit backup is only one of many reasons xmp sidecars may be preferred over embedded.

                                 

                                PS - another way (besides xEmP plugin) to get xmp sidecars for DNGs: make them read-only. Lightroom will report a failure if you try to save xmp into them, but Bridge will write the xmp sidecar that you want. Lightroom will read these xmp sidecar files - it just won't write them.

                                • 15. Re: Would other people like the option to write data to a side car for DNGs?
                                  john beardsworth Community Member

                                  Missing lots of Lightroom work - see list above.

                                  • 16. Re: Would other people like the option to write data to a side car for DNGs?
                                    areohbee Community Member

                                    Mark Alan Thomas wrote:

                                     

                                    Second-rate in what way?

                                    xmp instead of a true catalog backup would be 2nd-rate*, however having fresh xmp along with a catalog backup is 1st-rate.

                                     

                                    *xmp only includes dev-edits & photo-metadata, not everything in the catalog.

                                     

                                    Personally, I back up my catalog once a week, or more, but I save xmp several times each day.

                                     

                                    So, catalog-backup + fresh-xmp might set me back an hour or two, but not a day or a week...

                                     

                                    Yes, one could backup their catalog several times each day instead of xmp, but that requires restarting Lightroom, and takes several minutes for medium-sized catalogs, plus more frequent catalog backups still doesn't solve the problem of backing up a problem catalog. - xmp does (more so) solve that problem.

                                     

                                    Rob

                                    • 17. Re: Would other people like the option to write data to a side car for DNGs?
                                      Mark Alan Thomas Community Member

                                      Well, I for one wasn't talking about not backing up the catalog. I just think it would be handy if there were an option for Lightroom to treat DNGs the way it treats my NEFs when I have the "automatically write changes to XMP" option enabled. Surely checking this checkbox doesn't affect the integrity of the catalog.

                                      • 18. Re: Would other people like the option to write data to a side car for DNGs?
                                        DdeGannes Community Member

                                        So you are asking that Lightroom write the xmp data to the the DNG header and also to a .xmp sidecar file.?

                                        • 19. Re: Would other people like the option to write data to a side car for DNGs?
                                          JimHess CommunityMVP

                                          Then the solution is just to use the original NEF files.  I don't want to start an argument here, but I still think converting to DNG is pretty much senseless.  They promote it as an open format that will always be supported.  Why will we lose support for the original files?  We have it now, why will we lose it?  They promote it as a format that will load faster in the develop module in Lightroom.  I compared the other day, converted some files to DNG, and they took longer to load in the develop module than my original raw files.  Reduced size?  Not enough to even be significant, in my opinion.  Use the original raw files, you can have your XMP files, and everything works the way it should.

                                          • 20. Re: Would other people like the option to write data to a side car for DNGs?
                                            areohbee Community Member

                                            JimHess wrote:

                                             

                                            the solution is just to use the original NEF files

                                            Indeed, except for one thing: there is no way to revert your DNGs back to NEF, without those photos being considered new photos in Lightroom.

                                             

                                            PS - the same problem exists for RGB files (and virtual copies), for which there is no equivalent solution.

                                             

                                            Rob

                                            • 21. Re: Would other people like the option to write data to a side car for DNGs?
                                              JimHess CommunityMVP

                                              Rob Cole wrote:

                                               

                                              JimHess wrote:

                                               

                                              the solution is just to use the original NEF files

                                              Indeed, except for one thing: there is no way to revert your DNGs back to NEF, without those photos being considered new photos in Lightroom.

                                               

                                              Rob

                                              That merely validates my original argument.  Therefore, why convert to DNG in the first place?

                                              • 22. Re: Would other people like the option to write data to a side car for DNGs?
                                                areohbee Community Member

                                                Hi Jim,

                                                JimHess wrote:

                                                 

                                                That merely validates my original argument.

                                                To be clear: I wasn't trying to provide a counter-argument.

                                                 

                                                 

                                                JimHess wrote:

                                                 

                                                Therefore, why convert to DNG in the first place?

                                                I agree, people who want sidecars for raws, should consider not converting to DNG (if it's sufficiently important to them I mean).

                                                 

                                                PS - For those who've already converted to DNG, whether or not they regret it, the only way I know to have DNG sidecars is via xEmP.

                                                 

                                                 

                                                Cheers,

                                                Rob

                                                • 23. Re: Would other people like the option to write data to a side car for DNGs?
                                                  Jao vdL Community Member

                                                  >people who want sidecars for raws, should consider not converting to DNG (if it's sufficiently important to them I mean).

                                                   

                                                  That's exactly the solution I use to avoid constant backing up of raw files with automatic backup software such as Apple's Time Machine and every other automatic backup solution out there. I manually make a second backup of the raw files and I don't want backups of my raw files every hour or so but xmp files make sense as they are small anyway. So the solution is to just keep them in proprietary format and you'll be fine. I would really like to see a checkbox option to write sidecars outside of RGB files (tiff, psd, jpeg) too as Rob also notes. There you can't keep Lightroom from writing in the files. Really annoying. I don't want ANY program to touch my originals, even if it is to write in the metadata area of the file.

                                                  • 24. Re: Would other people like the option to write data to a side car for DNGs?
                                                    john beardsworth Community Member

                                                    Equally, if you do want to take advantage of DNG don't be put off by Lightroom not writing sidecars that provide a second-rate backup. Just fine tune your backup procedures as described above.

                                                    • 25. Re: Would other people like the option to write data to a side car for DNGs?
                                                      richardplondon Community Member

                                                      johnbeardy wrote:

                                                       

                                                      Equally, if you do want to take advantage of DNG don't be put off by Lightroom not writing sidecars that provide a second-rate backup. Just fine tune your backup procedures as described above.

                                                      Say you want to write out XMP, as added insurance against Catalog corruption (or pilot error).

                                                       

                                                      You regularly backup your Catalog also, and that is what you would likely go back to in the event of a widespread problem, since that is  the first-rate information with full History, Collections, Virtual Copies etc.

                                                       

                                                      But the externally written XMP is for protection and recovery of your most recent individual edits (to all kinds of image), made since that last available valid Catalog backup - it may also be of course, for communicating LR edits or other metadata to Bridge, or to some 3rd party DAM.

                                                       

                                                      You also have a regular backup routine for your images folders. So how complex and selective does that need to be?

                                                       

                                                      • You want to backup a proprietary Raw the first time it appears, but not subsequently, and this is not changing timestamp due to XMP.
                                                      • You want to backup a DNG the first time it appears, but not subsequently, and this is changing timestamp due to XMP.
                                                      • You want to backup a camera JPG the first time it appears, but not subsequently, and this is changing timestamp due to XMP.
                                                      • Tiff or PSD made from the above for Photoshop editing: you will want to backup initially, and then to re-backup whenever the timestamp changes due to editing the file contents, but not to re-backup whenever the timestamp changes due to writing XMP only.

                                                       

                                                      • If your exported JPG occurs inside this setup, and assuming you want to back that up at all, you will want to include this when it first appears, and to refresh the backup if over-written by an update re-export (with a new timestamp), but this file is not changing due to XMP.
                                                      • Your backup specification needs to reflect as many of the above contradictory requirements as you can manage.
                                                      • Separate XMP files belonging to your proprietary Raw files: optional, and easily distinguished.

                                                       

                                                      Contrast that with an OPTION where all external XMP writing by LR (or ACR) occurred as sidecar files only.

                                                      • Your proprietary Raw, camera JPG, DNG would not be changing timestamp due to LR metadata.

                                                       

                                                      • Your TIFF or PSD would change timestamp if the contents were edited in PS, but would not change timestamp due to LR metadata.
                                                      • Your exported JPG would change timestamp if overwritten / updated.
                                                      • All your metadata-writing would be concentrated into one class of files - XMP sidecars - which you could include, or not, in the backup as desired.
                                                      • perhaps, editing of Virtual Copies might then get the same short-term protection as their respective master copies can.
                                                      • So your backup specification is then probably as simple as: "backup all new or changed files except (or, including) .XMP files"
                                                      • 26. Re: Would other people like the option to write data to a side car for DNGs?
                                                        john beardsworth Community Member

                                                        Not exactly a fair contrast, is it? In practice it's not so complicated to fine tune your backup to pick up new DNGs (etc) and the catalogue, and adopting a sensible  backup strategy is likely to be more rewarding than hoping Adobe will switch resources to changing its long-established practice.

                                                        • 27. Re: Would other people like the option to write data to a side car for DNGs?
                                                          richardplondon Community Member

                                                          johnbeardy wrote:

                                                           

                                                          Not exactly a fair contrast, is it? In practice it's not so complicated to fine tune your backup to pick up new DNGs (etc) and the catalogue,

                                                          I was just pointing out: the requirements for the (etc) are different than they are for the DNGs. The (etc) includes Photoshop edit files, for example, which do change from time to time. So you have to program in separate backup behaviours (not just inclusions) for different file types, which is IMO a little complicated. And if the same filetype is involved but under two different roles you have to then forcibly separate these, in order to have the writing-out of XMP not give false positives as to what in fact does need to be backed up.

                                                           

                                                          and adopting a sensible  backup strategy is likely to be more rewarding than hoping Adobe will switch resources to changing its long-established practice.

                                                          The case can be made for practical benefits to some users, from turning on such an option; it is not purely academic and arbitrary. Of course there will still, on balance, be greater practical benefits for some other users, from leaving such a new option turned off.

                                                           

                                                          It might be that such a choice will in fact not become available; but it is surely not illogical to discuss it provisionally.

                                                           

                                                          It may be that some ACR users would be interested in the same options and behaviour, in terms of their interactions with LR and also with yet other software using the XMP sidecar file method. For one thing, it might be welcome if ACR could maybe see - and even make its own - virtual copies. Clearly ACR is going to have to start looking for the presence of such sidecars in situations where it currently may not; but this does not seem to me either completely unfeasible, or ridiculous in functional terms.

                                                          • 28. Re: Would other people like the option to write data to a side car for DNGs?
                                                            Mark Alan Thomas Community Member

                                                            I can't use the original raw files, because these DNGs were created from my old Canon 10D, and I didn't use the "embed original" option. Nowadays, with my Nikon, I don't bother to convert to DNG (though I do think the format has merits), so this is just an issue pertaining to my old Canon raw files. And actually, it only barely even qualifies as an "issue".

                                                             

                                                            I'd be fine with a checkbox which toggles where a DNG's XMP data is written, either within the dng or as a sidecar (or both). Just seems like having the option would be nice.

                                                            • 29. Re: Would other people like the option to write data to a side car for DNGs?
                                                              john beardsworth Community Member

                                                              But it is probably little more than a moot point.

                                                               

                                                              Is it really worth diverting development resources from other features to provide an option that provides little more than second-rate backup? I doubt it, but let's say we do go further and sidecars will include enough of the stuff like VCs, flags, history (optional at this level too?). Let's say we don't want to put resources into making the catalogue itself more robust or boosting its recovery capability. Let's ignore metadata like assignment to collections, books and slideshows on the basis that they can't be read elsewhere (huh, I thought we were talking backup value if the catalogue fails?), and we figure out how to write stacks metadata satisfactorily (folders + collections?). Or do we ignore the demand for Bridge and other Adobe apps to read LR-applied stacking? Flags as well? So probably not everything. Well, that means extra cost going into other teams, other features being canned in those apps, and internal energy going into whatever co-ordination and agreement needs to occur in a company without a dictatorial vegetarian CEO. Maybe we're smart and we're going to shave some cost out of the ACR compatibility by merging VC into sidecar metadata? OK. What about the third party angle and apps which read the embedded metadata and never expect sidecars? We also need to resolve how LR will read the metadata they have written.

                                                               

                                                              Even if you take quite a hard line on these details and can accept whatever may hit the fan, you're still into spending a lot of resources. On something that won't sell more than a few extra boxes? How feasible is that?

                                                              • 30. Re: Would other people like the option to write data to a side car for DNGs?
                                                                richardplondon Community Member

                                                                Suggesting nothing more here, really, than extending what is already done for proprietary Raw files, to DNG as well - with precisely the same content and similar implications, no more than that.

                                                                 

                                                                XMP is always about the individual image only, and not about its context within a Catalog, so these other attributes mentioned are not so relevant or even sensible IMO. For recovery, if the image was previously in your Catalog, you can carry on using its Collections etc. You are just re-loading some more recent editing on top of that.

                                                                 

                                                                Virtual Copies are a special case of writing the master file's metadata. Sometimes one needs to make a change to non-exposed metadata (such as, with LensTagger) and the only way to do that, is externally since LR does not provide internal "hooks" by which to do this. It is possible to take a roundabout route - switch copy and master, write metadata, modify externally, read metadata, switch copy and master back again - but it's awkward. And your most recent changes to VCs are definitely toast if anything bad happens. The risk of this last one is easy to overstate, I would completely agree there - but there's more than just one reason why people might want the ability to write (some) metadata out and to read that back in again.

                                                                • 31. Re: Would other people like the option to write data to a side car for DNGs?
                                                                  john beardsworth Community Member

                                                                  Then I simply return to my line of it being more a placebo than a backup, and to fining tune one's backup strategy in a DNG workflow. Also you can't dismiss collections because they are collected in the left panel - one might struggle with the idea of writing the collection object itself into xmp, but the image is individually assigned to a collection and other non-Adobe apps have written similar information to xmp.

                                                                   

                                                                  In any case, if the camera makers had adopted a DNG-style standard format 10+ years ago, would we really care that sidecars had never been dreamt up? They're a historical accident, and people grew used to them, but most people who use DNGs understand that having the metadata embedded in the file is one of the format's advantages.

                                                                  • 32. Re: Would other people like the option to write data to a side car for DNGs?
                                                                    Mark Alan Thomas Community Member

                                                                    This is a funny discussion. There's nothing to argue about here, yet people manage to argue about it anyway. Further proof that the natural state of human beings is to be in conflict.

                                                                     

                                                                    Me, I'd appreciate the option of that checkbox.

                                                                    • 33. Re: Would other people like the option to write data to a side car for DNGs?
                                                                      areohbee Community Member

                                                                      Mark Alan Thomas wrote:

                                                                       

                                                                      Me, I'd appreciate the option of that checkbox.

                                                                      And, it would be super-easy for Adobe to do it (and yes: I am qualified to say that). But I agree with John Beardy about one thing: Adobe may never do it - so it's worth considering plan B...

                                                                       

                                                                      PS - My theories about why Adobe might not do it are available upon request.

                                                                       

                                                                      Rob

                                                                      • 34. Re: Would other people like the option to write data to a side car for DNGs?
                                                                        DdeGannes Community Member

                                                                        Quote " In any case, if the camera makers had adopted a DNG- style ........ format's advantages"

                                                                        We may still be using ACR 3.6. The competition is driving the rapid advancement of the raw software processing technology.

                                                                        • 35. Re: Would other people like the option to write data to a side car for DNGs?
                                                                          john beardsworth Community Member

                                                                          Mark Alan Thomas wrote:

                                                                           

                                                                          This is a funny discussion. There's nothing to argue about here, yet people manage to argue about it anyway.

                                                                          Probably because you won't see that sidecars were an exception and only intended for proprietary raw files? Sidecars for file formats that don't need them - now that's funny!

                                                                           

                                                                          Adobe aren't going to put effort into a placebo backup. As Rob says, Plan B.....

                                                                          • 36. Re: Would other people like the option to write data to a side car for DNGs?
                                                                            john beardsworth Community Member

                                                                            DdeGannes wrote:

                                                                             

                                                                            Quote " In any case, if the camera makers had adopted a DNG- style ........ format's advantages"

                                                                            We may still be using ACR 3.6. The competition is driving the rapid advancement of the raw software processing technology.

                                                                            You're confusing software processing and file formats (in any case the DNG format has progressed since it was introduced). But the basic point is that if camera makers had always used a publicly-documented standard format, we would never have needed sidecars and people wouldn't have become so hooked on them.

                                                                            • 37. Re: Would other people like the option to write data to a side car for DNGs?
                                                                              Keith_Reeder Community Member

                                                                              Mark Alan Thomas wrote:

                                                                               

                                                                              Further proof that the natural state of human beings is to be in conflict.

                                                                               

                                                                              No it isnt!

                                                                               

                                                                              ()

                                                                              • 38. Re: Would other people like the option to write data to a side car for DNGs?
                                                                                Jao vdL Community Member

                                                                                Now I have told you once!

                                                                                 

                                                                                Sent from my iPhone

                                                                                1 2 Previous Next