6 Replies Latest reply: Aug 10, 2011 1:50 PM by davidpope007 RSS

    "Date Modified" for all files being changed if "Automatically write to XMP" is on

    davidpope007

      Recently upgraded from LR3 (v3.4.1) from LR2 on OSX 10.6.8 and have always had Catalog Settings > Automatically write changes into XMP turned on.

       

      When browsing existing JPG files in my Library (no Develop changes, no keywording, no Presets, no Import), LR3 is writing to disk — i.e., when I look at files in Finder, almost every viewed file’s “Date Modified” is being set to today’s date and time. (It actually creates a .swp file, then changes it's name back to .jpg)

       

      This is really bad, as it's making it impossible for me to use Finder to figure out when I last worked with a file, it is triggering needless Time Machine and Backblaze backups, and unnecessarily churning my disk.

       

      If I turn off "Automatically write..." this behavior stops. Per David Marx at thelightroomlab.com, I tried turning off this preference, manually doing a "Save Metadata to File" for all files, letting that complete, then turning the preference back on. This does not solve the problem.

       

      Per a suggestion at photoshop.com, I used ExifTool to see what changes LR was writing to a sample file; from the diff below, you can see that LR is adding a bunch of new fields as well as moving other fields around. But my point is that LR3 should never overwrite a file on disk if all I am doing is browsing thru them.

       

      Is anyone else seeing this? Any ideas would be greatly appreciated!


      -- David

      diff Exif5609_original Exif5609_update
      2c2
      < FileName: DSC_5609_original.JPG
      ---
      > FileName: DSC_5609_update.JPG
      5,6c5,6
      < FileModifyDate: 2009:11:27 21:32:54-08:00
      < FilePermissions: rwxr-xr-x
      ---
      > FileModifyDate: 2011:08:07 22:06:47-07:00
      > FilePermissions: rw-r--r--
      27a28,29
      > ShutterSpeedValue: 1/200
      > ApertureValue: 7.1
      55d56
      < SerialNumber: 3209521
      75d75
      < Lens: 18-200mm f/3.5-5.6
      185,188d184
      < UserComment:
      < SubSecTime: 00
      < SubSecTimeOriginal: 00
      < SubSecTimeDigitized: 00
      211a208,299
      > XMPToolkit: Adobe XMP Core 5.2-c004 1.136881, 2010/06/10-18:11:35
      > CreatorTool: Ver.1.00
      > MetadataDate: 2011:08:07 22:06:47-07:00
      > SerialNumber: 3209521
      > LensInfo: 18-200mm f/3.5-5.6
      > Lens: 18.0-200.0 mm f/3.5-5.6
      > ImageNumber: 26634
      > RawFileName: DSC_5609.JPG
      > SavedSettingsName: Import
      > SavedSettingsType: Snapshot
      > SavedSettingsParametersVersion: 6.4.1
      > SavedSettingsParametersProcessVersion: 5.0
      > SavedSettingsParametersWhiteBalance: As Shot
      > SavedSettingsParametersIncrementalTemperature: 0
      > SavedSettingsParametersIncrementalTint: 0
      > SavedSettingsParametersExposure: 0.00
      > SavedSettingsParametersShadows: 0
      > SavedSettingsParametersBrightness: 0
      > SavedSettingsParametersContrast: 0
      > SavedSettingsParametersSaturation: 0
      > SavedSettingsParametersSharpness: 0
      > SavedSettingsParametersLuminanceSmoothing: 0
      > SavedSettingsParametersColorNoiseReduction: 0
      > SavedSettingsParametersChromaticAberrationR: 0
      > SavedSettingsParametersChromaticAberrationB: 0
      > SavedSettingsParametersVignetteAmount: 0
      > SavedSettingsParametersShadowTint: 0
      > SavedSettingsParametersRedHue: 0
      > SavedSettingsParametersRedSaturation: 0
      > SavedSettingsParametersGreenHue: 0
      > SavedSettingsParametersGreenSaturation: 0
      > SavedSettingsParametersBlueHue: 0
      > SavedSettingsParametersBlueSaturation: 0
      > SavedSettingsParametersFillLight: 0
      > SavedSettingsParametersVibrance: 0
      > SavedSettingsParametersHighlightRecovery: 0
      > SavedSettingsParametersClarity: 0
      > SavedSettingsParametersDefringe: 0
      > SavedSettingsParametersHueAdjustmentRed: 0
      > SavedSettingsParametersHueAdjustmentOrange: 0
      > SavedSettingsParametersHueAdjustmentYellow: 0
      > SavedSettingsParametersHueAdjustmentGreen: 0
      > SavedSettingsParametersHueAdjustmentAqua: 0
      > SavedSettingsParametersHueAdjustmentBlue: 0
      > SavedSettingsParametersHueAdjustmentPurple: 0
      > SavedSettingsParametersHueAdjustmentMagenta: 0
      > SavedSettingsParametersSaturationAdjustmentRed: 0
      > SavedSettingsParametersSaturationAdjustmentOrange: 0
      > SavedSettingsParametersSaturationAdjustmentYellow: 0
      > SavedSettingsParametersSaturationAdjustmentGreen: 0
      > SavedSettingsParametersSaturationAdjustmentAqua: 0
      > SavedSettingsParametersSaturationAdjustmentBlue: 0
      > SavedSettingsParametersSaturationAdjustmentPurple: 0
      > SavedSettingsParametersSaturationAdjustmentMagenta: 0
      > SavedSettingsParametersLuminanceAdjustmentRed: 0
      > SavedSettingsParametersLuminanceAdjustmentOrange: 0
      > SavedSettingsParametersLuminanceAdjustmentYellow: 0
      > SavedSettingsParametersLuminanceAdjustmentGreen: 0
      > SavedSettingsParametersLuminanceAdjustmentAqua: 0
      > SavedSettingsParametersLuminanceAdjustmentBlue: 0
      > SavedSettingsParametersLuminanceAdjustmentPurple: 0
      > SavedSettingsParametersLuminanceAdjustmentMagenta: 0
      > SavedSettingsParametersSplitToningShadowHue: 0
      > SavedSettingsParametersSplitToningShadowSaturation: 0
      > SavedSettingsParametersSplitToningHighlightHue: 0
      > SavedSettingsParametersSplitToningHighlightSaturation: 0
      > SavedSettingsParametersSplitToningBalance: 0
      > SavedSettingsParametersParametricShadows: 0
      > SavedSettingsParametersParametricDarks: 0
      > SavedSettingsParametersParametricLights: 0
      > SavedSettingsParametersParametricHighlights: 0
      > SavedSettingsParametersParametricShadowSplit: 25
      > SavedSettingsParametersParametricMidtoneSplit: 50
      > SavedSettingsParametersParametricHighlightSplit: 75
      > SavedSettingsParametersSharpenRadius: +1.0
      > SavedSettingsParametersSharpenDetail: 25
      > SavedSettingsParametersSharpenEdgeMasking: 0
      > SavedSettingsParametersPostCropVignetteAmount: 0
      > SavedSettingsParametersGrainAmount: 0
      > SavedSettingsParametersLensProfileEnable: 0
      > SavedSettingsParametersLensManualDistortionAmount: 0
      > SavedSettingsParametersPerspectiveVertical: 0
      > SavedSettingsParametersPerspectiveHorizontal: 0
      > SavedSettingsParametersPerspectiveRotate: 0.0
      > SavedSettingsParametersPerspectiveScale: 100
      > SavedSettingsParametersConvertToGrayscale: False
      > SavedSettingsParametersToneCurveName: Linear
      > SavedSettingsParametersCameraProfile: Embedded
      > SavedSettingsParametersCameraProfileDigest: D6AF5AEA62557FCE88BC099788BBD3CC
      > SavedSettingsParametersLensProfileSetup: LensDefaults
      > SavedSettingsParametersToneCurve: 0, 0, 255, 255
      > IPTCDigest: d41d8cd98f00b204e9800998ecf8427e
      228,230d315
      < SubSecCreateDate: 2009:11:27 21:32:54.00
      < SubSecDateTimeOriginal: 2009:11:27 21:32:54.00
      < SubSecModifyDate: 2009:11:27 21:32:54.00

        • 1. Re: "Date Modified" for all files being changed if "Automatically write to XMP" is on
          clvrmnky Community Member

          Much discussed in other threads. If the XMP being written to is /in/ the file (as it is for non-proprietary files) as opposed to a sidecar file, then the file IO being done is going to update the modified timestamp.

           

          Lr by and large does not write out anything to the referenced files unless you make a gesture that indicates you want this to happen, the great exception being what happens when this (by default, disabled) setting is enabled. In that case, every metadata change is an implicit gesture to write changes to the XMP block.

           

          The follow-on issues you describe are some of the side-effects of doing this.  The other alternative is for Lr to touch the file back to the original timestamp, which has its own problems.

           

          Consider, first, whether you need this setting enabled. Do you need to transmit the state of the image to other XMP aware apps regularly?  If so, then you have to balance this with other considerations.

           

          But, like I said, this is much discussed here. Perhaps someone else has other workarounds.

           

          But, what you are seeing is expected, as far as I know.

          • 2. Re: "Date Modified" for all files being changed if "Automatically write to XMP" is on
            davidpope007 Community Member

            Thanks for your prompt reply.

             

            What is strange is that this did not happen in Lightroom 2 and I had the same "Automatically write XMP" setting on.

             

            I could understand LR3 updating the XMP with new fields if I made some other changes to the file (develop, keywords, etc.) but why should the act of just looking at a file trigger a disk write?

             

            I will search again in these forums (found nothing the first time) for prior discussions on this issue.

             

            Your comment that "every metadata change is an implicit gesture to write changes to the XMP block" is quite puzzling to me, since by viewing a file I didn't request any changes to the metadata!

             

            Regards and thanks again,

             

            -- David

            • 3. Re: "Date Modified" for all files being changed if "Automatically write to XMP" is on
              davidpope007 Community Member

              Thinking about this some more and looking at that diff file, it is as if Adobe moved to a new toolkit for managing XMP:

               

              > XMPToolkit: Adobe XMP Core 5.2-c004 1.136881, 2010/06/10-18:11:35
              > CreatorTool: Ver.1.00

               

              and as a result added a bunch of new fields to "LR3 XMP" (as compared to "LR2 XMP"), e.g.:

               

              > SavedSettingsName: Import
              > SavedSettingsType: Snapshot
              > SavedSettingsParametersVersion: 6.4.1
              > SavedSettingsParametersProcessVersion: 5.0
              > SavedSettingsParametersWhiteBalance: As Shot
              > SavedSettingsParametersIncrementalTemperature: 0

              etc etc.

               

              Then when LR3 loaded my old LR2 images into memory, it "dirtied" the in-memory copy of the file by adding in these new LR3 XMP fields. Then, because I had "Automatically write XMP" on, it said "I better write these changes to disk".

               

              Yuck. As a former software engineer, this is very bad software engineering.

               

              It should wait until the user dirties the file (via Develop, keywords, etc.) before presuming to add a bunch of metadata fields that are unique to the new version of LR3.

               

              -- David

              • 4. Re: "Date Modified" for all files being changed if "Automatically write to XMP" is on
                ssprengel CommunityMVP

                You want to rely on the system-level file modification date as part of your workflow and that won't work because LR also makes changes to the contents of the file unrelated in time to your changes.  It's not a bug or a horrible design decision, just a difference of opinion about how things should work.  I'm sure Adobe had a good reason to do things this way, not just the mess people up. 

                 

                It would make sense that LR would want to update the fields in the file, itself, for performance reasons or for code simplicity, so it's easier to synchronize the folder or otherwise check if there have been any outside changes by another program and not have to do the metadata version upgrade in memory every time the file is looked at. 

                 

                I'm surprised that LR doesn't update the metadata in every file in your database, slowly, in the background, not just files you click on, but maybe it was a performance hit every time people upgraded their LR version so Adobe chose to do it different and avoid the bad press about performance being bad in the new version.

                • 5. Re: "Date Modified" for all files being changed if "Automatically write to XMP" is on
                  clvrmnky Community Member

                  davidpope007 wrote:

                   

                  Then when LR3 loaded my old LR2 images into memory, it "dirtied" the in-memory copy of the file by adding in these new LR3 XMP fields. Then, because I had "Automatically write XMP" on, it said "I better write these changes to disk".

                   

                  Yuck. As a former software engineer, this is very bad software engineering.

                   

                  It should wait until the user dirties the file (via Develop, keywords, etc.) before presuming to add a bunch of metadata fields that are unique to the new version of LR3.

                   

                   

                  Well, I'm a current software developer, and this is, really, a perfectly reasonable thing to do. It is a reasonable trade-off for a convenient feature required by a small subset of users.

                   

                  Yes, in most cases the in-memory copy should "never" be dirtied unless the user makes a gesture of some sort, but like I said earlier, this option (once set by the user) sets up the situation where this gesture becomes implicit. This is a clear trade-off for the sake of convenience. And if the XMP is out of date and needs to be updated en masse, so be it.

                   

                  The fact is, there is no easy way around this. Do we save up /every/ dirty buffer somehow until you make a gesture that /might/ require the XMP to be up-to-date before acting on that gesture? Now we have to worry about unflushed buffers if something goes wrong and the app exits. Do we save the buffers to the DB? Now we have to block some calls to make another blocking call to flush some or all of those to DB, and then write some or all of it out to one or more files. In what order? What if there is a gesture to have X files with up-to-date XMP and some or all of those are in unflushed buffers, unflushed DB writes or we have to wait for the DB.

                   

                  As you can see, this is a transactionality nightmare, and the easiest and safest thing to get what the user wants (i.e., up-to-date XMP for the purpose of talking to a third-party XMP aware app) is to simply update the sidecar or XMP block in an atomic manner using the correct file IO. The file will have to change at some point, so it may as well be now.

                   

                  In fact, the only solution that isn't a tranactioanlity nightmare, and one that is a feature request already, is to force Lr to always use sidecar files, even for filetypes with XMP blocks. I wouldn't hold my breath for this one, though.

                   

                  Regardless, the first thing we should do is evaluate if we even need this option enabled. If so, then we need to evaluate why we are relying on the file modified timestamp, because this is usually a good sign that something in the workflow is not ideal. I would use this opportunity to inquire into what, exactly, you need from the app, and why the current behaviour is causing you a problem. Because if the answer is primarily that you don't like the inelegance of the solution, then I'm afraid you aren't going to get much sympathy. If you need the functionality this option gives you, then there are side-effects that you will have to accept for this convenience.

                   

                  Anyway, I'm not expecting to convince you otherwise, since you seem to have made your mind up. I will finish by saying that decades of experience has taught me that absolute rules are rarely absolute, and the entire software industry is a trade-off because users require functionality that can run contrary to received wisdom. The type and extent of those compromises only change with the target audience for the app.

                  • 6. Re: "Date Modified" for all files being changed if "Automatically write to XMP" is on
                    davidpope007 Community Member

                    clvrmnky wrote:

                    davidpope007 wrote:

                     

                    Then when LR3 loaded my old LR2 images into memory, it "dirtied" the in-memory copy of the file by adding in these new LR3 XMP fields. Then, because I had "Automatically write XMP" on, it said "I better write these changes to disk".

                     

                    Yuck. As a former software engineer, this is very bad software engineering.

                     

                    It should wait until the user dirties the file (via Develop, keywords, etc.) before presuming to add a bunch of metadata fields that are unique to the new version of LR3.

                     

                    Well, I'm a current software developer, and this is, really, a perfectly reasonable thing to do. It is a reasonable trade-off for a convenient feature required by a small subset of users.

                     

                    Yes, in most cases the in-memory copy should "never" be dirtied unless the user makes a gesture of some sort, but like I said earlier, this option (once set by the user) sets up the situation where this gesture becomes implicit. This is a clear trade-off for the sake of convenience. And if the XMP is out of date and needs to be updated en masse, so be it.

                     

                    The fact is, there is no easy way around this. Do we save up /every/ dirty buffer somehow until you make a gesture that /might/ require the XMP to be up-to-date before acting on that gesture? Now we have to worry about unflushed buffers if something goes wrong and the app exits. Do we save the buffers to the DB? Now we have to block some calls to make another blocking call to flush some or all of those to DB, and then write some or all of it out to one or more files. In what order? What if there is a gesture to have X files with up-to-date XMP and some or all of those are in unflushed buffers, unflushed DB writes or we have to wait for the DB.

                     

                    As you can see, this is a transactionality nightmare, and the easiest and safest thing to get what the user wants (i.e., up-to-date XMP for the purpose of talking to a third-party XMP aware app) is to simply update the sidecar or XMP block in an atomic manner using the correct file IO. The file will have to change at some point, so it may as well be now.

                     

                    [Thanks to both of you for your detailed replies. I am aware of the need for tradeoffs so when you say the approach taken is quite reasonable, I do believe you. I also apologize in advance for the length of the following and am extremely aware of the time it must have taken you to compose the above replies, but I'm going to add a bit more, if only for my own piece of mind and in hopes of coming up with a solution for my workflow.]

                     

                    From my naive point of view, I was expecting the answer to be simply "don't raise the XMPDirtyFlag upon reading in a file". Obviously if your architecture requires you to "upgrade to latest XMP format" upon read, and another part of the system auto-detects "out of date XMP", then it's going to write those changes to disk.

                     

                    But it didn't need to be designed that way. LR obviously has mechanisms to know when a user has made a change to XMP so it is able to write XMP changes to disk only when necessary.

                     

                    The promise (to me) of "Automatically Write XMP changes to disk" was to auto-save my changes, and not those made for any internal (i.e., XMP versioning related) changes.

                     

                    Perhaps the premise is that it is LR3's job to update an individual file's XMP to the latest version so that other XMP-aware apps can make use of it? I would argue that those third-party XMP-aware apps already have to know how to deal with all prior versions of XMP, so LR3 should just leave well enough alone.

                     

                    You asked if my problem with your approach was that it was "inelegant"; not at all, it is based on my own perception of what I need from my workflow, so let me describe that so maybe we can find a better way:

                     

                    * Part of the appeal of LR to me is that it preserves my original file as it came off the memory card, allowing me to move to a different workflow/toolset in 2025 if I choose to do so

                    * However, with all of changes contained in a single database file, I'm concerned about rare (but possible) corruption, so to mitigate this risk, I let LR backup my database weekly and it's also backed up continuously in the cloud via Backblaze

                     

                    * Even with backups of the database, there is still a chance that I could lose changes made to individual files (e.g., LR corrupts the DB and I have to go back to last week's DB)

                    * Thus the appeal of the "auto-write to XMP" flag -- that way critical changes (develop, crop, keywords) are saved on a per-file basis; I liked the "automatic" part of this (as opposed to a manual save) because then I don't have to teach others in my family how to manually save XMP changes

                     

                    * A nice side-effect of this setting is now when I use Finder to find a file and double-click on it to edit it in Photoshop, all my develop changes are right there; (in other words, I like the flexibility of not having to fire up LR in order to just invoke PS from within it); also when I use Bridge I see all the keywords there

                     

                    * So with LR2, I had gotten used to what I thought was the best of all worlds -- autosave of changes at the file level via XMP + raw negatives untouched (i.e., Date Modified == the date I took the picture); this allows me to use operating-system-level tools -- Finder -- to locate/search for files

                     

                    * Now I upgrade to LR3 and I'm finally now understanding that a concept "XMP versioning" is going to result in changes to many, but not all my files. (That's something else that's annoying about this issue -- I open up the Grid and browse a folder of files, and only seemingly random ones I've cursored over seem to get written to disk -- if it's so urgent that LR3 update the XMP, then it should do it for all the files in the catalog or at least in that directory)

                     

                    Here's a screenshot from Finder of what I see everytime I look at this folder:

                    Screen shot 2011-08-10 at 1.42.42 PM.png

                     

                    * So now I have to assume that each new version of XMP and/or LR is going to touch my files on disk. Sigh.

                     

                    * What I don't like about this is that it is ruining the promise of "untouched raw negative". Yes, the image data is untouched -- which I agree is most of the benefit; but the file has been touched.

                     

                    * Perhaps you might empathize a bit more if you imagined that someone went thru all your source code or Word files and randomly changed the date to "today" because you upgraded compilers or moved to Word 2011.

                     

                    I agree all of this would be solved by having an XMP sidecar file for JPGs, but you indicate that's not going to happen.

                     

                    You've also alluded to the solution of "resetting the Date Modifed" to it's original value -- which I believe is what Finder does when you move or copy a file -- but that that is fraught with issues as well. I believe you when you say there are issues, but again the naive part of me wonders why that soultion would be so bad...

                     

                    I just thought of another potential solution -- turning on Date Created in Finder -- but it turns out that's changed, too.

                     

                    I am really at a loss as to what to do and would welcome your suggestions.

                     

                    Thanks again and kind regards,

                     

                    -- David