13 Replies Latest reply on Jul 31, 2015 6:36 PM by benn brown

    Image ratings not read from XMPs after scanning for metadata updates

    matija1492

      Hello,

       

      I am having issues with LR CC 2015.0.1 on Windows 7 SP1.

       

      Summary:

      When scanning for metadata updates, everything from the XMP files is correctly loaded except for image star ratings. (color labels and develop settings are loaded correctly)

      This applies only to one specific set of images. Testing on other sample images works correctly!

       

      Here is my workflow:

      • I copied a set of images (a folder with 3 subfolders) to my PC and Mac
      • I gave star ratings in Photomechanic 5 on the Mac
      • I imported all photos into Lightroom CC on both computers (both computers are running the latest versions&patches of LR and Win 7 and OS X respectively)
      • In LR, I modified develop settings on Mac and further refined image ratings
      • I saved the metadata for all images
      • I took the XMPs and transferred them to PC
      • I scanned for metadata changes from LR
      • LR correctly imported the develop settings and color labels, but did not import image star ratings

       

      I transfered XMPs three times just to be sure, but nothing worked. I also gave ratings directly on PC and then overwritten the XMP files with the ones from Mac. This completely reset all ratings to zero. It didn't ignore them (leave them at 2 stars) -- it acted as if it read 0-star ratings from XMPs.

       

       

      I also opened XMPs from Mac to check their content, and sure enough, they all contain correct ratings. I will paste just the beggining of one such XMP here:

       

      <x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.6-c011 79.156380, 2014/05/21-23:38:37    ">

      <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">

        <rdf:Description rdf:about=""

      xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/"
      xmlns:photomechanic="http://ns.camerabits.com/photomechanic/1.0/"
      xmlns:xmp="http://ns.adobe.com/xap/1.0/"
      xmlns:aux="http://ns.adobe.com/exif/1.0/aux/"
      xmlns:tiff="http://ns.adobe.com/tiff/1.0/"
      xmlns:exif="http://ns.adobe.com/exif/1.0/"
      xmlns:dc="http://purl.org/dc/elements/1.1/"
      xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/"
      xmlns:stEvt="http://ns.adobe.com/xap/1.0/sType/ResourceEvent#"
      xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/"

         photoshop:DateCreated="2015-05-10T13:32:35.00+02:00"

         photoshop:SidecarForExtension="CR2"

         photoshop:EmbeddedXMPDigest="21F70DD88A3440DBFA2464737BD08EE8"

         photomechanic:ColorClass="0"

         photomechanic:Tagged="False"

         photomechanic:Prefs="0:0:1:000012"

         photomechanic:PMVersion="PM5"

         xmp:Rating="2"

         xmp:ModifyDate="2015-05-10T13:32:35.00"

         xmp:CreateDate="2015-05-10T13:32:35.00"

         xmp:MetadataDate="2015-05-15T16:41:18+02:00"

      ...


      So the data is clearly in there.

      To verify the issue, I took another set of images and performed the same test (skipping the Photomechanic ratings and using just LR) and everything worked correctly. The ratings were read after scanning for metadata updates.

       

      What might be the reason that LR CC won't read just the ratings of one particular set of images? Could it be that initial Photomechanic ratings somehow interfere with this process? I must emphasize that LR Mac correctly reads Photomechanic star ratings. I'll also repeat that the star ratings were further refined in LR after importing.

       

      How to go about fixing it? I would hate to have to manually enter hundreds of ratings again.

       

      Thanks!

        • 1. Re: Image ratings not read from XMPs after scanning for metadata updates
          johnrellis Most Valuable Participant

          Try selecting one of the problem pics in the Windows LR and doing Metadata > Read Metadata From File, rather than synchronizing the folder and selecting the Scan For Metadata Updates.  I've suspected in the past (but couldn't verify) that scanning was not always working properly, and I've definitely observed instances where scanning produced wrong results for finding missing photos (as compared to Library > Find All Missing). 

           

          If Read Metadata From File doesn't fix it, why don't you upload to Dropbox (or similar) one of the problem pics and its .xmp from Windows LR, and post the link here.  We can put the .xmp under the microscope to see if there is a subtle incompatibility or anomaly.

          • 2. Re: Image ratings not read from XMPs after scanning for metadata updates
            matija1492 Level 1

            Hi John,

             

            thanks for the new ideas. I did as you suggested and used "read metadata from file" command. Unfortunately, it didn't work, the results is the same - no star ratings are read.

             

            However, upon closer inspection, I came upon a new insight. After either "scan for metadata updates" or "read metadata from file" LR Win seems to have deleted the rating information from XMP file! To verify, I've overwritten the existing XMP (which had no rating info) with original XMP from Mac which does have rating. After reading metadata, sure enough, line reading "xmp:Rating="2"" was gone!

             

            I have uploaded one sample RAW file and both versions of XMP here:

            https://dl.dropboxusercontent.com/u/86972561/Temp/XMP_issues.zip

            I renamed the XMPs to more descriptive names so they need to be renamed back to work properly. The "Mac" XMP is the original containing the rating which should be read, and the other XMP is the results of "read from file" in LR Win.

             

            Can you reproduce the issue on your side?

            These steps should suffice:

            1. import only CR2 file
            2. copy XMP (Mac) to same folder
            3. read metadata from file and see what happens

             

            Any insights would be appreciated. Thanks!

            • 3. Re: Image ratings not read from XMPs after scanning for metadata updates
              johnrellis Most Valuable Participant

              I'm able to reproduce the problem with LR 5.7.1 and LR 6, Windows 8.1 and OS X 10.10.3.   I'll dig into it more to see what's going on.

              • 4. Re: Image ratings not read from XMPs after scanning for metadata updates
                matija1492 Level 1

                It's kind of comforting that you're able to reproduce the issue. Thanks for helping!

                • 5. Re: Image ratings not read from XMPs after scanning for metadata updates
                  johnrellis Most Valuable Participant

                  I have more understanding of what's going on:

                   

                  The .cr2 and the .xmp files both contain ratings:

                   

                  .cr2: rating = 0

                  .xmp: rating = 2

                   

                  I haven't found any specification for how this is handled, but I'm inferring that it's handled in a way similar to that recommended by the Metadata Working Group for resolving discrepancies between IPTC metadata embedded in a file and also stored in a .xmp sidecar.  My guess here is that this line in the .xmp file:

                   

                  photoshop:EmbeddedXMPDigest="EE74E1C6A44E10D4CAC648FE110E243E"

                   

                  is intended to represent a numeric digest (a checksum) of the contents of the XMP metadata in the .cr2 file.  However, it doesn't match the checksum of the current XMP contents of the .cr2 file, so LR says, "Ah, it must be that the someone modified the XMP in the .cr2 file after this .xmp sidecar was created.  So if I see an XMP metadata field that's both in the .cr2 and the .xmp, I'll prefer the value in the .cr2, since it must be more recent."   Thus, LR uses the rating=0 from the .cr2 rather than from the .xmp.    Changing the digest to "AD0ACF660F79AC49FD004B2DD3BC0461" causes LR to read the rating from the .xmp sidecar.

                   

                  The XMP embedded in the sample .cr2 doesn't contain the usual audit trail of which program added rating=0, so I can't help you there.   But it got added at some point.

                   

                  One way to fix this is to use the free Exiftool to delete the rating from all your .cr2 files:

                   

                  exiftool -xmp:rating= *.CR2

                   

                  Exiftool will automatically make backup copies of all files it modifies, in case anything goes wrong.  (But Exiftool is extremely robust, even with raw files from mainstream manufacturers.)

                  • 6. Re: Image ratings not read from XMPs after scanning for metadata updates
                    matija1492 Level 1

                    Hi,

                     

                    I can verify that your findings indeed are correct. Deleting embedded CR2 data as you suggested enabled LR to read metadata from sidecar files. (I will add that scan for metadata updates didn't work, but forcing "read from file" did.)

                     

                    However, something here doesn't make sense. To clarify what, here's my workflow in short:

                    1. shot images, no ratings in-camera
                    2. culled in latest Photo Mechanic (Mac) - added ratings to some files
                    3. imported in LR Mac (ratings are still visible)
                    4. refined ratings (ie. changed some ratings through LR) with some post-processing
                    5. saved metadata, copied to PC

                     

                    Photo Mechanic and Lightroom CC are new in my workflow (I used to use only LR 5.x on Mac and PC). Since you reproduced the issue on both Lightroom 5.7.1 and CC, it's highly probable that Photo Mechanic must be the offending party.

                     

                    What I don't understand is following: Photo Mechanic doesn't use sidecar files, so it must be writing directly into CR2s. It does this successfully since LR correctly imports the ratings. What then could cause the CR2 files to have 0 rating embedded within, when PM wrote non-zero ratings to many files?

                     

                    I find all of this a bit surprising - I was under the impression that many professionals use PM for quick culling and that it offers seamless integration with the rest of Adobe applications. I'll do some more investigation and try to contact PM for further support.

                     

                    I really appreciate all your effort! If you have any further theories on this issue, I'd be glad to hear them, but it appears to me that LR is at no fault here.

                     

                    Thanks again!

                    • 7. Re: Image ratings not read from XMPs after scanning for metadata updates
                      johnrellis Most Valuable Participant

                      You mentioned that this problem only occurred on one set of files, but not others.  Perhaps you accidentally used Photo Mechanic to write 0 ratings to that set of files after you had imported them into LR (and LR created .xmp files). 

                      • 8. Re: Image ratings not read from XMPs after scanning for metadata updates
                        DdeGannes Adobe Community Professional

                        Be aware that Lightroom by default reads and rights metadata to the Lightroom Catalog file.

                        However you can choose the option to write info into an .xmp file or to the file header of a DNG, TIFF or JPEG file. However it does not store all the info that is stored into the Catalog File. See the link below for details.

                        What Is Not Included In Lightroom XMP Files - Lightroom Fanatic

                        In addition Lightroom does not automatically read from the .xmp file, you have to specifically select the option for this to happen.

                        If you are importing a particular file into Lightroom that has .xmp data Lightroom in this process will read the info from the .xmp file.

                        • 9. Re: Image ratings not read from XMPs after scanning for metadata updates
                          matija1492 Level 1

                          John,

                          last week I made changes to my workflow - I used LR CC and Photo Mechanic both for the first time. When I mentioned the other set of files which didn't have this problem, I meant a control group where I only used LR CC to give ratings (without PM) and then copied XMPs to PC. I just wanted to isolate the problem since I made two significant changes to the workflow.

                           

                          There is no chance that I used PM to overwrite ratings after import into LR. I used it exactly once, for initial culling. I haven't run it since.

                           

                          Using Exiftool I analyzed a random file from my camera and realized that all my RAWs from 5D Mk3 contain "rating=0". This does make sense as camera supports in-camera ratings, so it would be logical that all ratings are 0 when the image is recorded. This rating within CR2 naturally persists throughout image lifetime if only LR is used, so LR must know how to handle this mismatch in order to still have non-zero ratings without modifying the raw file itself.

                          This gives credence to your findings of checksum mismatch, although I'm not sure it brings me any closer to the solution.

                           

                          Could you clarify how you read the checksums? I'd like to do some testing before contacting PM to shorten the process.

                          • 10. Re: Image ratings not read from XMPs after scanning for metadata updates
                            matija1492 Level 1

                            DdeGannes,

                            I set all my catalogs to write changes into XMP automatically. Additionally, to make sure, I hit Ctrl+S before copying XMPs to be certain that any pending metadata writes are executed before. As discussed above, in this case I tried scanning for metadata updates as well as forcing read metadata from file in LR.

                            • 11. Re: Image ratings not read from XMPs after scanning for metadata updates
                              johnrellis Most Valuable Participant

                              Interesting -- I downloaded a sample 5D MK III raw file and saw the rating=0 entry in the XMP metadata.  It does make sense, considering that the EXIF metadata section doesn't have any officially defined fields for rating.

                               

                              Looking at your workflow more carefully, it's clear that what you're doing -- copying the .xmp sidecars without the corresponding raws to the PC -- won't work.  When you don't copy the raws and sidecars together, you can get a sidecar whose digest doesn't match the XMP metadata in the raw, and in that case LR always prefers the metadata in the raw file.

                               

                              However, I think Photo Mechanic has the ability to write .xmp sidecars: Preferences | Camera Bits, Inc.. In which case, you should be able to copy the sidecars from Mac to PC and then reread the metadata back into LR PC (as long as you never modify the raw files themselves).

                               

                              --------------------------------------------

                               

                              Here's a more detailed look at your workflow and LR's use of digests:

                               

                              1. Copy a raw file to the Mac and PC.  It contains rating=0 in its XMP metadata.

                               

                              2. On the Mac, assign rating=5 using Photo Mechanic.  This overwrites the XMP metadata in the raw file.

                               

                              3. Import the raw file into LR Mac, creating a .xmp sidecar containing digest( rating=5 ).  (Remember, the digest encodes the value of the rating field in the raw file.)

                               

                              4. Import the raw file into LR PC, creating a .xmp sidecar containing digest( rating=0 ) (the raw file on the PC has rating=0).

                               

                              5. Copy the .xmp sidecar containing digest( rating=5 ) from Mac to PC. 

                               

                              On PC, you now have a raw with rating=0 and a .xmp sidecar with digest( rating=5 ). Since the sidecar's digest doesn't match the actual value of the rating field in the raw file, LR always prefers the value in the raw file.  This rule is based on the assumption that you always keep the raw and sidecar together, but you've violated that assumption.

                              • 12. Re: Image ratings not read from XMPs after scanning for metadata updates
                                matija1492 Level 1

                                I investigated and you're right. There are two ways to rectify my problem:

                                 

                                1. Copy RAWs+XMPs back to PC
                                2. Turn off "modify RAW files" option in Photo Mechanic, so it just writes XMPs. (apparently it creates XMPs anyway, but up until now it modified the RAW files itself in addition to creating XMPs) This means that RAWs on Mac and PC remain identical so I can just move XMPs and have everything work out, no digest mismatch.

                                 

                                I'm really grateful for the second solution since it's orders of magnitude faster and more efficient to just move XMPs than gigabytes of RAWs as well.

                                 

                                Thanks again!

                                • 13. Re: Image ratings not read from XMPs after scanning for metadata updates
                                  benn brown

                                  Hi Matja1492

                                  I have the same issue, and also can solve it by copying the RAW+XMP back to Pc.  However, I don't understand the 2nd option - Turn off modify RAW files..."  as if you do this I can't rate the images - can you explain for me please?

                                   

                                  thansk

                                  Benn