21 Replies Latest reply on Mar 2, 2017 1:50 PM by johnrellis

    How to use XMP sidecar files for JPEGs in Lightroom

    Mark_A

      Hello, I am a new Lightroom user who has recently migrated from Aperture.

      I am working on a referenced catalogue of nearly 150K images. The file types I am working with are RAW, RAW+JPEG, JPEG, TIFF, PNG, PSD and Videos. I am using a mac pro (late 2013), OS X El Capitan and Lightroom/Photoshop CC.

       

      My workflow includes a lot of metadata editing and a few adjustments. For reasons of back-up efficiency and data safety I will like to find a way to store and sync all the changes I make on any file within Lightroom, on an XMP sidecar file.

      Essentially I will like to find a way to get Lightroom to edit and store metadata on all files in the way that it treats the RAW files.

      Now, i know that there is no way to do this in Lightroom at the moment, which is why I am bringing this issue to the community so that we can discuss workarounds.

       

      Essentially I need to work out how to do three things:

       

      1. Export metadata from JPEGs, TIFFs,PNGs and PSDs (videos would be pushing it I imagine) in XMP sidecar files

      2. Get Lightroom to read the metadata from the XMP file instead of the JPEG

      3. Automatically sync any modification of the metadata on these files.

       

       

      What I got so far,

      1. Found an export plugin which exports JPEGs with an xmp sidecar file. No option to only export the xmp or automatically export in next to the files in question, but still good enough. The plugin in question is:

      http://www.capturemonkey.com/xmpextract/

      Please let me know if you know of any others

       

      Sidenote / question:

      4. I have discovered with disappointment that when I changed my catalogue settings to automatically write changes on XMP, all my metadata were written on the files that are not RAW. So, xmp were created for all RAW images, but any file that is not RAW, now has the metadata written on it. I also found this to be irreversible. I understand that it is a similar function as an XMP, but written internally in the file but this is something I didn't want and didn't get the chance to chose. This feels a bit strange for Lightroom, so please, do correct me if I have done or perceived something wrong here.

       

      I know I have a few questions there, which is why I numbered all separate issues! Thank you all in advance

        • 1. Re: How to use XMP sidecar files for JPEGs in Lightroom
          F. McLion Adobe Community Professional & MVP

          To 4.: You can turn that off. I'm actually not sure - need to test first because I don't work with jpg - if this check mark only refers to develop setting or all metadata.

          The box "Automatically write changes into XMP" should be CHECKED ON!

          • 2. Re: How to use XMP sidecar files for JPEGs in Lightroom
            dj_paige Level 9

            Writing changes to XMP includes the current edit settings (but not the history sequence of changes), and most metadata. However things like collections, virtual copies, pick flags and a few other things are never written to XMP.

             

            To the original poster ... you seem to want Lightroom to do things that it was not designed to do, and Lightroom won't do those things. As far as I can see, you either need to write your own plugins or find someone who can do so for you; or find other software that does meet your needs, or change your requirements.

            • 3. Re: How to use XMP sidecar files for JPEGs in Lightroom
              Mark_A Level 1

              F.McLion

              It seems to write most metadata (IPTC,XMP,EXIF) but not all catalogue items as dj_paige mentions specifically. Turning it off, didn't 'un-write' the metadata from the JPEGs. I can revert back to my back-up for the original versions of these files, that's not a problem. It is just peculiar that it does that without a warning. I was wondering if anyone else came across this issue.

               

              dj_paige

              Fair point! I may have to get someone to write one or two plug-ins, I know! But I read discussions for the same issue from 2009 and struggling to find the solutions. Someone may have written a plug-in already or there may be external softwares to include in workflow to satisfy some of my requirements. Any ideas and suggestions are more than welcomed!

              • 4. Re: How to use XMP sidecar files for JPEGs in Lightroom
                dj_paige Level 9

                It is just peculiar that it does that without a warning. I was wondering if anyone else came across this issue.

                I wouldn't use the word "peculiar", and I wouldn't use the word "issue". This is how it works. This is how it was designed.

                • 5. Re: How to use XMP sidecar files for JPEGs in Lightroom
                  dj_paige Level 9

                  You might want to look at the freeware ExifTool. Perhaps it can create the xmp files you want (and perhaps not).

                  • 6. Re: How to use XMP sidecar files for JPEGs in Lightroom
                    Mark_A Level 1

                    dj_paige Just trying to understand the logic behind it as it will inform my workflow and practise. thanks for your input

                    • 7. Re: How to use XMP sidecar files for JPEGs in Lightroom
                      robgendreau Level 3

                      Probably a bummer of a task within Lr. Adobe is kinda about standards; since it's standard to include metadata in JPEGs, TIFF, DNG, etc, that's what it will do. That being said, there is that plugin, or the one "set capture time" in which you could run an exiftool command (see here for the variety of XMP sidecar options: Metadata Sidecar Files). But that's the easy part.

                       

                      The hard part is what to do next. If exporting, done and done. But if you add another keyword, you'd have to manually generate another sidecar. But Lr won't read them. You'd have to use exiftool or something else (and usually the something else uses exiftool anyway) to write the XMP back into the JPEG (not sure if exiftool can also do that with non-JPEGs), and then a "read from..." could get the info into Lr's database.

                       

                      Overall, very fiddly, non automated, and probably more likely to cause loss of data.

                       

                      But, if you wish to persist with the plan, then use Mylio. Mylio uses XMP sidecars as a means to pass data back and forth between devices so it doesn't have to pass the whole file (clever; maybe something you need to do?). So it creates sidecars...even for JPEGs and DNGs. These are pretty much fully compatible with Lr BUT it requires a two step process going FROM Mylio to Lr: You issue a command in Mylio to write metadata changed in Mylio to the FILE, then in Lr do a "read from file" and boom, there it is in Lr. In the other direction, it works more automatically: Mylio auto reads from the JPEG, writes to the sidecar, sends the sidecar to your NAS, iPad, or whatever, and boom! the changes (like keywords, crop, B&W, WB) show up on the other devices.

                       

                      Or try Graphic Converter. It has a bunch of built-in metadata tools and scripts, and can copy out an XMP sidecar and copy it back in.

                      • 8. Re: How to use XMP sidecar files for JPEGs in Lightroom
                        trshaner Adobe Community Professional & MVP

                        System level file management does not allow having two files with the exact same name in the same folder. So what do you name the XMP files for a CR2 file, and any combination of JPEG, PNG, PSD, TIFF, and DNG file copies with the same file name?

                         

                        The primary issue is legacy compatibility:

                         

                        LIGHTROOM / ACR Have the possibility to store the xmp metadata outside DNG, jpeg etc file to be backup efficient | Photo…

                         

                        • 9. Re: How to use XMP sidecar files for JPEGs in Lightroom
                          robgendreau Level 3

                          You can have two files with the same filename, but different extensions. So IMG1234.cr2 and IMG1234.xmp is how it's done by Adobe.

                           

                          Most of the the extensions I use with Lr that create new files from the RAW add something to the filename, so IMG1234-edit.tiff for example. Or IMG1234-pano.TIFF, or IMG1234_DxO.dng. One doesn't create a second RAW; if you were to duplicate IMG1234.cr2 for some reason in the same folder the OS wouldn't let you do it, so the problem for XMP solves itself.

                           

                          All the other file types store XMP internally, and that's standard, so that is how they have compatibility over time. So the problem for duplicate sidecars just doesn't arise. I do not see how generating atypical XMP sidecars for those TIFFs, DNGs, etc promotes compatibility when almost everything ignores them. As noted with the example of Mylio, they do have their uses. But they are a kludge, basically; other files don't store info externally. And one always has to keep them together. A problem with IMG1234.cr2 and IMG1234.xmp is that if one gets moved you've got big problems. Especially if another camera created a IMG1234.cr2. Or IMG1234.orf. Or the same camera does.

                           

                          The thread you refer to is mainly about the problem of backup; if the RAW is the same it's easier to keep backing up changes in the sidecar. For the same reason Mylio uses them. Be a great idea for a photo-specific backup application. But with the cheap cost of storage, and the fact many people just archive the RAW and store changes in the Lr database and/or unarchived working copies, there isn't much call for that application apparently. And some of us prefer DNGs for their error checking even so. Mr. Beardsworth mentioned that in his reply in the thread.

                          • 10. Re: How to use XMP sidecar files for JPEGs in Lightroom
                            trshaner Adobe Community Professional & MVP

                            robgendreau wrote:

                             

                            You can have two files with the same filename, but different extensions. So IMG1234.cr2 and IMG1234.xmp is how it's done by Adobe.

                            IMG1234.CR2 gets an IMG1234.XMP file and a matching in-camera IMG1234.JPG gets an IMG1234.XMP file. That can't be done since they have the same exact name! This is just one of many examples of issues that you could workaround, but imagine the complaints when things don't work as expected especially with existing XMPs

                             

                            At the link I posted Rob Cole explains the issues and what would need to be done:

                             

                            I also acknowledge that the problem is more complicated than it seems at first.

                             

                            As currently defined:

                             

                            MyFile.RAW and MyFile.JPG and MyFile.TIF.. would have the same xmp filename.

                             

                            I realize the problem is solvable, but due to it's impact on other apps etc, would be a can of worms.

                             

                            Bottom-line: Adobe screwed up (yeah: that's my opinion) by not including the original extension in the xmp filename, but it's a screw up which would be hard to rectify at this point. Easy I mean, in a closed/self-contained environment, but legacy compatibility must also be maintained, so other apps aren't broken.

                             

                            Don't get me wrong, I still vote for a remedy, and in another decade (probably two) we'd all laugh about the problems it caused at first..

                             

                            That said, I think the fact that Adobe chose NOT to include the original extension in the filename represents a firm commitment to go with embedded xmp in all cases where it's supported, and think of sidecars as an unfortunate exception. If I'm right, then the fight for sidecars is a losing battle - hope I'm wrong.

                             

                            FWIW: there *are* some apps which use sidecars for all file-types, and *do* include the extension in the filename. It works rather well, but these non-standard sidecars do not interoperate with apps which only support standard sidecars (like Adobe's). Note: these apps do NOT support embedded xmp.

                             

                            When both embedded and non-embedded must be supported, the problem is exacerbated - for example: there would not only be the potential issue of metadata conflict between xmp and Lr catalog, but embedded xmp and non-embedded xmp too...

                             

                            I can see why Adobe would not want to touch this problem with a 10-foot pole.

                            • 11. Re: How to use XMP sidecar files for JPEGs in Lightroom
                              robgendreau Level 3

                              The JPG only "gets" an XMP if you use something like exiftool to create it. No camera does it. Cameras don't even create sidecars for RAWs. You'd have to import both RAW+JPG. Without renaming them. And then use some application to create JPEG sidecars. And if you're doing that via exiftool, you'd just use exiftool to rename the JPEGs. So still no problem. That's a lot of rare contingencies to get to a problem. And as you can see by that thread, about two people a year complain about it.

                               

                              Using an XMP sidecar with a file that has XMP embedded is such a hack that I can't imagine a user that can't also figure out how to rename files to accommodate it. And again, why? Unless you have a workflow like Mylio's, what's the point?

                              • 12. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                trshaner Adobe Community Professional & MVP

                                let's stay on subject with the OP's request and analyze your solution:

                                 

                                "But, if you wish to persist with the plan, then use Mylio. Mylio uses XMP sidecars as a means to pass data back and forth between devices so it doesn't have to pass the whole file (clever; maybe something you need to do?). So it creates sidecars...even for JPEGs and DNGs. These are pretty much fully compatible with Lr BUT it requires a two step process going FROM Mylio to Lr: You issue a command in Mylio to write metadata changed in Mylio to the FILE, then in Lr do a "read from file" and boom, there it is in Lr. In the other direction, it works more automatically: Mylio auto reads from the JPEG, writes to the sidecar, sends the sidecar to your NAS, iPad, or whatever, and boom! the changes (like keywords, crop, B&W, WB) show up on the other devices"

                                 

                                Here's the OP

                                 

                                My workflow includes a lot of metadata editing and a few adjustments. For reasons of back-up efficiency and data safety I will like to find a way to store and sync all the changes I make on any file within Lightroom, on an XMP sidecar file.

                                Essentially I will like to find a way to get Lightroom to edit and store metadata on all files in the way that it treats the RAW files.

                                Now, i know that there is no way to do this in Lightroom at the moment, which is why I am bringing this issue to the community so that we can discuss workarounds.

                                 

                                1) In order to use Mylio to create XMP files for non-raw files (JPEG, TIFF, PSD, etc.) LR must write AND keep updated the XMP metadata contained in the actual image file: Setting Up Lightroom for Mylio Interoperability - Mylio

                                 

                                Mylio Photo Organizer - FREE, Premium and Max

                                • 13. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                  robgendreau Level 3

                                  My suggestion is to not use that workflow.

                                   

                                  Mylio can write Lr-compatible changes itself to sidecars. In other words,  you can edit a JPEG or RAW in Mylio, adding say keywords or making it B&W or cropping and changing exposure. When done Mylio by default ONLY changes the XMP sidecar, not the JPEG, DNG or CR2.  To repeat, Mylio can write the XMP. You don't need Lr to do that. Lr can ignore the XMP. So it doesn't get written into files themselves.

                                   

                                  Lr would ignore the changes unless and until you tell Lr to read the metadata, which it could do from the RAW CR2. For the JPEGs and DNGs, Lr doesn't read sidecars. So in Mylio you write the changes to the image, then tell Lr to read that. And yes, that would trigger a backup. But one need not get to that point if one doesn't want to import, as it were, the changes you made in Mylio into Lr. Just stick with the paragraph above.

                                   

                                  If one doesn't use Lr, the economical way to back up is to not write changes to images, but store it all in Lr.

                                   

                                  And BTW one can write XMP data back into image files with exiftool I believe. Or Mylio. I could lose my whole Lr catalog and almost all the info is stored in my image files, and in those XMP sidecars. And Lr CAN read the XMP files Mylio creates (it just doesn't read sidecars for ANY file that isn't supposed to have a sidecar). Unfortunately, one does have to pay for it. Such is capitalism. But exiftool is free. And does batches. You can use it to do much of what Mylio does, along with maybe rsync.

                                   

                                  And yes, any time you turn on your computer and do anything there's risk. Another reason to just take advantage of cheap storage and use something like Time Machine to back up the whole thing and not worry about it.

                                  • 14. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                    LewisLevin

                                    You are somewhat abusive.  His need is reasonable because some non-destructive editing applications will not update xmp embedded in jpeg or tiff files and create/update sidecars instead.  It is OK to admit you weren't aware of this. 

                                    • 15. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                      robgendreau Level 3

                                      I don't think he's abusive to point out that Lr will simply not do what the OP wants it to do, i.e. store metadata in sidecars to JPEG.

                                       

                                      Nor is it abusive to point out that Lr will write changes into JPEGs, since the OP must have selected the setting to automatically write such settings into JPEGs, as another poster noted. The default is to store them in Lr's database.

                                       

                                      And I have offered other workarounds, but in point of fact using sidecars with JPEGs is rather esoteric and uncommonly used, the preference in practice to be to either include that data in files or store it in a database.

                                       

                                      I'm a bit unsure about what you mean by your second sentence; are you saying that some non destructive editors write sidecars to JPEG? like Mylio? I think Aftershot Pro can, but IIRC they are proprietary and they wouldn't work with Lr in any case.

                                      • 16. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                        LewisLevin Level 1

                                        Seems like the tone was strong:  just use the product the way it was meant to be used…

                                         

                                        Capture One writes sidecars for edited jpg’s and tiff’s.  Capture One will not alter an “original” even if there is a well-defined standard for the structure of the embedded metadata.  Good?  Bad?  —it’s what they do.  I am not aware of any other product that takes this stance.

                                         

                                        I wrote a python script to copy 6 fields (could be expanded) from xmp to image or image to xmp.

                                         

                                        I think a lot of people, mostly who try to use Capture One sessions, end up with this problem.  It’s not fair to tell them they are wrong.  Tell the guys in Denmark they are wrong—many have—and they ignore it.

                                        • 17. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                          fulldecent2

                                          Per Adobe, you CANNOT have external XMP for JPG

                                           

                                          Quick Tip – Saving Changes to Files in Lightroom « Julieanne Kost's Blog

                                           

                                          FYI, this used to be possible with Lightroom and the new "feature" is undocumented in production release notes.

                                           

                                          Here is proof in case there is doubt:

                                           

                                          Screen Shot 2016-05-27 at 7.10.26 PM.png

                                           

                                          And don't laugh about my workflow, I was stupid back then.

                                          • 18. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                            DdeGannes Adobe Community Professional

                                            I have followed this thread with interest and to keep myself informed.

                                            My present workflow with Lightroom does not include the use of xmp files. When I adopted Lightroom as my major application for managing and processing my files "mainly" raw files from my digital cameras, I fully embraced the Lightroom default procedure which stores all info in the Catalog file. I do not use the option to write to amp.

                                             

                                            This is different to the Photoshop / Adobe Camera Raw procedure which stores the info in the file or an xmp sidecar for the raw files. In addition at the time I was also using two or three other raw processing applications which also created sidecar files. So I decided to keep my Lightroom workflow kiss (keep it simple sweet), the info is in the Catalog not in the file. I have not used Adobe Camera Raw since version 2.4. Presently use Photoshop CC Creative Cloud. (Mainly PS and LR)

                                             

                                            Sidecar files can be useful if need to share etc, however I discard them when they have served their purpose.

                                             

                                            If I need to do additional work in Photoshop or another editing application the "edit in function" works just fine for me.

                                            • 19. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                              ssprengel Adobe Community Professional & MVP

                                              I do everything in LR, but I automatically-create and keep my XMP files.  This is just in case the catalog gets corrupted, then my XMPs will at least have all the current settings.  This has saved me once or twice.

                                              • 20. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                                R L Hess

                                                I am disappointed by the "this is the way it works, live with it" attitude.

                                                 

                                                There is a very good reason to have all metadata written to sidecar files and that is to maintain file integrity.

                                                 

                                                For example, I create a heap of TIFFs from a scanner as my preservation master for an archive. I then clean up the files to make an access copy. Good archival practice it so keep checksums of all master files to make sure they haven't changed, but once I write the XMP data to the TIFF file (rather than a sidecar) then the checksum will show the file has changed.

                                                 

                                                Of course, we could find a tool that would create checksums for just the image data chunk and not the metadata, but if my experience with audio files and BWF Metaedit is any indication of how well that might not work, I don't hold out much hope.

                                                 

                                                As to the file system rule, FastSum gets around this very nicely with
                                                myfile.nef.md5
                                                myfile.jpg.md5
                                                myfile.tif.md5

                                                 

                                                If you don't like two periods, you could change the first period to perhaps a hyphen, but that would affect sort order a bit.


                                                [EDIT]

                                                I came to this thread looking about how to un-do reading the XMP metadata when opening a file in PhotoShop. Finally, I found that there is a setting for that in Edit Preferences Camera Raw which solved my problem. In that way, one TIFF file can be both the preservation master and the access master, depending on what you do with the XMP data. That still doesn't solve the backing up of the entire file every time you change the XMP data, but if you use RSYNC then the whole file is not actually copied.

                                                 

                                                I think perhaps the purest method for image archives, considering Adobe's reluctance to address this, is to only keep the XMP data in the LR database and write access file TIFs as well as JPGs, but that is wasteful of space.

                                                 

                                                Good points about the history NOT being written to the XMP data. That is too bad.

                                                [/EDIT]

                                                 

                                                Thanks!

                                                 

                                                Richard L Hess
                                                Audio Tape Restoration (and sometime image restoration)

                                                • 21. Re: How to use XMP sidecar files for JPEGs in Lightroom
                                                  johnrellis Most Valuable Participant

                                                  Richard,

                                                   

                                                  Please add your me-too vote and details of why you want sidecars for non-raw files to the feature request in the official Adobe feedback forum: LIGHTROOM / ACR Have the possibility to store the xmp metadata outside DNG, jpeg etc file to be backup efficient | Photo…  Adobe product developers pay attention to that forum but rarely participate here.  While I think it's unlikely Adobe will change this before the next ice age, you never know -- maybe it will catch the attention of a new product manager.

                                                   

                                                  Previous discussion in that feature request (and this thread) focused on how to introduce non-raw sidecars while maintaining backward compatibility. I just added a simple proposal of how to do that.