34 Replies Latest reply: Sep 9, 2012 1:42 PM by emjaynie4 RSS

    XMP sidecar files

    DigPeter 2 Community Member

      I download raw files with Breeze Systems' Downloader Pro.  This creates XMP sidecar files.  I then process the files in LR4.0, including adding IPTC data like title, description.  When I export as jpg to a folder on my hard drive, theXMP file are exported also.      On the assumption that the jpg file has all the metadata embeddsed in the image file, I have no use for the XMP file with the exprted images.  How can I prevent the XMP files being exprted?

        • 1. Re: XMP sidecar files
          Hal P Anderson Community Member

          You can just delete them using Windows Explorer or Mac Finder. Befpre you delete all of them, try renaming one as a test to make sure that you'll get what you're expecting when you do the export with the .XMP files gone.

           

          Hal

          • 2. Re: XMP sidecar files
            areohbee Community Member

            Never heard of such a thing. Generally, xmp files only accompany proprietary raw files when exported in original format. Never heard of an xmp sidecar being exported with a jpeg. Has anybody else ever heard of such a thing?

            • 3. Re: XMP sidecar files
              Hal P Anderson Community Member

              Rob,

               

              Actually, not, but I took the OP's statement at face value.

               

              Hal

              • 4. Re: XMP sidecar files
                CatOne Community Member

                Yes, Rob.  Actually, if you have Aperture export masters, and select "Export XMP sidecars" it will export XMP sidecar files alongside ALL your "master" files (which could be JPEG, RAW, or TIFF).  For non-RAW files, Lightroom won't read the sidecars, thus any metadata you're trying to transfer over from Aperture to JPEG needs a different workflow depending on the file type.  Which stinks :-)

                 

                Maybe Breeze acts the same way.  I don't know what the XMP spec says about this, but it looks like there are a couple programs that do the same thing on the export side of things. 

                • 5. Re: XMP sidecar files
                  areohbee Community Member

                  OK, so the xmp files were exported from Breeze, *not* Lightroom - right?

                  • 6. Re: XMP sidecar files
                    DigPeter 2 Community Member

                    Thanks everyone.

                     

                    Without the background information, the question is whether it is possible to export image files from LR without any associated sidecar files?

                     

                    The background is that Breeze douwnloader creates the XMP files.  After processing the raw image files in LR, the sidecar files are then exported along with the converted jpg files.

                     

                    I woould rather not have to go to the trouble of deleting the XMP files.

                    • 7. Re: XMP sidecar files
                      martin-s Community Member

                      I woould rather not have to go to the trouble of deleting the XMP files.

                      Why don't you just use LR to download your images then?

                      • 8. Re: XMP sidecar files
                        JerryGerber Community Member

                        DigPeter 2 wrote:

                         

                        Thanks everyone.

                         

                        Without the background information, the question is whether it is possible to export image files from LR without any associated sidecar files?

                         

                        The background is that Breeze douwnloader creates the XMP files.  After processing the raw image files in LR, the sidecar files are then exported along with the converted jpg files.

                         

                        I woould rather not have to go to the trouble of deleting the XMP files.

                        This is still not making sense to me. It sounds like you are exporting from LR, and (as already pointed out) LR does not export xmp files with jpegs or tiffs. 

                         

                        I suggest you try importing using LR instead of Breeze Downloader Pro.  The problem should dissappear.

                         

                        Of course, your original XMP files associated with the raws will remain in the folder with the raw files.

                         

                        Jerry Gerber

                        • 9. Re: XMP sidecar files
                          richardplondon Community Member

                          I've just tried it with a Pentax PEF, where Lightroom had already written metadata to an XMP sidecar. Selecting "original" as the format when exporting, LR 4.1 did not include a copy of that XMP file alongside the Raw file, at the Export destination. I wondered if it might have done that.

                           

                          Edit: I stand completely corrected. There is an XMP there, but it is marked as a protected operating system file and therefore normally invisible even if you have selected to display hidden files. I'm on Windows Vista. XMP files in other circumstances are completely standard visible files.

                           

                          I only discovered this when I tried to manually make a new XMP file of that same name. I am not using Breezebrowser or anything similar, just LR.

                           

                          I am getting no XMP when exporting a JPG, regardless whether the original Raw file that is derived from, has an XMP saved alongside it or not.

                          • 10. Re: XMP sidecar files
                            areohbee Community Member

                            Hi DigPeter 2,

                             

                            Let me see if I understand what you are saying:

                             

                            1. You are importing raws into Lightroom which include xmp sidecar files created by Breeze (alongside the raws, same folder, same base filename, different extension).

                            So far so good?

                             

                            Cases:

                             

                            (it sounds like this is what you are saying, but that's hard for me to imagine):

                             

                            2A. You are exporting, from Lightroom, jpegs to a separate directory, and that directory is also having xmp sidecar files as a result of the export?

                            If so, this is a Lightroom behavior which I am not familiar with.

                             

                            Or is it

                             

                            2B. You are exporting jpegs from Lightroom to the same folder that contains the raws, so now you have a raw, a jpeg, and the original xmp, all in the same folder (the xmp being extraneous at this point).

                             

                            If so, then everything makes sense.

                             

                            You see where the confusion is? - Many of us are not familiar with the case of exporting jpegs + xmp sidecar from Lightroom, and I although it sounds like you are saying that is what's happening (case 2A), I have to assume I'm understanding that wrongly, and that what's actually happening is case 2B, or something. Either that or there is a hole in my understanding of Lightroom.

                             

                            ???

                             

                            Rob

                            • 11. Re: XMP sidecar files
                              DigPeter 2 Community Member

                              I find that the Breeze package suits my workflow in the the intial downloading and filtering of my images.

                              • 12. Re: XMP sidecar files
                                areohbee Community Member

                                Still, is Lightroom exporting the xmp sidecars when you export the raw files to jpeg from Lightroom, or are the xmp's coming soley from Breeze? I'd like to clear this up, if you don't mind.

                                 

                                Are you exporting to the source folder? Maybe that's where the confusion is. If so, then the xmp's originally written by Breeze will be overwritten by Lightroom whenever you save metadata, if you save metadata. And if you delete them, they will reappear when you save metadata, if you save metadata.

                                • 13. Re: XMP sidecar files
                                  DigPeter 2 Community Member

                                  Your point 1  -  correct.  This what Breeze downloader does.  After download each image has a raw (rw2) file and aXMP file.

                                   

                                  Your 2A is the correct interpretation.  In LR, I edit the raws, add title and description, then export to a different folder.  BOTH jpeg and XMP files are exported.by LR.

                                   

                                  How i get to this point is immaterial.  The XMP files are exported and I would like to know if I can set LR up to stop this.

                                  • 14. Re: XMP sidecar files
                                    DigPeter 2 Community Member

                                    Thanks richardplondon - please see my rep[y above to Bob Cole.

                                    • 15. Re: XMP sidecar files
                                      areohbee Community Member

                                      DigPeter 2 wrote:

                                       

                                      Your 2A is the correct interpretation.  In LR, I edit the raws, add title and description, then export to a different folder.  BOTH jpeg and XMP files are exported.by LR.

                                       

                                      How i get to this point is immaterial.

                                       

                                      How you get to this point is *VERY* "material", to me, since before today, if anybody ever asked me if this was possible, I'd have said "No way no how", and it sounds like you are telling me I'm wrong, and I want to know if I'm wrong.

                                       

                                       

                                      DigPeter 2 wrote:

                                       

                                      The XMP files are exported and I would like to know if I can set LR up to stop this.

                                       

                                      Since I can not imagine how this is possible in the first place, I have no idea how to set Lr up to not do it.

                                       

                                       

                                      It sounds like you are being bitten by a strange bug in Lightroom, although it's entirely possible this is just some new thing I've never heard of...

                                       

                                      Sorry I'm not more help, but right now I'm really questioning my sanity, and yours too if you want the truth.

                                       

                                      Rob

                                      • 16. Re: XMP sidecar files
                                        DigPeter 2 Community Member

                                        Thanks Bob - I am sorry if I have offended.  I am writing from my perspective.  I am no an expert in LR, only using it for the editng & conversion process.  I can assure you that I am as sane, but no more so, as the next person!  There is no doubt that the converted jpeg and XMP files are exported together.  This did not happen in LR3.

                                         

                                        What is the meaning of richardplondon's phrase "export as original" ?  This is not an option that I can see in the export dialog.

                                        • 17. Re: XMP sidecar files
                                          Hal P Anderson Community Member

                                          Here's a snip from the Export Dialog:

                                          Export One File_2012-07-07_14-49-47.jpg

                                          "Original" means to make a copy of your master file, without Lightroom modifications.

                                           

                                          Hal

                                           

                                          p.s. I'm with Rob on this one--I can't imagine LR exporting a .XMP file when you export a JPEG that is created from a raw.

                                          • 18. Re: XMP sidecar files
                                            areohbee Community Member

                                            Hi DigPeter 2,

                                             

                                            My name is Rob, not Bob. I am not offended, but I am interested in getting to the root of this issue. Ya see, I do fancy myself to be somewhat of an expert in Lr, yet I don't have a clue what's going on.

                                             

                                            Most quasi-experts in Lightroom would agree, there is no way for Lightroom to export an xmp file, except as a raw sidecar when exporting as original. Thus the dilemma many of us are having in this thread.

                                             

                                            Given that you don't see the "export as original" option, I'm even more perplexed. This would be under File Settings / Image Format - still don't see it? I've been assuming you had JPEG selected for Image Format, no?

                                             

                                            At the moment, my theory is:

                                            * Lightroom is seeing the xmp as "foreign", and is so dumping it out with the export, since it doesn't know what might be in it, or whether its needed...

                                             

                                            If that theory is correct, the solution is simply to save metadata in Lightroom, which will overwrite the now-obsolete (hopefully) xmp in native format, and then stop exporting it. But, it's just a theory at the moment.

                                             

                                            Rob

                                            • 19. Re: XMP sidecar files
                                              DigPeter 2 Community Member

                                              Thanks for that - I have this set permanently to jpeg, so had not noticed this option.

                                              • 20. Re: XMP sidecar files
                                                john beardsworth MVP

                                                You say there's no doubt but I really think you should check exactly what you're doing. Lightroom should not export XMPs for JPEGs as you describe, and AFAICS it doesn't. Are you certain you aren't exporting these JPEGs back to the folders where the XMP files were originally stored?

                                                 

                                                "Export as Original" is in the Export dialog, File Settings, Image Format, Original. That doesn't appear to do what you're describing either.

                                                • 21. Re: XMP sidecar files
                                                  DigPeter 2 Community Member

                                                  Hi Rob ,

                                                   

                                                  Thanks for the trouble you are taking on this.  I am not a technical guy, so cannot comment on what you say.  I had not seen the 'original' option, as I have it set permanently on jpeg.  As far as I can make out, the xmp file is redundant.  I have just moved some to a different folder from the raw files and the editing info and all metadata is unaffected, both in the raw state in LR and in the exported jpeg.  So, I reckon the problem is with the Breeze suite and I will eliminate the XMP files before processing importing into LR.   I might even experiment with downloading using LR.

                                                   

                                                  By the way, I do not 'save' metadata as a discrete action.  It seems to embed itself automatically into the image file as I enter data in the panel.  Am I missing something?

                                                   

                                                  Once again, thanks

                                                   

                                                  Peter

                                                  • 22. Re: XMP sidecar files
                                                    DigPeter 2 Community Member

                                                    johnbeardy wrote:

                                                     

                                                    Are you certain you aren't exporting these JPEGs back to the folders where the XMP files were originally stored?

                                                     

                                                     

                                                    Yes - with my work flow, the raw and xmp files are downloaded into an external hard drive.  The converted files are exported to the internal drive.

                                                    • 23. Re: XMP sidecar files
                                                      areohbee Community Member

                                                      Hi again (thanks for getting my name right )

                                                       

                                                      So did you test my theory? In other words, if you save metadata via Lightroom, do the xmp's cease to be exported with the jpegs?

                                                       

                                                      Lr's save metadata feature can be accessed from Metadata Menu -> Save Metadata to File - this will overwrite the Breeze generated xmp file with an Lr generated xmp file, so only do it if you've already got all the Breeze generated metadata into Lightroom. If my theory is correct, this will be a simple solution to your problem, and I will have learned something new .

                                                       

                                                      ?

                                                       

                                                      R

                                                      • 24. Re: XMP sidecar files
                                                        DigPeter 2 Community Member

                                                        Rob Cole wrote:

                                                         

                                                        So did you test my theory? In other words, if you save metadata via Lightroom, do the xmp's cease to be exported with the jpegs?

                                                        I have deleted all my XMP fiies, but will test again when I get back form holiday in a couple of weeks.  Should have a load of new images by then.

                                                         

                                                        I was not aware of the save facility for metadata or of its sigificence.  I will adopt this practice in future. 

                                                         

                                                        I will return to this on my return.

                                                         

                                                        Cheers

                                                         

                                                        Peter

                                                        • 25. Re: XMP sidecar files
                                                          areohbee Community Member

                                                          Standing by... (thanks Peter).

                                                          • 26. Re: XMP sidecar files
                                                            emjaynie4 Community Member

                                                            Hi Rob Cole, I've just read this thread hoping to find an answer to my problem, which I partially did, but it didn't work.  Here's my issue:   I'm gradually getting up to speed with LightRoom, and today decided to import my raw files (both NEF and CR2 files as I was using both cameras for a wedding yesterday) ... and try going through the whole selecting, deleting, batch renaming process in LR (instead of iPhoto which I only did because I was used to it ... nothing else though, I never edited anything in iPhoto, that got done in LR).

                                                             

                                                            So after my delete 'n' select process, I discovered how to get the renaming how I wanted it, etc. etc., then I exported the originals onto an external drive, and ... well this is why I came to this thread ... they exported with the XMP files.

                                                             

                                                            After reading this entire thread I tried the Save Metadata to File option, then exported a few to see how they would arrive in their destination folder, and alas, I still had the XMP files.  

                                                             

                                                            Is this normal when exporting renamed originals to a new folder?  Any help would be appreciated.  Thanks!

                                                            • 27. Re: XMP sidecar files
                                                              Cornelia-I Community Member

                                                              Hi emjaynie4,

                                                               

                                                              Yes, it is normal: whenever you have

                                                              a) set the metadata option to save to file

                                                              and

                                                              b) a proprietary raw file format (such as NEF and CR2, opposed to DNG)

                                                              LR cannot save anywhere else but into a newly created sidecar xmp-file. Because it does not know the proprietary formats, so way too dangerous to try to write anything into it.

                                                               

                                                              JPG and DNG are file formats which have a space reserved inside them, where to receive the metadata. This is the only reason why there is no separate xmp-file for these formats.

                                                               

                                                               

                                                              So if you hate xmp-sidecars, but want to stick with NEF and DNG, you need to disable the save-to-metadata option.

                                                              Then only the LR catalog will know them.

                                                              No problem, as long as you do proper backups of the catalog including optimizing, saving the backups somewhere else than your internal drive.

                                                               

                                                              Saving metadata into files/sidecars is just a safety net, that you find them again outside LR, in other programs capable of reading metadata.

                                                               

                                                              Cornelia

                                                              • 28. Re: XMP sidecar files
                                                                Jim Wilde Community Member

                                                                emjaynie4 wrote:

                                                                 

                                                                So after my delete 'n' select process, I discovered how to get the renaming how I wanted it, etc. etc., then I exported the originals onto an external drive, and ... well this is why I came to this thread ... they exported with the XMP files.

                                                                 

                                                                After reading this entire thread I tried the Save Metadata to File option, then exported a few to see how they would arrive in their destination folder, and alas, I still had the XMP files.  

                                                                 

                                                                Is this normal when exporting renamed originals to a new folder?  Any help would be appreciated.  Thanks!

                                                                 

                                                                If when you exported you set the Image Format (in the File Settings panel of the Exp[ort Dialog) to "Original", then that is exactly what you should expect, i.e. a COPY of the original Raw file (NEF or CR2) PLUS an XMP sidecar file which will contain all the associated metadata, e.g. develop edits.

                                                                 

                                                                But is exporting in the original format what you really need? After all, there will be a limited number of applications that will be able to read and correctly interpret the Raw+XMP combination. If you intend to send those exported files to the client, or put them on a web-site for the client's viewing, you may need to rethink the Image Format for those exports. Certainly JPG or TIFF would be a more usual export file format.

                                                                • 29. Re: XMP sidecar files
                                                                  emjaynie4 Community Member

                                                                  Thanks for your explanation, Cornelia ... that helps a little, considering I never even knew what an XMP file was before today (although I've heard of sidecar files).

                                                                   

                                                                  After exporting said files onto my external drive, when I looked down the list of exported files, I noticed that about 20 of them in a row have the DNG extension ..... that is, they're not in NEF or CR2 format, nor do they have an XMP file, just the DNG.   These were from the same batch, and I did nothing different with these, either in the camera or in LightRoom.   Can you explain that one to me???

                                                                   

                                                                  Thanks again!

                                                                  • 30. Re: XMP sidecar files
                                                                    emjaynie4 Community Member

                                                                    But is exporting in the original format what you really need? After all, there will be a limited number of applications that will be able to read and correctly interpret the Raw+XMP combination. If you intend to send those exported files to the client, or put them on a web-site for the client's viewing, you may need to rethink the Image Format for those exports. Certainly JPG or TIFF would be a more usual export file format.

                                                                    Yes, I need to export in original format because I want the raw files titles to match the jpeg file titles ... I title all my files before sending them to my clients, as image/file names are too confusing when viewed in a list.   After I title/rename my files and export the raws, then I export them all in Jpeg format also, for the client.

                                                                     

                                                                    In regards to whether they can read them, certain clients only get Jpeg files, but if I'm contracted to do a job as a 3rd party, the contract-or (so-to-speak) will get all the files, and I happen to know this particular one work with LR.  

                                                                     

                                                                    I hope that clarifies why I bother to export in original/raw format.  Just a little history here: but what prompted me to explore this aspect of LR was that while importing to iPhoto (which I only use for viewing and batch naming), I noticed that some of the images were in an unrecognizable format and couldn't be imported ... no particular ones, both NEFs and CR2s ... and after trying to trick iPHoto using another method of importing, it still didn't want to import those files, so I decided to bite the bullet and start using LR from scratch.

                                                                    • 31. Re: XMP sidecar files
                                                                      Cornelia-I Community Member

                                                                      You might have clicked on "Copy as DNG" when you imported your images. It can be quire easily confounded with your prbably usual "Copy", which is the second option in the import dialog.

                                                                      DNGs are not created by their own...

                                                                       

                                                                      A DNG is a raw format, which is public (Adobe is trying to get it recognized as standard like PDF for documents). When you choose to convert your proprietary files will be read as far as Adobe can and translated into this public DNG format.

                                                                      There are lots of rumors that you may miss some data in doing so. Certainly you say good-bye to the option to use Nikon-software for NEF or Canon-software for CR2 with the DNG-files.

                                                                      Converting to DNG only makes sense if you want to stick to Lightroom or other Adobe software or other software which is open source.

                                                                      Or if you want to anticipate that Nikon might stop to support NEF / Canon to support CR2. No joke if you think several decades ahead.

                                                                       

                                                                      Here is my choice:

                                                                      I always "Copy as DNG" during import, and have a second copy to a network storage drive. The second copy will get the same renamed name, but stays in proprietary format.

                                                                      So I have a DNG-version on my internal drive, and the NEF/CR2 on the network storage. I leave them there as a safety backup, or if I ever would want to use Nikon-/Canon-software on them. I know that this would be starting any development from afresh. (I have never yet done so.)

                                                                      On my internal drive I do every LR work on the DNGs, saving metadata into them, but not continuously, just by hitting <ctrl>s when I think it worthwhile: after develop module and after keywording/filling capture+title, rating.

                                                                      When I am done with most of the work, I move the images within Lightroom to another network storage, to free up my internal drive.

                                                                      Plus I appy several storage backups, of course.

                                                                      The DNGs are the images I consider relevant for long-term storage. Saving metadata into them is just an extra safety belt in case of hardware failure of if I would want to abandon Lightroom some day, so the catalog data need to go somewhere else.

                                                                       

                                                                      Cornelia

                                                                      • 32. Re: XMP sidecar files
                                                                        Jim Wilde Community Member

                                                                        That's fine...but it also explains why you see the XMP files after export, which I doubt you can avoid. It has nothing to do with the "Save Metadata" option, as that relates only to the original files, not the exported files.

                                                                         

                                                                        The reason you don't see an XMP file associated with the DNG exported files is because being an open, documented file type (unlike the proprietary RAW files) Adobe is quite happy to write all the XMP metadata into the file header of the exported DNG (this also applies to other open formats such as JPG and TIFF).

                                                                        • 33. Re: XMP sidecar files
                                                                          Cornelia-I Community Member

                                                                          Might I suggest one book about Lightroom?

                                                                          Spend (the equivalent of) not quite 15 British Pound and get an electronic copy of Victoria Bamptons Missing FAQs4:

                                                                          http://www.lightroomqueen.com/books/adobe-lightroom-4-missing-faq/

                                                                           

                                                                          You will find concise discussions of all sorts of organisation aspects, in an easily readable format. you can go for specific searches, or read complete chapters or even from cover to back.

                                                                           

                                                                          As hinted in my previous answer you can get the renaming done during import automatically with the tick for "Make a second copy to", so no need to export as originals later.

                                                                          Unless you do not yet know the future filenames during import.

                                                                          LR does not do anything with your images, but write referencing records about them into its own database=catalog.

                                                                           

                                                                          If you want a client to read your develop results from an NEF you need to give the xmp-file as well, otherwise he would only see the raw data as interpreted by his means of viewing them in his software, with a different interpretation possibly. In JPG files the results are included, which is why anybody can view them in a colour-managed program as you intended them.

                                                                           

                                                                          Cornelia

                                                                          • 34. Re: XMP sidecar files
                                                                            emjaynie4 Community Member

                                                                            As hinted in my previous answer you can get the renaming done during import automatically with the tick for "Make a second copy to", so no need to export as originals later.

                                                                            Unless you do not yet know the future filenames during import.

                                                                            LR does not do anything with your images, but write referencing records about them into its own database=catalog.

                                                                             

                                                                            Interesting info ... thanks Cornelia, and I will get the book.

                                                                             

                                                                            I wouldn't even consider naming my files during import, because a third of them will probably be chucked out and I don't know which ones they will be till I see them, hence no point in naming them until I've made my final choices.

                                                                             

                                                                            Also, I ONLY send Jpegs to my personal clients for the very reason you stated.   And I keep the RAWS.

                                                                             

                                                                            The only people who ever gets the RAWS are other photographers who contract me to work for them, then I check to see what programs they use.