31 Replies Latest reply: Dec 27, 2011 5:47 PM by web-weaver RSS

    backing up takes almost an hour

    jimtron Community Member

      It routinely takes almost an hour to back up my main LR 3 catalog, which has over 300,000 images in it. Is that normal? It's pretty annoying having to wait for that when I shut down my computer at night. I'm optimizing the catalog right now, but IIRC I've optimized before and still had the long wait.

       

      I'm on a Mac Pro dual core 2.66 with 5GB ram, OS X 10.6.4.

        • 1. Re: backing up takes almost an hour
          hamish niven Community Member

          pain isnt it? long backups and testing integrity and then optimizing,

          Try unchecking the test and optimize for week days, and only doing that once a week.

          catalog.jpg

           

          This speeds up my tiny 79,000 image library catalog substantially.

           

          Or finish work an hour earlier to allow the back up time whilst you have a mug of ovaltine or horlicks or hot chocolate as a nightcap.

          Or dont back up at night, rather turn on the computer before breakfast,open and then close lightroom and allow it to back up whilst you have coffee and croissants, read the paper and watch the dogs playing in the garden.

          This is not a workaround, rather you have a HUGE catalog that takes time, this is more time management

          • 2. Re: backing up takes almost an hour
            jimtron Community Member

            Thanks for the reply, I'll give that a try. This is probably a stupid question, but where are those backup prefs? I see in preferences where you can choose backup frequency, but I don't see the preference pane you attached.

            • 3. Re: backing up takes almost an hour
              Hal P Anderson Community Member

              jimtron wrote:

               

              Thanks for the reply, I'll give that a try. This is probably a stupid question, but where are those backup prefs? I see in preferences where you can choose backup frequency, but I don't see the preference pane you attached.

               

              That's the dialogue box that comes up when the backup is about to start. Not a pref.

               

              Hal

              • 4. Re: backing up takes almost an hour
                jimtron Community Member

                Hmm, OK, I've been unchecking those items for faster backups, but today I checked them both since it's been a while, and my backup has been going on for over 5 and a half hours. Is that normal? This is with RC 3.2.

                 

                eta: According to the Mac Activity Monitor LR is using from 3.8 to over 4GB of memory--is that high for backing up? I've got 5GB total. CPU % for LR w while backing up averages 2.5-3%.

                • 5. Re: backing up takes almost an hour
                  hamish niven Community Member

                  how large is your database?

                   

                  I'd do one more overnight back up and then contact Adobe and see what they have to say on the matter.

                   

                  Other checks -

                  hard drive space?

                  fragmentation?

                  OS windows or Mac?

                   

                  Other than that, 5 hours is crazy, I  get peeved with 10 mins, but my catalogue is only 80,000 images and 1,000 are virtual images...

                  • 6. Re: backing up takes almost an hour
                    jimtron Community Member

                    I've got over 300,000 images, so it's a big catalog, but 5 hours (actually it's still going and approaching 6) seems like too much. Also, I forgot to add, it's slowing down my entire system--and I don't have any heavy duty apps like Final Cut running.

                     

                    My startup drive, where the LR cat resides, has 180GB free. How can I check for disk fragmentation? I'm on a Mac Pro.

                    • 7. Re: backing up takes almost an hour
                      hamish niven Community Member

                      Youno issue with mac and fragmentation, its a PC thing.

                       

                       

                      300,00 images, how many Gb is the file?

                      Click Lightroom - Catalog settings and you get this dialog that indicates the size in Gb of your catalog

                       

                      Screen shot 2010-08-27 at 08.31.38.png

                       

                      Backing up does seem to slow things, but 6 hours is not so lekker.

                       

                      I'd talk to Adobe about this,  try and PM Melissa Gaul - on the lightrom team

                      http://forums.adobe.com/people/MelissaJ.G.

                       

                      Sorry man, well out of my domain there.

                      Good luck

                      • 8. Re: backing up takes almost an hour
                        jimtron Community Member

                        My catalog is 4.37gb. I force quit LR last night when it was in the 6th hour of backing up and crippling my system. Today launched the catalog and optimized (without backing up) and it didn't take long. I wonder if it's the integrity check that's getting hung up. I'll do some more testing and contact Adobe support.

                        • 9. Re: backing up takes almost an hour
                          jimtron Community Member

                          Still having trouble with this. Backing up takes a very long time, plus it slows down my entire system. I've got 9GB of Ram, but when backing up LR other applications run significantly slower. Anyone else having trouble with this?

                           

                          I'm using 3.3 now. Also, I just PM'd Melissa Gaul.

                          • 10. Re: backing up takes almost an hour
                            web-weaver Community Member

                            Yes, back-up hangs up sometimes.

                            I have about 25,000 images in LR.

                            I made these observations:

                            1) The progress bar hangs when it is almost at the end. If I suspect it hangs I hit the enter key, a dialog box comes up saying that LR was not able to finish the back-up. Mostly it is fine on the second try.

                            2) The hang-ups occur when I have done a lot of work and not done a back-up for a few days. When I backup at least once a day, hang-ups don't occur.

                            3) I have never timed it exactly. I guess my back-up takes about 10 min.

                             

                            Skipping optimization of the catalog is certainly an option when the back-ups take too much time.

                            But I wonder if it's not worth the wait. I have a hunch that a not optimized catalog might slow LR down by quite a bit in its work.

                            WW

                            • 11. Re: backing up takes almost an hour
                              clvrmnky Community Member

                              I'm on a Mac with a decent sized catalogue -- 100K images or so (though I don't know the actual size at the moment.) Backing up takes only minutes.

                               

                              5-6 hours for a few GB of data is too long, and suggests there is a problem of some sort.

                               

                              I apologize of this has already been answered:

                               

                              Are you writing the backup to a local drive? Is this a RAID drive?  Is the drive (or drives) ok? Do you have any auto-backup or anti-virus service running? Most Mac users eschew real-time AV software, but it's worth asking.

                               

                              This sounds like a drive controller or adapter failing back to some tiny RW rate.

                               

                              Try this: while the backup is running slow, take a "sample" of the Lightroom process from the Activity Monitor.  Take a few samples (say, 3-5) every 2-3 minutes or so. Save these samples and either supply them to Adobe or share them here or both.  I'd like to see what it is actually doing.

                              • 12. Re: backing up takes almost an hour
                                jimtron Community Member

                                clvrmnky wrote:

                                 

                                I'm on a Mac with a decent sized catalogue -- 100K images or so (though I don't know the actual size at the moment.) Backing up takes only minutes.

                                 

                                5-6 hours for a few GB of data is too long, and suggests there is a problem of some sort.

                                 

                                I apologize of this has already been answered:

                                 

                                Are you writing the backup to a local drive? Is this a RAID drive?  Is the drive (or drives) ok? Do you have any auto-backup or anti-virus service running? Most Mac users eschew real-time AV software, but it's worth asking.

                                 

                                This sounds like a drive controller or adapter failing back to some tiny RW rate.

                                 

                                Try this: while the backup is running slow, take a "sample" of the Lightroom process from the Activity Monitor.  Take a few samples (say, 3-5) every 2-3 minutes or so. Save these samples and either supply them to Adobe or share them here or both.  I'd like to see what it is actually doing.

                                Yes, local drive, installed inside my Mac Pro. Not RAID. No sign of trouble from the drive--I run DiskWarrior on it a few times. Auto-backup--yes, Time Machine--maybe that's causing trouble? I'll try disabling Time Machine next time I backup LR.

                                 

                                I'll try to track what's going on with Activity Monitor and post it here.

                                • 13. Re: backing up takes almost an hour
                                  EyefulFrance

                                  I'm afraid I have no answer, but was wondering if you got one, and found out what the problem was? I'm having a similar issue, long looooong back ups. If you got any good feedback, would really appreciate your sharing my way. Thanks!

                                  • 14. Re: backing up takes almost an hour
                                    web-weaver Community Member

                                    I have only 20,000 images so can't really say how long backup ought to take for 300,000 images.

                                    But two things to consider: The backup files are quite large, so make sure that there's enough space on the drive where your back-ups are.

                                    Occasionally I delete older back-ups. The way I work, I see no reason to keep back-ups older than a month because restoring the catalog that much back in time doesn't make sense for me.

                                    Also, if you are on a Windows machine, defrag the hard drives - where the "working" catalog sits and where the back-up catalog sits - often. Particularly if your catalog is large.

                                     

                                    WW

                                    • 15. Re: backing up takes almost an hour
                                      jimtron Community Member

                                      Still having this problem, and it's very frustrating. I've reported this to Adobe--twice I think. Backing up (along with optimizing and testing) takes 4 hours or more, and pretty much cripples my system. I've got over 250gb of free space on my startup drive (where the catalog is located) and 9gb memory. LR 3.3, OS X 10.6.7, Mac Pro 2.6.

                                       

                                      I don't like to leave my computer on all night, and if I quit LR and backup/optimize while I'm working at the computer it's almost unusable. I've had my Mac checked thoroughly by Apple and don't have any problems with other programs (including Photoshop and Final Cut Pro).

                                       

                                      Activity Monitor shows that LR is using over 4GB of memory when backing up.

                                       

                                      Any ideas?

                                      • 16. Re: backing up takes almost an hour
                                        Pete Marshall Community Member

                                        jimtron wrote:

                                        . I've got over 250gb of free space on my startup drive (where the catalog is located) and 9gb memory..

                                         


                                         

                                        Any ideas?

                                        Are you actually backing up to the same drive as where the catalog is located? If so this isn't the best of ideas. It is far better, for obvious reasons, to backup to a separate drive than where your original catalog is located. It will also be faster in most cases.

                                        • 17. Re: backing up takes almost an hour
                                          jimtron Community Member

                                          I'll move my LR backups to another drive--I've got another internal drive I can use.

                                          • 18. Re: backing up takes almost an hour
                                            jimtron Community Member

                                            Pete said: "Are you actually backing up to the same drive as where the catalog is  located? If so this isn't the best of ideas. It is far better, for  obvious reasons, to backup to a separate drive than where your original  catalog is located. It will also be faster in most cases."

                                             

                                             

                                            Oh wait, no I'm not. I do have my LR catalog on my startup drive, but my backups are on a separate, internal drive. Is there any problem with having my catalog on the startup drive?

                                            • 19. Re: backing up takes almost an hour
                                              Geoff the kiwi MVP

                                              jimtron wrote:

                                               

                                               

                                              Oh wait, no I'm not. I do have my LR catalog on my startup drive, but my backups are on a separate, internal drive. Is there any problem with having my catalog on the startup drive?

                                              Should be no problem are you still having the time consuming issue?

                                              • 20. Re: backing up takes almost an hour
                                                jimtron Community Member

                                                Yeah, backup is still really slow and also slows other applications down. And most other operations are slow. This is with my main, very large catalog. I've been using a temporary catalog to edit after shoots, and then importing into my big catalog.

                                                • 21. Re: backing up takes almost an hour
                                                  jimtron Community Member

                                                  ambienttroutmask wrote:

                                                   

                                                  jimtron wrote:

                                                  . I've got over 250gb of free space on my startup drive (where the catalog is located) and 9gb memory..

                                                   


                                                   

                                                  Any ideas?

                                                  Are you actually backing up to the same drive as where the catalog is located? If so this isn't the best of ideas. It is far better, for obvious reasons, to backup to a separate drive than where your original catalog is located. It will also be faster in most cases.

                                                   

                                                  I see in LR preference where to set the catalog location, but how do I setup LR backups if I want to keep them on a different drive?

                                                  • 22. Re: backing up takes almost an hour
                                                    Geoff the kiwi MVP

                                                    When you go to back up this window appears:

                                                    Picture 1.png

                                                    and you choose the location with the Choose button.

                                                    • 23. Re: backing up takes almost an hour
                                                      jimtron Community Member

                                                      Thanks, don't know how I missed that!

                                                      • 24. Re: backing up takes almost an hour
                                                        jimtron Community Member

                                                        Still frustrated with this, I was hoping LR updates would improve it. Backup and optimization taking hours is annoying, but even more bothersome is that everything on my Mac gets dramatically slowed when backing up. I know I could do this at night, but I'd prefer not to leave my computer on all night. Is it normal for LR backup to slow my entire system down?

                                                         

                                                        Update on my system:

                                                         

                                                        • Mac Pro dual core 2.66, 9GB ram
                                                        • OS X 10.7.2, LR 3.6
                                                        • Running LR on 1TB internal drive with 593GB free
                                                        • 25. Re: backing up takes almost an hour
                                                          web-weaver Community Member

                                                          jimtron,

                                                          Yes, backing up the catalog is pretty intense for my system too but it should not take hours. My backup takes about 10 min.

                                                          See here what you can do to optimize the performance of LR: http://kb2.adobe.com/cps/400/kb400808.html

                                                           

                                                          With only one drive LR will not work as fast and efficiently as if you had more drives.

                                                          For security reasons it would be better anyways if your catalog backup were on a different drive - even if it was just an USB key.

                                                          WW

                                                          • 26. Re: backing up takes almost an hour
                                                            jimtron Community Member

                                                            web-weaver, thanks for the reply. I have read those optimization suggestions and am following most of them (running LR in 64 bit mode, keeping a lot of free disc space, optimizing often, increasing raw cache, etc). I do have multiple drives--could you elaborate on speeding things up using multiple drives? I've switched to backing up to a different drive and it hasn't made a difference.

                                                            • 27. Re: backing up takes almost an hour
                                                              web-weaver Community Member

                                                              When LR is slow part of the problem is in my opinion that the times for seeking, reading and writing to and from the hard drive add up over time. When you use LR, the hard drive(s) are involved in (a) running the system, (b) running LR, (c) reading from and writing to the LR catalog, (d) reading from and writing to the LR cache, and (e) reading from the image file and - if you have checked this option - write to xmp.

                                                              If all of these tasks are done from one and the same hard drive they can only be executed one after the other to the effect that there will be a queue of tasks for this one hard drive that gets longer and longer and LR slows down. Remember there is only one head on each drive that needs to do all the reading and writing and in between jump to the new location.

                                                              The solution is to spread the tasks over several hard drives, so that the tasks - or at least some of them - can be executed simultaneously.

                                                               

                                                              For these reasons I have the following setup:

                                                              LR "resides" on the C-drive - I think it's called "My Mac" on a Mac; my LR catalog is on the E-drive, while the LR cache is on the D-drive. These three are all internal drives.

                                                              My images are - except for a few - on external hard drives - but one caveat for externals: USB 2.0 is too slow, and the loading of an image from an USB 2.0 external takes forever. Fire Wire (1394a / b) is OK, and eSata is excellent and the best choice for external drives with images. I have no experience with USB 3.0 but I think eSata is faster than USB 3.0.

                                                              In fact speed wise I cannot tell a difference between my eSata external drive and my internal drives - but I haven't done bench tests.

                                                               

                                                              I have also done the following: I opted for rtendering 1:1 previews during import and for never discarding previews. Thus previews load faster in the Develop Module.

                                                              And I have set my cache to a generous 75 GB.

                                                              Does this help?

                                                               

                                                              You also have to remember that with a dual core running at 2.66 GHz your system is at the lower end of systems that can run LR smoothly. So you have to "help" LR with optimal system settings.

                                                              WW

                                                              • 28. Re: backing up takes almost an hour
                                                                jimtron Community Member

                                                                Yeah, I realize hard drive speed has a lot to do with it. I hope to upgrade to SSD's one of these days, but since I have over 380,000 photos it's not cost effective yet.

                                                                 

                                                                My LR catalogs live on my startup drive, which is internally installed in my Mac Pro. The link you provided on optimizing performance recommends keeping the preview cache with its catalog, which I do.

                                                                 

                                                                My original photo files are on a few drives, the most recent and commonly used is installed internally. I backup LR to a separate drive.

                                                                 

                                                                I also render 1:1 previews at time of import. My raw cache is 50gb; I'll try 75. Does anyone know how high I can go without negative effects? I have 9gb of memory and over 500gb free space on my startup drive (where LR cats are stored).

                                                                 

                                                                Thanks, that does help. I realized that my computer is pretty slow, but it still doesn't seem right that my system gets crippled during backup. And when I'm not using LR, I never have any problems with my system (and I use PS, Final Cut Pro, and tons of other programs).

                                                                • 29. Re: backing up takes almost an hour
                                                                  web-weaver Community Member

                                                                  You wrote:

                                                                  "The link you provided on optimizing performance recommends keeping the preview cache with its catalog, which I do."

                                                                  The wording chosen in the link is a bit confusing as they call it the previews cache file, when from the context it becomes clear that they task about the previews file - the .lrdata file. This .lrdata file should be together with the catalog (the .lrcat file).

                                                                  But the .lrdate file is not the same as the cache of which we can set the size and location when we set the Cache in > Edit >Preferences >File Handling tab.The latter is a temporary cache that LR uses to store previews -  temporarily. And when I suggested to put the Cache on a different drive than the catalog, I meant the cache that is set in >Preferences > File Handling tab. I did not mean the .lradate file.

                                                                   

                                                                  You say you have 380,000 photos. No wonder your backup takes hours. I have about 25,000 and my .lrcat file is 9.5 GB. So your backup should take at least 15 times longer than mine.

                                                                  I also noticed that the more often I do a backup, the less time it takes. Keep in mind that part of the backup process is the optimization of the catalog. In my experience it can take more time than the backup itself.

                                                                  WW

                                                                  • 30. Re: backing up takes almost an hour
                                                                    jimtron Community Member

                                                                    Ah, thanks for the clarification regarding the cache files. I do have my raw cache on a separate (internal) drive, not my startup drive where my catalog resides.

                                                                     

                                                                    So do you think it's normal to for the backup and optimization to take hours and cripple my system, due to the large number of files and slowish computer? I've heard that it shouldn't be a problem to have a catalog with hundreds of thousands of images, but I haven't heard about min. system requirements either. I do backup and optimize at least once a week.

                                                                    • 31. Re: backing up takes almost an hour
                                                                      web-weaver Community Member

                                                                      I don't know if it's normal for the backup of a catalog with 380,000 images to take hours, but to me it sounds reasonable under the circumstances and considering how long my backup takes (about 10 min). I think that the backup duration is probably more symptomatic for the speed of your system that it is a problem of LR.

                                                                      As far as I know you are the first one who has so many images in the LR catalog - I don't know of anyone else (but that doesn't mean there isn't anybody). Yes, Adobe says that it shouldn't be a problem to have all these images in one catalog. The only caveat I've read from Adobe is that "somewhere between 100,000 and 1,000,000 images your computer will run out of address spaces".But when it's said "it's not a problem" that means that it's possible; it doesn't necessarily mean it'll go fast. And you seem to have no other problems than the long duration of the backup process.

                                                                      Have a look how big your catalog file is - it must be in the range of 150 GB. This will take a while for your system to backup.

                                                                      WW