12 Replies Latest reply on Aug 24, 2016 7:22 AM by J0NNO

    Lightroom Catalogue Backup Speed and Size Issues

    J0NNO

      Hi All,

       

      We are trying to optimise our workflow and was after some advise...

       

      Originally we backup to a NAS unit via Gigabit connection (capable of close to 100MB/s sustained speeds) and were finding the Catalogue backup to take too long.

       

      We have recently purchased a new PC to get things working as fast as possible but still face speed issues generally with Lightroom but currently focusing on backup speed issues for now and will get onto the editing side a little later after we have spend more time trying to tweak and tune things further.

       

      New System and Equipment Overview:

      - Intel i7 6700K

      - 32GB DDR4 RAM

      - 512GB Samsung Pro PCIe M.2 SSD Drive

      - GTX960 GPU

      - Windows 10 Pro 64Bit (Latest Updates and drivers installed)

      - Adoble Lightroom 2015.5.1 (Latest build as of today: 12-May-2016)

      - Sony A7Rii RAW files (Compressed format around 45MB per file) - mentioned for general LR performance development processing speed issues

       

       

      Catalogue backup size and timed testing

       

      2.9GB - Original Catalogue Size

       

      928MB - Lightroom ZIP - 1Min 45Sec

      - Copy - 25 Seconds

      - Optimizing - 19 Seconds

      - Compressing - 80 Seconds

       

      640MB - Windows ZIP

      - Compressing - 81 Seconds

       

      73MB - WinRAR

      - Compressing -  28 Seconds

       

      File Copy test

      - 2.9GB Catalogue copy (read/write) from C Drive to C Drive with re-read of file to perform CRC File verification test: 28 Seconds

       

       

      * All testing above done from Drive C to the same Drive C

      * No system resources were flooded/pushed during the tests above with resource monitor (eg, CPU, RAM or Disk all reported under 25% usage other than WinRAR which did push the CPU further but still not all the way to 100%)

       

      Summary:

      - Lightrooms longes process during backup is the compression process, which also happens to produce the worse file output size? 928MB

      - Windows internal Zip processed the same catalogue down to a 640MB ZIP file (also does not push the PC at all in terms or resources)...

      - WinRAR produced far better results reducing the backup time by almost 3x and file output size by over 12x compared to Lightroom!

       

      Questions

      - Why does it seem LR Catalogue, compression and processing is currently rather inefficient or have I overlooked something?

       

      - Is there any way to speed LR up and get better performance and compression as it seems the LR Catalogue is a highly compressible database?

       

      - Is there any chance Adobe will be improving efficiency with their LR database and product as it does seem even with the latest Intel CPU's and PCIe SSD drives the product still does not run fast and is not snappy in most tasks... it still feels slow, heavy (and large in size)..etc..  I was hoping for a much improved performance jump with this new system but it seems LR is just 'slow'...

       

      We also have a copy of Capture One 9 we use for tethered capture (as it just works out of the box with no effort at all) but have never used it for editing as we have invested in LR for many years and all staff know Lightroom so would prefer to stick with it for now but it does seem LR is not optimising the product at all in many regards?

        • 1. Re: Lightroom Catalogue Backup Speed and Size Issues
          ManiacJoe Adobe Community Professional

           

          2.9GB - Original Catalogue Size

           

          928MB - Lightroom ZIP - 1Min 45Sec

          - Copy - 25 Seconds

          - Optimizing - 19 Seconds

          - Compressing - 80 Seconds

           

           

          If your LrCat files is really 2.9 gig in size, these times seem reasonable but not great.

          The major wait time would normally be in the "integrity test" phase before the actual backup starts.

          • 2. Re: Lightroom Catalogue Backup Speed and Size Issues
            J0NNO Level 1

            Hi ManiacJoe

             

            Thanks for the reply, may I ask why you think it's reasonable for the Lightroom results?

             

            Going on above testing Lightroom is almost 3 times slower than WinRAR and also puts out a backup file that is over 12 times larger.

             

            This is not a matter of being a small 5-15% speed different, currently it seems to be over 1,000% improvement in file size and close to 200% faster compression times... thats huge!)

             

            It seems Lightroom currently does not push the system and uses very little system resources during this process... almost like they are using an old generation compression engine that just can't use modern PC's resources correctly? (I'll continue to do some more testing to see what else I can find but open to any idea's if I've missed something but on face value this is significant)

            • 3. Re: Lightroom Catalogue Backup Speed and Size Issues
              dj_paige Level 9

              Lightroom isn't just creating a zip file, it is doing important things such as testing the integrity of the catalog and optimizing it. So to simply compare the times isn't relevant.

               

              You would be very wise, in my opinion, to allow Lightroom to do this and accept the time and file size. If you don't, you run the risk of future catalog malfunction/slowdown/corruption.

              • 4. Re: Lightroom Catalogue Backup Speed and Size Issues
                J0NNO Level 1

                Hi dj_paige,

                 

                Thanks for the reply, I've checked into this more to ensure I'm doing the testing correctly and this is what I've managed to confirm so far:

                 

                Backup with NO Integrity or Optimize:

                Copy Process: 10 Seconds

                Compressing: 70 Seconds

                Total Time: 80 Seconds

                 

                 

                Backup with Integrity and Optimize:

                Integrity Check: 3 Seconds

                Copy Process: 10 Seconds

                Optimise: 19 Seconds

                Compressing: 70 Seconds

                Total Time: 102 Seconds

                 

                 

                Original Database Size: 2.91GB

                LR Compress Output Backup file Size: 934MB

                Time Taken to Compress File: 70 Seconds

                 

                 

                WinRAR "ZIP" Backup Size: 714MB  (using default settings of 32KB Dictionary Size)

                Time Taken to Compress File: 24 Seconds

                 

                WinRAR "RAR" Backup Size: 75MB  (using default settings of 4096KB Dictionary Size)

                Time Taken to Compress File: 28 Seconds

                 

                WinRAR "RAR5" Backup Size: 55MB (using default settings of 32MB Dictionary Size)

                Time Taken to Compress File: 30 Seconds

                 

                What I'm basically trying to point out the following points:

                 

                1) LR's ZIP compression method seems to be very inefficient and perhaps a more modern compression method is better?

                - As you can see from the results above Light Room produces a 934MB backup file were as the more modern WinRAR 5 standard can get the same database down to 55MB!

                 

                2) LR's compression speed is also very slow

                - You will also noticed LightRoom currently takes about 70 Seconds on a system running PCIe attached SSD drives with the latest Intel i7 CPU on a catalogue of just under 3GB in size... The same process on a standard SSD drive or mechanical based system that many still use would be longer!

                 

                3) I do not have any issue with the Integrity Check of the database taking: 3 Seconds nor the copy process taking: 10 Seconds..etc the main issue I have is with the compression side of things and it's efficiency LB is achieving in terms of both time and output file size.

                 

                I'm raising the point in hope that Adobe might start looking into more optimised says to perform backups as mentioned before currently the system is just about idling whilst Lightrooom performs many of the tasks listed above showing it's not capable of using systems resources efficiently. This is not the only area LR seems to be slow but it's a very good example of inefficiencies one would hope Adobe will address in future builds/versions.

                • 5. Re: Lightroom Catalogue Backup Speed and Size Issues
                  dj_paige Level 9

                  J0NNO wrote:

                   

                  I'm raising the point in hope that Adobe might start looking into more optimised says to perform backups ...

                  Then you should tell Adobe that. File a formal feature request.

                  Photoshop Family Customer Community

                  • 6. Re: Lightroom Catalogue Backup Speed and Size Issues
                    J0NNO Level 1

                    Hi dj_paige,

                     

                    Thanks for the reply and link to the "Community powered support for Photoshop Family", so is that the forum/site Adobe monitors? And does it mean Adobe does not monitor this Adobe forum, I just assumed going directly to Adobe.com could be the best place to post topics such as this to get others feedback and if consensus on the findings Adobe would check into things as well?

                     

                    This is my first post on the forum so apologies if I have things posted in the wrong website, but boy Adobe does not make things easy, I originally tired to call them to discuss this before posting on this website and gave up now it seems I have the wrong forum as well... Not your fault and appreciate your help but wish Adobe made things easier.

                     

                     

                    • 7. Re: Lightroom Catalogue Backup Speed and Size Issues
                      J0NNO Level 1

                      Hi dj_paige,

                       

                      Just wanted to let you know I've copied the details above over to the other forum and will see how things go... thanks for the heads up!

                      Lightroom: Catalogue Backup Speed and Size Optimisation Please! | Photoshop Family Customer Community

                      • 8. Re: Lightroom Catalogue Backup Speed and Size Issues
                        J0NNO Level 1

                        Just wanted to provide an update I posted the details at the other more official Adobe forum for the developers to consider and unfortunately nothing has happened, speaking to a few other Lightroom users I did get told not to waist my time as a few mentioned they had also tried to report various issues in the past and it does not really seem to do anything so they stopped posting...

                         

                        Little sad really, paying for software licensing, updates and renew licencing and still get ignored by the vendor? Bit of a poor show Adobe! I also tried to call on this as well as some other performance issues (that was a complete waist of time and effort as well!)

                         

                        I really do hope Adobe pickup their game with Lightroom as I like the product but just wish it ran smoothly and efficiently.

                        • 9. Re: Lightroom Catalogue Backup Speed and Size Issues
                          dj_paige Level 9

                          So, my comments in reply #3 are irrelevant to your opinion?

                           

                          Lightroom backups are doing things that slow it down, but they are doing this for your own protection! A backup in which the database has lost its integrity is not a backup, it is a piece of trash. The optimizing of the database also slows down the backup but makes Lightroom run faster! These are all good things to do!

                          • 10. Re: Lightroom Catalogue Backup Speed and Size Issues
                            J0NNO Level 1

                            Hi dj_paige,

                             

                            Thanks for the reply, No I don't think it's irrelevant but I think I did reply to your comments at post #4 which you might have just forgotten as it was a while back back and I was just updating any other users interested in case they wanted to know how things were going after you helped point me in the direction of a forum that Adobe developers monitor more that's all :-)

                             

                            But just to re-cap I do understand backup very well (I actually work in the data backup and storage industry and have done so for almost 20 years now so I do have good understanding about databases, open files, integrity, optimisation, indexing..etc  and I've actually watches the adobe processes, disk I/O transactions on the system of both target and source and it's how I roughly calculated the break down below to be as accurate as possible within reason...etc)

                             

                            Backup with NO Integrity or Optimize selected in Lightroom:

                            Copy Process: 10 Seconds   (This was the time it took Adobe to copy the LR catalouge)

                            Compressing: 70 Seconds    (This was the time it took for Adobe compression process to run!)

                            Total Time: 80 Seconds

                             

                             

                            Backup with Integrity and Optimize:

                            Integrity Check: 3 Seconds

                            Copy Process: 10 Seconds

                            Optimise: 19 Seconds

                            Compressing: 70 Seconds

                            Total Time: 102 Seconds

                             

                            As you can see from the above I did factor in the integrity, copy, optimise processes and I did also monitor them at the OS system level to ensure what LR was reporting back to me was infact true as I also understand software does also do other processes it does not tell users about but in this case LR is not doing much more than it reports to users of any major impact that I observed during monitoring the threads and I/O operations on the system

                            • 11. Re: Lightroom Catalogue Backup Speed and Size Issues
                              dj_paige Level 9

                              (I actually work in the data backup and storage industry and have done so for almost 20 years now so I do have good understanding about databases, open files, integrity, optimisation, indexing..etc  and I've actually watches the adobe processes, disk I/O transactions on the system of both target and source and it's how I roughly calculated the break down below to be as accurate as possible within reason...etc)

                              Glad to hear that. I suspect that there are people on the Lightroom team who have similar qualifications, and they have valid reasons (which I do not know) for engineering the Lightroom backup the way they did. Adobe personnel almost never explain the reasons why some things are engineered the way they are, I doubt you will ever find out why the decisions were made.

                               

                              If you have stated the problem at the link I gave, that's all you can do for now, and hope that this is something that is improved in the next full release of Lightroom. There's no guarantee that because you have documented the problem at the other site, that Adobe will do anything, they have to make decisions on how best to use their resources and sometimes things that seem extremely important to one user (or many users) are not things that they choose to work on.

                              • 12. Re: Lightroom Catalogue Backup Speed and Size Issues
                                J0NNO Level 1

                                Hi dj_paige

                                 

                                Appreciate your reply and time taken to assist and point me in the right direction :-) Hope someone internally notices it as being in the data backup industry it's important we get good reliable backups and performing Lightroom backups a few times a day as well... if they can make them run multiple times faster and also a fraction of the output size I'm sure many other users would also do them more often and store more of them as well (eg, on every exit of LR or at least once a day)...

                                 

                                I've spoken to many people that don't backup at night because LR 'takes too long' and they will do it when they have time.. I try educate them backup backup backup but some people seem to want to always learn the hard way, however it's one of the reasons I posted these findings originally because I think it's so important to backup often and I think if Adobe use a modern compression algorithm that will essentially make it many times faster and substantially smaller and easier for users in a busy world more of them will backup more often as it won't consume as much time nor disk space...

                                 

                                Like you said we have all done what we can and hopefully someone at Adobe pickup the idea and run with it for a future release as from a coding point it should be such a simple update process perform (I expect 1 person could do the physical code change in under 10min + testing + merging + compiling + further testing + + +  so it does add up as you need to ensure it's solid but I suspect the actual issue might be Adobe having to pay a license fee to use a compression algorithm such as RAR which is producing outstanding results and Adobo probably don't want to pay , however there is still a range of options for them if that is the case.

                                 

                                Any way hope they mange to consider it and we can all benifit from more backups more frequently as our work in progress database I think is now around 8-9GB i believe (rather crazy really!) and doing backups I can feel the backup of users! it's slow.. really slow and I was considering writing something myself (thats another story all together though but I've promised myself not to go down that path and first force myself to find time to evaluate Capture One in more detail first as there is things I want to know about that produce other than tethering which is what we use it for at the moment... never enough time in the day!)

                                 

                                Night, dj_paige and thanks again for pointing me in the right direction to the required Adobe forum :-)  your assistance has been great