17 Replies Latest reply on Oct 14, 2016 6:52 PM by alanterra

    Lightroom slow deleting collection set. Advice?

    alanterra Level 1

      Just to add to the general chorus of complaints about Lightroom being slow.

      Most of the complaints are about developing or rendering being slow, but mine is about static (or "dumb") collections.

      I have been trying to fix metadata on my photos, so have been running Jeffrey Friedl's Metadata Explorer on my entire database, to figure out which metadata are correct and which are not. This has given me some collection sets whose collections total 100,000 or more photos.

      Deleting these collection sets takes a long time. I don't know how long. I started one last night, and it had finished by this morning. Today I tried to delete a second one, and after one hour, this is where the progress bar is:

      Screen Shot 2016-10-11 at 13.50.30.jpg

      I have been optimizing my catalog multiple times per day, but Lightroom is still slow.

      Any advice? Perhaps I should just bite the bullet and make a new catalog.

      A

        • 1. Re: Lightroom slow deleting collection set. Advice?
          dj_paige Level 9

          I doubt that the benefits of starting a new catalog will offset the disadvantages. The speed of most operations in Lightroom does not depend on the size of the catalog.

           

          Other than the speed of deleting collection sets, what else is slow?

          • 2. Re: Lightroom slow deleting collection set. Advice?
            alanterra Level 1

            So, if I understand you, you don't think that hours to delete a collection set of 100,000 items is unusual?

             

            I have been mucking around with my metadata, writing it using exiftool and then reading it in Lightroom. In one case I found it took 3+ hours to read the metadata on 500 photos and then another half hour for the updates to actually be applied to the photos (ie, be reflected in the catalog). Note, this was not updating previews, it was just "Read Metadata from Photos".

             

            Creating these collections of 100,000+ photos is also very slow. Jeffrey's Metadata Explorer finds the sets in question in a quite reasonable amount of time, but then when you try to create a collection or collections based on the criteria, again you are in for a multi-hour wait. And this operation does not trigger a progress bar in Lightroom.

             

            A

            • 3. Re: Lightroom slow deleting collection set. Advice?
              dj_paige Level 9

              alanterra wrote:

               

              So, if I understand you, you don't think that hours to delete a collection set of 100,000 items is unusual?

              I guess you don't understand me. I said the speed of "most" operations, not every single operation, does not depend on catalog size. And this particular slow operation may or may not depend on catalog size, it seems like a bug or a hardware problem to me, not a catalog size issue.

               

              Nothing you have said convinces me that catalog size is the problem. There are people with catalogs of over 600,000 images and they are not reporting these problems.

              • 4. Re: Lightroom slow deleting collection set. Advice?
                alanterra Level 1

                I was asking for advice. "Seems like a bug" I guess is helpful, but what sort of bug? Obviously not shared by those people with 600,000 images. Where would I look for this bug? Hardware problem?, well I get read/write speeds of over 400 Mb/s on my Lightroom disk, and I have a pretty fast processor.

                 

                When I suggested going to a new catalog, I wasn't suggesting making a smaller catalog, I was suggesting making a new catalog. I suspect that this will take a long time, but some have suggested that it might help. I was asking for advice from those who might know more than I do about how to speed up these operations.

                 

                I might try the new catalog solution after the latest delete collection finishes, and will report here if I do.

                • 5. Re: Lightroom slow deleting collection set. Advice?
                  johnrellis Most Valuable Participant

                  While developing my plugins, I've noticed that when the plugins are creating very large individual collections (several thousand pics or more), the rate at which pics are added to the collection slows down more and more, the larger the collection.  (Technically, it appears that LR is using an inappropriate "n-squared" algorithm, where the time is proportional to the square of the size of the collection.)  But I noticed this slowdown only when doing the collection operations via the plugins -- when done in the user interface, they were very fast.

                   

                  But perhaps you tripping over a similar situation.   What if you deleted individual collections within the collection set, one at a time?  That might be painful, but if LR is using an n-squared algorithm, it might go much, much faster than deleting an entire collection set.

                  • 6. Re: Lightroom slow deleting collection set. Advice?
                    Abambo Level 4

                    To add my 2 cents: I don't think that deleting such large collection sets is a common task. So I'm quite sure that nobody inside of Adobe ever questioned the algorithm they use.

                    • 7. Re: Lightroom slow deleting collection set. Advice?
                      alanterra Level 1

                      Thanks, John, for the thought.

                       

                      I did the minimal testing, and found that deleting items from a collection is (approximately) linear in the number of items, and is the same whether you delete the items from the collection, delete the collection, or delete a collection set with two collections.

                       

                      The rate of deletion on my computer is 33/second or 2,000/min, which would translate into about an hour to delete a collection of 100,000 items. My final times for deleting large collections (about 110,000 items) were about 1hr40, so if there is non-linearity, it is certainly less than an n-squared effect.

                       

                      I guess the answer is that (at least on my computer), Lightroom doesn't deal with static collections well. This is unfortunate for me, as I often use Jeffrey Friedl's data explorer to partition my photos using criteria that the Grid view Library Filter does not offer.

                       

                      I was going to build a new Lightroom database to see if somehow that loosened things up, but it looks like that operation will take a weekend or longer, and I can't do that right now.

                      • 8. Re: Lightroom slow deleting collection set. Advice?
                        alanterra Level 1

                        Patient: "Doctor, it hurts when I do this."

                        Doctor: "So, don't do it."

                        • 9. Re: Lightroom slow deleting collection set. Advice?
                          johnrellis Most Valuable Participant
                          I often use Jeffrey Friedl's data explorer to partition my photos

                          Right, that's a typical example of how a number of plugins use large collections.  My Any Filter plugin has to estimate the time needed to create a large collection and ask the user if she really wants to do that.

                           

                          I did some quickie timings of adding and deleting from a collection via the user interface on a catalog with 26K photos.  I observed clear n^2 behavior for deleting but not adding:

                          For a collection of size 110K, it's easy to see how the rate could drop to 18 photos/sec (your observed rate).

                           

                          I ran out of time, so I didn't fully measure whether the rate is also a function of catalog size. I did do one measurement on a smaller catalog of size 7.3K: Deleting a collection of size 7.3K went at 1604 photos/sec, which is roughly comparable to the rate of a similar size collection in the 26K catalog.   This suggests the rate may not be a function of catalog size, just collection size.

                           

                          Of course, this doesn't help you with your practical problem.  You might discuss with Friedl.  One possibility would be for Data Explorer to provide an option to use a custom metadata field rather than collections to represent the photos in each bucketed value.  Then the user would use the Library Filter bar rather than the Collections panel to display sets of photos with the same bucketed value. 

                          • 10. Re: Lightroom slow deleting collection set. Advice?
                            alanterra Level 1

                            Well, that is eye-opening. My rates of deletion were 30/second for collections of 2,000 and 4,000 items in a catalog of 110,000 items. You are getting delete rates 60x faster than I am for medium-sized collections. I suspect that there is something wrong with my catalog, although it is possible that my multiple keywords and many custom metadata fields are the real issue, or perhaps there is a catalog size issue that you couldn't measure.

                             

                            FWIW, I have gone back and forth between using your AnyFilter and Friedl's Data Explorer -- I can't remember exactly why I used which one when.

                             

                            Your idea for Data Explorer is a good one, I'll contact Jeffrey.

                            • 11. Re: Lightroom slow deleting collection set. Advice?
                              johnrellis Most Valuable Participant
                              I suspect that there is something wrong with my catalog, although it is possible that my multiple keywords and many custom metadata fields are the real issue, or perhaps there is a catalog size issue that you couldn't measure.

                              The catalog I tested has about 1200 keywords and many custom metadata fields from a number of plugins.  I'd bet that deletion also depends on catalog size, but that's just an educated bet.

                              • 12. Re: Lightroom slow deleting collection set. Advice?
                                alanterra Level 1

                                Well, I made a new catalog (through the File:Export as Catalog... command) and now deletes of 100,000 or so items take 20 seconds or so, instead of an hour and a half. So that was worthwhile.

                                 

                                Thank you for your timing chart -- it certainly motivated me. And, for what it is worth, now my catalog is much faster than yours. I wonder if cruft builds up in a catalog and saving as a new catalog clears it up. (If I could do this for my own body....)

                                 

                                For anyone who is following this because they have similar issues, I don't know why but creating a new catalog through File:Export as Catalog... takes minutes, while creating a blank catalog and then importing from the old catalog takes days. I don't know why there should be such a difference -- perhaps it has something to do with bringing over the previews.

                                • 13. Re: Lightroom slow deleting collection set. Advice?
                                  johnrellis Most Valuable Participant

                                  Fascinating.  I can't imagine what might have been happening in your catalog that an export would fix.

                                   

                                  I tried optimizing my catalog and then an exported copy -- neither changed the delete speed.

                                  • 14. Re: Lightroom slow deleting collection set. Advice?
                                    Abambo Level 4

                                    I suppose that Adobe very often learns how users use the software. It's not an easy task...

                                     

                                    BTW: The doctor's advice is sometimes the good one... But that wasn't what I said.I only said that large collections are not that common... I didn't say that you should not do it...

                                    • 15. Re: Lightroom slow deleting collection set. Advice?
                                      alanterra Level 1

                                      Hi John

                                       

                                      Just a little follow-up. First, an engineer at Adobe is looking into this problem, so I have hopes that we will figure out what is happening.

                                       

                                      I can offer some additional data.

                                       

                                      You were getting delete speeds of 2,000 – 800 records/sec. on a 26K database.

                                       

                                      I was getting speeds of 20 – 30 records/sec on a 120K database. After I exported the database, I was getting extremely high speeds, but on trying to replicate that, I am now seeing speeds of 2,000 records/sec on a newly exported database. This is on deleting a collection of about 120K items that I created in the database before exporting it.

                                       

                                      But the interesting thing is, if I create one new collection set with two large collections in it (totaling about 120K items) in the exported database, the deletions after that slow down significantly -- to around 100 records/sec. So it seems that it is the process of making large collections that causes the subsequent deletions to slow down.

                                       

                                      Of course, I have no idea if this generalizes to anyone else's Lightroom database. I'll let you know if I hear anything from Adobe.

                                       

                                      A

                                      • 16. Re: Lightroom slow deleting collection set. Advice?
                                        johnrellis Most Valuable Participant
                                        But the interesting thing is, if I create one new collection set with two large collections in it (totaling about 120K items) in the exported database, the deletions after that slow down significantly -- to around 100 records/sec. So it seems that it is the process of making large collections that causes the subsequent deletions to slow down.

                                        Another one of your posts suggested this hypothesis: I think you may have a large number of photos synced with LR Mobile / Web. I know from developing another one of my plugins (Any Comment) that synced photos involve a number of other tables in the catalog database.   It could be that these tables aren't properly indexed, which accounts for the dramatic slowdown when deleting from large collections.

                                        • 17. Re: Lightroom slow deleting collection set. Advice?
                                          alanterra Level 1

                                          I don't use LR Mobile, and the thing is -- when you create a new database and export it, all the links to your published collections are severed. And remember it only gets slow when you create a collection -- until you create your first new collection, it is fast. So I'm pretty sure it isn't a missing index.

                                           

                                          My guess, and I literally know almost nothing about databases, is that it is indices being maintained while the delete operation is going on. I have seen discussions about how in many cases sqlite works better if you delete an index and then recreate it after a big operation. But, honestly, I'm out of my depth.

                                           

                                          I decided to clean up my file and delete some collections I no longer needed -- it took 5 hours.