Skip navigation
ocroquette
Currently Being Moderated

Lightroom processes for ages after withProlongedWriteAccessDo

May 21, 2012 4:05 AM

Tags: #plugin #performance #keywords #sdk3 #withprolongedwriteaccessdo

Hi,

 

I am considering a switch from Expression Media to Lightroom, and I wrote for that a Lightroom plugin that imports all my metadata from an EM XML file. Functionaly, it works great.

However, I have a performance issue problem: if I run the plugin on my whole photo collection (>30.000 items), then Lightroom gets crazy with the CPU and the memory after once my code returns.

The processing time seems exponential. If I run the plugin with 1.000 items, the return is almost instantaneous. On 5.000 items, it takes several minutes. With 30.000 items, it still wasn't finished after 30min, I killed Lightroom.

 

My guess is that Lightroom does some kind of post-processing, or maybe that's because Lightroom prepares the "undo".

Is that a known problem ?

 

Here is the skeleton of my code. Once doStuff2 returns, Lightroom does who knows what, and then doStuff returns too. This time seems exponential with the number of database changes my code causes.

 

 

 

function doStuff2(context, progress)
     
     local catalog = LrApplication.activeCatalog()
     local photos = catalog:getTargetPhotos()

     parseXml()

     for completed, photo in ipairs( photos ) do 
          if progress:isCanceled() then return end
          
          photo:addKeyword(...)
          setMetadata(...)
end

function doStuff()
     local catalog = LrApplication.activeCatalog()
     
     catalog:withProlongedWriteAccessDo( 
          {
               title="Import from Expression Media",
               func=doStuff2,
               caption="Initializing plugin",
               pluginName="Expression Media Import",
               optionalMessage = "The plugin will first read your Expression Media XML file and then import the data into Lightroom for "
                    .. ( # catalog:getTargetPhotos() ) .. " photos"
          })
     
end

import 'LrTasks'.startAsyncTask( doStuff )

 
Replies
  • Currently Being Moderated
    May 21, 2012 4:32 AM   in reply to ocroquette

    Lightroom can really get bogged down when doing large numbers of catalog updates as a single transaction.

     

    I suggest limiting to 1000 photos a whack. I think Lr4 is worse than Lr3 in this regard, or at least it is for me anyway...

     

    Rob

     
    |
    Mark as:
  • Currently Being Moderated
    May 21, 2012 4:50 AM   in reply to ocroquette

    You're welcome ocroquette.

     

    Only way I know is to return from the "with-do" function, and then go back in...

     

    Hint: Use withWriteAccessDo instead of the "prolonged" version.

     
    |
    Mark as:
  • Currently Being Moderated
    May 21, 2012 6:05 AM   in reply to Rob Cole

    Hi,

     

    As Rob said the official way to choose the commit point is to close a "with-do", and then start a new one for the next batch of transactions. This technique is useful when batching large numbers of changes to the catalog (as you are) or when making changes that only become visible after a commit (e.g. creation collections or changing their contents).

     

    Matt

     
    |
    Mark as:
  • Currently Being Moderated
    May 21, 2012 9:34 AM   in reply to Matt Dawson

    While every database technology has various resource issues with long-running, voluminous transactions, I wonder in this case whether the problem might not be with the relational database (SQLite) but rather LR's undo facility. 

     

    A quick experiment: Try catalog:withPrivateWriteAccessDo() instead. From the SDK:

     

    Provides write access to custom fields defined by your plug-in. Use this instead of withWriteAccessDo() if you are only modifying metadata for your plug-in and do not want to add the operation to the undo stack.

     

    The documentation is ambiguous, not differentiating between the intended use and actual semantics. But I'd guess that the only difference between withWriteAccessDo() and withPrivateWriteAccessDo() is that the latter doesn't add all the catalog changes to the LR application undo stack.

     

    If that quick change doesn't work, then I agree with Rob that breaking up the changes into smaller batches will give much better overall performance.

     
    |
    Mark as:
  • Currently Being Moderated
    May 22, 2012 12:14 AM   in reply to ocroquette

    ocroquette wrote:

     

    I was almost done, but now I have to refactor my code to process batch of 1.000 files instead of processing the 30000 files at one, and that's not a trivial task

    It's a pisser alright, and although not trivial, it's not too difficult either.

     

    I've been lobbying with Adobe to remedy this. It's one thing for it to take a long time, but a worse problem is that it can over-consume ram and hang the system.

     

    I decided to build the incremental catalog updating into my core methods (not yet released):

     

    http://forums.adobe.com/thread/1009149

     

    Even splitting the update into chunks, Lightroom gets slower and slower as it goes. For me, it usually makes it through if I leave it run long enough, but it's a mystery why it gets sooooooo slow after a time...

     

    Rob

     
    |
    Mark as:
  • Currently Being Moderated
    May 24, 2012 1:35 PM   in reply to ocroquette

    Just curious -- did you try catalog:withPrivateWriteAccessDo()? 

     
    |
    Mark as:
  • Currently Being Moderated
    May 24, 2012 1:37 PM   in reply to ocroquette

    The LrKeyword methods, along with most methods that access objects represented in the catalog, are very slow.  But at least they got about 6 times faster in LR 4 compared to LR 3.  LrPhoto has batch methods for getting metadata from many photos at once, but there's no equivalent for keywords.  I think it's a big deficiency in the SDK.

     
    |
    Mark as:
  • Currently Being Moderated
    May 24, 2012 1:39 PM   in reply to ocroquette

    Adobe wants all bugs and feature suggestions for Lightroom to be posted here:

     

    http://feedback.photoshop.com/photoshop_family/products/photoshop_fami ly_photoshop_lightroom

     

    Those few of us developers active here generally start the subject lines with "SDK: ".

     
    |
    Mark as:
  • Currently Being Moderated
    May 26, 2012 4:47 AM   in reply to ocroquette

    Hi Olivier,

     

    ocroquette wrote:

     

    I think I also found a bug. The following code:

         local keyword = catalog:createKeyword("Test" , { }, true, nil )
         local children = keyword:getChildren()

    Produces the followin error:

    An internal error has occured
    bad argument #2 to 'format' (number expected, got string)
    

     

    It it possible you will need to close a transaction (i.e. a with...Do call) between the first and second command, as the second one is accessing a change to the catalog that might not have been committed yet. This could be causing the bug you are observing. Does your workaround involve something like this? If so it might not actually be a bug and instead be operating "as designed", but not documented clearly enough to be easily understood by those new to the SDK.

     

    Matt

     
    |
    Mark as:
  • Currently Being Moderated
    May 26, 2012 5:03 AM   in reply to Matt Dawson

    Very possibly.

     

    I've been really stoked with my new multi-phase catalog accessor - strongly recommend writing something like it for yourselves (see source code in other thread, or contact me for fresh stuff).

     

    Here's an example from my most recent plugin:

     

    local s, m = cat:update( 10, "Assure Folder Collection Set", function( context, phase ) -- 10 is seconds to get in.

        -- function can take parameters, but not used here.

        if phase == 1 then

            set = catalog:createCollectionSet( folder:getName(), parent, true )

            return false -- keep going

        elseif phase == 2 then

            leafPhotos = folder:getPhotos(false)

            if #leafPhotos > 0 then

                return false

            else

                return true -- done

            end

        elseif phase == 3 then

            coll = catalog:createCollection( str:fmt( '[^1]', folder:getName() ), set, true )

            app:logVerbose( "Created/assured \"Folder Leaf Photos\" collection ^1 in '^2'", folder:getName(), set:getName() )

            return false

        elseif phase == 4 then

            local collPhotos = coll:getPhotos()

            if #collPhotos > 0 then

                local collPhotoSet = tab:createSet( collPhotos )

                for i, photo in ipairs( leafPhotos ) do

                    if not collPhotoSet[photo] then

                        toAdd[#toAdd + 1] = photo

                    end

                end

            else

                toAdd = leafPhotos

            end

            if #toAdd > 0 then

                coll:addPhotos( toAdd )

            end

            return true

        end

    end )

     

    Note: each phase is a new transaction.

     

    In your case Oliver:

     

    local keyword

    local children

    local s, m = cat:update( 10, "Keyword business", function( context, phase )

         if phase == 1 then

              keyword = catalog:createKeyword( "Test", {}, true, nil )

              return false -- continue to 2nd phase.

        elseif phase == 2 then

              children = keyword:getChildren()

              -- do something to catalog with children

              return true -- or nil.

         end

    end )

     

    This would not make sense if nothing further involved catalog writes (in which case just exiting a simple single-phase with-do method would suffice), but you get the idea, no?

     

    R

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points