4 Replies Latest reply on Feb 21, 2013 10:09 PM by maruthid

    How to use batch mode in cq dam - new questions added

    maruthid Level 1

      Hi I am doing DAM migration. I am using custom code to pick the image and create asset by calling Assetmanager.createAsset which creates dam assets.

      This code I am calling through the scheduler which will run for every 0.5 hr. Around 1000 images we are trying to push in each cycle. I came acrosss batchMode for saving all the images per batch so that it will reduce the migration time But I am not sure how to use it.

      .setBatchMode(true); isBatchMode are the methods available in Asset api.

      Please share your thoughts on this.


      Some more Point I want to ask.

      1.What is the best appraoch when we are pushing images to DAM

           Stopping the workflows push all the images and run the work flows. (Is it possible to do this?)

           Do the both simultaniously

           Do it in the bath (i.e push bunch of images and then run the workflows)


      2.When i am migrating images (from file system which is residing in the same box where cq is running )it is almost taking 0.5 hr for 1000 images (avg size of image is 100kb). Is there any possibility to      improve this process.


      3. When I am adding images I am adding subassets to some of the images so when ever I add subasset cq is creating new version of the asset which is not required for me. What is the best approach to avoid      creating too many versions for images in DAM. By doing this is there any performance improve in the system.





        • 1. Re: How to use batch mode in cq dam - new questions added
          Jörg Hoh Adobe Employee

          Regarding the batch mode. You're import uses a session object as a connection to the repository. When you have called ".setBatchMode(true);" on your asset, any changes are not directly persisted on the session, but it's up to you to call "session.save()". This allows you to control the amount and the size of the transaction. As a transaction always has a certain overhead, it might be more performant to reduce the total number of transactions (= session.save() calls) and to increase the amount of work persisted with each transaction.

          You need to experiment a little bit with the amount of changes for each transaction. If you try to put too much changes into the transaction, it might blow up your JVM, as the changes to be saved are stored temporary in memory, and you might run into an Out-Of-Memory. My personal recommendation would be call every 10 assets the session.save() or so.


          Regarding the other questions: to reduce the time for the import itself it's best to temporarily disable the DAM asset update workflow; after the import re-enable it and touch all imported assets.


          Performance: Run a profiler to find out the bottleneck. Optimizing the transaction size might be a big factor here.




          • 2. Re: How to use batch mode in cq dam - new questions added

            how about using webdav and push your images using webdav client

            • 3. Re: How to use batch mode in cq dam - new questions added
              maruthid Level 1

              Hi Vijay,


              Initially we though of using Webdav but we need to do some process like creating subassets which is not possible through webDav.

              • 4. Re: How to use batch mode in cq dam - new questions added
                maruthid Level 1

                Hi Jorg,


                Thank you.

                I will try out batch saving option.

                I have few more doubts.

                1. Today I tried for moving aroung 5k images through the scheduler in my local machine. It went smoothly with out much issues but as usual some of the renditions are failing for some of the sub assets. This is about 3 for every 10 images. Renditions are generating properly for all assets it is failing in case of sub assets only. Along with this I am getting version exceptions. Below are errors which I am facing very frequently during migration dry run. This seems to be beacuse of the non synchronized execute methods in WF steps. What is your call on these issues how can we avoid these kind of errors.


                20.02.2013 19:16:00.082 *ERROR* [JobHandler: /etc/workflow/instances/2013-02-20/model_1361367956960012000:/content/dam/<IMAGE PATH>/subassets/<Image Name>/jcr:content/renditions/original] com.day.cq.dam.core.impl.AssetImpl setRendition: cannot set new rendition [cq5dam.thumbnail.140.100.png] for asset [<IMAGE PATH>/subassets/<Image Name>]:  javax.jcr.version.VersionException: Unable to perform operation. Node is checked-in.


                20.02.2013 19:17:01.660 *ERROR* [JobHandler: /etc/workflow/instances/2013-02-20/model_1361368020884400000:/content/dam/<IMAGE PATH>/jcr:content/metadata] com.day.cq.dam.core.impl.handler.xmp.NCommXMPHandler Stack Trace: java.lang.Exception: Unable to create revision.

                Caused by: javax.jcr.RepositoryException: Unable to update item: item.save()

                Caused by: javax.jcr.InvalidItemStateException: Item cannot be saved because it has been deleted externally: item.save()


                20.02.2013 19:19:08.666 *WARN* [JobHandler: /etc/workflow/instances/2013-02-20/model_1361368144852589000:/content/dam/<IMAGE PATH>/subassets/<Image Name>/jcr:content/renditions/original] com.day.cq.dam.core.process.ExtractMetadataProcess unexpected error occurred during metadata extraction. Cause: Unable to perform operation. Node is checked-in. javax.jcr.version.VersionException: Unable to perform operation. Node is checked-in.



                2. Can you share about experience in DAM migrations. What is the avg time it takes for migrating about 1 to 1.2 lac images which are aroung 60    gig. We thought of running the scheduler (this.scheduler.addPeriodicJob(jobName, job, config, period, canRunConcurrently);)for every 0.5 hr in  that case it will go on for 3 days.