17 Replies Latest reply on Mar 2, 2010 3:59 PM by Harm Millaard

    Memory usage by ImporterProcessServer

    Howard Hoffman Level 1

      I have a Premiere project with about 1000 clips.  When Premiere opens the project, ImporterProcessServer will show a VM size about 1000 MB.

       

      I created a test sequence in Premiere with a single clip and chose File -> Export -> Media.  This eventually launches Media Encoder.  Once I started the job queue in AME, ImporterProcessServer will keep growing in VM size until it about doubles to 2000 MB.

       

      I am suspecting ImporterProcessServer does not share clip data used by Premiere and AME.  Worse, when I exit Premiere, its VM usage does not decrease.  Only when I quit both application did ImporterProcessServer disappear.

       

      Is there any setting I could change?  Or any practice that can help to reduce the memory usage?

       

      Thanks!

        • 2. Re: Memory usage by ImporterProcessServer
          Wil Renczes Adobe Employee

          Ironically, you're kind of describing the intent of ImporterProcessServer.

           

          In CS4, the push was to increase the ability of a 32 bit app to use memory in a 64 bit environment.  So the media handling routines were pushed into this separate process, and ImporterProcessServer is shared between various apps, so if there's some files in use by one of the Adobe apps, it'll stay resident in memory until all references to media are released.  The idea being, now, PPro, AME, and this separate process each can access 2 gigs of memory independently, so overall the apps have a bigger memory footprint to play with than they did in CS3.

           

          If you're finding that you're running out of memory, there is one pref in Premiere where you can toggle between optimizing usage for memory vs performance.  Try flipping it to memory.  But IIRC, ImporterProcessServer tries to determine how much memory it'll use based on what's available.  I don't think there's an explicit user control for it in CS4.

          • 3. Re: Memory usage by ImporterProcessServer
            Howard Hoffman Level 1

            The memory size observation seems to suggest that the memory usage is not shared b/w Premiere and AME.


            • 4. Re: Memory usage by ImporterProcessServer
              Wil Renczes Adobe Employee

              How do you figure?  I don't understand your logic here.

               

              ImporterProcessServer has an internal frame cache, so as you start to use more footage, it will grow until it hits a certain cap & saturates.

              • 5. Re: Memory usage by ImporterProcessServer
                Howard Hoffman Level 1

                If it's shared, once Premiere project is opened, ImporterProcessServer will have all the media loaded.  When AME starts to encode the sequence in the project, there memory size of ImporterProcessServer should remain roughly the same, not double.

                • 6. Re: Memory usage by ImporterProcessServer
                  Wil Renczes Adobe Employee

                  You might try checking what the memory usage is in Task Manager while you actively have Premiere in focus and are scrubbing the timeline. The difference is whether the files are in use or not.

                   

                  When you minimize Premiere, the application issues a 'quiet' call to all the attached media, which tells the Importer plugins to release any holds on files & memory usage (the idea being, say I'm alt-tabbing to another app, don't hog all the memory).  Once AME loads the sequence in question & starts referencing the media during the render, it brings everything back online & memory starts to climb again.

                   

                  Now, that doesn't account for a couple of things - it's probably different from media type to media type.  Not all codecs behave the same, and some require more memory than others.  And I won't rule out that it's possible one particular media type has a bug and isn't letting go of memory when it should.  (Any 3rd party video cards / plugins?)

                   

                  Anyway, feel free to disbelieve me, but I can definitely vouch that media references are pooled between the apps.

                  • 7. Re: Memory usage by ImporterProcessServer
                    Howard Hoffman Level 1

                    Not that I don't believe you.  I am just thinking there is some bugs.

                     

                    Scrubbing in Premiere timeline only result in the increase size of Premiere, not ImporterProcessServer

                    • 8. Re: Memory usage by ImporterProcessServer
                      Howard Hoffman Level 1

                      Here is a quick test.

                       

                      Clean start, no  ImporterProcessServer

                       

                      Open Premiere project, once done  ImporterProcessServer at 600 MB; exit Premiere.  Eventually,  ImporterProcessServer disappear

                       

                      Open AME, open a sequence in the above project,  ImporterProcessServer at 600 MB; exit,  ImporterProcessServer disappears

                       

                      Open Premiere project,  ImporterProcessServer at 600 MB; Then open AME for one sequence in this project,  ImporterProcessServer at 1.2GB

                       

                      I'd think this is a bug.

                      • 9. Re: Memory usage by ImporterProcessServer
                        Harm Millaard Level 7

                        Some call this logic. When using AME, it duplicates the required files temporarily to the C drive in order to perform it's encoding process in the background and to alow you to continue in the foreground.

                        • 10. Re: Memory usage by ImporterProcessServer
                          Howard Hoffman Level 1

                          I am confused.  I thought the purpose of ImporterProcessServer is to share the media used b/w different applications.

                           

                          What files are copied?

                          • 11. Re: Memory usage by ImporterProcessServer
                            Harm Millaard Level 7

                            Consider this:

                             

                            You have a PR project file with file A and B on your X drive. You send those to AME for encoding.

                             

                            AME now makes a temp copy of A and B to drive C and starts encoding. Meanwhile you can continue editing your project with A and B, using the original files on drive X.

                             

                            After encoding AME deletes the A and B copies and you are still editing the original A and B files.

                            • 12. Re: Memory usage by ImporterProcessServer
                              Howard Hoffman Level 1

                              What types of file A and B are you referring to?  Project/Sequence file?  The project/sequence file itself is small, no where near 600MB.  The media files should be shared b/w the original and the copy, right?

                              • 13. Re: Memory usage by ImporterProcessServer
                                Harm Millaard Level 7

                                Those can't be shared because what must AME do with file A for encoding, while you have changed in- and outpoints of A while AME was getting ready to encode? Encoding and editing both require exclusive rights to a file and if you can't give exclusive rights without stopping the other application, which by definition is excluded if the other has exclusive rights, you have to make a temporary copy. One of the dilemmas of multi-tasking.

                                 

                                It really is pretty simple if you think about it.

                                 

                                You have one apple (file). First you pick it up and cut off a piece (editing), then you give the apple to your wife, so she can have a piece (encoding). While she is eating her piece you grab the apple and cut off another piece and hand it back to your wife. She complains that you took a piece of her apple and it now different than when she last had it in her hands, so she is in doubt where to take the next piece from. A new apple, the left side or the right side of the existing apple....

                                 

                                Confusion as you know spells disaster in computer terms. These are really dumb animals that can not even count to two. One is the limit.

                                • 14. Re: Memory usage by ImporterProcessServer
                                  Howard Hoffman Level 1

                                  What you described is pretty clear.  Premiere and AME maintains different data in-memory.  So we should observe memory usage for each application to grow by themselve as the project gets loaded.

                                   

                                  However, I thought ImporterProcessServer was to share some data b/w different applications, such as the media/clip itself since they don't change (effects are on top of it).  If ImporterProcessServer has to maintain separate in-memory data for each application, what's the point for its existence?

                                   

                                   

                                  • 15. Re: Memory usage by ImporterProcessServer
                                    Harm Millaard Level 7

                                    Will or Dennis are much more qualified than I am to answer the intricacies of the Adobe programs, but remember that Adobe only uses references to the original files, not the original files in the non-destructive approach they have used for ages. That means that when starting the encoding with AME, it merely copies all references in your timeline to a new spot, where AME can access and work on without disrupting your workflow in PR. Now, if you delete 3/4 of your clips or timeline, that will only reflect on your memory usage after the references have been 'released' by saving and possibly restarting the application. But something that drastic is not likely to occur I think.

                                     

                                    I can understand you want to understand what happens. I'm rather curious myself, but here my approach is to let the program do it.

                                     

                                    You know there are three kinds of people: Those who make things happen, those who watch things happen and those who wonder what happened.

                                     

                                    In this case I leave the first group to Adobe, consider myself in the second group and let noobs join the third group.

                                     

                                    No offense intended.

                                    • 16. Re: Memory usage by ImporterProcessServer
                                      Howard Hoffman Level 1

                                      I guess the dicussion deviated from where I started.  All I wanted is to figure out if this is a bug.  I have seen ImporterProcessServer blows out at the 2GB memory limit on my 32-bit box twice on the same project which normally consumes only 600 MB.

                                      • 17. Re: Memory usage by ImporterProcessServer
                                        Harm Millaard Level 7

                                        Howard,

                                         

                                        Unfortunately (or should I say fortunately) I never encountered that error, but then I have 12 GB installed, not 2 GB so that may be the difference.

                                        CS3 had memory leaks, that were largely repaired in the latest updates, but still...

                                        CS4 did not have those problems AFAIK.