23 Replies Latest reply on Oct 6, 2010 8:03 AM by Todd_Kopriva

    Multiprocessor settings and hard drive set up?

    Semlom

      Hello everyone,

       

      I'm new to AE so I'd appreciate a little help with a few settings, the preferences are a little more complex than the familiar Photoshop and Illustrator ones.

       

      I'll list my PC specs and hopefully it'll make it easier to answer.

       

      Intel i7 975 Extreme (4 cores)

      6GB DDR3 Triple Channel Ram

      64 bit OS (Vista)

      |

       

      Hard Drive Set-Up

      X2 150GB WD Velociraptor (set RAID 0) This is the main drive that the operating system and all the programs are installed.

      X1 150GB WD Velociraptor (This drive is blank and is used solely has a scratch disk in Photoshop and Illustrator)

      X1 1TB WD Caviar Black (used as storage)

      X2 750GB External Seagate Drives. (Storage)

       

      The graphics card is an Asus GTX 295 (Very powerful Multi-GPU. 1792 MB)

      |

       

      Basically I'd like an expert opinion on what the best settings would be for my system, mainly the Memory & Multiprocessing settings. Should I use all the cores or should I leave some for other programs. I know this question is dependent on how I would be using AE but I don't know the answer to that because I've only just started using it, therefore I don't know if I'll be bouncing from Illustrator and Photoshop... I think I'll be doing a mix of everything but mainly motion graphics - is it common practice to jump from Photoshop to After Effects? etc.

      |

       

      I'd also like to know what would be the best way of working when it comes to hard drives. How should I use my hard drive set up to get the most out of AE?

       

      Hope this makes sense!?

       

      Cheers

       

      Ps. I'm using After Effects CS4. I'm also upgrading the RAM to 12GB... (I know 12GB is an overkill but I want to future proof myself.)

       

        • 1. Re: Multiprocessor settings and hard drive set up?
          Mylenium Most Valuable Participant
          I know 12GB is an overkill but I want to future proof myself.

           

           

          No, it isn't. It is actually the only sensible way to use multiprocessing. AE does barely use scratch disks and really is dependent on physical RAM. with only 6GB therefore you could at best hope to use 2 additional background isntances, but using just 1 is closer to the truth. Each instance should ideally at least have 2GB of RAM, so if you do the math, you see the problem. Regarding the disks, it depends on what you do. a few rules:

           

          - Footage input and footage output for rendering should not be on teh same drive to prevent bottleneck situations. With MP enabled, even a RAID will suffocate eventually, depending on the amount of clips needing to be loaded by each instance as well as the frames being written.

           

          - The conformed media cache defaults to your user home directory, which already is your busiest directory on your entire system (apart from the Win dir). Move it too a different location. Watch it closely, as it can grow indefinitely large when working with compressed sources all the time. Purge it regularly from the prefs. The location should not matter that much, but the previous point also applies.

           

          - The temporary disk cache used for accelerating RAM previews and realtime playback should be on your fastest disk, but also in a dedicated directory. The files will be deleted when you quit AE, so no worries about keeping care of them. The maximum size can be set in the prefs.

           

          One comment regarding your GTX 295: It may be necessary to turn it into single GPU mode or else OpenGL features in AE may not be available. the latest NVidia drivers allegedly avoid that, but one never knows...

           

          Mylenium

          1 person found this helpful
          • 2. Re: Multiprocessor settings and hard drive set up?
            Semlom Level 1

            Thanks for taking the time to answer my question Mylenium. It's got to be a habit writing the "I know it's an overkill" line, I'm mainly Photoshop and Illustrator which I'm sure you know runs easily on 6GB. I've just this minute purchased another 6GB so hopefully this will suffice. Would you say 24GB is an overkill? I only ask because my motherboard can accommodate this amount of RAM. However, I'd have to replace all my 2GB strips with 4GB strips.

            Regarding footage, should I run the input footage from the RAID and then output to my 1TB drive or vice versa? Which would benefit more from the RAID, input or output? The RAID drives run at 10,000 rpm (each) just to let you know.

             

            Do I have to use the same drive with the Conformed Media Cache or would you recommend I use one of my other internal drives? Regarding these settings, should I change the "Database" folder or just the "Cache" folder? What does the "Enable Disk Cache" check box do? Is that the temporary disk cache you mentioned?

             

            I'm sorry for all these questions but it's difficult to find any information that's absolute, I'd rather get the opinion of an expert rather than trying to sift through heaps of information myself.

             

            Many thanks

             

            Luke

            • 3. Re: Multiprocessor settings and hard drive set up?
              Adolfo Rozenfeld Adobe Employee

              Thanks for taking the time to answer my question Mylenium. It's got to be a habit writing the "I know it's an overkill" line, I'm mainly Photoshop and Illustrator which I'm sure you know runs easily on 6GB. I've just this minute purchased another 6GB so hopefully this will suffice. Would you say 24GB is an overkill? I only ask because my motherboard can accommodate this amount of RAM. However, I'd have to replace all my 2GB strips with 4GB strips.

               

              On a quad core machine, AE can use up to 16 GB of RAM (4 GB per core) when using the "Render Multiple Frames Simulataneously" feautre. Having more than that could help you run other applications with less disk swapping, for example.

               

               

              Regarding footage, should I run the input footage from the RAID and then output to my 1TB drive or vice versa? Which would benefit more from the RAID, input or output? The RAID drives run at 10,000 rpm (each) just to let you know.

              Definitely, use the fastest drive for input. Output is unlikely to happen at a speed that pushes the hard drive, unless you do practically nothing in your Comp

               

              Do I have to use the same drive with the Conformed Media Cache or would you recommend I use one of my other internal drives? Regarding these settings, should I change the "Database" folder or just the "Cache" folder? What does the "Enable Disk Cache" check box do? Is that the temporary disk cache you mentioned?

              The Conformed Media Cache can be on an internal drive. It's mostly used for converting compressed audio formats (like MP3) to PCM/uncompressed audio.

              The Disk Cache is only used for spacebar play. When AE uses the disk cache, that part of the timeline shows a blue bar. AE avoids using the disk cache when it thinks it won't speed up things. It's better to use for disk cache a fast drive that's differrent from the one you use to store source files.

              More info on RAM and disk caches in AE Help.

              • 4. Re: Multiprocessor settings and hard drive set up?
                Mylenium Most Valuable Participant
                Would you say 24GB is an overkill? I only ask because my motherboard can accommodate this amount of RAM. However, I'd have to replace all my 2GB strips with 4GB strips.

                 

                You can never have enough! If your mobo supports it, then that's a good thing, but naturally you will have to weigh the extra power consumption and heat production as well as how often you will actually throw hevavy projects at your machine. Of course it depends a lot on your work and based on the info you provided, indeed it does not seem necessary. I'd definitely upgrade to 12GB, though. That seems a reasonable amount and from personal experience, when doodling at home, both in AE and my 3D programs, it's just nice being able to keep multiple programs open, if nothing else. In addition to the better stability, that more than anything else is the biggest benefit of 64bit and lots of RAM to me. You may also find yourself getting used to workflows where you copy&paste across programs (paths as masks for instance) and then that point also becomes improtant.

                 

                Regarding footage, should I run the input footage from the RAID and then output to my 1TB drive or vice versa? Which would benefit more from the RAID, input or output? The RAID drives run at 10,000 rpm (each) just to let you know.

                 

                I have to slightly contradict Mr. Rozenfeld on that one. While generally input is more critical, on the output end it's quite possible to just as well choke up things. The point here realyl is, which output types and file formats you use in combination with what effects and features. On a Core7i with decently optimized plug-ins like e.g. Particular 2 it's quite well possible to quickly render even complex projects and at that point, writing multiple images simultanously may put some stress on your disks, especially in 32bpc mode with large HD files. You will not notice any such thing with most movie formats, though - because the frames need to be appended sequentially, most of them cause the MP render to throttle down, i.e. wait for the BG instances to finish in the correct order.

                 

                What does the "Enable Disk Cache" check box do? Is that the temporary disk cache you mentioned?

                 

                Yupp, it is. See the explantion in the help as pointed out by the previous poster.

                 

                On a general note you should be aware of one thing: while we can provide some rough guidance, some of these settings may need to be tweaked and changed on a per-project basis. If. e.g. you render with lots of input footage, the cache can considerably speed up things, but the number of background processes should be reduced to reduce I/O stress. On the otehr hand, simple titles and slates with just text will not benefit from cahing at all and allow you to use all cores for BG rendering. It's realyl a bit fiddly at times, but with experinece, you will find good combinations for your work, eventually.

                 

                Mylenium

                • 5. Re: Multiprocessor settings and hard drive set up?
                  ShawnMichael

                  OK. I'm confused.  You said "On a quad core machine, AE can use up to 16 GB of RAM (4 GB per core) when using the "Render Multiple Frames Simulataneously" feautre. Having more than that could help you run other applications with less disk swapping, for example"

                   

                  But you also said: http://forums.creativecow.net/thread/2/957712#957714

                   

                  This is an i7 machine we are talking about here.  So do we turn MP on or off?

                  • 6. Re: Multiprocessor settings and hard drive set up?
                    Adolfo Rozenfeld Adobe Employee
                    I have to slightly contradict Mr. Rozenfeld on that one. While generally input is more critical, on the output end it's quite possible to just as well choke up things. The point here realyl is, which output types and file formats you use in combination with what effects and features. On a Core7i with decently optimized plug-ins like e.g. Particular 2 it's quite well possible to quickly render even complex projects and at that point, writing multiple images simultanously may put some stress on your disks, especially in 32bpc mode with large HD files. You will not notice any such thing with most movie formats, though - because the frames need to be appended sequentially, most of them cause the MP render to throttle down, i.e. wait for the BG instances to finish in the correct order.

                     

                    No problem with contradicting, but for example 32 bpc would only be an issue if you also output to 32 bpc files without any form of data squeezing, which is not the most common case, is it?

                    The RQ spitting frames at more than 70 Megabytes per second (typical sustained speed of an average hard drive nowadays) would be quite rare Not impossible, but unlikely unless you are targetting very high-end formats. For a 1080 8 bpc file, that would be about 8-9 frames per second!

                    • 7. Re: Multiprocessor settings and hard drive set up?
                      Adolfo Rozenfeld Adobe Employee

                      OK. I'm confused.  You said "On a quad core machine, AE can use up to 16 GB of RAM (4 GB per core) when using the "Render Multiple Frames Simulataneously" feautre. Having more than that could help you run other applications with less disk swapping, for example"

                       

                      But you also said: http://forums.creativecow.net/thread/2/957712#957714

                       

                      This is an i7 machine we are talking about here.  So do we turn MP on or off?

                      Shawn: That thread in Creative Cow was before 9.02 was released. 9.02 addressed a couple of issues with the i7 range of processors. So, make sure you update to 9.02 and then enable MP if you want.

                      • 8. Re: Multiprocessor settings and hard drive set up?
                        ShawnMichael Level 1

                        That's awesome!!!! Thanks!

                         

                        BTW: does the update also fix the "Locking Existing Frames" situation? (Sorry, I'm not trying to hijack this thread. I Promise)

                        • 9. Re: Multiprocessor settings and hard drive set up?
                          Semlom Level 1

                          The 9.02 update doesn't fix the issue with how AE sees the cores on the i7, as far as I know the update fixes a few bugs with some of the CC effects. Technically the i7 has 4 cores with 8 threads so I'm not sure how AE handles the hyper threading under 64bit Vista. According to the Adobe website AE counts the virtual CPU's so it should recognise hyper-threading and display the i7 has having 8 cores.

                           

                          Quote

                          " After Effects can also use multiple processes to accelerate rendering of a single frame. This form of multiprocessing can take advantage of hyper-threading. The number of CPUs that After Effects reports in the Multiprocessing Preferences dialog box for accelerating rendering within a single frame counts the virtual (logical) CPUs available due to hyper-threading."

                           

                           

                          I'm seeing only 4 cores in AE with my i7 975, possible 2.8GB RAM per core. Hyper-threading is enabled in the BIOS and the operating system, Vista 64bit, sees 8 cores (in Task Manager and Device Manager). I read elsewhere on the internet that i7 on Macs have no problem with this and will display 8 cores within the preferences in AE.

                           

                          Can anyone actually confirm running a i7 under 64Bit Vista and seeing 8 cores in the preferences?

                           

                           

                           

                           

                          • 10. Re: Multiprocessor settings and hard drive set up?
                            ShawnMichael Level 1

                            I'm seeing the same thing.  Vista 64bit sees 8 cores.  AE only sees the 4.  That's with the update.

                            • 11. Re: Multiprocessor settings and hard drive set up?
                              Adolfo Rozenfeld Adobe Employee

                              Actually, that's the right thing for AE to do, Shawn.

                              Your i7 processor has four physical cores and it uses a technology called hyperthreading, which makes the system see another four logical or virtual cores. In other words, they are not real, physical cores.

                              The kind of render-farm-in-a-box model AE uses needs real, physical cores. If it saw the logical/virtual cores as real ones, AE would launch a rendering instance for each of them, they would take a lot of RAM and finally be counterproductive for rendering speeds. In other words, you don't want AE to treat virtual cores as if they were physical cores.

                              Note that AE can benefit from hyperthreading by using other multiprocessing optimizations it has. It's just not a great idea to launch separate rendering instances for virtual cores with the render multiple frames feature.

                              In fact, for some Xeon-based machines (recent Mac Pros, for example), AE could report the virtual cores as physical cores, and under those circumstances telling AE not to use those cores could be the best idea.

                               

                               

                              The issues 9.02 addressed with i7 processors were about reliability/stability, not about treating virtual cores as physical ones. Because they are not that.

                              • 12. Re: Multiprocessor settings and hard drive set up?
                                Adolfo Rozenfeld Adobe Employee

                                Semlon: The paragraph you quoted is contrary to what I heard so far.

                                I´ll try to get a statement about this. Because the important thing is not if AE reports or not the virtual cores, but if launching separate rendering instances for virtual cores would provide any benefit or be directly counterproductive.

                                Note that if AE really treated the four virtual cores as actual cores (ie, 8 cores in total), you'd need at least 16 GB of RAM (as always, 2 GB per rendering instance) for the backround instances to work correctly.

                                • 13. Re: Multiprocessor settings and hard drive set up?
                                  Adolfo Rozenfeld Adobe Employee

                                  OK, Semlon.

                                  No need for a statement.

                                  I re-read the quote you posted and it says something very similar to what I just posted in a previous message in this thread,

                                  AE can and does take advantage of hypethreading by using all physical and virtual cores when using multiprocessing optimizations to render a single frame. This happens independently from the "Render Multiple Frames Simultaneously" settings. For the latter feature, you wouldn't get much or any advantage from launching rendering instances for the virtual cores. And RAM requirements would be much, much higher.

                                  The last bit of the quote is not "selling" the fact that the virtual cores appear in some cases in the multiprocessing preferences, but explains why that could happen.

                                  I've only seen the virtual cores appear in MP preferences in Xeon-based systems, but that's not a good thing per se. You would probably want to turn off those virtual CPUs if they appeared there.

                                  As I said, AE will use them but not in the context of this specific feature.

                                  • 14. Re: Multiprocessor settings and hard drive set up?
                                    Steve Whatley

                                    Lots of great answers here, but after months of trying to figure these questions out for Production Houses in NYC, I got a excellent response from the AE engineers.

                                     

                                    here it is: "leave the Multiprocessor option OFF - Leave the Memory and Multiprocessing option to the default settings"

                                     

                                    It bears repeating that in most all cases, the default setting works great. The trade-off for turning it on (mentioned before by others) is pushing and pulling frames. This is the same reason a lot of people never used Watch Folders and Network Rendering....it was and is a great solution, but only when the number of layers and complexity of the project make exchanging files worthwhile.

                                     

                                    Again, according to the After Effects engineers, 9 out of 10 times it should be left OFF.

                                     

                                    Here is my "personal" rule of thumb....if you are waiting for screen redraw of the comp window that takes more than 6 seconds after you move the Timeline Indicator....trying Render Multiple Frames Simultaneously is worth the effort. Saying that, you need to test this by mapping out your similar projects and seeing when and if it helps, this will give you a basis on whether to use it.

                                     

                                    my 2 cents

                                     

                                    Steve

                                    • 15. Re: Multiprocessor settings and hard drive set up?
                                      Adolfo Rozenfeld Adobe Employee

                                      Steve: What AE engineer told you that the best thing is to leave multiprocessing off? Where did you read that AE engineers say that 9 out of 10 times it should be left off? I think the most likely thing they told you is, when you enable multiprocessing, leave everything else in the Memory and Multiprocessing preferences at default values.

                                      • 16. Re: Multiprocessor settings and hard drive set up?
                                        Semlom Level 1

                                        Your i7 processor has four physical cores and it uses a technology called hyperthreading, which makes the system see another four logical or virtual cores. In other words, they are not real, physical cores.

                                         

                                        That's exactly what I said in a previous post!? We all know the i7 has 4 cores and 8 threads.

                                         

                                        The kind of render-farm-in-a-box model AE uses needs real, physical cores. If it saw the logical/virtual cores as real ones, AE would launch a rendering instance for each of them,

                                         

                                        I'm sorry but to me the quote, which is from the Adobe website, seems to elucidate that fact that AE will see the 8 threads on the i7 and display them as 8 cores in the Multiprocessing dialogue box. It clearly states that AE "counts the virtual (logical) CPUs available due to hyper-threading" I'd also like to point out the thread on CreativeCow.net - you acknowledge this very problem. And I quote " With 12 GB of RAM (once the update ships), you may want to use 6 of your 8 cores (you can set the multiprocessing preferences to leave 2 CPUs free)... This should still give you screaming fast performance." That quote is referring to the i7.

                                         

                                        I've only seen the virtual cores appear in MP preferences in Xeon-based systems, but that's not a good thing per se. You would probably want to turn off those virtual CPUs if they appeared there.

                                         

                                        I bet that's under OSX too. When I said, and I'm paraphrasing here, that on a Mac, 8 cores are visible in AE, that's on a normal i7 system running on OSX, not a Xeon based system. The new Xeon processor uses the Nehalem architecture anyway. You say above that "it's not a good thing" and "we should turn off the virtual CPU's" but you're also saying that the updated version of AE will give "screaming fast performance" and we should use 6 of our 8 cores, providing we have 12GB of ram, and leave 2 cores free!? I'm confused?!

                                         

                                        I'm not on a witch hunt here, I'm only trying to understand how AE works on a i7 platform. It seems to me that the problem could be Vista related, more so 64 Bit Vista. I'll be interested to see how AE runs under Windows 7 when it is officially released.

                                         

                                        The current 8-core system on the Mac Pro has 2 quad cores so that's 16 threads/virtual cores. I know people have had trouble running this system with programs like Pro Tools, just wondering how AE would handle this pseudo 16 core.

                                        • 17. Re: Multiprocessor settings and hard drive set up?
                                          Steve Whatley Level 1

                                          I hope this does not offend, but I don't know who you are.

                                           

                                          Would you mind updating me?

                                           

                                          Steve

                                          • 18. Re: Multiprocessor settings and hard drive set up?
                                            ShawnMichael Level 1
                                            Note that AE can benefit from hyperthreading by using other multiprocessing optimizations it has. It's just not a great idea to launch separate rendering instances for virtual cores with the render multiple frames feature.

                                            I did some of my own tests on projects and found this to be true, in my case.  With 4 cores turned on in MP, I watched the performance monitor in Windows task manager.  The other 4 Virtual cores were working hard as well.  Sort of sharing the load as if I had MP turned off.  The total usage would peak at around 50%.  Not the same as the good ol' days of Nucleo Pro with everything running 100%.  But, it was something.

                                             

                                            However, I also found that rendering time on simple projects, such as a video layer with a basic color effect on it, was actually MUCH slower with MP on.  By turning it off and letting the cores all share the rendering on one frame at a time, render time was cut nearly in half.

                                             

                                            24sec video...

                                            4 cores in MP: 1min35sec render

                                            MP off: 52sec

                                             

                                            In much more complex projects of 100+ layers, MP would usually turn itself off, citing an incompatible effect or something. I never figured that one out.

                                             

                                            The issues 9.02 addressed with i7 processors were about reliability/stability, not about treating virtual cores as physical ones. Because they are not that.

                                            That was definitely the problem I was having before the update.  It would start to render and then freeze after about 7 frames. The update fixed that.

                                             

                                            In conclusion, what I was hoping for was a replacement for Nucleo Pro.  In my old CS3 system, running with dual quad Opterons and 32bit XP Pro, Nucleo was a dream. It was instant start-up, fast preview, fast render.  Sadly, Nucleo just won't run well on my new system (i7, 64bit Vista, CS4).  And it appears that the MP function in AE is not any better.

                                            • 19. Re: Multiprocessor settings and hard drive set up?
                                              Adolfo Rozenfeld Adobe Employee

                                              I hope this does not offend, but I don't know who you are.

                                               

                                              Would you mind updating me?

                                              Hello, Steve.

                                              Why would that offend me?

                                              I am working for the After Effects team, acting as liaison between the product team and users out there in a number of ways.

                                              I am a essentially a passionate AE user who has been hired to do this after collaborating with the AE team for several years.

                                              • 20. Re: Multiprocessor settings and hard drive set up?
                                                Adolfo Rozenfeld Adobe Employee

                                                I'm sorry but to me the quote, which is from the Adobe website, seems to elucidate that fact that AE will see the 8 threads on the i7 and display them as 8 cores in the Multiprocessing dialogue box. It clearly states that AE "counts the virtual (logical) CPUs available due to hyper-threading"

                                                Where does the quote mention i7, specifically?

                                                Also, again, this last bit explains that logical cores appear due to to hyper-threading, ie because of hyperthreading. It's not selling this as a performance point, and that's why it probably doesn´t rush to say "may appear". If they don't appear, no problem, because this specific feature was not designed to spawn rendering instances for virtual cores.

                                                The missing part of the quote already explained the advantages hyper-threading can provide for other optimizations in the application.

                                                 

                                                 

                                                And I quote " With 12 GB of RAM (once the update ships), you may want to use 6 of your 8 cores (you can set the multiprocessing preferences to leave 2 CPUs free)... This should still give you screaming fast performance." That quote is referring to the i7.

                                                Point me to that thread. I am pretty sure I am talking about a Nehalem-based Mac Pro there.

                                                 

                                                When I said, and I'm paraphrasing here, that on a Mac, 8 cores are visible in AE, that's on a normal i7 system running on OSX, not a Xeon based system.

                                                There are no i7 Mac Pros. They are all Xeon, ie Nehalem for the last revision.

                                                • 21. Re: Multiprocessor settings and hard drive set up?
                                                  Steve Whatley Level 1

                                                  Adolfo,

                                                   

                                                  Does this reply go only to you?

                                                   

                                                  Steve

                                                  • 22. Re: Multiprocessor settings and hard drive set up?
                                                    Adolfo Rozenfeld Adobe Employee

                                                    No, Steve.

                                                    This is posted to the forum.

                                                    If you need to ask or say something privately, you can use private messaging in the forum site.

                                                    • 23. Re: Multiprocessor settings and hard drive set up?
                                                      Todd_Kopriva Level 8

                                                      The After Effects CS4 (9.0.3) update fixes the "locking existing frames" problem.