Skip navigation
Currently Being Moderated

Slow Render with High-End System?

Apr 24, 2013 12:34 AM

Tags: #premiere_pro #playback #performance #render #aftereffects #speed #performance_issue

Im currently working on a (in my opinion) high-end Windows system for video editing. The system is about 2 years old and has cost a fortune in that time. So Im expecting significantly better speed. So heres my problem:

Im working primarly in Premiere Pro and After Effects. All the media I work with are as a video imported Jpeg Sequences. Often I have multiple Sequences (up to 7 or 8) overlayed and tweaked with Dissolves and plugins like Twixtor. I also use the Adobe Dynamic Link from After Effects to Premiere and vice versa. All the footage is currently in 1080p but in future I will want to render 4K. Im aware that a 4K workflow is probably a pain in the *** so Im surely going to edit offline with 1080p. However  I cant get any real-time playback with all my sequences. I ALWAYS have to render a preview to watch my edits. I dont know if Im just having too high expectations for my system, but Im kinda sure there has to be an issue for this lack of performance. Maybe the Dynamic Link is slowing my system down?

 

System Specs:

Model : HP Z400 Workstation 103C_53335X

Mainboard : HP 0B4Ch

System BIOS : HP 786G3 v03.15 10/29/2010

RAM : 12GB ECC DIMM DDR3

Processor : Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz (4C 3GHz/3GHz, 2.13GHz IMC, 4x 256kB L2, 8MB L3)

Chipset:

     HP X58 I/O Hub 2x 2.4GHz (4.79GHz)

     HP Core Desktop (Bloomfield) UnCore 2x 2.4GHz (4.79GHz), 3x 4GB ECC DIMM DDR3 1GHz 192-bit

Graphic card : NVIDIA Quadro 4000 (8CU 256SP SM5.0 950MHz, 512kB L2, 2GB 2.81GHz 256-bit, PCIe 2.00 x16)

Harddisks:

      4x WDC WD2002FAEX-007BA0 (1TB, RAID10/SATA600, SCSI-4, 3.5", 7200rpm) : 932GB (C:)

     Intel Raid 1 Volume (4TB, RAID, SCSI-4) : 4TB (D:)

     HL-DT-ST BD-RE BH10LS30 (SATA150, BD-RE, DVD+-RW, CD-RW, 4MB Cache) : k.A. (E:)

 

 

Thank you very much in advance for your help and I apologize for any grammatical mistakes since english is not my main language.

 
Replies
  • Currently Being Moderated
    Apr 24, 2013 3:24 AM   in reply to wualeyyy

    There are a number of issues here, some on the hardware side and some with your source material, and finally in how these interrelate. But first request is to run the PPBM5 Benchmark to see how your system performs in comparison to others. That can give clues as to what is going on here.

     

    1. Source material is sequences of JPEG files. That is both computationally hard on the CPU / memory system, due to the heavy compressions, and on the disks, due to the number of files to be accessed.
    2. Your system is essentially similar to an i7-930 only packaged as the more expensive Xeon. It is not overclockable.
      • It has only 12 GB memory, and it uses the slower ECC memory. A distinct disadvantage. 24 GB non-ECC would help significantly.
      • The C: drive is a single disk and the D: drive is a raid1 array, thus essentially almost as fast as a single disk, so it boils down to having only two volumes that are as fast as two single conventional disks. Each around 100 - 120 MB/s transfer rate.
    3. The disk setup is minimal for regular video editing, three separate physical disks are recommended, and with the high volume of JPEG's to be accessed on a single HDD with its low access times and all the mechanical head movement that is a definite factor to slow you down.
    4. Disk fragmentation and the fill rate can be further factors that can cause serious delays.
    5. The Quadro 4000 does not really help here. Twixtor is not accelerated and the Quadro 4000 is very slow in comparison to modern cards that cost only a fraction.
    6. With the number of tracks and the I/O demands for all these files, your CPU and memory are completely bogged down. It is just too much for such a system.

     

    Just my $ 0.02

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 24, 2013 4:34 AM   in reply to wualeyyy

    Valid questions, but first we need to know from what camera the source material originated, in what format it was shot and the reasons for ingesting them as JPEG sequences. You also mention at least 2K export, but how do you plan to deliver that on what medium. In short tell us a bit more about your workflow.

     

    As to upgrading your system, my initial reaction, without knowing your workflow, would be no, not really necessary yet. Well, if you are talking about 4K, then a definite yes, but then the question of available budget arises. I have an older system, Harm's Beast, based on the i7-920, that still does a pretty decent job at rank #48 in the benchmark I linked you to before and that I asked you to do.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 24, 2013 5:49 AM   in reply to wualeyyy
    Maybe the Dynamic Link is slowing my system down?

    They definitely contribute. See this good old thread on Dynamic Link workflow. Pay attention to Todd's comments.

     

    On a side note: you don't have to replace your dynamically linked comps in PrPro timeline every time you rendered DIs. You may set them as proxies for the dynamically linked comps right inside your AE project(s).

     

    However, don't expect miracles and avoid the work flaw (no typo) like this stubborn kid established: with hundreds of dynamically linked comps it's not going to work. At least in real time. For example, a 30 min timeline built out of 935 dynamically linked comps, which are just original footages wrapped in their own compositions and hence are equivalent to DIs set as proxies inside AE project(s), it becomes almost unresponsive and with my spec takes around 27 hours to render...

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 24, 2013 6:32 AM   in reply to wualeyyy

    Harm is way more qualified to work with you, but I do have one question that I have not yet seen addressed.

     

    I suggest that using JPEG sequences in Premiere Pro is not that tough on a PC, or at least, mine doesn't seem to mind them at all.

     

    However, what is the frame size of the images? Are you panning and scanning around the large frame inside Premiere Pro, or could you (or do you) resize them to fit into the 1080p sequence ahead of time - before you import them?

     

    If you are panning around, that's fine. But if you are not, then cutting the images down to size before you import them could save a lot of bandwidth issues.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 24, 2013 6:38 AM   in reply to wualeyyy

    OK. But now what I asked exactly. What is the frame size? 1920X1080 or something much larger. I don't see where you mentioned the camera, so I can't guess at a frame size. My images are 4608X3456 the way I normally shoot. So cutting then down to fit a 1080p timeline reduces them significantly.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 24, 2013 6:41 AM   in reply to wualeyyy

    Twixtors process of interpolating motion between frames is really slow on every system.  There maybe be other minor contributing factors, but every high end machine I have used it one slows done to a snails pace. Its the nature of the beast.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 24, 2013 6:46 AM   in reply to wualeyyy

    I would contact the software develope for their advice. They know the product better then most.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 24, 2013 9:34 AM   in reply to wualeyyy

    Thanks for your workflow explanation. It clears up the questions I had, but at the same time does not give a solution to your problem. I admit I have not tried it, so I'm not sure it will work or is any faster, but try it out with some stills to see it it helps:

     

    • Use .PNG instead of .JPG. I have found using PNG instead of JPG works much better on websites, maybe the same applies to PR.
    • Numbered stills are often used with the .TIFF format. Maybe that is an alternative to JPG or PNG.

     

    Getting a third disk will certainly help, if you use one disk for your stills and one for your AE compositions, assuming a typical timeline will alter stills and AE compositions with a high regularity. That way the access times will be distributed among the two data disks, improving performance. Definitely make it a habit to defrag your disks on a daily basis with so many smaller files.

     

    You currently have one memory bank in use for your 3 x 4GB ECC memory. Adding three more sticks  to up memory to 24 GB will also give a nice performance boost, however I admit that I don't know if HP has attractive pricing on them. An extra disk can always be ported to a new system in the future, but that does not apply to memory, so you may have a high depreciation cost over the remaining life of your system.

     

    Eventually, you may have to go for a dedicated raid controller and a number of SSD's in an array to significantly lower access times and improve transfer rates with these huge amounts of files. If you were to create a 5 disk SSD raid3 for instance, giving you 4 times the capacity of a single SSD, giving you average access times below 1 ms, versus the 14 ms you now have, and improving your average transfer rates from 120 MB/s to around 1500 MB/s.

     

    For a new system I would definitely look at one that is overclockable, since Twixtor can benefit greatly from the higher clock speeds of OC'ed systems.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 24, 2013 9:59 AM   in reply to Harm Millaard
    • Use .PNG instead of .JPG. I have found using PNG instead of JPG works much better on websites, maybe the same applies to PR.
    • Numbered stills are often used with the .TIFF format. Maybe that is an alternative to JPG or PNG.

    It's always about finding optimal 'quality/file size/render time' ratio.

    In my testings JPEG 2000, PNG, TGA, TIFF, DPX provide similar quality.

    TGA, TIFF, DPX have about identical render time. For JPEG 2000 and PNG it's up to 1.3 and accordingly up to 2.25 times higher.

    JPEG 2000 provides smallest file size, whereas for PNG it's up to 1.3 times larger, and for TGA, TIFF and DPX - up to 3.7 times.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 25, 2013 1:10 AM   in reply to wualeyyy

    For some background on raids, see To Raid or not to Raid, that is the question...

     

    Your C: volume for OS & programs is a standard single disk, your D: volume is a 4 drive mirrored raid1, that gives you twice the capacity and twice the speed of a single drive.

     

    That means that all your source files, your intermediate AE compositions, your media cache, your preview files and your exports need to be read from and written to this single volume. Take into consideration that this volume is only half duplex, meaning that it can only read at one time and that action must be completed before writing can start. It is like a single lane road with traffic lights on both sides. Traffic can only flow in one direction at a time and traffic from the other side has to wait.

     

    If I remember correctly the road up to Isenfluh and Sulwald is such an example. Never a problem when traffic is light, but with lots of traffic, delays become inevitable and that is what happens with all your files, many tracks, many compositions and effcts. You encounter traffic jams. The solution is to increase the number of lanes, like was done in Bern at the Ost Ring. It took quite some time though and there were still delays till the roadworks were finished.

     

    In your situation spreading your traffic (file accesses) across multiple lanes (raid volumes)  can reduce traffic jams.

     

    ECC memory is always slower than non-ECC memory, because of the error correction bits that need to be added and checked. In addition ECC memory can often have slower speed specs than non-ECC memory.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 25, 2013 1:19 AM   in reply to wualeyyy
    So I guess Jpeg 2000 would be the way to go then?

    That's not so straightforward and highly depends on both your rig and your project.

    If saving on a disk space overweighs performance factors, then you may go with JPEG 2000.

    With complex compositing, where multiple layers with time consuming effects are involved, your decision may be driven by savings on render time, which in turn depend on how noticeable they are on your machine.

     

    Since you're testing everything instead of just jumping to a conclusion, you are on the right track.

     

    Keep in mind, that optimising your workflow time-wise may involve pre-rendering your compositions in After Effects and rendering on the background via either Adobe Media Encoder or BG Renderer script.

    which preset you choose when starting a new project in Premiere Pro?

    General answer is here.

     

    Since you are going to work on image sequences entirely, while delivering for a cinema, pick one from 2K or 4K 24p preset or build a custom sequence with this settings. Do not use DSLR 1080 25p - it doesn't serve your needs from any angle you described so far.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 25, 2013 1:50 AM   in reply to wualeyyy

    It is all explained in the raid article I linked to.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 25, 2013 1:54 AM   in reply to wualeyyy
    in terms of preset I am looking for a preset to work with offline footage.

    You normally build your proxies on your project settings, not vise versa...

    The final online render will of course be in 2K 25p or 4K 25p.

    I'm confused. 25 fps is for PAL delivery, not for a cinema...

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 26, 2013 8:42 AM   in reply to wualeyyy

    As far as I understand, you don't shoot video, you take photos (probably timelapses). Hence, all those Canon or DSLR presets don't serve you - they are for video files, which come from a camera.

     

    Define your delivery intentions: whether you're going to deliver just for a cinema or e.g. for a Blu-ray as well.

     

    Ask theaters for the spec: with which frame rate you must deliver (with high probability in just 24p).

    Build your project around these requirements.

     

    If you need to deliver with different frame rates, test if built-in tools please you and preserve 'film look'. Check out this thread in After Effects Forum.

     

    DO NOT start from 1080p so as to upscale to 4k on export. Rather edit in 4k and downscale to 1080p on export if need be.

     

    If you have issues with realtime playback while working on 4k image sequences, make them offline (what you're currently doing), link to lower resolution proxies, select your clips in the timeline, right-click and choose 'Scale to Frame Size'. Reverse back when done.

     

    Keep in mind that when you replace a clip or a group of clips in PrPro timeline with AE composition, 'Scale to Frame Size' flag results in creating nested composition in After Effects and Collapse Transformation switch turned on for this layer within the master comp. See e.g. this thread.

     

    Hope, this clarifies things a bit.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 26, 2013 9:35 AM   in reply to wualeyyy

    Valentin,

     

     

    I have always called a raid10 a solution for the paranoid in a hurry. It takes 4 drives to give you the capacity and performance of two disks, but gives you security by the mirroring.

     

    Before going into your specific situation, allow me to tell something about volumes and drives, because they can be confusing and at the same time they are very important for optimal performance of a system.

     

    • Single disk, not partitioned is 1 disk = 1 volume to the OS
    • Single disk, partitioned is 1 disk = multiple volumes (not a good idea BTW)
    • Multiple disks in one raid array is Many disks = 1 volume
    • Multiple disks in one raid array with partitions is Many disks = multiple volumes (not a good idea either)
    • Each volume has a distinct drive letter for access.

     

    Partitioning is a thing of the past and should not be used at all on modern systems.

     

    You have to think about volumes more than about number of disks. In my current system I have 27 different physical disks but only 4 volumes. In the old one I have 17 disks and 5 volumes.

     

    Now that we are clear what we are talking about, volumes with distinct drive letters, we can address your situation.

     

    You have TWO volumes, C: (single disk) and D: (4 disks in raid10). Spreading the load across two volumes is more demanding and gives slower performance than using more volume, unless one or more volumes are very fast, as I tried to explain in a previous reply (remember Isenfluh/Sulwald?). If you add a SSD as you intend, you have increased the number of volumes to 3, which will definitely help performance, because SSD's are faster than conventional disks, the pagefile can be stored on the SSD, so all your performance will go up.

     

    Compare your setup with mine with rough estimated figures:

     

    VolumeValentinHarmTransfer rate ValentinTranfer rate Harm
    C:1 HDD1 SSD125 MB/s450 MB/s
    D:4x Raid101 SSD250 MB/s450 MB/s
    E:NA21x Raid30NA2,700 MB/s
    F:NA1 HDDNA150 MB/s

     

    These figures are indicative, but do show where the major differences are. In my experience disk setup is overlooked quite often, but has a huge impact on a system's responsiveness. It is the weakest (slowest) link in the chain, afterall and with your workflow, doubly so.

     

    But in your specific case there is something else, and that is your disappointing hardware MPE score. 100 seconds is extremely slow, even for a Quadro 4000. I would be quite normal to see a score around 8 - 9 seconds on such a system, well maybe around 12 - 13 seconds with your ECC memory, but 100 is way too slow. Some background services or processes are interfering with the hand-over from memory-GPU-VRAM-GPU-memory. This can be caused by a myriad of things, but a good starting point would be the BlackViper list of services to set to manual or disabled and taking a closer look at the processes running with Process Explorer. There should normally be less than 50 processes running.

     

    Hope this helps.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 26, 2013 7:56 PM   in reply to Harm Millaard

    I have always called a raid10 a solution for the paranoid in a hurry.

     

    I think I qualify. I have started worrying about my two RAID0 setups lately and it is probably time to get one of those automatic web backup subscriptions. The problem is that I would need a RAID controller to put in even one more disk so my seven disks (5 volumes) will have to do for now.

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points