Im currently working on a (in my opinion) high-end Windows system for video editing. The system is about 2 years old and has cost a fortune in that time. So Im expecting significantly better speed. So heres my problem:
Im working primarly in Premiere Pro and After Effects. All the media I work with are as a video imported Jpeg Sequences. Often I have multiple Sequences (up to 7 or 8) overlayed and tweaked with Dissolves and plugins like Twixtor. I also use the Adobe Dynamic Link from After Effects to Premiere and vice versa. All the footage is currently in 1080p but in future I will want to render 4K. Im aware that a 4K workflow is probably a pain in the *** so Im surely going to edit offline with 1080p. However I cant get any real-time playback with all my sequences. I ALWAYS have to render a preview to watch my edits. I dont know if Im just having too high expectations for my system, but Im kinda sure there has to be an issue for this lack of performance. Maybe the Dynamic Link is slowing my system down?
Model : HP Z400 Workstation 103C_53335X
Mainboard : HP 0B4Ch
System BIOS : HP 786G3 v03.15 10/29/2010
RAM : 12GB ECC DIMM DDR3
Processor : Intel(R) Xeon(R) CPU W3530 @ 2.80GHz (4C 3GHz/3GHz, 2.13GHz IMC, 4x 256kB L2, 8MB L3)
HP X58 I/O Hub 2x 2.4GHz (4.79GHz)
HP Core Desktop (Bloomfield) UnCore 2x 2.4GHz (4.79GHz), 3x 4GB ECC DIMM DDR3 1GHz 192-bit
Graphic card : NVIDIA Quadro 4000 (8CU 256SP SM5.0 950MHz, 512kB L2, 2GB 2.81GHz 256-bit, PCIe 2.00 x16)
4x WDC WD2002FAEX-007BA0 (1TB, RAID10/SATA600, SCSI-4, 3.5", 7200rpm) : 932GB (C:)
Intel Raid 1 Volume (4TB, RAID, SCSI-4) : 4TB (D:)
HL-DT-ST BD-RE BH10LS30 (SATA150, BD-RE, DVD+-RW, CD-RW, 4MB Cache) : k.A. (E:)
Thank you very much in advance for your help and I apologize for any grammatical mistakes since english is not my main language.
There are a number of issues here, some on the hardware side and some with your source material, and finally in how these interrelate. But first request is to run the PPBM5 Benchmark to see how your system performs in comparison to others. That can give clues as to what is going on here.
Just my $ 0.02
Thank you very much fo your precise answer. If possible I would like to ask some follow-up questions:
1. What other Image Format than Jpeg format do you recommend to work with? Or should I first render all the sequences seperately and then reimport them? Also, would it help working offline with only 720p or even SD Footage?
2. Would you recommend upgrading my pc? If so, where would I invest the most to get the maximum performance. If building a new PC, what could you recommend? What would be essentially for a new pc to do these tasks?
3. What disk setup could improve my current system and what disk setup would you recommend for a new pc?
Also I maybe have to say that the final export will be 2K at minimum. So it is important to have a really good and stable workflow without losing time prerendering files.
Valid questions, but first we need to know from what camera the source material originated, in what format it was shot and the reasons for ingesting them as JPEG sequences. You also mention at least 2K export, but how do you plan to deliver that on what medium. In short tell us a bit more about your workflow.
As to upgrading your system, my initial reaction, without knowing your workflow, would be no, not really necessary yet. Well, if you are talking about 4K, then a definite yes, but then the question of available budget arises. I have an older system, Harm's Beast, based on the i7-920, that still does a pretty decent job at rank #48 in the benchmark I linked you to before and that I asked you to do.
All the source material are still images from a 5D Mark II in RAW format. (CR2)
About my workflow:
We have over 600'000 RAW files stored on a NAS which I imported into Lightroom. In Lightroom Im gonna organize those files for short sequences. I then export maybe 1000 photos from Lightroom and convert them to Jpeg or whatever file format you suggest is better. Then I import them to Premiere as image sequences. But I also work with stills in Premiere. From Premiere, some sequences get tweaked by Twixtor and many other effects. Especially different dissolves are essentially. Also some of the sequences get imported to after effects via dynamic link. Finally theyre all gonna be exported as a feature length movie. In the end there will be a project with hundreds of image sequences and stills, with multiple layers, all tweaked with effects from either Premiere Pro, After Effects (and maybe DaVinci Resolve).
Currently I am testing things out and thought Id start working with compressed 1080p Jpegs to make a short Trailer/Teaser for the project. The goal is to show the final film in a cinema. So the final export should be cinema ready and at least 2K. But Im not yet familiar with what file format has to be used for cinema distribution, since we are currently in the test phase.
It is important for this project that my workflow is stable and fast enough for realtime-playback, because we are also limited in our time. In terms of budget I cant really tell a price range right now. We can for now just leave that aside and evaluate what would be the best solution.
4K is of course an option which would be awesome, but not initally necessary. The final 4K render could also be done externally with another company.
Right now I dont have the time to run the Benchmark you suggested because Im currently rendering out my Jpeg sequences. But I will do that as soon as possible and get back to you with my results.
Maybe the Dynamic Link is slowing my system down?
They definitely contribute. See this good old thread on Dynamic Link workflow. Pay attention to Todd's comments.
On a side note: you don't have to replace your dynamically linked comps in PrPro timeline every time you rendered DIs. You may set them as proxies for the dynamically linked comps right inside your AE project(s).
However, don't expect miracles and avoid the work flaw (no typo) like this stubborn kid established: with hundreds of dynamically linked comps it's not going to work. At least in real time. For example, a 30 min timeline built out of 935 dynamically linked comps, which are just original footages wrapped in their own compositions and hence are equivalent to DIs set as proxies inside AE project(s), it becomes almost unresponsive and with my spec takes around 27 hours to render...
Harm is way more qualified to work with you, but I do have one question that I have not yet seen addressed.
I suggest that using JPEG sequences in Premiere Pro is not that tough on a PC, or at least, mine doesn't seem to mind them at all.
However, what is the frame size of the images? Are you panning and scanning around the large frame inside Premiere Pro, or could you (or do you) resize them to fit into the 1080p sequence ahead of time - before you import them?
If you are panning around, that's fine. But if you are not, then cutting the images down to size before you import them could save a lot of bandwidth issues.
OK. But now what I asked exactly. What is the frame size? 1920X1080 or something much larger. I don't see where you mentioned the camera, so I can't guess at a frame size. My images are 4608X3456 the way I normally shoot. So cutting then down to fit a 1080p timeline reduces them significantly.
Twixtors process of interpolating motion between frames is really slow on every system. There maybe be other minor contributing factors, but every high end machine I have used it one slows done to a snails pace. Its the nature of the beast.
Thanks for your workflow explanation. It clears up the questions I had, but at the same time does not give a solution to your problem. I admit I have not tried it, so I'm not sure it will work or is any faster, but try it out with some stills to see it it helps:
Getting a third disk will certainly help, if you use one disk for your stills and one for your AE compositions, assuming a typical timeline will alter stills and AE compositions with a high regularity. That way the access times will be distributed among the two data disks, improving performance. Definitely make it a habit to defrag your disks on a daily basis with so many smaller files.
You currently have one memory bank in use for your 3 x 4GB ECC memory. Adding three more sticks to up memory to 24 GB will also give a nice performance boost, however I admit that I don't know if HP has attractive pricing on them. An extra disk can always be ported to a new system in the future, but that does not apply to memory, so you may have a high depreciation cost over the remaining life of your system.
Eventually, you may have to go for a dedicated raid controller and a number of SSD's in an array to significantly lower access times and improve transfer rates with these huge amounts of files. If you were to create a 5 disk SSD raid3 for instance, giving you 4 times the capacity of a single SSD, giving you average access times below 1 ms, versus the 14 ms you now have, and improving your average transfer rates from 120 MB/s to around 1500 MB/s.
For a new system I would definitely look at one that is overclockable, since Twixtor can benefit greatly from the higher clock speeds of OC'ed systems.
- Use .PNG instead of .JPG. I have found using PNG instead of JPG works much better on websites, maybe the same applies to PR.
- Numbered stills are often used with the .TIFF format. Maybe that is an alternative to JPG or PNG.
It's always about finding optimal 'quality/file size/render time' ratio.
In my testings JPEG 2000, PNG, TGA, TIFF, DPX provide similar quality.
TGA, TIFF, DPX have about identical render time. For JPEG 2000 and PNG it's up to 1.3 and accordingly up to 2.25 times higher.
JPEG 2000 provides smallest file size, whereas for PNG it's up to 1.3 times larger, and for TGA, TIFF and DPX - up to 3.7 times.
Thanks for your insight in file formats. So I guess Jpeg 2000 would be the way to go then?
Since it looks like you have some experience with this, could you tell me which preset you choose when starting a new project in Premiere Pro? I am currently using the default DSLR 1080 25p preset.
Thanks for your answer Harm. One thing I dont yet understand is what you say about my disk setup. (Be patient because Im really an amateur when it comes to RAIDs etc.)
I have at this time 5 disks in my computer. One is functioning as one disk alone, the C: drive. The other 4 are assembled in a RAID 1 with 4TB in total, the D: Drive. So if Im storing all my media, project files, and exports on my RAID 1 wouldnt that make adding an additional disk redundant?
Also If Im upgrading my RAM you advise me that I should go for non-ECC memory? Why is non ECC faster? And would it even work on my mobo? Since I have a server type mobo.
My last question:
If I would upgrade my RAM to 24GB, add maybe two (or three) SSDs to work in a RAID as my D: Drive (if thats possible) and optimize my file sizes, would you expect my projects to playback smoothly? And would it be worth the money? Maybe I also have to say that the current system Im working on has cost altogether about 6-7000 bucks. (Price in Swiss Francs; its probably more expensive here than in the US)
For some background on raids, see To Raid or not to Raid, that is the question...
Your C: volume for OS & programs is a standard single disk, your D: volume is a 4 drive mirrored raid1, that gives you twice the capacity and twice the speed of a single drive.
That means that all your source files, your intermediate AE compositions, your media cache, your preview files and your exports need to be read from and written to this single volume. Take into consideration that this volume is only half duplex, meaning that it can only read at one time and that action must be completed before writing can start. It is like a single lane road with traffic lights on both sides. Traffic can only flow in one direction at a time and traffic from the other side has to wait.
If I remember correctly the road up to Isenfluh and Sulwald is such an example. Never a problem when traffic is light, but with lots of traffic, delays become inevitable and that is what happens with all your files, many tracks, many compositions and effcts. You encounter traffic jams. The solution is to increase the number of lanes, like was done in Bern at the Ost Ring. It took quite some time though and there were still delays till the roadworks were finished.
In your situation spreading your traffic (file accesses) across multiple lanes (raid volumes) can reduce traffic jams.
ECC memory is always slower than non-ECC memory, because of the error correction bits that need to be added and checked. In addition ECC memory can often have slower speed specs than non-ECC memory.
So I guess Jpeg 2000 would be the way to go then?
That's not so straightforward and highly depends on both your rig and your project.
If saving on a disk space overweighs performance factors, then you may go with JPEG 2000.
With complex compositing, where multiple layers with time consuming effects are involved, your decision may be driven by savings on render time, which in turn depend on how noticeable they are on your machine.
Since you're testing everything instead of just jumping to a conclusion, you are on the right track.
Keep in mind, that optimising your workflow time-wise may involve pre-rendering your compositions in After Effects and rendering on the background via either Adobe Media Encoder or BG Renderer script.
which preset you choose when starting a new project in Premiere Pro?
General answer is here.
Since you are going to work on image sequences entirely, while delivering for a cinema, pick one from 2K or 4K 24p preset or build a custom sequence with this settings. Do not use DSLR 1080 25p - it doesn't serve your needs from any angle you described so far.
Thanks. Pre-rendering could really help optimizing my workflow. But in terms of preset I am looking for a preset to work with offline footage. The final online render will of course be in 2K 25p or 4K 25p.
My workflow should definetly be optimized for performance. Disk space is not important, since I have about 4 NAS with 30TB space in total.
I will look further in those presets and gonna do some render tests with a multi-layered twixtor sequence with JPEG 2000, JPEG and TIFF.
in terms of preset I am looking for a preset to work with offline footage.
You normally build your proxies on your project settings, not vise versa...
The final online render will of course be in 2K 25p or 4K 25p.
I'm confused. 25 fps is for PAL delivery, not for a cinema...
I'll have to have a look at this whole 24p 25p thing, I said 25p because PAL is standard in Europe.
However, I would like to ask you for some more advice if you find the time. Could you give me any tip about the preset I should configure in order to speed up my workflow. (Assuming I am working with JPEG or TIFF) I dont have any idea which codec I should use to make my workflow faster. Thanks.
As far as I understand, you don't shoot video, you take photos (probably timelapses). Hence, all those Canon or DSLR presets don't serve you - they are for video files, which come from a camera.
Define your delivery intentions: whether you're going to deliver just for a cinema or e.g. for a Blu-ray as well.
Ask theaters for the spec: with which frame rate you must deliver (with high probability in just 24p).
Build your project around these requirements.
If you need to deliver with different frame rates, test if built-in tools please you and preserve 'film look'. Check out this thread in After Effects Forum.
DO NOT start from 1080p so as to upscale to 4k on export. Rather edit in 4k and downscale to 1080p on export if need be.
If you have issues with realtime playback while working on 4k image sequences, make them offline (what you're currently doing), link to lower resolution proxies, select your clips in the timeline, right-click and choose 'Scale to Frame Size'. Reverse back when done.
Keep in mind that when you replace a clip or a group of clips in PrPro timeline with AE composition, 'Scale to Frame Size' flag results in creating nested composition in After Effects and Collapse Transformation switch turned on for this layer within the master comp. See e.g. this thread.
Hope, this clarifies things a bit.
I have always called a raid10 a solution for the paranoid in a hurry. It takes 4 drives to give you the capacity and performance of two disks, but gives you security by the mirroring.
Before going into your specific situation, allow me to tell something about volumes and drives, because they can be confusing and at the same time they are very important for optimal performance of a system.
Partitioning is a thing of the past and should not be used at all on modern systems.
You have to think about volumes more than about number of disks. In my current system I have 27 different physical disks but only 4 volumes. In the old one I have 17 disks and 5 volumes.
Now that we are clear what we are talking about, volumes with distinct drive letters, we can address your situation.
You have TWO volumes, C: (single disk) and D: (4 disks in raid10). Spreading the load across two volumes is more demanding and gives slower performance than using more volume, unless one or more volumes are very fast, as I tried to explain in a previous reply (remember Isenfluh/Sulwald?). If you add a SSD as you intend, you have increased the number of volumes to 3, which will definitely help performance, because SSD's are faster than conventional disks, the pagefile can be stored on the SSD, so all your performance will go up.
Compare your setup with mine with rough estimated figures:
|Volume||Valentin||Harm||Transfer rate Valentin||Tranfer rate Harm|
|C:||1 HDD||1 SSD||125 MB/s||450 MB/s|
|D:||4x Raid10||1 SSD||250 MB/s||450 MB/s|
|E:||NA||21x Raid30||NA||2,700 MB/s|
|F:||NA||1 HDD||NA||150 MB/s|
These figures are indicative, but do show where the major differences are. In my experience disk setup is overlooked quite often, but has a huge impact on a system's responsiveness. It is the weakest (slowest) link in the chain, afterall and with your workflow, doubly so.
But in your specific case there is something else, and that is your disappointing hardware MPE score. 100 seconds is extremely slow, even for a Quadro 4000. I would be quite normal to see a score around 8 - 9 seconds on such a system, well maybe around 12 - 13 seconds with your ECC memory, but 100 is way too slow. Some background services or processes are interfering with the hand-over from memory-GPU-VRAM-GPU-memory. This can be caused by a myriad of things, but a good starting point would be the BlackViper list of services to set to manual or disabled and taking a closer look at the processes running with Process Explorer. There should normally be less than 50 processes running.
Hope this helps.
I have always called a raid10 a solution for the paranoid in a hurry.
I think I qualify. I have started worrying about my two RAID0 setups lately and it is probably time to get one of those automatic web backup subscriptions. The problem is that I would need a RAID controller to put in even one more disk so my seven disks (5 volumes) will have to do for now.