13 Replies Latest reply on Jan 20, 2011 6:14 AM by Scott Chichelli

    Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?

    Harm Millaard Level 7

      Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?


      Life used to be so simple when shooting video on a tape based camera. You shot your material, captured it for editing and stored your precious original footage on tape in a safe and dry place. Sure, it took time to capture, but the big advantage was that if you had a computer or drive failure, you would still have the original tape so everything could be recreated.


      Now with tapeless workflows we have the significant advantage of much faster import of the original footage. Connect the flash card or disk drive to the computer over USB and copy the data to a HDD on the computer, ready for editing. The data on the flash card or disk drive can then be erased, so you can reuse it for more shots. But, like Johan Cruyff has said repeatedly, every advantage has its drawback. In this case it simply means that you no longer have the original material to fall back on, in case of computer or drive failures. That is a very unpleasant and insecure feeling.


      The easy anwser to that problem is backups. Backup of the original media, backup of projects and backup of exports. This often means a bundle of externals for backup or NAS configurations. One thing is clear, it requires discipline to make regular backups and it costs time, as well as a number of disks. Four as a minimum: 1 for media, 1 for exports and at least 2 for projects. Note: This is excluding a backup drive for OS & programs.


      There are different backup strategies in use. Some say backup daily and use one disk for monday, one for tuesday, and so on.  Others say one disk for the first backup, the second for the second backup, then the first again for an incremental backup, etc. and once weekly a complete backup on a third disk. Whatever you choose, be aware that shelf live of a disk is far less than tape. There are horror stories everywhere about ball-bearings getting stuck after some time and without original tapes, you better be safe than sorry, so don't skimp on backups.


      What is the relevancy of all this? I thought this was about Sandy Bridge and other PC's.


      It is and let me try to explain.


      Card based cameras are for the most part DSLR and AVCHD type cameras, and we all know how much muscle is required to edit that in a convenient way. Adobe suggests in the system requirements to use raid configurations for HD editing and practice has shown that raid arrays do give a significant performance boost and improve responsiveness, making for a nicer editing experience. The larger the project and the longer the time-line, the more a raid array will help maintain the responsiveness.


      One thing you would not do is using a raid0 for projects, media and exports, even if you have backups. The simple reason is that the chance of disk failure multiplies by the number of disks in the raid0. Two disks double the chance of disk failure, three disks triple the chance, four disks quadruples the chance, etc.


      Remember: Disaster always strikes when it is most inconvenient.


      Imagine you have been working all day on a project, you decide to call it a day and to make your daily backup, but then the raid fails, before you made your backup. Gone is all of today's work. Then take into consideration the time and effort it takes to restore your backups to the state it was in yesterday. That does not make you happy.


      Another thing to avoid is using a software or mobo based parity raid, for the simple reason that it is slooowww and puts a burden on the CPU, that you want to use for editing, not house keeping.


      For temporary or easily recreated files, like the page-file, media cache, media cache database and preview files, it is very much advised to use a raid0. It makes everything a lot snappier and if disaster strikes, so what? These are easily recreated in a short time.


      This was a general overview of what is required with tapeless workflows. Now let's get down to what this means in terms of system design.


      Two approaches or train of thoughts


      KISS: Keep it stupidly simple or LOVE: Laughing over video editing


      The first one, the most economic one, is to use a system with 3 or 4 disks internally and 4 or more backup disks.


      A typical disk setup can look like this:

       

      18-Jan-11 11-40-06.png

      This is a perfectly sensible approach if one does not have large or complex projects, long time-lines and is willing to take the risk of occasionally losing a whole days work, between backups. Many hobbyists and consumers fall in this category.

       

      The KISS approach keeps it stupidly simple. The drawback is that there is no logical way to add more disks or storage. The discipline, diligence and effort required for regular backups make it far from a laughing matter. In fact it can quickly become a bore. Add to that the fact that the disk setup is simple but not very fast, so less suited for situations where lots of clips are involved, multi-cam is a regularly recurring situation or lots of video tracks are involved.

       

      A number of video editors want more from their system than the occasional platonic KISS, they want to really LOVE their system, which lead to the other train of thought.


      This is more costly than the KISS approach, but you all know a fiancée or wife is more costly and dear than the occasional kiss on the cheek by an old friend.


      Let's start with a typical disk setup. It may look like this:

      18-Jan-11 12-26-35.png

       

      Two striking differences in comparison to the KISS approach:


      1. Much easier disk organization and more disks and thus more space.
      2. It requires a hardware raid controller, causing a higher investment cost. It is like an engagement ring. You don't get LOVE for free, one of the guiding principles of the oldest trade in the world.


      These are easy statements to make, but what are the benefits or advantages, that you would fall in LOVE with such a system, and what are the drawbacks? Think back to Johan Cruyff's adage.


      The only drawback is cost. The advantages are multiple, easier organization, more speed, more storage, snappier editing, no jerkiness, lesser requirements for regular backups and - this is the major benefit - hardly a chance of losing a day's work in case of a drive failure. Keep in mind that a parity raid keeps all your data intact in case of a drive failure, so lessens the need for up-to-date backups.


      We all know, we get what we pay for: "If you pay peanuts, you get monkeys. OTOH, if you pay money to monkeys, you get rich monkeys". But in this case you get what you pay for, a much better editing experience with a much easier workflow.


      Using a parity raid (be it raid 3/5/6/30/50/60) you get security, ease of mind that you are protected against losing precious media, that you need not worry about the last time you made a backup, that the editing you did today may be lost and you save valuable time editing and a lot of aggravation because of a much more responsive system.


      How does this all relate to Sandy Bridge and other PC's?


      First of all, the price difference between a Sandy Bridge / P67 platform and an i7-950+ / X58 platform is very small. Of course the new architecture is slightly more expensive than the older one, but the differences are small, almost not worth talking about.
      So what are the differences? Look below:

       

      18-Jan-11 14-37-58.png

      The first thing to keep in mind is that the Sandy Bridge is the successor of the i7-8xx CPU and as such it is much more evolutionary than revolutionary. The CPU power has increased significantly over the i7-8xx due to new architecture and a smaller production process (32 nm), but in essence all the capabilities have remained unchanged. Same memory, same PCI-e lanes, same version, same L3 cache and no support for dedicated raid controllers.


      It is great that the processor performs much better than the older i7-8xx CPU's, almost achieving the level of the i7-9xx range of processors, but is still limited:


      The Sandy Bridge is unsuitable for anything more than a KISS system.


      Why? Because it lacks the required PCI-e lanes to accomodate more than a 16 x PCI-e nVidia card with CUDA support to enable hardware MPE acceleration and the integrated graphics are not supported by CS5.


      You may wonder if that is a bad thing. The plain and simple anser is NO. It is a great processor, it delivers great value for money, is a solid performer, but it has its limitations. Intel had a reason to position this CPU as a mid-level CPU, because that is what it is, a mid-level performer in comparison to what is to come.


      The term mid-level performer may seem strange when compared to the old generation of i7-9xx CPU's, because they perform almost equally well, but keep in mind that there is a generation difference between them.


      So what about the i7-9xx and X58 platform?


      It still is going strong. About the same performance as a Sandy Bridge, with only the much more expensive hexa-cores clearly in the lead, both performance and price wise. The quad cores deliver about the same value for money.  The main difference however is the platform that allows a dedicated raid controller to be installed, thus making it the platform of choice for those who want to go from a passing KISS to true LOVE.


      And what lies ahead?


      Sandy Bridge E on the Waimea platform (X68). Now that is revolutionary. More than double almost everything a processor can offer: double the cores, double the PCI-e lanes, triple the memory, more than double the L3 cache, increase the PCI-e support from 2.0 to 3.0, etc...


      This is why Intel calls this a high-end CPU / platform.


      So what now?


      If you prefer a KISS approach, choose either a Sandy Bridge/P67 or an i7-950+/X58 platform.


      If you wonder whether in the future you may need multi-cam more frequently, edit more complex projects and longer timelines or even progress to RED, look at KISS/LOVE solutions, meaning the i7-950+/X58.


      If you can't have downtime, time pressure is high, delivery dates to clients are critical or you edit highly complex projects, lots of multi-cam situations or lengthy time-lines, choose a LOVE solution, an i7-950+/X58 platform.

       

      18-Jan-11 13-21-37.png


      If you have the time to wait till Q4/2011, Sandy Bridge E/Waimea looks to be worth the wait.

       

      Hope this gives you some more insight into recent and future developments and helps you make wise investment decisions.

        • 1. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
          Scott Chichelli Level 3

          HI Harm,

          nice article but i can read between the lines.

           

          and frankly have to disagree with your biggest point.

          Raid 0.

           

          while it would be nice if everyone had your money and could afford a big raid 3,5,6 raid array most do not.

           

          90% of systems i sell look like this (and on PPBM)

           

          OS 1 drive

          Media 2 drives in raid 0

          Render 2 drives in raid 0

           

          other have just 3 drives.

          the rest have the large raid 5,6

           

          fail rate or data loss is very rare and we can have issues with the big raids as well. not data loss but drives dying, 8 drives is 8 times the potential for failure)

          you know as well as anyone how long it takes to rebuild an array after a drive dies..

           

          Data loss can oocur on a large raid as well dirty power, power spike, catestrophic etc..

          99% of the time you will know a drive is on the way out before it happens... so so much for working all day and poof it gone..

           

          lastly you can buy Sandy Bridge boards that allow for a real raid card making your kiss statement incorrect. Eric had already showed you this..

           

          frankly as hard as we try few understand the benefits of a raid 3,5,6 AND external back up...

          if it were up to me everyone would have raid 5.. FYI i can do raid 5 onboard for the very budget but cautious.

           

          Scott

          ADK

          • 2. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
            framednoise

            I really don't see why there's so much fuss over this. Different tools for different needs and budgets.

             

            Sandy Bridge socket 1155 is an excellent platform for those who don't want/need or can't afford a more seriously configured system. Need a raid or two? Well, there are plenty of pci-e x4-raid-cards with support for up to 8 drives. How many average Premiere Pro users will actually ever buy more drives than that for a raid? Many standard P67-based motherboards have one x16 AND one x4 pci-e port, not shared, providing enough bandwidth (in theory 1.6GByte/s vs 3.2 for x8) for all but the most extreme cases. I have yet to see numbers of how a x8 + x8 system would fare in PPBM5 either, but my guess; not that much worse.

             

            A proper built 2600K-based system is excellent value for the money. Sure, the coming Sandy Bride E-platform will be a lot better, but if you're buying today I really don't see the point in putting your money on a platform that's going to obsolete by Q4 anyway (ie, 1366). Unless you're going with the 980x you likely won't get better performance anyway, providing you're using the same disk set-up. From what I can see PPBM5 seems to favor huge and fast raids, in terms of raw computing power the 2600K beats all other quad cores on the market today (yes, i7-950, I'm looking at you) and there will certainly be faster CPUs for the 1155 socket in the future, something you can't say for the 1366 socket...

             

            If you're a normal (average) user, editing AVCHD and DSLR-material and doing Photoshop work, you can't go wrong with a Sandy Bridge system as an upgrade from an older (pre socket 1366) platform. Can't really see the point in buying the i7-950 today unless you actually need that extra disk performance with +8 drives.

             

            /daniel.

            • 3. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
              Harm Millaard Level 7

              Although there are technically four PCIe lanes open for PCIe expansion  cards on the LGA 1155 platform (after accounting for onboard USB3,  PCIe-to-PCI bridge and additional SATA/IDE controllers), the few  motherboards that have PCIe x4 slots actually share the bandwidth of  that slot with the PCIe x1 slots on those same boards. This means that  ANY card plugged into a PCIe x1 slot would have forced the x4 slot to  run in x1 mode. That hurts the performance of most discrete hardware  RAID controllers. The only other place to put a RAID card would have  been the second PCIe "x16" graphics slot (which is bifurcated from the  main PCIe x16 slot, and thus both slots would have been forced to run in  x8 mode). This does hurt MPE GPU performance.

               

              A few high-end LGA 1155 motherboards have onboard PCIe lane repeaters.  Unfortunately, while they create additional PCIe lanes, they do not  change the total bandwidth of the CPU's integrated PCIe  controller. So instead of 16 PCIe lanes operating at full PCIe 2.0  bandwidth, you now have 32 PCIe lanes that are artificially restricted  to PCIe 1.0 bandwidth -- 2.5 GT/s instead of 5.0 GT/s.

              • 4. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
                Jim_Simon Level 9

                I think Harm makes a very good point here, and I'm glad he did.

                 

                My own plans after upgrading to AVCHD this year were to configure a RAID 3 for media.  (Without that tape backup, doing any less is just asking for trouble, in my opinion.)  The starting point for this seems to be the Areca ARC-1210, which is a PCI-E 8x card.  Using a Sandy Bridge mobo, there appears to be no way to get both my GTX 460 and the Areca running at full speed.

                 

                For me, that's an issue.

                • 5. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
                  pauncho1

                  I'm upgrading from an AMD 3800+, cutting with Vegas 7 Pro. Usually shoot DSLR or HDV, sometimes P2, EX or RED. I have ridiculously cheap access to Macs, FCP/FCS, all kinds of software.

                   

                  I've been agonizing over this for the last month, was originally hoping the UD7 mobo was the solution, read the read about the NF200/PCIe issue a few days ago, http://www.dvinfo.net/forum/non-linear-editing-pc/489424-i7-980x-now-wait-sandybridge-2.ht ml- and still decided to go for a 2600k. 

                   

                  My preference is to treat my video footage the same way as my digital imagery: I make (at least) duplicate back ups of everything before reformatting the cards, never delete the back ups, and only worry about the day-to-day stuff at night. Unless I'm rendering or involved in other long processes, in which case I'll back up the work in process the next day. If I am under a really really tight deadline I might back up as I go.

                   

                  Yes, a RAID might make it easier, but I'm paranoid enough to prefer a slower, safer backup. You can always duplicate, and usually improve upon, a days work, but you can never get back original footage you lost. I have only ever had one hard drive die on me (a few enclosures crapped out, though)- it took a couple of (mostly unattended) hours to rectify. As a matter of act, I've had far more loss/damage from tapes than from hard drives.

                   

                  I ordered the UD7, 2 F4s and 4 F3Rs, understanding I will probably want to upgrade to SBE when it comes out, or maybe next year. The 2600k/mobo/RAM will likely hold its value better than a 950/X58, likely because of the marketplace as much as merit.

                   

                  The UD7 / RAID card issue is in it's early days, there may be a solution/mitigation. Probably not. But if I really really need a RAID card, then I probably really really need a 980, NAS, etc etc.

                   

                  But Harm still rocks!

                  • 6. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
                    hellerbrewing

                    So where does Ivy Bridge (LGA 1356) fall in intel's arrangement if Sandy Bridge E (LGA 2011) is going to be the new enthusiast platform?

                    • 7. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
                      Scott Chichelli Level 3

                      one is desktop (1366 replacement) the other is Xeon..

                      the Xeons will have quad channel ram not the desktop

                      • 8. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
                        hellerbrewing Level 1

                        So the 1356 is the enthusiast and the 2011 is the new server

                        platform?

                        • 9. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
                          Scott Chichelli Level 3

                          http://pc.watch.impress.co.jp/video/pcw/docs/360/112/6.pdf

                           

                          a good article on present Sandy Bridge

                          http://www.xbitlabs.com/articles/cpu/display/sandy-bridge-microarchitecture.html

                           

                          most notably page 3

                          The second important improvement in the early pipeline stages deals with the branch prediction unit...

                          According to preliminary estimates, the branch prediction correctness in Sandy Bridge improved by more than 5% compared with the predecessor.

                           

                          If we compare all above described innovations, we will see that the microarchitecture of computational cores in Sandy Bridge processors has been modified more than significantly. These innovations are undoubtedly serious enough to be regarded as dramatic modifications rather than simple fixing of Nehalem’s bottlenecks.

                           

                          Scott

                          • 10. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
                            Jim_Simon Level 9

                            the Xeons will have quad channel ram not the desktop

                             

                            Hmm.  According to other sources, the 6 and 8 core Sandy Bridge-EN Desktop models will also have 4xDDR3

                            • 12. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
                              Jim_Simon Level 9

                              Thanks Scott.

                               

                              Here's what I saw as the relevant quote from the top link.

                               

                              "The highest speed 8-core CPUs with up to 150W TDP should, however, be reserved for the high-end Socket LGA2011. With more power and ground lines to support 40 PCIe v3 lanes and four DDR-1600 memory channels per socket, as well as dual QPI 8 gigatransfers/sec links, the 8-core, 20MB L3 cache Sandy Bridge-based Xeons should have sufficient system bandwidth to feed even the highest workloads."

                               

                              It specifies that CPUs for the 2011 socket will have 4 memory channels.  The same is verified in your second link.  But neither necessarily precludes an i7 from using the 2011 socket.  And at the following link, we do see hints of planned Desktop (i7) 6 and 8 core models for the 2011 socket using 4 channels.

                               

                              http://en.wikipedia.org/wiki/Sandy_Bridge_%28microarchitecture%29

                              • 13. Re: Tapeless workflows and Sandy Bridge or other PC's: KISS or LOVE?
                                Scott Chichelli Level 3

                                i have an email out to an intel engineer.. getting too many conflicting answers

                                looking around the net.. i had found the one you linked to but it does not call out the socket (we now know to be 1156)

                                and clicking thru his references i found nothing to support the quad channel on anything but Xeons. i felt it a little rushed..

                                i am under constant NDA with Intel so i may not be able to say much when i do get an answer..

                                 

                                his is due to pop around here in a month but i dont want to wait..