Skip navigation
Currently Being Moderated

Using External HDD's for Video Editing

Jan 28, 2011 7:24 AM

Tags: #external_hard_drive

External HDD’s (Hard Disk Drives) are everywhere, and quite popular. These allow one to increase the storage capacity of a system greatly, quickly and easily, for not that much money. We now have many sizes, and many connection types. Some connection types are better than others, if one is doing more than off-loading data to archive it, and perhaps free up internal HDD space. With the right connection type, one can also edit to/from the external HDD’s. Again, some connection types are better than others. Note: for editing Video, the physical HDD’s need to be at least 7200 RPM.

 

For simple archiving, any connection type will work well. In that process, files are just written to the external HDD, and when needed, read from the external HDD. The problem comes in, when one tries to edit to/from the external HDD. Most current computers are faster than the slower connection types can manage. The OS is firing commands for reads/writes too fast. The OS expects rapid response, and can overwhelm the external HDD. When one adds in an NLE (Non Linear Editor) program, there is a great deal of HDD activity, and this requires many reads and writes, and requires them quickly. The files are very large too. Suddenly, the I/O (Input/Output) sub-system is overwhelmed, but the requests keep coming.

 

Here is a real-world look at the data transfer of some common connection types:

 

Average transfer rates in MB/s for different interfaces:

 

  • USB2:    20 - 25, depending on other USB devices sharing the same bandwidth
  • FW400:  30 - 35
  • FW800:  50 - 60
  • USB3:    65 - 80, depending on other USB devices sharing the same bandwidth
  • eSATA: 100 -140
  • SATA:   100 - 140

 

Raid0:    0.9 x N disks over a single disk

Raid3/5: 0.8 x (N-1) disks over a single disk for read, 0.6 x for write. ICHR10 figures are a bit lower than hardware controllers.

 

Note: “FW” refers to FireWire, which might also be referred to as IEEE-1394a (FW-400) or iLink, and as IEEE-1394b (FW-800). The connectors are different, as are the cables. Also, one needs the appropriate controllers. Most computers have FW-400/IEEE-1394a/iLink controllers on the MoBo and connectors available for the cables. For FW-800/IEEE-1394b, one usually needs to add a controller, either on a PCI card, or via PCMCIA, or ExpressCard. More on controllers later.

 

As one can see, the USB 2.0 connection is slower than the rest. Problems can arise, when trying to edit to/from such an external HDD. They can be plagued with read/write errors, as they cannot maintain the data transfer, required by the OS and by the NLE program. When the allowed time has expired, based on the OS’s expectation of data transfer, it stops the process, and issues a read/write error. In a worst-case scenario, the OS will have routed the write data into a buffer, waiting for the external HDD to catch up. These buffers are finite in their capacity to store the data, while the OS is waiting. It does not take too long, before the dreaded “Delayed write failure” rears up. This can lead to a catastrophic failure of the external HDD, as the data is only partially written, and the File Allocation Tables are not completely written, causing the loss of ALL data on the external HDD. At that point, all data on the external HDD is lost. It might be possible to do a recovery with a program, like Stellar-Phoenix, but it is not cheap, not is it fail-safe. Depending on the value of the files lost, one might hire a data recovery service, but these are anything but cheap. Probably best to just do a complete low-level Format of the external HDD and then do an OS Format.

 

Also, while speaking of Format, most external HDD’s come from the factory, Formatted to FAT-32, an older Format type. Besides being slower, FAT-32 has a max. file size of about 4GB, and AV files are often much larger. One needs to CONVERT the Format to NTFS for the PC, or to the Mac OS’s Format, to allow for the larger files, and to increase the speed a bit. On the Mac/PC front, a PC can read/write to FAT-32, or NTSF. However, a Mac can ONLY read/write to a PC external HDD, Formatted to FAT-32. If one is migrating the HDD’s between Mac’s and PC’s, the external needs to remain in FAT-32 with its limitations. Just one consideration, when setting up external HDD’s for use with an NLE program.

 

For editing Video, I have found that the connection types should be considered this way:

 

  • USB 2.0 - archiving for storage only
  • FW-400 - archiving for storage, and light editing to/from (just very slow)
  • FW-800 - archiving for storage, and regular editing to/from (fairly fast)
  • eSATA - archiving for storage, and regular editing to/from (about as fast as an internal SATA)
  • I have not tested USB 3.0 in editing situations, so cannot comment on their acceptability.

 

Obviously, a SATA internal is ideal for all tasks.

 

Now, back to controllers. The real-world throughput of an external can be diminished due to the controller, and the number of chips available. Many MoBo’s have 1 - 2 USB 2.0 chips, and 1 - 2 FW-400 chips. One needs to check the number of chips provided. One might have multiple connectors, either on the computer’s case, or direct from the MoBo. These connectors might go to multiple controller chips, or might all feed into one. The same can be said for add-on cards, PCMCIA cards, or ExpressCards. The number of connectors does not guarantee the number of chips.

 

The more externals, and peripherals that are connected to the controller chip, the lower the data transfer rates. This holds for hubs especially. One might have various externals and peripherals plugged into various connectors, but everything being fed through a single controller chip. Add to that, the ability to daisy-chain multiple externals through the In/Out connectors on most external HDD’s, and overall performance falls even more. USB 2.0 and FW-400 suffer most from these multiple externals limitations, but FW-800 can too, just not to the same degree.

 

When one is using an external to edit to/from, there is another consideration. When peripherals with mass storage capabilities are plugged into a computer, a drive letter is assigned by the OS. Most NLE programs, like PrE/PrPro rely on the drive letter to link to the Assets. They do this via an absolute path, including that drive letter. Depending on how many other peripherals are plugged in, and the order that they are plugged in, that drive letter assignment might differ, session to session. When using external HDD’s, one should go into the OS’s Disk Management console and assign a fixed drive letter for that individual external HDD. This would hold for however many one is likely to use, and each should have a unique drive letter. I suggest marking each external with the drive letter that the user assigns to it, say Z:\. Then, whenever Z:\ is plugged in, it will always be seen as Z:\. This way, the NLE can keep up with where the Assets are located, starting with the drive letter. If one is migrating Projects between computers, they will repeat this exact process in the OS of each computer.  Note: when doing the migration, ALL Assets, Scratch Disks, and the Project file, MUST be included on that external, or one will spend a good deal of time tracking down the Assets.

 

External HDD’s are great devices, but some considerations need to taken into account, before one just plugs them in, and begins editing.

 

Hunt

Branched to a new discussion.
 
Replies
  • Currently Being Moderated
    Feb 6, 2012 1:51 AM   in reply to Bill Hunt

    Video editing is actually one of the most like apps in many iphone these days. But recently many longtime consumers of Apple's Final Cut Pro non-linear video editing software were upset when the business upgraded to Final Cut Pro X last summer. The software has been a staple for video editors, pro and novice, since 1998. But the new edition is incompatible with files from old versions of the software, and the interface has been entirely redesigned. Apple released upgrade 10.0.3 Tues in an attempt to correct some of the troubles. Resource for this article: New update for FCP X addresses user complaints.

     
    |
    Mark as:
  • Currently Being Moderated
    Jul 5, 2012 7:34 PM   in reply to Bill Hunt

    I'm undecided on how to set up a raid 0 not sure if internal hard drives would be best or docked external hard drives? which would work best in video editing.. I'm looking for speed and safety which from my understanding it's one or thee other?... I have a Mac pro 2.8 quad-core intel xeon 10.6.8 with 7gb of ram I do all editing with adobe.

    I don't have a huge budget so i'm looking to know whats necessary for smooth work flow.

     

    Thanx in advance!

     
    |
    Mark as:
  • Currently Being Moderated
    Dec 20, 2012 3:23 PM   in reply to Bill Hunt

    What about using Mac Drive for windows?  I have a Black Magic hyperdeck studio that records to SSD drive (they have to be formatted hsf+ which is a mac format).  I will be docking these in an eSata dock but need to be able to read them with my windows machine.  Does anyone know if Mac Drive slows down the read/write speed?  I could always transfer from the SSD to the internal but was hoping to skip that step and edit directly from the SSD.

     
    |
    Mark as:
  • Currently Being Moderated
    Jul 12, 2013 9:26 AM   in reply to Bill Hunt

    BILL

     

    Have a question.   I am going to be loading CS6 MASTER COLLECTION on a new SONY L-SERIES DESKTOP.  The question I have is this - No 2nd internal hard drive for this computer.  Will need to use an external.  Will be doing ESATA or SATA 4TB.  The PC doesn't not come with a ESATA but has ThreeUSB3 connections.  Can I go from the external ESATA connection to one of those USB3's & not die a screaming death while editing in PRIEMERE?  The prcessor is a QUAD CORE 3.80.

     
    |
    Mark as:
  • Currently Being Moderated
    Nov 23, 2013 6:39 PM   in reply to Bill Hunt

    Hello Bill Hunt
    This is indeed a very good idea, this technology is also currently being accepted by more and more people will become a new trend, I believe a later time technology will become increasingly mature. I do not know when to use an external hard disk to edit video files in my original software can use it? And my DR. iPhone Data Recoveryhttp://www.driphonedatarecovery.com/software can use it?


     
    |
    Mark as:
  • Currently Being Moderated
    Feb 26, 2014 4:27 PM   in reply to Bill Hunt

    Hello Hunt

     

    V.useful post for someone new to HDD issues like myself. Thanks.

     

    Newbie question:

    I have 2 internal drives on the "built" system I bought a few years back.

    c drive: Samsung SpinPoint F1642 640 GB.

    d drive: Storage 2: Samsung SpinPoint T166 500 GB

     

    As I understand it, the aim is to minimise risk of data loss by "spreading" reading/writing of data amongst as many drives as possible.

     

    So I should read from one drive and write to the other?

     

    I have cs5.5 using premiere (music video editing) and soon aftereffects (compositing)

     

    When funds allow an additional HDD, is there any reason why it would have to be physically external to the machine? My Gigabyte GA-P55A-UD3R motherboard has slots for at least 1 internal drive.

     

    If you say yes external is needs to write end video what do you think of the Lacie 301543 D2 Quadra 2000 GB External? My motherboard has an eSATA-300 interface.Logical choice?

     

    Thanks again.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 28, 2014 4:29 AM   in reply to Bill Hunt

    Thanks Hunt,  That was a comprehensive and immensely useful response.

    Peace

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)