10 Replies Latest reply on Aug 11, 2011 10:15 AM by ECBowen

    NAS or similar for collaborative workflow

    Colin Brougham Level 6

      Without going into too much in-depth detail about the situation I'm in, I'm looking to set up two Premiere Pro workstations to access media stored on one central unit. Media is ProRes422 and DVCPRO50; one system will access the unit to do really basic stuff--logging, transcription, selects--and the other will do more of the heavy lifting, such as it were. These systems and storage will never see a drop of HD--these will be SD only projects.

       

      Ignoring the possible headaches caused by sharing projects and media--and paying close attention to the bottom line--what sort of solutions can I consider here? I'd love a rippin' FC network and storage setup, but that ain't happenin'. What about something like this: QNAP TS-659-PROII-US Turbo NAS ? I'd access it via USB3.0 or eSATA from the primary editing machine (this is pretty simple editing--2-3 streams of SD) and then via the GigE ports (both at the same time, maybe?) from the logging machine. Most likely, the storage unit would not be heavily accessed by both machines simultaneously, if at all--it's more a convenience to have them shared than anything else.

       

      If this could work, what sort of drives and what would be a recommended RAID level to use? It looks like it's capable of many different configurations. If this box would definitely not be sufficient, what else could I look at to support this kind of workflow? As mentioned, budget is a concern: I want the right tool for the job, however. I do not need to shoot for the moon here, though, since this is straightforward documentary work, all in SD, and absolutely no HD.

       

      Thanks y'all...

        • 1. Re: NAS or similar for collaborative workflow
          Harm Millaard Level 7

          Colin,

           

          The NAS solution you envisage is extremely difficult. You best solution is to go for 10 Gb/s iSCSI with support for jumbo frames (because of the 6-fold reduced packet overhead), but you need to make sure that the NAS, switch and both PC's support that. Simple 1 Gb/s ethernet is too slow and teaming is difficult, because all components need to support 802.3AB protocols (I think) with LACP and you need to run multiple processes and different ports on each PC to use teamed connections on each PC, because of the shared TCP/IP connectivity.

           

          NAS in a distributed environment like video editing is usually out of the question. SAN with its much higher price tag is the better solution.

           

          You may be better off with dual storage solutions that are synchronized daily or even more times per day.

          1 person found this helpful
          • 2. Re: NAS or similar for collaborative workflow
            ECBowen Most Valuable Participant

            You can edit off these units via the Gbit if you don't mind the delay between switching clips and start of playback. If you use the iSCSI protocol, the delay is less but still present in Adobe. You would deal with the Delay more than bandwidth issues with SD material on this unit.

             

            Eric

            ADK

            1 person found this helpful
            • 3. Re: NAS or similar for collaborative workflow
              Colin Brougham Level 6

              Thanks Harm--I can see the dollar signs piling up

               

              Any thoughts on building your own (preferably economical) SAN?

              • 4. Re: NAS or similar for collaborative workflow
                Colin Brougham Level 6

                Thanks Eric--that doesn't sound too bad. The offline box will be doing pretty barebones work on the files, so a bit of a delay is not a major problem. I'll take a look at the iSCSI stuff; does that require much additional hardware or software on the host PC? Or is that built into Windows 7 now (seem to recall that)? I'm vaguely recalling us having a similar conversation a year or so ago... so hazy...

                • 5. Re: NAS or similar for collaborative workflow
                  jasonvp Level 3

                  Wow.  Some mis-information abound in this thread. :-)

                   

                  It is still very difficult for a PC-based solution to make real good use of a 10GigE NIC without a LOT of internal kernel hacking.  Yes, 10GigE is becoming more and more available, but the vast majority of their use is smaller than a single GigE NIC.  10GigE switches are still fairly expensive, as well.

                   

                  Using jumbos is a good idea, and GigE supports jumbos (and has, long before 10GigE existed).  The challenge is the managed switch you'll need to make it work.  You're run-of-the-mill piece of crap Linksys purchased at NewEgg isn't going to cut it.

                   

                  802.3ad NIC teaming is very easy to do assuming you have a switch that can support it.  The hooks for it are already in every major OS out there, without the need for add-on anything.  Once you've teamed a pair of NICs, it looks to the OS (and all applications) as a single 2GigE pipe.  No extra processes or anything like that are necessary.  But again, this requires a managed GigE switch.

                   

                  Using a NAS with iSCSI will look and act fairly similarly to a SAN running over fiber channel.  In other words: to the client device, it appears as a block device... another disk drive.  You CAN NOT SHARE iSCSI filesystems between client computers.  Like a SAN block device, an iSCSI block device is literally treated like a disk on the OS.  No workie that way.

                   

                  If you want to share data like this, you need a NAS.  You need a filesystem on one system that can be exported to another (set of?) system.

                   

                  jas

                  • 6. Re: NAS or similar for collaborative workflow
                    ECBowen Most Valuable Participant

                    "You CAN NOTSHARE iSCSI filesystems between client computers. Like a SAN block device, an iSCSI block device is literally treated like a disk on the OS. No workie that way."

                     

                    Actually that is incorrect as well. The iSCSI protocol makes the volume a direct attached storage volume to the assigned client. To share that volume, you simply share it out like any internal drive that are set to shared on the network. Those volumes then use the standard file sharing protocol instead of the iSCSI protocol as far as network layers/packets go. So the host system of the volume will have a lower latency connection to the volume where the systems that hit the shared directories or volume will have the standard network latency when dealing with shared drives. That is the only real difference between the protocols other than theoretically the iSCSI is also suppose to be faster overall but it normally does not work out that way.

                     

                    There are also Virtual iSCSI programs out there that allow shared use of a iSCSI volume but that is an entirely different scenario and commonly used by blade servers.

                     

                    The Microsoft iSCSI client is now built into the OS so you dont require 3rd party software.

                     

                    Eric

                    ADK

                    • 7. Re: NAS or similar for collaborative workflow
                      jasonvp Level 3

                      ECBowen wrote:

                       

                      Actually that is incorrect as well. The iSCSI protocol makes the volume a direct attached storage volume to the assigned client. To share that volume, you simply share it out like any internal drive that are set to shared on the network.

                       

                      I think you may have missed my point.  If you have a storage server running the iSCSI protocol, you can only use the iSCSI protocol to connect to that exported block device once.  One client.  Re-exporting it via the network to another client was not what I meant; of course you can do that.  But why bother, at that point?  Just configure both clients to connect to the server via SMB or NFS.  Then they can share all the data they want without various levels of indirection/redirection.

                       

                      jas (I do this stuff for a living :-))

                      • 8. Re: NAS or similar for collaborative workflow
                        Harm Millaard Level 7

                        Jas,

                         

                        If you have a storage server running the iSCSI protocol, you can only use the iSCSI protocol to connect to that exported block device once. 

                         

                        That is exactly why you need multiple processes and different ports on each PC to make it work with two or more devices and it is far from easy to achieve that in a reliable way. From a practical point of view, you are correct, but it can be solved to work like Colin wants, only is it worth all the time and effort to make it work? I doubt it. It really is a huge undertaking.

                        • 9. Re: NAS or similar for collaborative workflow
                          ECBowen Most Valuable Participant

                          Actually I did not misunderstand, I was clarifying your post so others did not misunderstand. The post did not state it was not recommended. It stated you could not share iSCSI files systems which is not accurate. It would be easily misunderstood as you cannot share a volume that is connected via iSCSI. Also the communication between the client and the host does not change for the system accessing a share versus the Host. Basically the Host is the server at that point so that communication remains the same as if both systems would be accessing a server whether Windows or Linux with Samba. However if you setup one of the editing stations as the host because you don't require a server for only 2 clients then atleast 1 of the editing stations has the streamlined packets regarding latency. The SAN/NAS unit can be connected to a switch directly instead of 1 of the system ports which should help moderate the communication in that situation. Remember the original post is only dealing with a very small network. Your scenario is harder to implement and is used for a larger infrastructure.

                           

                          Eric

                          ADK

                          • 10. Re: NAS or similar for collaborative workflow
                            ECBowen Most Valuable Participant

                            BTW forgot to mention allot of the decent or better NAS units that support iSCSI also support both SMB and NFS. So you can set them up that way and eliminate 1 level latency point. Check on the Specs at the manufacturer website.

                             

                            Eric

                            ADK