19 Replies Latest reply on Aug 12, 2011 3:08 PM by ECBowen

    HD-SDI data rate (uncompressed 10-bit 4:2:2)

    Bill Gehrke Most Valuable Participant

      I have a new BlackMagic HyperDeck Shuttle, the output file format is a MOV file that is uncompressed 1920 x 1080 10-bit 4:2:2.  When I put this output clip into Premiere here is what the properties of the clip show.

       

      Mov-BM-Characteristics.jpg

      What bothers me is the "Average Data Rate" of 90.8 MB/second.  Is this what I should be seeing?  I was under the impression that uncompressed HD 10-bit 4:2:2 required much higher.data rates something like 180 MB/second and thereby necessitating RAID arrays for Premiere.

       

      Does anyone have uncompressed HD data to compare to my results?  The data is complete as I did also record the regular AVCHD data and have both in a timeline

        • 1. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
          Frédéric Segard Level 2

          It is true that you don't have the full rate of uncompressed. Usually uncompressed 4:2:2 10-bit @ 1920x1080 29.97fps is about 160MB/s. I can't really help in figuring out if something is set incorrectly yet, as I'm getting my own system with a Decklink in about 10 days. My current guess is there is a compression setting in the capture process that is incorrect, or Decklink's uncompressed codec is actually losslessly compressed, like Avid's DNxHD and Apple's ProRes 422 HQ. But I've encoded a clip on my current Mac with FCP and got these figures for you to reference.

          1 person found this helpful
          • 2. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
            Bill Gehrke Most Valuable Participant

            Thank you very much Frederic!  I have additional data from Sony that I am doing everything correct on their end,  Still waiting to hear from BlackMagic on comments from them.  This Hyperdeck Shuttle is a very new device (and I am a complete newbie with uncompressed) and since support probably really has to come from Australia the time difference is affecting a response.

            • 3. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
              jkosmicki Level 2

              I don't have the HyperDeck but I use BlackMagic cards with my edit systems and I use BlackMagic uncompressed AVIs a lot. Here are the properties of a 1080p 23.976 10bit clip I have (didn't have a 29.976fps clip):

              datarate.png

              90.8MB/s definitely seems wrong.

              • 4. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                Bill Gehrke Most Valuable Participant

                I just went back to check my Premiere CS5.5 presets and it sure looks correct.  Besides I assume this would not be the cause anyway.

                 

                BM-Presets.jpg

                • 5. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                  jkosmicki Level 2

                  No, I wasn't saying 60i was incorrect. I meant that the data rate of 90.8MB/s looks incorrect compared to my 24p clip at 127MB/s.

                  • 6. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                    ECBowen Most Valuable Participant

                    Please list the cameras or decks model numbers used for the uncompressed capture so I can compare something and see.

                     

                    Eric

                    ADK

                    • 7. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                      Frédéric Segard Level 2

                      124MB/s for 24p is more in-line with the data rate of uncompressed.

                       

                      I just checked the HyperDeck Shuttle's specs and manual, and it does not seem to losslessly compress in any way. Their figures of 12.5 minutes on one128GB SSD falls in-line with true uncompressed.

                       

                      Your clip duration of 18:20 should have an approximate file size of 176GB, not 97.7GB. So obviously, something is not right. Was the file transcoded in anyway? Was it a straight copy?

                      • 8. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                        Bill Gehrke Most Valuable Participant

                        No, I was not implying that I just wanted to make double sure I had chosen the correct settings.

                        • 9. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                          Bill Gehrke Most Valuable Participant

                          Frederic,

                           

                          I only expected 10-12 minutes on that 120 GB SSD.  When it went beyond that I immediately suspected something was wrong.  I have just downloaded upgraded firmware for the Shuttle and I guess I will have to retry my test.  The other possibility is that my SSD is only what I would call a second generation SSD and now days we have third generations with the newer SandForce SF-2281 controllers (and most now a SATA III also).

                           

                          Well I called BM and the screen grabs are on their way and this appears to be a new one for them so it may be a while before we can close this on off.

                          • 10. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                            Bill Gehrke Most Valuable Participant

                            Frederic, to answer you question no that was the raw file I pulled the SSD out and mounted it on my Areca controller in the editing machine.  A straight copy.

                            • 11. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                              Frédéric Segard Level 2

                              Bill,

                               

                              From what I gathered on the BMD documentation, of course tech support will shed some light on the situation, the Shuttle would not properly work if the SSD was not fast enough. Or course, no were does it mention data reduction to compensate for the slower speeds. But I'd expect if you get one of their recommended SSDs, or one that matches in specs, you should have no issues. What SSD model are you using? Have you run a benchmark on it to test the write speeds? Anything above 190 MB/s sustained writes (for non-compressible files) should do the trick. Alas, you also need to check how your SSD handles compressible and non-compressible files. Most disk benchmarking utilities don't benchmark non-compressible files adequately. And cheaper SSDs write half as slow for non-compressible files.

                              • 12. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                                ECBowen Most Valuable Participant

                                Unless Blackmagic included variable bit rate algorithms or different codecs into the shuttle then a disk unable to keep up would show that problem with dropped frames. Bill has not detected any drop frames as of now. Also the benchmark testing Bill did included the Blackmagic Diskspeed test which does use uncompressed files and had a far higher read and write score than 90MB/s. It is unlikely the SSD speed causing this since it would show other anomalies. Now it could be the SandForce controller running the compression anyway on the data addressing to the drive since that is what Sandforce drives do. That would be the only likely reason related to the SSD drive that could cause that without anomalies in the video.

                                 

                                Eric

                                ADK

                                • 13. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                                  Frédéric Segard Level 2

                                  Yeah, I also think it's unlikely that the shuttle has variable bitrate recording capabilities, and even less so doing it automatically according to the installed SSD's speed. If it did, BMD would market it as a "feature" prominently in their brochures. The main reason I suspected it "could" have used a lossless compression was because of the fact it did not drop any frames, and that he could record beyond 12.5 minutes!

                                   

                                  I suspect the speed of the SSD might be an issue while interacting with the Shuttle's overall functionality. But it remains a mystery as to how. Tech support will surely shed some light on the situation. But Eric, I didn't see Bill's benchmark results on this post, how do you know his SSD goes well beyond 90MB/s?

                                   

                                  Plus, SandForce compression does not remotely account in the alteration of the bitrate and duration. He does have 18+ minutes in there, not 12.5 max.

                                  • 14. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                                    ECBowen Most Valuable Participant

                                    http://www.enterprisestorageforum.com/technology/features/article.php/3930601/Real-Time-Da ta-Compressions-Impact-on--SSD-Throughput-Capability-.htm

                                     

                                    " This process takes some time and computational resources prior to writing it to the storage media.

                                    Once the data is compressed, it is placed on the blocks within the SSD. However, things are not quite this simple. To ensure the drive is reporting the correct amount of data stored, presumably the uncompressed data size is also stored in some sort of metadata format on the drive itself (maybe within the compressed data?). This means that if a data request comes into the controller with a request for the number of data blocks or size of a data block, the correct size is reported.

                                    But since the data has been compressed, the amount of data that is written is less than the uncompressed data. Less data is written to the storage media, which means less time is used, which means faster throughput. The amount of time used to write the data is proportional to the size of the compressed data (i.e., the compressibility of the data), which drives the throughput performance. "

                                     

                                    Now my thoughts are what if the Hyperdeck controller is not instructing/reporting to the Sandforce to ammend the actual size to the original size before the compression or if the Sandforce just ignores that part because this is a non standard controller in the Shuttle or atleast the firmware.

                                     

                                    Eric

                                    ADK

                                    • 15. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                                      Frédéric Segard Level 2

                                      Ah, OK, I understand better what you were saying earlier. It explains a lot on how the SSD can efficiently compress and transfer the data, all the while giving more recording time on a single 128GB unit. But compression has to be treated as fast as the data is coming in, otherwise you'd get some sort of buffer (or cache) overflow, probably resulting in dropped frames or some other undesired anomaly. Also, it does not explain the alteration of the content's reduced bitrate at 90MB/s. This is something that can only be done on the Shuttle's side, not the SSD. And ONLY IF the Shuttle's capabilities include variable bitrate and/or codec compression of any kind. And nowhere in the documentation they state that. It would be a nifty optional feature to promote, if you asked me!

                                       

                                      90MB/s is about half of what uncompresssed 4:2:2 10-bit should be. That's a 2:1 ratio. I don't know of any disk compression techniques that can squeeze uncompressible data that much without loss in integrity of the data itself. I would expect that ratio from a visually lossless codec compression.

                                       

                                      Were both theorizing here. But I'd laugh so hard if tech support says it's due to some stupid switch on the side of the Shuttle that switches the thing to some sort of compressed codec!  hehe

                                       

                                      Message was edited by: Frederic Segard

                                      • 16. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                                        ECBowen Most Valuable Participant

                                        Well actually my thoughts included the bit rate as well. if Adobe just read the size of the file and the timecode metadata for length of video then it may give a reduced bitrate reading for that file which would explain the 90MB/s. As you state this is further conjecture but it does make sense.

                                         

                                        I have not opened one I have on the shelf yet to check the unit out but you never know with new hardware what to expect for settings and such.

                                         

                                        Eric

                                        ADK

                                        • 17. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                                          Bill Gehrke Most Valuable Participant

                                          Frederic, Eric was my first consult and I did send him some benchmarks so here they are, they happen to sell the same Mushkin Callisto that I have.

                                           

                                          But first let me tell you the convoluted trail.  BM requires you to format the drive to HFS+.  BlackMagic Disk Speed Test will not run on a PC unless it is a PC format.  I did a secure erase to set the SSD to all zeros, then I formatted to HFS+ and found that their disk test gave me a BSD.  Backing up I then formatted to NTFS and ran the BM DST test.  Now their test never stops running and bounces all over the place on testing without any indication of what is really happening so what you see here is a probable best case screen grab.  I also ran AS SSD Benchmark.  I also tried running off my Intel ICH10R and my Areca 1880

                                           

                                          BMD-DST-good.jpg

                                          I am to embarrassed to show the bad ones.  I have never seen such large excursions, probably due to the unique nature of SSD's.  I will say this above data was not influenced by cache which gave initial readings in the Gigabyte range as evident in the AS SSD Sequential test below.

                                           

                                          AS-SSD-Muskin.jpg

                                          If you believe this benchmark I of course do not have the write speed required but what one do you believe.

                                          • 18. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                                            Frédéric Segard Level 2

                                            Man! If I wasn't laughing so hard, I'd probably have a stupid comment to say about these charts!

                                             

                                            I'm really at a loss. One says you can, and royally so. And the other, translates in wanting to go back home, crying and kicking helplessly and shamelessly under the kitchen table....(oh, wait, I do have a stupid remark to say after all)

                                             

                                            Can you borrow a fast SSD form someone, and see what happens?

                                            • 19. Re: HD-SDI data rate (uncompressed 10-bit 4:2:2)
                                              ECBowen Most Valuable Participant

                                              You might want to look at other results for the AS SSD benchmark and different drives people are posting and compare them before you say that including some Vertex 3 drives Blackmagic recommends.

                                               

                                              Eric

                                              ADK