31 Replies Latest reply: Nov 28, 2013 3:06 PM by Trevor.Dennis RSS

    What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama

    Noel Carboni Community Member

      Just did a Photomerge of 25 shots...

       

      Note that over 100 GB (yes, you read that right, GIGAbytes) of scratch disk was required, not to mention easily filling my 16 GB RAM.

       

      BigPanoResources.jpg

       

      -Noel

        • 1. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
          Noel Carboni Community Member

          Sigh.  I merged the layers into one, and cropped most of the transparency off the edges.  Then I selected the remaining small blocks of transparency and expanded the selection 4 pixels, and called up Edit - Fill - Content Aware. 

           

          By that time Photoshop had reduced its memory usage down to about 6 GB, and on starting the CAF operation it started marching up to max again.  It was really chewing up resources for a while, but now it's been sitting at 15 GB or so used with what looks like just one core in a tight loop.  I don't think it's going to complete, as I'm just not seeing any I/O or other activity from the Photoshop.exe process.  The progress bar shows the Content Aware Fill got about 80% of the way done before going into this loop.  The scratch files still total 100+ GB, though they haven't been modified for a few hours.

           

          CAFNearlyComplete.jpg

           

          PSLooping.jpg

           

          I have other things to do, and I'm going to just let this sit.  I'll see if it ever finishes...

           

          Chris/Adobe, you folks REALLY need to work on making your Gee Whiz features more robust in the next version release.

           

          -Noel

          • 2. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
            Trevor.Dennis CommunityMVP

            Goodness.  What sort of job need so many pixels?  The bigest image size I have ever done was for one of those revolving signs at Nelson Airport, and that was only 1.5Gb on the computer

             

            http://farm4.staticflickr.com/3515/4018990152_ffe69899d0_z.jpg?zz=1

             

            Unless you absolutely need a 21 foot wide print at 300dpi, then wouldn't it be better to downsize the files before merging them?

            • 3. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
              Noel Carboni Community Member

              Yes, exactly.  It was an experiment.  Mostly I was trying to determine the amount of scratch space Photomerge needs (which is huge), but I figured I'd finish it at the size shot.  You're absolutely right that outside of it being a curiosity no pano really needs to be that large.

               

              -Noel

              • 4. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                Trevor.Dennis CommunityMVP

                John Doogan is an Adobe Ambasador in New Zealand, and I attended a one day seminar he ran in the early days of CS4.  During this he stitched a three row, by four (or five, I can't remember) photomerge on an Apple laptop in real time, and it did not seem to take any time at all!   IIRC this was with Canon 1DsMK2 files.   I can't hot link the image, but this is it.   I was pretty impressed.  That was either 12 or 15 16Mb files on a tiny laptop.

                • 5. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                  PECourtejoie CommunityMVP

                  Trevor, maybe did John Doogan wisely (for a presentation) choose to make a panorama with 8 bit files... 16 bits ones are huge when layered...

                  Noel, I guess that on a non-test operation, you would do a CAF side by side, or with areas of similar colors, rather than on the whole file?

                  • 6. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                    Trevor.Dennis CommunityMVP

                    One of the things I took away from that John Doogan seminar was that he told us he never opened RAW files as 16bit unless he had a problem with 8bit - like banding in subtle tonal gradients in skies etc.   That changed my attitude - the guy is an Adobe Ambassador after all, and this was back in the days I was using 32bit Windows XP with Gb RAM installed but not accessible.

                    • 7. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                      D Fosse Community Member

                      This is not directly comparable, because I started with resampling an existing flat psd up to 75678 x 4576. Then I punched five 1000 x 1000 px holes in it, and called up content aware fill. Didn't have to say much; it knew what it had to do.

                       

                      After 8 minutes, there was a faint smell of burning plastic.

                       

                      After 10 minutes, the computer case started to take on a deep amber hue. Very nice color BTW.

                       

                      After 14 minutes the desk started to melt, so I had to prop the thing up with some empty metal trays that I have been collecting blown lightbulbs in. What a mess that was.

                       

                      Now it's 18 minutes, and I'm not quite sure what to do next. Is there an "abort" button somewhere? Can this thing be remote controlled? Because I really think I should leave the building soon...

                      • 8. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                        D Fosse Community Member

                         

                         

                        (actually I killed it in Task Manager after 24 minutes. It was clearly stuck around the 80% mark. But it was fun trying...)

                        • 9. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                          Noel Carboni Community Member

                          Thanks for the feedback, guys.  And for the smiles, Dag.  I found it alarming when my system sped up its fans to provide additional cooling, even though they didn't nearly reach "airport takeoff" level.  Hardly anything stresses it to do that.

                           

                          Actually, I do prefer to work with 16 bits/channel files because that gives me plenty of pixel accuracy to do further edits, but indeed I would normally convert to smaller sizes before doing such a pano.  It turned out that I was thinking of doing so, but simply forgot and so turned the task into an experiment to see if it would succeed.  I was impressed that it did, actually, make a very nice stitching job at the full size, even though the image is substantially water, which traditionally doesn't stitch well.  Kudos to the Photomerge developers.

                           

                          I think I'll retry it at, say, half the size now, just for fun.

                           

                          -Noel

                          • 10. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                            Noel Carboni Community Member

                            OK, good results all around starting over with my Canon EOS-40D raw files:

                             

                            Phase 1, Photomerge

                             

                            • Conversion of 25 raw files to (downsized resolution of) 3072 x 2048 x 16 bits/channel PSD files: 62 seconds.
                            • Photomerge of these 25 files into 37841 x 2291 pixel pano181 seconds.

                             

                            Memory usage only went up to 11.9 GB, scratch disk usage was 33 GB.

                             

                            Phase 2. Image Finishing:

                             

                            • Merge Visible was instantaneous, no delay.
                            • Save of 1.3 GB .PSB file (with Flate tweak to Photoshop to circumvent compression):  7 seconds
                            • Edit - Puppet Warp to remove a slight bow in the horizon:  Interactive, no delay, then the transform took 10 seconds to complete after [OK].
                            • Rough crop with Crop Tool:  Interactive.
                            • Selection of transparent edge remnants and expansion of the selection:  Interactive.
                            • Edit - Fill - Content Aware succeeded in 64 seconds (and did a NICE job).
                            • Image - Adjust - Shadows/Highlights:  Interactive.
                            • Save again into PSB file:  5 seconds.
                            • Filter - Sharpen - Smart Sharpen: 14 seconds to complete after [OK].
                            • Image - Image Size, downsize to 30000 pixels in the long dimension:  2 seconds.
                            • File - Save As - JPEG, 5 seconds.

                             

                             

                            Here's the (large) JPEG result, if you're curious to see it:

                             

                            http://Noel.ProDigitalSoftware.com/ForumPosts/JacksonvilleRiverPano.jpg

                             

                            Moral of the story:  Work within your RAM capabilities!

                             

                            -Noel

                            • 11. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                              Rick McCleary Community Member

                              Last year, I completed a project for the board room of a new corprate building. The architect wanted a 60' wall covered with a hi-res panorama image of a landscape. Trim size: 60' x 8'. (Yes, that's sixty feet wide!!) Because it's an indoor piece, we had to use enough output resolution for it to be viewed up close - 12" away. After some testing, final decision was to create the file at 150ppi resulting in the final file dimension of: 108,000 pixels x 14,400 px.

                               

                              The image was a stitched panorama of 81 files from a Canon 5Dmkii - 3 rows of 27 images each. Camera was mounted on a Really Right Stuff pano rig. RAW files were processed in Camera Raw and output as Level 10 JPEGs.

                               

                              Photoshop was absolutely not up to the task of stitching that many images together. I used AutoPanoPro. The actual rendering of the file took about an hour, but came out almost perfectly. There were a few (less than five) stitching errors to correct by hand in Photoshop. Layered file in PS was about 65Gb. Flattened file that was sent for output was about 6Gb. Output was done by a nicely color managed billboard company.

                               

                              Hardware setup is simple:

                              MacPro 3.1 (8-core 2.8GHz)

                              16Gb RAM

                              300Gb dedicated scratch disk

                               

                              It was a wild project. Client is tickled pink.

                              • 12. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                Noel Carboni Community Member

                                Wow, a 1.5 gigapixel image.  Sounds like a great result, Rick.  Nice work!  I'd love to see something like that in person.

                                 

                                Rick McCleary wrote:

                                 


                                108,000 pixels x 14,400 px
                                Photoshop was absolutely not up to the task of stitching that many images together

                                 

                                Based on my experience above (with the document format I used) it's clear that Photoshop "goes virtual" on a 16 GB system somewhere above 80 MP and below 320 MP, so it's not surprising that you found the above to be true.  It's not a miser with resources.

                                 

                                I imagine on a system with a LOT more RAM it would probably breeze through a much larger image, as mine did when I lowered the size.  Note to self:  Put 48 GB or more in next workstation. 

                                 

                                -Noel

                                • 13. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                  PECourtejoie CommunityMVP

                                  Rick, were the files 8 or 16 bits? And how much RAM was allocated to Ps?

                                  Was it printed in one piece? on what printer? Trough a RIP, I guess?

                                  • 14. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                    PECourtejoie CommunityMVP

                                    Noel, please do not look here: http://macperformanceguide.com/blog/2012/20120122_2-diglloydHuge-MacPro.html (I think that the WIF on 16 Gig memory sticks is very low)

                                    • 15. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                      Noel Carboni Community Member

                                      Why not?    Apple makes great workstations.  If I was an OSX man I'd definitely have a Mac Pro.  As it is, Dell makes workstations on the PC side that are every bit as good, and so I don't have to switch OS affiliations just yet. 

                                       

                                      But if Lloyd is using 48GB I guess I'll have to get more than that. 

                                       

                                      -Noel

                                      • 16. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                        Rick McCleary Community Member

                                        8 or 16 bits?

                                        8 bits.

                                        And how much RAM was allocated to Ps?

                                        80% (11Gb). This seems to be critical, i.e., not allocating too much to PS - that would starve the OS.

                                        Was it printed in one piece? on what printer? Trough a RIP, I guess?

                                        Not sure what brand - I think Epson. In any case, it was a 60" ink jet. Printed on outdoor vinyl by a company that outputs large graphics for 40' trucks and 45' billboards. Definitely through a RIP.

                                         

                                        FYI - Here's a short video that gives you an idea of what the piece looks like:

                                        http://vimeo.com/33180956

                                        • 17. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                          PECourtejoie CommunityMVP

                                          I was teasing you, not on the Mac platform at all, but on the amount of memory he has... I hope he creates panoramas everyday to justify the expense...

                                          • 18. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                            PECourtejoie CommunityMVP

                                            Rick, I don't remember... some plug-ins are using Photoshop's memory, others not (Camera Raw?), So I don't know how much memory should have been allocated in your case. (even if you got it working through a dedicated program)

                                            If they run outside of memory, how much RAM can they use... could plug-ins allocate more than 4gigs if they run outside of Ps's memory?

                                            Chris Cox, could you state if that is correct?

                                            • 19. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                              Noel Carboni Community Member

                                              You've done a good job of it, too.    Now let's see, I'm up to 96 GB RAM, RAID array of SSDs (probably at least 4 x 512 GB)...  Hudechrome has me up to 2 x hex core Xeons...  Shtarkel and D.Fosse have me up to 2 x Dell U3011s...

                                               

                                              Where's that lotto ticket...  I just know I must have won...

                                               

                                              -Noel

                                              • 20. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                PECourtejoie CommunityMVP

                                                Noel, such a luxury system needs to be breaked in. Send it to me, so that I'll be able to make sure it works correctly. Thourought tests, no more than 6 months, ok?

                                                • 21. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                  Noel Carboni Community Member

                                                  Sure, then based on your report and more recent technology I can then move up to a really powerful system and you can just keep it.

                                                   

                                                  Luxury, eh?  Fits right in with Canon 'L' lenses.

                                                   

                                                  -Noel

                                                  • 23. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                    psb Community Member

                                                    is there a calculation method to determine the size of a scratch disc, if you would have for example a

                                                    300dpi rgb-image, 10.000 x 10.000 pixels and 10 to 50 layers (also in 16bit) with 6 history steps?

                                                     

                                                    i am currently working with a external dedicated scratch disc for photoshop cs6, USB3.0, 512 GB SSD, always empty, (390MB/sec read, 300MB/sec write)

                                                    and i ask myself if a 250GB SSD would do it, too..

                                                     

                                                    thanks

                                                    • 24. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                      JJMack CommunityMVP

                                                      I would think you would be best off to make the ssd your primary Photoshop scratch disk and add your external disk as a second scratch disk in case the ssd fills.  Myself I keep All my images on an external USB3 disk use my ssd as Photoshop first scratch disk and my internal hard as it second scratch disk.  The files Photoshop writes to scratch disk are temp files when you close Photoshop they are deleted.  SSD scratch disk are best for there is no head seeking and rotational latency. Though users make disk raid 0 using SSD I do not know if there is any performance gain for with SSD there are no arms heads and rotating platters.

                                                      • 25. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                        Noel Carboni Community Member

                                                        Since having set up the above test, I've a more powerful system, with 4 SSDs wired together into a nearly 2 TB RAID 0 array that serves all functions in the computer save for backup storage and storage of very low access files.  Everything's on it - system, applications, photo data, swap, Photoshop scratch...  And everything runs super fast.  Makes sense - everything uses the 4 x SATA III connections simultaneously.  RAID is an underappreciated configuration.  I also now have quite a lot of RAM and normally configure for 200 history steps..

                                                         

                                                        To more or less directly answer your question, psb, multiply height x width x data depth x 3 or 4 (RGB or RGB plus transparency) x layers x history steps.

                                                         

                                                        RAM is good, more RAM is better.  When the RAM runs out, Photoshop scratch space is used.

                                                         

                                                        -Noel

                                                        • 26. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                          JJMack CommunityMVP

                                                          I do not know when or what Photoshop writes to scratch disk space. I have 40GB of ram on my machine and Photoshop it configured so it can use something like 38GB of ram.

                                                           

                                                          I have observed when running Photoshop Ram usage at like 6 GB  where 4GB was in use before Photoshop started and at the same time have seen Photoshop scratch files in excess of 6GB with 32GB of RAM available for use of which Photoshop is allowed to us 30GB of that. Also when Photoshop does allocate RAM it will not return it for use till Photoshop end execution.  Photoshop manages the ram it allocates.

                                                          • 27. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                            Noel Carboni Community Member

                                                            I've seen it allocate scratch files but not write anything to them.  Maybe it just prepares ahead a good bit.

                                                             

                                                            -Noel

                                                            • 28. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                              psb Community Member

                                                              thanks for this noel: "To more or less directly answer your question, psb, multiply height x width x data depth x 3 or 4 (RGB or RGB plus transparency) x layers x history steps."

                                                               

                                                              the number i get in my calc. is "864.000.000.000"

                                                               

                                                              i agree. i see the read/write light blinking of my scratch disc ssd (external), but the ram is not filled at all. (only 30% filled)

                                                              efficency is at 100% all the time. so noel is right, it seems to just prepare. more in depth details could only give photoshop developers,

                                                              i would be very interested in those deep technical aspects.

                                                               

                                                              @ jjmack: i have a pcie-ssd in my imac, which runs at 750/500 MB/sec read/write, i decided to use this internal ssd as my boot volume and work volume, where my few photoshop working files are temporarily and not as scratch disc, as i read, you could better use seperate discs for each purpose, like noel does. the external usb3 ssd 512GB is currently my only scratch disc (always empty of course), and i am constantly watching the istatmenus, if cs6 writes or reads onto the scratch disc.

                                                               

                                                              @noel: very interesting you did a 4 x raid0 with ssd´s over thunderbolt. i imagined same way, but the "pegasus j4" is too pricey in the moment for me. (regarding the 4 ssds at least)

                                                              may i ask, of what purpose is your internal ssd in your mac (or pc?) when you use the ssd raid0 array for all important purposes regarding photoshop?

                                                              • 29. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                                JJMack CommunityMVP

                                                                Nole use raid 0 he has one logical drive where data is striped across several ssd. He has one huge boot disk and everything goes there except backup.

                                                                • 30. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                                  Noel Carboni Community Member

                                                                  Not Thunderbolt, but SATA III.  I have a PC workstation, and bought a PCIe RAID controller card from HighPoint, a 2720SGL model.  I use only 4 of the 8 connections.  This setup runs Windows and Photoshop and everything like a dream.  Everything that does I/O access is just instantaneous, and it just doesn't load up - I can run any number of things.  It's blessedly quiet.

                                                                   

                                                                  Latest performance tests...

                                                                   

                                                                  ATTO 11-17-2013 Windows 8.1.png

                                                                   

                                                                  DiskSpeedGraph.png

                                                                   

                                                                  This second benchmark is a "real world" test with reading/writing and random access occurring.

                                                                   

                                                                  -Noel

                                                                  • 31. Re: What Resources It Takes to Stitch 75678 x 4576 Pixel Panorama
                                                                    Trevor.Dennis CommunityMVP

                                                                    Some of the raid controllers guys like Harm on the Prem Pro Hardware forum use, cost more than some folk pay for a decent computer, so it is reasuring to hear you are doing it OK with such an affordable card.  It is also reasuring that you are apparently getting that performance without too much noise.  Even with MSI Afterburner doing its best to keep the fans under control, my big system box is definitely not quiet!  :-(