10 Replies Latest reply: Aug 16, 2012 6:48 AM by trshaner RSS

    HotPixels show up after several LR processing steps

    starrysky01

      Dear all,

       

      I am having two problems with Nikon Raw Processing and LR4.1.

       

      1st: I am producing and processing content for ESO (European South Observatory), especially TimeLapses. Those sequences consist of hundreds of Images in hundreds of folders. Source files are either from D3s, D4, D700, D7000 or the like.

       

      Several development changes are applied to single key images in a sequence, and those later applied to all images in that sequence. Problem is that after a certain amount of development changes (that are written in to the XMP sidecar files with LR4), hot pixels start to show up in the images that were not visible before.

       

      Seems like the oversaturated pixel information (Hot Pixel) that is always stored within the RAW images, is lost after a certain amount of development changes.

       

      2nd: is there a known trick how to force LR4 into updating ALL thumbnails after development changes, including those, that are outside the currently visible grid view?

       

      It is very time consuming to manually page up and down the grid to have LR4 updating the thumbnails on each page (of for example a folder of 1200 images) so that one gets a preview of the TL sequence with the current development changes in Library View.

       

      Thanks for any help

      Chris

       

      Bildschirmfoto 2012-08-10 um 11.57.59.png

        • 1. Re: HotPixels show up after several LR processing steps
          -Agfaclack- Community Member

          what do you mean with HOTPIXELS?

          are you sure it´s not the highlight clipping warning?

           

          can you show a screenshot of s single image with hotpixel visible?

           

           

          ps: in the 90s i build some stuff for your observatory. 

          • 2. Re: HotPixels show up after several LR processing steps
            starrysky01 Community Member

            Hi,

             

            no screenshot needed... With "hot pixels" I meant noise on chip during long exposures. oversaturated pixels, either yellow, white, green, blue etc etc. usually information about hot pixels is stored within the RAW file (as they are easily recognizable), but it seems that LR4 seems to loose that information after some processing runs.

             

            about the stuff: cool. did you visit them too when they incorporated them into the designs? if so? where have you been at? so far I've been to ALMA and Paranal but not La Silla.

             

            Here's a film from one of these trips: http://newswatch.nationalgeographic.com/2012/02/21/new-time-lapse-gives-rare-glimpse-at-at acamas-starry-nights/

             

            all the best

            chris

            • 3. Re: HotPixels show up after several LR processing steps
              -Agfaclack- Community Member

              yeah i know what hotpixels are.

              but i never noticed that they appear or disappear after editing in LR.

               

              but then i don´t have seen much of them.

               

               

              starrysky01 wrote:

               

               

               

              about the stuff: cool. did you visit them too when they incorporated them into the designs? if so? where have you been at? so far I've been to ALMA and Paranal but not La Silla.

               

               

              nope unfortunately not.

              i worked for the gauß observatory in göttingen (germany) and we build some meassuring equipment for la silla and the observatory del teide on teneriffa.

              • 4. Re: HotPixels show up after several LR processing steps
                -Agfaclack- Community Member

                i see you used the panogear panoramic head.. what do you think about it?

                • 5. Re: HotPixels show up after several LR processing steps
                  starrysky01 Community Member

                  For that production and others I used two Merlins with the additional MX2 controller of DynamicPerception. Basically they are sold as Merlin/Orion Heads by Skywatcher with GOTO controllers, and the MX2's add more functionality.

                   

                  Panogear on the other hand buys them OEM and brands them with the Panogear Controller, which is really a neat device for VR's (the Panogear Controller is BTW built by a German company ).

                   

                  However, the Panogear/Merlin/Orion Heads have two problems: Desert Dust can cause trouble on the Gears and they are low temp sensitive.

                   

                  And while DP's MX2 controller is an aweseome device, it is open source and can have bugs, also the cables they use are very sensitive to break etc. More than once the Merlins did not move at all, ruining shots, or moved choppy or started to move into a totally different direction. If they work they are nice, and with the MX2 you can even do 2-axis shots.

                   

                  However, these days I use them for VR's mostly, and for pans have changed to Vixen's new "Polarie". Great device for panning

                   

                  Here's the video and you will know some of those fill-in scenes: http://www.vixen.co.jp/en/lp/polarie_movie.html

                   

                  Cheers

                  Chris

                  • 6. Re: HotPixels show up after several LR processing steps
                    trshaner MVP

                    starrysky01 wrote:

                     

                    Hi,

                     

                    no screenshot needed... With "hot pixels" I meant noise on chip during long exposures. oversaturated pixels, either yellow, white, green, blue etc etc. usually information about hot pixels is stored within the RAW file (as they are easily recognizable), but it seems that LR4 seems to loose that information after some processing runs.

                    It's my understanding that only "manually mapped" hot/dead pixel information is contained in the raw file and this is usually done when the camera is manufactured or at a factory depot.

                     

                    Did you try increasing LR's Color and/or Luminance noise reduction settings? Astrophotography is a "worst case" scenario for making hot pixels visible (i.e. long exposure + high ISO + black background). Every digital camera has them, it's just a question of the ISO and exposure time setting required to make them visible.

                     

                    If you are still seeing a few hot pixels that don't respond to LR's noise reduction controls, try removing them at pixel level  zoom (8:1) using the adjustment brush at +100 Noise, Feather = 0, Flow = 100, Density = 100, and then 'Sync' the Local Adjustments> Brush settings across like images (i.e. same shoot and camera).

                    • 7. Re: HotPixels show up after several LR processing steps
                      starrysky01 Community Member

                      Hi Trshaner,

                       

                      yes, to all your suggestions. Tried them alreadym and it's in our workflow.

                       

                      With Astro-TimeLapse and/or panning motions it is even more complex. You immediately see the static hot pixels during a timelapse motion on a moving landscape.

                       

                      First you cannot use the adjustment brush on stars - it looks odd when they move trough and brush areas fill and empty with other stars during the Timelapse Motion. Second once you pan or have a dolly movement some brush areas "move" trough trees or other parts of the landscape. it looks very ugly. what you can do is to stop brushing the hot pixels there for the the couple of images, or do all by hand means adjusting the healing brush location outside the tree or other part of the landscape moving by.

                       

                      enclosed are some screenshots with brushed hot pixels, and the start of a pan and the end - the timelapse motion in this instance goes over 344 frames (not that much, but enough to work on for a couple hours).

                       

                      remains my question:

                       

                      I don't see those hot pixels during the first processing steps which are all synced over all images of a set (lighten darks, color correction, gradation curve etc). at some point during processing and developing they start to show up. And I can't remember that this was the case with LR3.

                       

                      any idea on how to force LR4 to update ALL previews also those outside the grid are still very appreciated.

                       

                      this here illustrates my problem:

                      (watch it until the end how it should look like on preview)

                       

                      https://vimeo.com/43257787

                       

                      pw: starry_lr4

                       

                      it got a bit better now with a current MBP 15" 2.6 Ghz Core i7 and SSD's and Mountain Lion (Lion was quite slower), but still previews outside the grid do not get updated...

                       

                      all the best

                      c

                       

                      illustration:

                      begin of panoramic timelapse move

                      Bildschirmfoto 2012-08-16 um 00.20.11.jpg

                      somewhere in the middle

                      Bildschirmfoto 2012-08-16 um 00.20.34.jpg

                      near the end

                      Bildschirmfoto 2012-08-16 um 00.18.43.jpg

                      hot pixel detail of last frame without heal brush

                      Bildschirmfoto 2012-08-16 um 00.19.57.jpg

                      • 8. Re: HotPixels show up after several LR processing steps
                        trshaner MVP

                        starrysky01 wrote:

                         

                        Hi Trshaner,

                         

                        yes, to all your suggestions. Tried them alreadym and it's in our workflow.

                         

                        With Astro-TimeLapse and/or panning motions it is even more complex. You immediately see the static hot pixels during a timelapse motion on a moving landscape.

                         

                        First you cannot use the adjustment brush on stars - it looks odd when they move trough and brush areas fill and empty with other stars during the Timelapse Motion. Second once you pan or have a dolly movement some brush areas "move" trough trees or other parts of the landscape. it looks very ugly. what you can do is to stop brushing the hot pixels there for the the couple of images, or do all by hand means adjusting the healing brush location outside the tree or other part of the landscape moving by.

                        OK, I see how masking out the hot pixels just moves the problem. You really need a way to detect them and remove them on each image by sampling nearest neighbors and replacing the data values. I am not sure why more hot pixels seem to crop up after making a few develop adjustments. If you can provide an image with XMP file (or DNG) for download I'll be happy to see if I can "reverse engineer" the cause.

                         

                        During capture production don't use 'live view' mode on the camera. It heats up the image sensor and will greatly increase the hot pixel levels. What is the typical per image exposure time and ISO setting?

                         

                         

                        starrysky01 wrote:

                         

                        Hi Trshaner,

                         

                        any idea on how to force LR4 to update ALL previews also those outside the grid are still very appreciated.

                         

                        this here illustrates my problem:

                        (watch it until the end how it should look like on preview)

                         

                        https://vimeo.com/43257787

                         

                        The only way I know how to rebuild previews in the Library module is to first select the images, and then go to toolbar Library> Previews> Render 1:1 Previews.

                        • 9. Re: HotPixels show up after several LR processing steps
                          starrysky01 Community Member

                          Hi,

                           

                          send me a PM or a note via Vimeo and I can forward you some sample images after our short holiday.

                           

                          We only use Live View once to focus on bright stars using LV maginfication at the begin of the set / night, afterwards all lenses focus rings are fixed for the rest of night with duct tape (if there are no huge temperature differences that would need a re-focusing or a different angle via zoom). After that initial focus setting, LV is off and cameras just run on a preset Interval (however it is interesting that low light specialist cameras like D4 actually are able to show brighter stars / constellations in LiveView, which is amazing).

                           

                          ISO settings are between 2500 and 6400, exposure times vary between 15 to 30 secs (depends if there is some moonlight or completely transparent starry sky like in Chile or African / Asian deserts).

                           

                          Noise depends on outside temperatures on location, it makes a difference if you capture images at a desert or during a low temp winter night.

                           

                          Regarding the "grid view update", to my experience it makes no difference if one sets "minimal" or "render 1:1 previews". Before LR I used Bridge a lot on XMP processing, and the way Bridge works on it's previews is much better. You get a clear information on what it does, at what preview out of X it currently is working and it processes ALL previews no matter if they are outside the current view. That is something which is needed urgently in LR4's next update.

                           

                          Hopefully Adobe reads this. We had contacted Adobe Europe about this, but got never much response.

                           

                          Cheers

                          C

                          • 10. Re: HotPixels show up after several LR processing steps
                            trshaner MVP

                            Some Adobe employees do monitor the posts here, tbut he best way to get Adobe's attention is to post the Library previews issue at the Photoshop Feature Request site:

                             

                            http://feedback.photoshop.com/photoshop_family

                             

                            I'll send you a PM concerning the test files and see if I come come up with some other suggestions.