Skip navigation
Currently Being Moderated

Info on lossy DNG?

Jan 10, 2012 5:25 PM

Tags: #export #dng #lossy

The idea of a (slightly) lossy, but full dynamic range, compressed image format is excellent, and long overdue. It seemed up to now that JPEG XR (previously HD Photo) would be the best choice, if only anyone agreed to use it. I was hoping LR would soon offer import/export of this format. The main advantage of a format such as this is much reduced storage space, full dynamic range preservation for further editing, and minimal loss of spatial detail.

 

It seems the new support for lossy DNG in LR beta 4 is a step in the right direction: I could convert all my raw files to lossy DNG on import, save roughly half the storage space (very important in online backup), and still have close to original fidelity, and full dynamic range, for processing.. And in briefly playing with it, it does look promising. But I can't find any real information on exactly what this format does, and I have roughly 98 questions about it, of which I'll ask only 4 here

 

  1. Is the image data linear / demosaiced, or original sensor format?
  2. Are exported images the edited / developed pixels, or the original pixel values, with the edits saved only in the CRS metadata?
  3. What is the compression algorithm? JPEG (DCT), wavelet, or some other Adobe tech?
  4. What is the quality loss of this compression, and why is there no quality / size selector in the export dialog?

 

Answers or pointers would be highly appreciated.

 
Replies
  • Currently Being Moderated
    Jan 10, 2012 6:03 PM   in reply to Fuchsteufel

    Thank you for your interest.

     

     

    To answer your questions:

     

     

    1. The image data is linear (scene-referred) and demosaiced, in the native color primaries of the sensor (e.g., RGB color filters for most cameras).

     

     

    2. I'm not sure what you mean by "exported images." If your input is a (non-DNG) raw file, and you are converting to a lossy compressed DNG, then the exported result will have linear (scene-referred) demosaiced image values (see #1, above), with image edits stored in metadata. If your input is a lossy compressed DNG, and you are converting to a standard output-referred format like TIFF or JPEG, the resulting pixel data will be output-referred (i.e., reflect whatever tone & color edits you've made). The edits you've made will also be saved as XMP in the metadata (unless you strip the metadata, of course).

     

     

    3. Standard JPEG.

     

     

    4. It is a fixed compression level, roughly equivalent to JPEG level 10. Remember, we are dealing with scene-referred data here (as opposed to output-referred data), so we have to keep the compression quality high enough to withstand the edits that photographers generally apply (e.g., tonal stretching, etc.).

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 10, 2012 8:24 PM   in reply to MadManChan2000

    1. The image data is linear (scene-referred) and demosaiced, in the native color primaries of the sensor (e.g., RGB color filters for most cameras).

     

    Eric,

     

    Umm, demosaiced, so a "linear raw"?

     

    Really, or was that a slight slip of finger on keyboard

     

    Regards,

     

    Sandy

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 10, 2012 8:27 PM   in reply to sandy_mc

    Correct, linear raw.  Effectively same as linear DNG, but lossy JPEG compressed (with appropriate mapping and encoding of the image data so that JPEG compression works effectively on it).

     

    Eric

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 10, 2012 8:45 PM   in reply to MadManChan2000

    Eric,

     

    Ok, thinking about it, I guess you have to demosaic to use a completely vanilla JPEG engine, which I suppose was the design goal. I'd expected something DCT based, but not exactly standard JPEG

     

    It's a bit of a bizare process though - add lots of pixels whose information is actually redundant so you can get fewer bytes at the end. Don't think I'll enabling that option

     

    Thanks,

     

    Sandy

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 11, 2012 12:05 AM   in reply to sandy_mc

    Eric,

     

    Thinking about this some more, what's the sample width here? I'm assuming significantly greater than 8-bits, right, given that it's linear data?

     

    Regards,

     

    Sandy

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 11, 2012 5:44 AM   in reply to Fuchsteufel

    The color components of the stored & compressed image data is 8-bit depth. 

     

    The linear scene-referred data (usually 16 bits after demosaicing) goes through a 1D perceptual mapping, then is quantized to 8 bits, then JPEG compressed.  The DNG spec will not require these steps (in fact the only new part is a new supported Compression tag value, and some additional tags to deal with the downsampled case), but it is what we do to ensure high-quality results.  An opcode (from the existing DNG 1.3 spec) is embedded into the file; this is used to invert the 1D mapping when reading in the image, thereby returning the stored image data to linear scene-referred encoding.

     

    Yes, the demosaic step generally expands 1 color channel to 3.  But, the quantization to 8 bit (after perceptual mapping) and JPEG compression more than makes up for that. 

     

    For example, a 5D Mark II full raw file is ~27 MB with lossless DNG compression (same with .CR2).  Lossy compressed DNG (full-res) is ~13 MB depending on scene content, ISO, etc.

     

    The two other advantages of this setup are:

     

    1. It makes it simpler to define downsampled (i.e., "small raw") versions of the image.  For example, the same 5D Mark II raw file described above can be saved as a 10-megapixel lossy compressed DNG (7 MB) or just screen-sized, like 2048 pixels on the long side (1.5 MB).

     

    2. It makes loading images much faster, primarily because it skips the early-stage raw processing (linearization, demosaic, etc.).

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 11, 2012 5:51 AM   in reply to MadManChan2000

    The linear scene-referred data (usually 16 bits after demosaicing) goes through a 1D perceptual mapping, then is quantized to 8 bits, then JPEG compressed.

    Eric,

     

    Ok, interesting.........

     

    8 bits

     

    Is "1D perceptual mapping" a complicated way of saying an sRGB (or similar) tone curve?

     

    Thanks,

     

    Sandy

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 11, 2012 6:09 AM   in reply to sandy_mc

    Hi Sandy,

     

    Yes, 8 bits.  You may be surprised how far this will take you!  Keep in mind the only reason that normal raw files need to be high bit-depth is because the data is stored with linear encoding.  You can get away with far less bits if you can use a non-linear encoding with appropriate dither.

     

    The way we've been testing this is by taking original raw files, making lossy compressed DNG versions of them, and developing the two pictures in Lr and then opening the results into Photoshop.  We toggle between the two, and try to determine visually which is which.  We do this at multiple visual scales (Fit View, 1:1, 2x, 4x, and even higher).  Then we do a diff between the two images to scrutinize the visual structure of the differences.

     

    I encourage you to give this a try, if you're interested in assessing the visual quality of the results.

     

    As for the mapping, yes it's just a tone curve.  But the tone curve isn't fixed -- it's image-dependent.  This is because we have to deal with the fact that most images don't use the full 12/14/16-bit space of the input data, so we need to stretch out the tones to fill the range (to avoid quantization problems when going down to 8 bits).  The perceptual encoding curve is similar to sRGB.  I think it's closer to L*, actually (need to check).

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 11, 2012 6:31 AM   in reply to MadManChan2000

    Eric,

     

    Actually, I wouldn't be at all surprised at how far 8 bits take you - I've spent lots of time with Leica M8's and M9's, and their compression scheme is 8-bit with a "tone curve" of sorts. Of course, to some extent, what saves the Leica scheme is that it's pre-demosaic, and the demosaic gives you what is in effect a dither. So I take your point on dither as well.

     

    Generally that works well - when you do run into issues, it's not normal images, it's when you're try to do lots of shadow recovery on an underexpsoed image, for example. Or when you correct vignetting.

     

    But returning to the new scheme, I take your point as regards the image dependent curve. But other than that, and call me cynical if you want, this sounds very much like a common or garden JPEG in a DNG wrapper. I can understand that as something that a camera can produce using the on board processor. But surely this isn't the recommended way to convert to DNG from a raw file???? Lossless compression gives similar compression, and well, is lossless

     

    Regards,

     

    Sandy

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 11, 2012 6:46 AM   in reply to sandy_mc

    Hi Sandy,

     

    I forgot about that -- yes, you should indeed already be familiar with the 8 bit result with appropriate encoding of image values.  Your point about the underexposed image is well taken, and that's why we "stretch" the tonal values (on a per-channel basis) to fill the nominal range, before doing the quantization to 8 bits (with dither).  This is just to minimize further problems when trying to do shadow recovery later.

     

    As I mentioned earlier, at a high level all we've really done is add a new supported Compression tag value -- namely, regular JPEG compression. 

     

    The fundamental difference between a lossy compressed raw DNG and a normal JPEG in a DNG wrapper is that the former is scene-referred, whereas the latter is output-referred.  With the lossy compressed DNG (scene-referred), you can do all the normal edits you're accustomed to with raw:  white balance, highlight recovery, changing camera color profiles, noise reduction, sharpening, lens corrections, etc.  With the normal JPEG in DNG wrapper (output-referred), you cannot -- or more accurately, those processes will generally not work as well as they do on raw (e.g., clipped highlights are already gone).

     

    So, a lossy compressed DNG gives you the full editing capability of raw, with a storage footprint that is usually much smaller than lossless compressed raw.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 11, 2012 7:05 AM   in reply to MadManChan2000

    Eric,

     

    Thanks for all the info -  I know this must be a hugely a busy time, so taking the time to explain this is much appreciated.

     

    We are going to have to agree to disagree on to what extent scene-referred vs output-referred is a significant advantage. Other than as regards white balance, it's all just image data, and in this case demosaiced, JPEG compressed, 8-bit data

     

    Regards,

     

    Sandy

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 11, 2012 7:21 AM   in reply to sandy_mc

    Hi Sandy,

     

    Indeed, data is data, but I think the degree to which one can process the data effectively will depend on its state.  Here are some concrete examples where I feel having the scene-referred data makes a big difference:

     

    1. Staying in the original camera space primaries (as opposed to storing data that's already transformed to a canonical space, like Adobe RGB or sRGB, like most camera JPEGs). 

     

    I see two advantages.  One is that since the data is not yet mapped/clipped to a smaller gamut space, the raw converter can then perform smooth gamut mapping (under user control) to determine how saturated colors are handled.  For example, red flowers that are often clipped in camera JPEGs are often not clipped in the original camera primaries (they only become clipped during the raw-to-output gamut mapping process, unless one takes care to avoid the clipping).  A second advantage is that chromatic aberration correction tends to work a lot better.  This is because of the spectral separation between the primaries.  (Once they are "mixed" into sRGB or Adobe RGB primaries, it is much harder to perform the correction.)

     

    2. Changing the white balance.  Yes, one can change the WB for any output-referred image, too, but since the tone curve used to render that data is often unknown, it is much harder to get good WB results without color crosstalk along the tonal range (e.g., light areas may appear neutral, but midtones will then take on a cast, or vice versa).  Furthermore, most output-referred data is already white-balanced, and any channels that become clipped in the process cannot be recovered if one wants to adjust the WB (see previous item on gamut mapping).

     

    3.  Noise reduction.  Of course, one can always do noise reduction on any image data format or type.  However, having (mostly unprocessed) scene-referred data establishes a much better mapping to physical noise models (e.g., how noise varies as a function of exposure), which we use to drive to the noise reduction algorithms.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 12, 2012 3:48 PM   in reply to MadManChan2000

    At some point I started scanning old slides in very high resolution and 16 bit color dept. Unfortunately I discovered that each image was using about 150Mb and thus I soon ran out of storage space. I was thinking about doing noise reduction on the original TIFF images and then converting them to 16-bit JPEG. Unfortunately LR3 had no support for 16-bit JPEG. Would the new lossy DNG format be suitable as a storage format for these images?

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 12, 2012 5:04 PM   in reply to M. Cederberg

    It depends on how important these images are.  For archiving purposes, if you don't want the storage overhead of 16-bit TIFFs, I'd suggest 8-bit TIFF with LZW or ZIP compression.

     

    To be honest, I'm not really comfortable with the idea of lossy compressed DNG for archival storage purposes, but then again, some people like the Canon S-RAW and M-RAW modes.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 13, 2012 12:47 PM   in reply to MadManChan2000

    These images are important - that is why I want to archive them. But in my experience scanned TIFF files do not compress very much. Also, from a quality point of view I would rather store my images in 7200 DPI with lossy compression than in 3600 DPI with lossless. In my tests that gives better results. Thus, I would very much like to see support for a lossy format with 16 bit color depth.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 13, 2012 1:26 PM   in reply to M. Cederberg

    If your TIFF images are already edited there is probably not going to be much benefit to keeping 16 bits.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 13, 2012 1:39 PM   in reply to Fuchsteufel

    Where would you see the advantage with DNG? Your file has the tone and white balance already set, so an 8 bit JPEG would do the same job. It still isn't what you are looking for, I can see that.

    But the choice between the new lossy DNG and JPEG seems to have a very obvious answer.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 13, 2012 1:49 PM   in reply to slarti3

    For folks interesting in converting from JPEG to DNG (i.e., using the DNG as a container/wrapper for their JPEGs), using lossy compressed DNG will be an improvement over the previous offerings.  That's because the DNG file will be approximately the same size as the JPEG (since we can now use JPEG compression instead of lossless compression), and also because the JPEG image data will actually just be copied over verbatim.  (i.e., when building lossy compressed DNGs from jpegs, we do not decompress the jpeg data, then recompress it into the DNG -- that would lose quality).

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 13, 2012 1:55 PM   in reply to Fuchsteufel

    Please forgive me for the bluntness of my question:

    Where is the advantage? Is JPEG not able to store the develop adjustments in the metadata anyway?

    Obviously I'm missing something big..

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 13, 2012 2:19 PM   in reply to slarti3

    Some people like to have all their images with the same file type. Personally, I'd rather be able to tell the difference, although it probably wouldn't make much difference one way or the other.

     

    So maybe we're both missing something big.

     

    Hal

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 13, 2012 2:20 PM   in reply to slarti3

    There are some advantages to using the DNG container, such as the use of checksums to verify image integrity.  Other DNG-specific metadata tags may also be useful for image processing.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 13, 2012 4:03 PM   in reply to MadManChan2000

    Eric,

     

    Thanks. I forgot about the checksums. Those could be worth a lot, indeed. What are the other DNG-specific tags?

     

    Hal

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 13, 2012 4:49 PM   in reply to Hal P Anderson

    DNG supports opcodes that can be used to perform calibration or optical corrections.  Normally you'd want to do this on raw data , but in some cases it can be valuable to do it on output-referred data, too.  (Sometimes raw isn't an option, like if you're scanning film.)

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 10, 2012 3:55 AM   in reply to MadManChan2000

    Hey Eric,

     

    This post ended up circulating around the office yesterday and I have to say I'm really happy with the prospect of a lossy DNG format. Nice work!

     

    I'm most interested in the effects of saving a resized lossy DNG, since it seems Nikon has left off sRAW/mRAW from their latest high-MP cameras. I was wondering whether shrinking a DNG makes LR's noise reduction more or less effective. I imagine you get some benefit due to noise averaging out during the resizing, but I'm concerned that LR's noise reduction algorithms may actually work best on full resolution data. Would it ever make sense to provide the option save the lossy DNG with some noise reduction already applied?

     

    Thanks, Ryan

     
    |
    Mark as:
  • Andrew Rodney
    1,389 posts
    Apr 16, 2009
    Currently Being Moderated
    Feb 10, 2012 6:38 AM   in reply to MadManChan2000

    MadManChan2000 wrote:

     

    1. The image data is linear (scene-referred) and demosaiced, in the native color primaries of the sensor (e.g., RGB color filters for most cameras).

     

    How do you come up with this native color primary information?

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 10, 2012 7:17 AM   in reply to Andrew Rodney

    Andrew, you can think of it as an untagged RGB file.  If someone gives you an untagged RGB image, you could consider that the "original" or "native" RGB primaries of that image.  As long as you don't apply a color transform (matrix, table, etc.) that mixes the color channels, you've preserved those original/native RGB values, right?  That's what we're doing here.

     
    |
    Mark as:
  • Andrew Rodney
    1,389 posts
    Apr 16, 2009
    Currently Being Moderated
    Feb 10, 2012 8:38 AM   in reply to MadManChan2000

    MadManChan2000 wrote:

     

    Andrew, you can think of it as an untagged RGB file.

    Thanks for confirming my initial assumptions about this. So like processing any raw data, the people writing the software have to make some (educated) assumptions about this ‘untagged’ data and move on.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 10, 2012 9:06 AM   in reply to Andrew Rodney

    Andrew Rodney wrote:

     

    Thanks for confirming my initial assumptions about this. So like processing any raw data, the people writing the software have to make some (educated) assumptions about this ‘untagged’ data and move on.

     

    No, why would you need to make assumptions? - the matrixes in the DNG tell you everything you need to know.

     

    Of course, you can override them, but you don't have to.

     

    Sandy

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 10, 2012 9:26 AM   in reply to sandy_mc

    Sandy is right ... I forgot to mention that the DNGs have color profiles embedded, so the DNGs themselves are not actually untagged.  My use of the word "untagged" was meant in terms of the color transform applied (i.e., there isn't any).

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 15, 2012 11:15 AM   in reply to Ryan D Mack

    Hi Ryan,

     

    The noise reduction algorithms in Lr 4 Beta are aware of the relationship between the image size of a downsampled, lossy compressed DNG (e.g., 10 MP) and the full-resolution original (e.g., 36 MP) and can therefore adjust its internal parameters accordingly.  This happens automatically, without any specific action needed from the user.  You can expect noise reduction to be extremely effective on high ISO, downsampled, lossy compressed DNGs.

     

    Hope this helps,

    Eric

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 15, 2012 9:03 PM   in reply to Fuchsteufel

    There is almost merit in using it.

    I'm still just going to buy additional harddrives and keep my images lossless, but man a load of work has gone into providing the lossy DNG's

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 15, 2012 9:59 PM   in reply to MadManChan2000

    I accidentally run the DNG lossy conversion twice over the same image..  Does Lightroom make another pass or does it ignore the file? if it does how damaging can it be?

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 16, 2012 6:58 AM   in reply to Simon Full

    Hi Simon,

     

    If you are converting a lossy compressed DNG to another lossy compressed DNG of the same image dimensions (i.e., no downsampling), then there is no loss of quality.  This is because we recognize this special case and simply copy over the compressed data stream as-is (verbatim).  Thus, we do not decompress & recompress, which would lose quality. 

     

    Of course, if you are generating a lossy compressed DNG of another image size (e.g., going from 10 MP to 4 MP), then of course the image characteristics will change because the image is getting smaller ...  

     

    Eric

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 16, 2012 9:26 AM   in reply to MadManChan2000

    Thanks Eric, really like this whole lossy idea.

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (2)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points