Lossless compression vs High Efficiency*

[bjanes]

I’m finding the z9 files have a lot of noise. I have started using lossless compressed and comparing them in both NX studio and Lightroom (as I believe Lightroom only supports lossless compressed) and found things slightly better but I still need to do more testing to convince my self I’m really not convinced that a z9 is any kind of improvement over a d850/500 in the autofocus department or file quality and there’s no real weight saving either. It seems we have come full circle

I have both the D850 and the Z9. Autofocus with the Z9 is much better than with the D850. LR does offer support for HE NEF and HE NEF*, but some say the support is not optimized. However, it works well for me with Windows. With respect to file quality, the D850 may be slightly better with dynamic range. Noise is about the same. See Bill Claff Photons to Photos. The Z9 has IBIS (in body image stabilization) that complements VR with the S line lenses and IBIS can be used with lenses lacking NR. The Z9 has live view histograms and many other usability features over the D850.

Bill

 

[BillW]

Bill,

This is excellent information. Walter on Nikon Cafe has posted a good update on what software can work with HE NEF.



Apparently ACR/LR are not optimized for HE*. I don't know what this means, but ACR/LR works fine with HE* on my windows machine. Perhaps this NEF HE* should be regarded as a beta release. If you are shooting in bursts, HE* enables many more files to be stored in the buffer. According to the Z9 reference manual Lossless NEF files are 55.1 MB and the buffer can hold 79 frames. This should be sufficient for most purposes. However HE* files are 33 MB and the buffer can hold 685 images.

Bill

Thanks for the link. I wonder how one knows whether Abobe has optimized for HE* or not. It would be nice to see what this conclusion is based on. Has Adobe said this? I’m not necessarily doubting it — I have no information one way or the other, beyond what I cited above.

It’s also interesting to think what might be the best way to test quality differences between lossless compressed and HE*. I suppose you can take the same shot in lossless compressed and HE* and then convert both in Nikon Studio NX and also convert both in Adobe ACR. I’m normally not much of a tester - beyond shooting what I like to shoot and seeing how things turn out. I’m interested to know what I might be losing by using HE* and what the circumstances are where that might be most apparent/important.

Since I got my Z9 in early January, I have shot several days of fishing bald eagles along the Mississippi (all Z9) and a week’s worth of shooting in Yellowstone (split between my Z9 and Z7II). And used the Z9 around the neighborhood on squirrels, songbirds and the like. Almost all the Z9 shots were taken in HE* and I have been pleased with them as converted in ACR and then further processed in PS with Topaz AI and Nik plug-ins.

 

[Nimi]

At least one feature is not supported by LRC on my 2019 Intel MacBook Pro: tethering.

 

[Pam Jensen]

As Steve said above (and you should listen to THAT Steve ) I believe LR and PS are only fully working with lossless comp. I also note that ON1 denoise tells me that my z9 Raw files are jpg so I don't know if it's converting them to jpg to work on them as it doesn't recognize the Z9 files yet or if is just being confused by the new file? (You can see my level of technical understanding in that description).

Thank you, I have not had this same experience. I shot HE* and experienced no issues in either photo mechanic or light room. PHotoMechanic had to make the files DNG. However, this is probably not a Z9 issue.
I’m on an older MAC. I have only done two shoots, not enough to be confident all is good tho and I do listen to Steve.

 

[bleirer]

The adobe info linked above says they have camera matching profiles for both, but an easy test would be to import the same photo, one with use camera settings set in preferences and anorher with use camera settings turned off in preferences. See if the sliders are different in each case.

 

[bjanes]

I just received my Z9! I am requesting feedback on real world experience on tradeoffs of lossless compression versus high efficiency*. And yes, I have read what little Nikon has to say about it.

Has anyone done objective comparisons on image quality differences?
What about software support? (I use Lightroom, Photoshop, and Topaz DeNoise, Sharpen Ai, and Gigapixel.)
Are there any post-processing performance caveats with using the compressed HE*?
Anything else to consider?

I did a preliminary evaluation of noise at base ISO comparing Z9 lossless compressed, Z9 HE*, and the Nikon D850. I posted the results here in another thread. Unfortunately, the link leads to the start of the thread rather than my post do please scroll down to my post.

Bill

 

[FB101]

I'm way ahead with LR (on Mac) by being part of the Pre-release program and I haven't seen anything about further Z9 support or updated support for HE*. My understanding is at least on the Mac, LR is processing the HE files with full support...they ain't going to make magic happen later on.

During my Z9 days I shot everything in HE* but never really compared to Lossless. On Sony I shoot Lossy Compressed, on Canon I shoot CRAW. I really have no issues (or maybe just turn a blind eye) with these Lossy RAW formats. I like full FPS, deeper buffers and more shots per card!!

I have had the dreaded lossy compression artifact appear once with the A1 and honestly, I shouldn't have taken the shot in the first place
I was following an osprey in flight and it flew right across the sun - the shots before the ones that were 100% overexposed, showed massive artifacts in the highlights around the bird - those highlights were not "blown" (next few frames, right into the sun were blown) but they had that weird banding all around the bird in the areas of massive contrast.

That's the one and only time I ever saw it and like I said, those shots were destined for the trash bin no matter what.

 

[DougGood]

I did a preliminary evaluation of noise at base ISO comparing Z9 lossless compressed, Z9 HE*, and the Nikon D850. I posted the results here in another thread. Unfortunately, the link leads to the start of the thread rather than my post do please scroll down to my post.

Bill

Thank you, Bill! Very informative.

 

[BillW]

I did a preliminary evaluation of noise at base ISO comparing Z9 lossless compressed, Z9 HE*, and the Nikon D850. I posted the results here in another thread. Unfortunately, the link leads to the start of the thread rather than my post do please scroll down to my post.

Bill

Thanks Bill. Looks interesting. One question. What is RawDigger measuring that it reports a mean value for? I take it this is a measurement of noise? Or of some item that correlates with noise?

I see that high efficiency raw* is the highest of the 3. Do you think that difference is enough to be visible in photos?

How do you think this would change, if at all, in absolute or relative terms, at higher ISOs? I recall that Thom Hogan said with a D850 that once you got above a certain ISO (I don’t recall now whether it was 400 or 800 or something else), there was little benefit to 14-bit files as compared to 12-bit files. When I am shooting wildlife, I am ususally at a higher ISO, given the shutter speeds I tend to want for BIF and other action. I have a separate bank set up for landscape that fixes ISO at the base, 64. In that bank, I am also set up to use lossless compressed. The size savings of high efficiency raw* are not material in landscape shooting, as a general matter, as I shoot far fewer frames. The smaller size of high efficiency raw* is nice when shooting bursts of BIF or other action.

 

[bjanes]

Thanks Bill. Looks interesting. One question. What is RawDigger measuring that it reports a mean value for? I take it this is a measurement of noise? Or of some item that correlates with noise?

I see that high efficiency raw* is the highest of the 3. Do you think that difference is enough to be visible in photos?

How do you think this would change, if at all, in absolute or relative terms, at higher ISOs? I recall that Thom Hogan said with a D850 that once you got above a certain ISO (I don’t recall now whether it was 400 or 800 or something else), there was little benefit to 14-bit files as compared to 12-bit files. When I am shooting wildlife, I am ususally at a higher ISO, given the shutter speeds I tend to want for BIF and other action. I have a separate bank set up for landscape that fixes ISO at the base, 64. In that bank, I am also set up to use lossless compressed. The size savings of high efficiency raw* are not material in landscape shooting, as a general matter, as I shoot far fewer frames. The smaller size of high efficiency raw* is nice when shooting bursts of BIF or other action.

Bill,

The standard deviation is the measure of noise. The mean pixel value is a measure of exposure and should be the same for all images. One of the best criteria for perceived noise is the SNR (signal to noise ratio). Poisson shot noise is the majority of noise in midrange exposure, and is proportional to the square root of the number of photons collected (at higher exposure pixel response nonuniformity-PRNU contributes, but let's keep things simple) . Accordingly noise is higher at larger exposure, but the SNR is higher. S is the signal and the noise is sqrt(S) so the SNR is S/sqrt(s), which simplifies to sqrt(S). For more information see Emil Martinec here.

Your questions prompt me to re-evaluate my analysis to include SNR. Engineers often express SNR is decibels so I include this in my revised chart.

You can only see EXIF info for this image if you are logged in.

I do not think that these differences are visually significant. The higher photographic dynamic range with the D850 as compared to the Z9 at base ISO as noted by Bill Claff is consistent with this since DR is a measure of noise. At higher ISOs, the differences largely disappear. At higher ISOs 12 bits is sufficient to encode the results, but I do not at what ISO this occurs. Bill Claff has data for 12 bits for the Z9 and you could look there.

I agree with your strategy for shooting wild life and landscape, but I think HE* is OK for landscape too.

Thanks for your comments,

Bill

 

[bleirer]

Bill,

The standard deviation is the measure of noise. The mean pixel value is a measure of exposure and should be the same for all images. One of the best criteria for perceived noise is the SNR (signal to noise ratio). Poisson shot noise is the majority of noise in midrange exposure, and is proportional to the square root of the number of photons collected (at higher exposure pixel response nonuniformity-PRNU contributes, but let's keep things simple) . Accordingly noise is higher at larger exposure, but the SNR is higher. S is the signal and the noise is sqrt(S) so the SNR is S/sqrt(s), which simplifies to sqrt(S). For more information see Emil Martinec here.

Your questions prompt me to re-evaluate my analysis to include SNR. Engineers often express SNR is decibels so I include this in my revised chart.



View attachment 33988

I do not think that these differences are visually significant. The higher photographic dynamic range with the D850 as compared to the Z9 at base ISO as noted by Bill Claff is consistent with this since DR is a measure of noise. At higher ISOs, the differences largely disappear. At higher ISOs 12 bits is sufficient to encode the results, but I do not at what ISO this occurs. Bill Claff has data for 12 bits for the Z9 and you could look there.

I agree with your strategy for shooting wild life and landscape, but I think HE* is OK for landscape too.

Thanks for your comments,

Bill

Hello Bill,

This is Bill talking to Bill about his comments to Bill about Bill's data on photons to photons. Thanks for the info, Bill.

Sincerely, Bill

 

[BillW]

Bill,

The standard deviation is the measure of noise. The mean pixel value is a measure of exposure and should be the same for all images. One of the best criteria for perceived noise is the SNR (signal to noise ratio). Poisson shot noise is the majority of noise in midrange exposure, and is proportional to the square root of the number of photons collected (at higher exposure pixel response nonuniformity-PRNU contributes, but let's keep things simple) . Accordingly noise is higher at larger exposure, but the SNR is higher. S is the signal and the noise is sqrt(S) so the SNR is S/sqrt(s), which simplifies to sqrt(S). For more information see Emil Martinec here.

Your questions prompt me to re-evaluate my analysis to include SNR. Engineers often express SNR is decibels so I include this in my revised chart.



View attachment 33988

I do not think that these differences are visually significant. The higher photographic dynamic range with the D850 as compared to the Z9 at base ISO as noted by Bill Claff is consistent with this since DR is a measure of noise. At higher ISOs, the differences largely disappear. At higher ISOs 12 bits is sufficient to encode the results, but I do not at what ISO this occurs. Bill Claff has data for 12 bits for the Z9 and you could look there.

I agree with your strategy for shooting wild life and landscape, but I think HE* is OK for landscape too.

Thanks for your comments,

Bill

Thanks for the explanation Bill. I remember Poisson distributions from my days (many years ago) as a math major.

 

[RichF]

Bill,

The standard deviation is the measure of noise. The mean pixel value is a measure of exposure and should be the same for all images. One of the best criteria for perceived noise is the SNR (signal to noise ratio). Poisson shot noise is the majority of noise in midrange exposure, and is proportional to the square root of the number of photons collected (at higher exposure pixel response nonuniformity-PRNU contributes, but let's keep things simple) . Accordingly noise is higher at larger exposure, but the SNR is higher. S is the signal and the noise is sqrt(S) so the SNR is S/sqrt(s), which simplifies to sqrt(S). For more information see Emil Martinec here.

Your questions prompt me to re-evaluate my analysis to include SNR. Engineers often express SNR is decibels so I include this in my revised chart.



View attachment 33988

I do not think that these differences are visually significant. The higher photographic dynamic range with the D850 as compared to the Z9 at base ISO as noted by Bill Claff is consistent with this since DR is a measure of noise. At higher ISOs, the differences largely disappear. At higher ISOs 12 bits is sufficient to encode the results, but I do not at what ISO this occurs. Bill Claff has data for 12 bits for the Z9 and you could look there.

I agree with your strategy for shooting wild life and landscape, but I think HE* is OK for landscape too.

Thanks for your comments,

Bill

I would like to repeat this, at different ISO. See if HE* noise is the same lossless a 1/3 or perhaps 2/3 stops lower (just a guess).

BTW is mu is the mean signal (of the raw unprocessed image) it should be the same (ideally) for the Z9 lossless and HE*. I can understand a difference between the D850 and Z9 but the Z9 compression modes should be the same.

 

[bjanes]

I would like to repeat this, at different ISO. See if HE* noise is the same lossless a 1/3 or perhaps 2/3 stops lower (just a guess).

BTW is mu is the mean signal (of the raw unprocessed image) it should be the same (ideally) for the Z9 lossless and HE*. I can understand a difference between the D850 and Z9 but the Z9 compression modes should be the same.

It should be the same. Maybe there is some data loss in the compression or perhaps there was a measurement error. I think the difference between the D850 and Z9 would disappear at higher ISO.

BTW, your byline suggests to me that you are the RIchF at GPS?

Nice to hear from you.

Cheers,

Bill Janes

 

[RichF]

It should be the same. Maybe there is some data loss in the compression or perhaps there was a measurement error. I think the difference between the D850 and Z9 would disappear at higher ISO.

BTW, your byline suggests to me that you are the RIchF at GPS?

Nice to hear from you.

Cheers,

Bill Janes

Yes. GPS