Stop Struggling With Shadow Recovery!

[Steve]

Ever notice that when you try to preform shadow recovery / pull shadows during post processing that sometimes it works and sometimes it’s a disaster? Did you know your shadow recovery success is directly related to your choices in the field?

In this video, we’ll talk about what shadow recovery actually does, why those areas get noisier, how to tell how noisy they’ll get, and – most importantly – what you can do in the field that will make a difference back home on the computer.

This 7 minute video will change how you think about shadows, ISO, and it’ll modify your field technique forever! Jump in!

(Note – at the 3.42 mark there’s a typo in the note above ISO equivalency that says 6000 that should say 16000 – I missed the “1” when I was making the caption.)

 

[Fishboy1952]

Thanks @Steve. I appreciate your lessons like this. It definitely helps me in my hobby. BTW, I do like the new intro.

 

[Irap]

Great explication and great tips

 

[Marcus Slade]

Thanks, Steve. I made a comment to the video essentially about ETTR and wondering if there is a high wire to be walked in trying to get as much shadow information as possible in the first place, prior to any post work? I have plenty of shots working with shadows where that's the case. Interested too in your comments about the non-ISO invariance of the D5 and D6, for instance, having recently added a D6.

 

[Steve]

Thanks, Steve. I made a comment to the video essentially about ETTR and wondering if there is a high wire to be walked in trying to get as much shadow information as possible in the first place, prior to any post work? I have plenty of shots working with shadows where that's the case. Interested too in your comments about the non-ISO invariance of the D5 and D6, for instance, having recently added a D6.

Best bet is to keep the ISO as low as possible and if you're under ISO 800 or so, shoot in 14 bit.

Note that if the camera is ISO invariant, ETTR using ISO doesn't really do much. So, keeping the image overly bright using ISO. vs dropping ISO so it looks correct - I pick the second choice every time. ETTR using ISO is a good way to diminish dynamic range without much (or any gain).

As for the D5/6, treat them the same as any other camera in the field - keep the ISO low and do what you can to avoid situations that would require an extreme shadow pull. Funny thing is with the D5/6, they actually seem to do better with shadow pulls at higher ISOs than they do at lower ISOs.

 

[kimball]

Great video as usual!

As I've dived into the weeds of ISO and camera electronics I've learned one surprising thing: the ISO standard does not specify how a camera should implement changes in its ISO setting. It can be done in hardware, software, analog processing, digital processing, or some eclectic mix of all of the above.

Fuji's 50mp medium format cameras like the GFX50S are a great example of this. As you raise ISO from 100 to 1600, the sensor hardware is clearly processing the signal proportionally to the ISO setting as one would expect. But it then stops at 1600. Now, the camera specs list the GFX50S as having a normal, non-extended ISO latitude of 100-12,800. So what happens between 1600 and 12,800 if you're shooting raw? Anything above 1600 is shot as if the sensor electronics were at ISO 1600, and that is the data that is embedded in the raw file. But then a tag is added to the raw file instructing the raw processor to boost lightness digitally to correspond to the actual chosen ISO setting of the shot. Pretty surprising stuff, and 100% allowed by the ISO standard. (proof in DPReview's review and photonstophotos.net PDR graphs)

The consequence of this: ISO is not and never has been a measurement of sensor sensitivity. It actually defines the sensitivity of the entire camera as a whole (when set to a particular ISO setting), where

input = amount of light passing the shutter
output = image lightness levels in a final, straight-out-of-camera image file (not a raw file!)
sensitivity = the relationship between input & output. For the same input, a camera setting that produces a larger output ("brighter") is said to be more sensitive, i.e. have a higher ISO.

Unfortunately the entire photographic community has been thinking about ISO performance in terms of sensor performance for years, and folks have kind of forgotten that officially ISO is merely a mapping of incoming light that has passed the shutter to final camera JPG/TIFF image lightness. Camera makers can implement that mapping however they see fit, either in camera hardware, camera software, or both.

 

[bleirer]

Great video as usual!

As I've dived into the weeds of ISO and camera electronics I've learned one surprising thing: the ISO standard does not specify how a camera should implement changes in its ISO setting. It can be done in hardware, software, analog processing, digital processing, or some eclectic mix of all of the above.

Fuji's 50mp medium format cameras like the GFX50S are a great example of this. As you raise ISO from 100 to 1600, the sensor hardware is clearly processing the signal proportionally to the ISO setting as one would expect. But it then stops at 1600. Now, the camera specs list the GFX50S as having a normal, non-extended ISO latitude of 100-12,800. So what happens between 1600 and 12,800 if you're shooting raw? Anything above 1600 is shot as if the sensor electronics were at ISO 1600, and that is the data that is embedded in the raw file. But then a tag is added to the raw file instructing the raw processor to boost lightness digitally to correspond to the actual chosen ISO setting of the shot. Pretty surprising stuff, and 100% allowed by the ISO standard. (proof in DPReview's review and photonstophotos.net PDR graphs)

The consequence of this: ISO is not and never has been a measurement of sensor sensitivity. It actually defines the sensitivity of the entire camera as a whole (when set to a particular ISO setting), where

input = amount of light passing the shutter
output = image lightness levels in a final, straight-out-of-camera image file (not a raw file!)
sensitivity = the relationship between input & output. For the same input, a camera setting that produces a larger output ("brighter") is said to be more sensitive, i.e. have a higher ISO.

Unfortunately the entire photographic community has been thinking about ISO performance in terms of sensor performance for years, and folks have kind of forgotten that officially ISO is merely a mapping of incoming light that has passed the shutter to final camera JPG/TIFF image lightness. Camera makers can implement that mapping however they see fit, either in camera hardware, camera software, or both.

Good points. I think ISO is standard in defining the brightness of the resulting jpeg and can tie into EV in that sense, but the wild west in raw. I wouldn't even use the term sensitivity except in terms of actual photons striking the sensor, as you said the sensor sensitivity never varies no matter what iso is used. Gain after exposure is maybe a better description.

 

[bleirer]

Ever notice that when you try to preform shadow recovery / pull shadows during post processing that sometimes it works and sometimes it’s a disaster? Did you know your shadow recovery success is directly related to your choices in the field?

In this video, we’ll talk about what shadow recovery actually does, why those areas get noisier, how to tell how noisy they’ll get, and – most importantly – what you can do in the field that will make a difference back home on the computer.

This 7 minute video will change how you think about shadows, ISO, and it’ll modify your field technique forever! Jump in!

(Note – at the 3.42 mark there’s a typo in the note above ISO equivalency that says 6000 that should say 16000 – I missed the “1” when I was making the caption.)

Great video! I love that Journey analogy, very accessible for us non-techies.

 

[kimball]

Good points. I think ISO is standard in defining the brightness of the resulting jpeg and can tie into EV in that sense, but the wild west in raw. I wouldn't even use the term sensitivity except in terms of actual photons striking the sensor, as you said the sensor sensitivity never varies no matter what iso is used. Gain after exposure is maybe a better description.

Yeah, sensitivity is a really confusing term in digital photography and clearly a lot of well-intentioned and educated folk think the physical sensor sensitivity is changing with ISO. I wish that term wasn't used to imply that, and I'm grateful that educators like Steve are pushing back on that misconception with really clear videos like these. Sadly the ISO standard does use the term sensitivity, but only in the context of the camera as whole (including the camera's software image processing pipeline) as it creates a final JPG image, not the sensor specifically.

The extra confusing part is that it's still totally valid to analyze how sensor + electronics performance changes with ISO. I just personally find it super interesting and valuable to remember that ISO doesn't require any hardware specifics at all. Analog gain, digital gain, etc. are all used at the option of the manufacturer. Their engineering teams are entitled to manage image quality at various ISO values as they see fit, and that is outside the scope of the ISO standard.

It's useful to imagine a future camera with effectively "noiseless" electronics. In such a camera, the only image noise present would be shot noise inherent in the incoming light itself. Such a camera would have no need for hardware-based implementation of ISO changes. It could manage changes to ISO just as Fuji's GFX50S does above 1600-- always shoot at "base" and merely put an ISO tag in the raw file to achieve proper image lightness for that ISO. This would keep the camera's dynamic range fixed at its maximum, thus remarkably preserving highlights in high ISO shots. One day...

 

[Marcus Slade]

Best bet is to keep the ISO as low as possible and if you're under ISO 800 or so, shoot in 14 bit.

Note that if the camera is ISO invariant, ETTR using ISO doesn't really do much. So, keeping the image overly bright using ISO. vs dropping ISO so it looks correct - I pick the second choice every time. ETTR using ISO is a good way to diminish dynamic range without much (or any gain).

As for the D5/6, treat them the same as any other camera in the field - keep the ISO low and do what you can to avoid situations that would require an extreme shadow pull. Funny thing is with the D5/6, they actually seem to do better with shadow pulls at higher ISOs than they do at lower ISOs.

Totally agree, Steve, as to keeping ISO as low as possible. However, at times in the past I have slavishly held ISO down to the point where i've lost shots because of inadequate shutter speed, believing the higher ISO would render the photo "unviewable". Agreed again on the ETTR which in of itself I try to avoid. Interesting, your point about shadow recovery being better at higher ISO levels with the D6. Given I got this for low light situations, I've had lots of early experience with high ISO situations where I've been pleasantly surprised at the retention of shadow detail and, in tandem with Topaz Denoise, remarkably impressed.

 

[kimball]

As for the D5/6, treat them the same as any other camera in the field - keep the ISO low and do what you can to avoid situations that would require an extreme shadow pull. Funny thing is with the D5/6, they actually seem to do better with shadow pulls at higher ISOs than they do at lower ISOs.

The D5/6 electronics dramatically reduce camera-induced noise (read noise) as ISO increases. Camera-induced noise is highest at base ISO, but isn't often seen because at base ISO it is swamped by the great SNR of the captured light. However if you have to increase base ISO shadows hard, due to a lack of light, you are likely to see it.

https://www.photonstophotos.net/Charts/PDR_Shadow.htm#Nikon D5,Nikon D6,Nikon D7000
https://www.photonstophotos.net/Charts/RN_ADU.htm#Nikon D5_14,Nikon D6_14,Nikon D7000_14

Compared to the D7000 (a great example of an ISO-invariant sensor), the D5/6 sensor has 5 times the read noise in DNs (Digital Numbers, the values recorded in a raw file) at base ISO. See the second link. But as ISO increases, the D5/6 sensor keeps it's read noise near base ISO DN values, while ISO-invariant sensors like the one in the D7000 increase proportionally with ISO, doubling with every stop, so you never feel like noise performance is improving with ISO on ISO-invariant sensors as it does on non-invariant sensors like the D5/6.

 

[IainD]

Is the best policy then to shoot with the lowest ISO setting you can to have good exposure parameters? And if we use auto iso should we consider reducing the upper limit of ISO we are willing to accept? It seems that this plan would succeed with any sensor, but maybe the D5 and D6 should start at a slightly higher baseline ISO. Not sure if I am getting this right.
Thank you,
I

 

[Steve]

The D5/6 electronics dramatically reduce camera-induced noise (read noise) as ISO increases. Camera-induced noise is highest at base ISO, but isn't often seen because at base ISO it is swamped by the great SNR of the captured light. However if you have to increase base ISO shadows hard, due to a lack of light, you are likely to see it.

https://www.photonstophotos.net/Charts/PDR_Shadow.htm#Nikon D5,Nikon D6,Nikon D7000
https://www.photonstophotos.net/Charts/RN_ADU.htm#Nikon D5_14,Nikon D6_14,Nikon D7000_14

Compared to the D7000 (a great example of an ISO-invariant sensor), the D5/6 sensor has 5 times the read noise in DNs (Digital Numbers, the values recorded in a raw file) at base ISO. See the second link. But as ISO increases, the D5/6 sensor keeps it's read noise near base ISO DN values, while ISO-invariant sensors like the one in the D7000 increase proportionally with ISO, doubling with every stop, so you never feel like noise performance is improving with ISO on ISO-invariant sensors as it does on non-invariant sensors like the D5/6.

That makes a ton of sense - I never really dug that deep into the reasons why the D5/6 were like that, I just accepted that they were optimized for higher ISOs (hey, even I get lazy sometimes ) Thanks!

 

[Steve]

Totally agree, Steve, as to keeping ISO as low as possible. However, at times in the past I have slavishly held ISO down to the point where i've lost shots because of inadequate shutter speed, believing the higher ISO would render the photo "unviewable". Agreed again on the ETTR which in of itself I try to avoid. Interesting, your point about shadow recovery being better at higher ISO levels with the D6. Given I got this for low light situations, I've had lots of early experience with high ISO situations where I've been pleasantly surprised at the retention of shadow detail and, in tandem with Topaz Denoise, remarkably impressed.

It really is a balance. What I see in the field is often people are using way more shutter speed or a fall smaller F/stop than what's required for the shot, thus driving their ISO up higher than they need it to be. At the same time though, ISO is there for a reason and I'd rather deal with a sharp, noisy photo than a soft, clean one any day.

 

[Steve]

Is the best policy then to shoot with the lowest ISO setting you can to have good exposure parameters? And if we use auto iso should we consider reducing the upper limit of ISO we are willing to accept? It seems that this plan would succeed with any sensor, but maybe the D5 and D6 should start at a slightly higher baseline ISO. Not sure if I am getting this right.
Thank you,
I

Yup, keep the ISO as low as you can as long as you're still getting the shutter speed and F/stop you require for the shot. As I mentioned to Marcus a second ago, I often see people using excess shutter speed and smaller F/stops than necessary. At the same time, you have to make sure you aren't hurting image quality by going too slow or too wide-open.

As for ISO caps, be careful there. Putting in a low cap that forces you to shoot slower shutter speeds or shoot wide open when you don't want to is problematic as well.

The bottom line is there's no free lunch, it's all a balancing act. Sometimes you just don't have the light to pull off a shot and no amount of rebalancing the equation can fix it. Although, sometimes you can overcome problems like that with a little creativity. When I'm doing BIF work and I'm faced with stratospheric ISOs to get the minimum required shutter speed, I'll often switch gears and drop to a low ISO and slow shutter speed and do some panning shots. So, there's often something you can do

 

[IainD]

Great! Thank you Steve. Extremely helpful info as always.
I

 

[jeffnles1]

good stuff

 

[ruley74]

So one thing I've yet to see an answer on is the slider adjustment in relation to stops of adjustment.... how many stops of adjustment is the slider actually capable of? So in LR, if you go 100% in slider, what is that in stops...?

This also seems program dependent, I feel like DxO PL sliders go further than LR in pushing/pulling....

 

[kimball]

So one thing I've yet to see an answer on is the slider adjustment in relation to stops of adjustment.... how many stops of adjustment is the slider actually capable of? So in LR, if you go 100% in slider, what is that in stops...?

This also seems program dependent, I feel like DxO PL sliders go further than LR in pushing/pulling....

Not easily quantifiable nor comparable. From a google search, and a DPReview poster I respect:

Re: Lightroom +100 shadow how many stops?: Retouching Forum: Digital Photography Review

Expert news, reviews and videos of the latest digital cameras, lenses, accessories, and phones. Get answers to your questions in our photography forums.

www.dpreview.com

Personally I'd focus less on quantifying it and more on whether it can provide the aesthetic result you're looking for from a reasonably exposed image.

 

[ruley74]

Not easily quantifiable nor comparable. From a google search, and a DPReview poster I respect:

Re: Lightroom +100 shadow how many stops?: Retouching Forum: Digital Photography Review

Expert news, reviews and videos of the latest digital cameras, lenses, accessories, and phones. Get answers to your questions in our photography forums.

www.dpreview.com

Personally I'd focus less on quantifying it and more on whether it can provide the aesthetic result you're looking for from a reasonably exposed image.

Agreed, and that's what I do... however we talk in stops of light to pull and push so it's still an interesting point to truely understand as a reference to.

I had a look at the thread and that's pretty obvious on what it effects but more simply put I feel it still can be as a maximum... i.e. the shadow slider will at maximum change the corresponding colour exposure by X.

 

[Marco74]

Dear Steve,

I have just seen your very clear and useful video. Great job as always.

In my experience with the new ML camera, I noted that in dark condition, so not only with a limited shadow zone to recovery, it is better to increase the ISO and not try to recover the light in post.
The Z6/Z6II keep the high ISO very well and look like that the AF works a way better with a proper expose image during the shot.

As an example, just after dawn, there is shallow light. If I shoot at 12800 ISO to keep low the digital noise, but I theoretically need a higher iso like 25600 or 32.000, it is a lot better to leave the camera to go up until that value than trying to recover the light in the post.
The noise at 32.000 is not so bad, but the most important thing is that all the photos at 32.000 are in focus meanwhile the underexposed photos at 12.800 quite often are completely out of focus because the AF of the new Z has problems to work without a strong contrast.

Do you have this feeling too? What do you think about it?

Marco

Quote Reply
Report

 

[jeffnles1]

So one thing I've yet to see an answer on is the slider adjustment in relation to stops of adjustment.... how many stops of adjustment is the slider actually capable of? So in LR, if you go 100% in slider, what is that in stops...?

This also seems program dependent, I feel like DxO PL sliders go further than LR in pushing/pulling....

I don't use LR so can't say for sure your answer. I do use Affinity Photo and Pixelmator Pro quite a bit as well as Nikon's software. The process I use is I zoom in the image on the shadow area I want to pull up. I go to 100% or maybe a little more. I adjust shadow, brightness, exposure etc. sliders until I start seeing unacceptable noise then back off until I can live with the noise. When the photo is viewed at "normal" size, if I think the shadows are still too dark or there is still much noise, I then use the delete key and move to the next image.
(edited to correct my horrible typing skills)

 

[Gail L]

I then use the delete key and move to the next image.

OK, that last line was funny!!

 

[jeffnles1]

OK, that last line was funny!!

Thanks. I'm glad you caught my humor. My attempts at humor are not always appreciated.

One of my "photography friends" once asked me what I thought was the single most important tool that I have used to improve my photography. My response was "the delete key." I am brutal with the delete key. Only 2 types of images I keep. 1) those that contain a special memory or photos of loved ones. 2) those which I think are worthy of printing (even if I may never print it). If the photo has motion blur, too much noise, missed focus, exposed incorrectly, or just not an interesting subject... to the trash bin it goes. If I ended up with several images of the same subject, I will keep the best one and if none of them meet my "keep it" criteria, they all go.

 

[Steve]

Dear Steve,

I have just seen your very clear and useful video. Great job as always.

In my experience with the new ML camera, I noted that in dark condition, so not only with a limited shadow zone to recovery, it is better to increase the ISO and not try to recover the light in post.
The Z6/Z6II keep the high ISO very well and look like that the AF works a way better with a proper expose image during the shot.

As an example, just after dawn, there is shallow light. If I shoot at 12800 ISO to keep low the digital noise, but I theoretically need a higher iso like 25600 or 32.000, it is a lot better to leave the camera to go up until that value than trying to recover the light in the post.
The noise at 32.000 is not so bad, but the most important thing is that all the photos at 32.000 are in focus meanwhile the underexposed photos at 12.800 quite often are completely out of focus because the AF of the new Z has problems to work without a strong contrast.

Do you have this feeling too? What do you think about it?

Marco

Quote Reply
Report

That's possible. Also, though, there's really no advantage to shooting underexposed anyway. If you shoot under at 12,800 and pull the overall image brightness up by a stop back home, it'll still look like 25,600. So, might as well shoot what you need in the field.