Z8/9 Auto Focus (C) w/Sub Detection 🐦+👁️=🧐

[Cameron T]

One hundred and twenty calculations per second may seem like a lot, but consider the speed our subjects might be approaching. Not even getting into airplanes and racing cars it's easy for a bird to be flying at us at 50 kilometers per hour (approximately 31 mph) and that's on the slow side as Ospreys have been clocked at 80 mph, Pronghorn Antelope can run at 60 mph and a Peregrin Falcon in a dive at close to 240 mph but dives aren't generally right towards us. Still, speeds can be fast even for live animals and a lot faster for other photo subjects.

At 50 kph coming directly at us the subject is moving nearly 11.5 cm in 1/120 of a second or roughly 4.5 inches between each AF calculation. That might not seem like much movement but when viewed through a long lens that's easily enough to be out of focus by the time the shutter is released based on the most recent AF calculation. So for many years the camera manufacturers have used predictive AF tracking algorithms to try to stay slightly ahead of subject movement to deliver as many crisp well focused images as possible for fast moving subjects. Even at 120 calculations per second there are many subjects that move fast enough that we'd want predictive AF tracking.

But yes, moving the mass of focusing elements is part of the question but not the whole question as even with instantaneous mechanical focus adjustment 120 calculations per second can't keep up with the fastest photography subjects if they're coming right at us.

In addition, moving the elements really is less of an issue at longer ranges due to increased dof. A slight miss isn't noticeable.

Example with made up numbers, your dof covers 20 units of stepper motor movement at distance x. Af tells it to go 5 units closer. It only goes 3. Not really noticeable.

At range y your dof covers 2 units. Camera tells it to move 5 units closer, it moves 3 again. Now your subject is oof.

The ops issue was focus jumped wildly and the focus area is much different, and not close.

 

[JoelKlein]

One hundred and twenty calculations per second may seem like a lot, but consider the speed our subjects might be approaching. Not even getting into airplanes and racing cars it's easy for a bird to be flying at us at 50 kilometers per hour (approximately 31 mph) and that's on the slow side as Ospreys have been clocked at 80 mph, Pronghorn Antelope can run at 60 mph and a Peregrin Falcon in a dive at close to 240 mph but dives aren't generally right towards us. Still, speeds can be fast even for live animals and a lot faster for other photo subjects.

At 50 kph coming directly at us the subject is moving nearly 11.5 cm in 1/120 of a second or roughly 4.5 inches between each AF calculation. That might not seem like much movement but when viewed through a long lens that's easily enough to be out of focus by the time the shutter is released based on the most recent AF calculation. So for many years the camera manufacturers have used predictive AF tracking algorithms to try to stay slightly ahead of subject movement to deliver as many crisp well focused images as possible for fast moving subjects. Even at 120 calculations per second there are many subjects that move fast enough that we'd want predictive AF tracking.

But yes, moving the mass of focusing elements is part of the question but not the whole question as even with instantaneous mechanical focus adjustment 120 calculations per second can't keep up with the fastest photography subjects if they're coming right at us.

Wow !
Thanks for the explanation. First time I encounter the term predictive AF tracking. It definitely makes sense.

How do we know if predictive AF tracking is engaged? For instance, in studio, I think I would want it off..

 

[Dinusaur]

This is a classic case of a busy background and a subject that is hard to identify. The camera will often struggle with this kind of scene since the bird is hard to recognize as a bird, and the background has distracting high contrast detail. I've run into this scenario in the past. When I have run into this issue, it was a specific subject and background causing the problem, and changing backgrounds made a big difference. That may not be possible - but could be an alternative with a busy background like this.

While you and I know this is a bird, it's really only the bright part of the bird on the top of the wings and the back of the bird's head that is used as a target. That's a shape to the camera, but not a bird shape. A silhouette of a dark bird and a clean background would be easy - but the shape is tough to recognize in the lighting available.

The key is whether the camera ever acquires focus, or does it achieve focus and then lose focus. If you are not achieving focus at all, I would try to use a smaller AF area. I would also make sure I was pre-focusing between the subject and the camera so the camera is moving from mid to far distance of focus. I don't want to allow any hunting from far to near. Wide small should do the trick in terms of focus area if Wide Large is not working. If you continue to struggle with this angle, I'd try turning off bird subject detection and using Animal subject detection rather than have a profile the camera does not recognize as a bird. If the camera is losing focus on the subject and picking up the background, I'd adjust tracking and make tracking stickier - or slower to change subjects.

Thanks for your thoughts on this, you are absolutely right about the challenging background conditions here. However, before the focus was lost there were 8 frames up to frame #4149 where the focus was maintained even though the bird was diving and coming towards me against the same challenging background. Then on frame #4150 (9th in the series) the focus jumped to the background. In a burst mode I would think that 8 good data points should be good enough to predict where the 9th data point will be in a simple linear extrapolation. I am sure that this scenario requires more than linear extrapolation because of the many numbers of focus points involved. That's most likely where the software/hardware limitations come into play as noted in some earlier posts.

I'll try your other suggestions about limiting the focus area - I did post a series of photos of a diving Osprey earlier with single point focusing and how good they were. Not sure how that would fare against the challenging background like this - it is worth a try. Thanks again.

 

[Dinusaur]

....

As a consequence, I tried to dissect the situation further by looking at the images AF information in exiftool and have included a portion of it here to spur further conversation. To recall, the sequence was comprised of 70-100 images shot continuously as the osprey was swooping in for a catch. I labeled the image taken immediately before the camera lost AF as "one" and provided several sequential oof images, labeled "two" through "four", and skipped the additional oof images until the camera regained AF several frames later in image "nine". The summary of information is listed in the table below:

​	Image one​	Image two​	Image three​	Image four​	Image nine​
Focus Distance​	49.17​	49.17​	49.17​	49.17​	49.17​
Lens Position​	1241​	1302​	1326​	1326​	1278​
AF Area X position​	3699​	3946​	3989​	4279​	4452​
Focus position H​	2L of Center​	1L of Center​	1L of Center​	2D of Center​	1R of Center​
AF Area Y position​	3355​	3129​	3000​	3226​	3549​
Focus position V​	3D of Center​	2D of Center​	1D of Center​	Center​	3D of Center​
AF Area Width​	344​	494​	494​	430​	430​
AF Area Height​	344​	494​	494​	430​	430​
Focus Result​	Focus​	Focus​	Focus​	Focus​	Out of Focus​

Looking at the data, there are several interesting observations. First, the focus distance is recorded as the same in all of the frames. I'm not certain whether this is a function of exif tool, a recording issue in the exif, or an indication of something more ominous, i.e. the camera/lens is not responding/recording properly. What I find more revealing is the lens position information. When the bird was in focus, the lens position was indicating "1241", it then jumped significantly to another position for the OOF frames and then returned to a reasonable compromise in frame "nine". The AF area X/Y information is difficult to interpret, although the focus position H/V didn't change that much suggesting that the subject was kept relatively centered in the frame. What I find more interesting is that the recorded "focus result" does not correspond to the actual image. The OOF images are regarded by the camera as being in "focus" while ironically, all of the images (not just image "nine") where the camera regains AF are regarded as "out of focus".

While one can look at the tabulated results and make some observations, I am left pondering as to what is really occurring. Is the camera and/or lens malfunctioning or miscommunicating? Is there a problem in the tracking software? I wish that I could reach out to a Sony engineer for further exploration.

Appreciate the thoughts, though short of some sort of lens/camera communication/operational issue, I tend to side with Steve's explanation involving predictive AF. Why? This AF error is repeatable and situational. The panning technique appears appropriate from the context that the subject remains large in the frame and is relatively in the same position, thereby minimizing user disturbance in the AF point. Also, if you look closely at the sequence - and my apologies for not posting more images though I am trying to be respectful of the forum - the AF plane doesn't appear to move to the bird's reflection in the wave, rather it is several feet in front of the bird where it remains for several frames and then it resets back onto the subject. One of the challenging aspects of deconstructing these sequences is the lack of information of the true AF point. The reliability of the AF point displayed in NxStudio remains debatable, however from the sequence presented, it doesn't appear to deviate significantly.

As a consequence, I tried to dissect the situation further by looking at the images AF information in exiftool and have included a portion of it here to spur further conversation. To recall, the sequence was comprised of 70-100 images shot continuously as the osprey was swooping in for a catch. I labeled the image taken immediately before the camera lost AF as "one" and provided several sequential oof images, labeled "two" through "four", and skipped the additional oof images until the camera regained AF several frames later in image "nine". The summary of information is listed in the table below:

​	Image one​	Image two​	Image three​	Image four​	Image nine​
Focus Distance​	49.17​	49.17​	49.17​	49.17​	49.17​
Lens Position​	1241​	1302​	1326​	1326​	1278​
AF Area X position​	3699​	3946​	3989​	4279​	4452​
Focus position H​	2L of Center​	1L of Center​	1L of Center​	2D of Center​	1R of Center​
AF Area Y position​	3355​	3129​	3000​	3226​	3549​
Focus position V​	3D of Center​	2D of Center​	1D of Center​	Center​	3D of Center​
AF Area Width​	344​	494​	494​	430​	430​
AF Area Height​	344​	494​	494​	430​	430​
Focus Result​	Focus​	Focus​	Focus​	Focus​	Out of Focus​

Looking at the data, there are several interesting observations. First, the focus distance is recorded as the same in all of the frames. I'm not certain whether this is a function of exif tool, a recording issue in the exif, or an indication of something more ominous, i.e. the camera/lens is not responding/recording properly. What I find more revealing is the lens position information. When the bird was in focus, the lens position was indicating "1241", it then jumped significantly to another position for the OOF frames and then returned to a reasonable compromise in frame "nine". The AF area X/Y information is difficult to interpret, although the focus position H/V didn't change that much suggesting that the subject was kept relatively centered in the frame. What I find more interesting is that the recorded "focus result" does not correspond to the actual image. The OOF images are regarded by the camera as being in "focus" while ironically, all of the images (not just image "nine") where the camera regains AF are regarded as "out of focus".

While one can look at the tabulated results and make some observations, I am left pondering as to what is really occurring. Is the camera and/or lens malfunctioning or miscommunicating? Is there a problem in the tracking software? I wish that I could reach out to a Sony engineer for further exploration.

Very interesting, thanks for sharing the data. I never thought of looking into these exif data but now as I look into them it provides some insight on how the tracking is taking shape in some of my images. I have a feeling that the focus distance, after the first acquisition, remains the same in a burst to within a certain limit, the Lens Position is adjusted to track the subject. After a certain subject distance, beyond a pre-set limit or if re-focus, the focus distance is re-set and used as a reference for the next set of lens position adjustments. Increasing lens position number in your table probably means that the subject is coming closer to you or front focusing. Since 2-8 are oof, front focusing is the main culprit here. Take a photo with focus at infinity (check your recently taken eclipse photo, if you have one) and at MFD and see how the number varies. Unfortunately, the data doesn't help eliminate the front focusing problem or change of sudden focus change.

 

[DRwyoming]

How do we know if predictive AF tracking is engaged? For instance, in studio, I think I would want it off..

There's no on-off switch, it's real time adaptive.

Basically the camera computes a series of AF solutions, if the AF doesn't change at all between computations then there's no predictive delta applied. If each subsequent AF calculation changes in the same direction by a certain amount then a predictive correction factor is applied to the subsequent AF calculation. So if something moved towards or away from you at a steady speed the prediction would compensate based on the delta between each proceeding AF calculation.

But in the general case things don't move at steady speed and they may change direction and stop or start again so a series of AF solutions is fed into an algorithm which is basically some form of filtering that takes into account a series of subsequent AF calculations and comes up with a correction term for the next AF calculation. Again if a bunch of AF calculations in a row are the same (e.g. a static subject) then the correction term is zero and there's no prediction applied.

Nikon doesn't publish the specifics of their AF prediction algorithms nor any specifics on the filtering applied from calculation to calculation but I've worked on many systems like this in various engineering applications and they're all more or less the same. Basically compute a series of single point solutions at each subsequent calculation point, measure the delta (change) between each successive computation and then use that series of time changing (or time invariant) calculations to create a correction factor for the upcoming interval but time filtered in such a way that the algorithm can self adjust as the object you're tracking speeds up, slows down, changes direction or stops moving all together.

There's no perfect tracking algorithm and no perfect predictive tracking algorithms but well designed systems can work pretty well across a wide range of tracking speeds with minimal overshoot or undershoot and can easily handle the cases where there's no motion and there's no real predictive part to the tracking (measurement to measurement delta is zero).

That lack of tracking perfection in every conceivable situation at every conceivable speed or change in speed is part of why sometimes AF misses a bit. For instance in takeoff shots where a bird goes from perched to flying it's not unusual to get some crisp shots but it's also not unusual to have the AF system miss on occasion for some or even all shots in a sequence as predicting motion, speed and relative direction from a standstill is a very hard challenge and sometimes when a bird launches into the air the camera misses and sometimes it nails the entire takeoff sequence. AF systems have gotten a lot better at handling situations like this in recent years but it's still a challenging scenario compared to tracking a subject traveling at a steady speed.

 

[ajrmd]

Very interesting, thanks for sharing the data. I never thought of looking into these exif data but now as I look into them it provides some insight on how the tracking is taking shape in some of my images. I have a feeling that the focus distance, after the first acquisition, remains the same in a burst to within a certain limit, the Lens Position is adjusted to track the subject. After a certain subject distance, beyond a pre-set limit or if re-focus, the focus distance is re-set and used as a reference for the next set of lens position adjustments. Increasing lens position number in your table probably means that the subject is coming closer to you or front focusing. Since 2-8 are oof, front focusing is the main culprit here. Take a photo with focus at infinity (check your recently taken eclipse photo, if you have one) and at MFD and see how the number varies. Unfortunately, the data doesn't help eliminate the front focusing problem or change of sudden focus change.

I've bench tested the lens (800 PF) at two different distances (near MFD and distant) and it is on the mark. Again, I wish that I had a connection to Nikon engineers who could better interpret the data including why the camera logic was indicating "focus" when it clearly was not and for the subsequent images which were in-focus, it recorded "out of focus" in the exif. Unfortunately, I am no closer to determining whether it is a predictive focus algorithm problem or a camera/lens issue. The assertion that the focus distance is recorded from the first frame in the sequence seems plausible and I'll look at other sequences to see if this is the case.

So, I just looked at a variety of BIF sequences using the same lens, 800 PF, as well as other lenses 180-600, 400 f/4.5, etc. All of the other sequences record distances changing as the subject is moving towards or away from the camera. We may be on to something, namely why did the camera not detect a subject changing distance? Odder still, the lens position was changing the entire time?

 

[Ken Miracle]

Thanks for your thoughts on this, you are absolutely right about the challenging background conditions here. However, before the focus was lost there were 8 frames up to frame #4149 where the focus was maintained even though the bird was diving and coming towards me against the same challenging background. Then on frame #4150 (9th in the series) the focus jumped to the background. In a burst mode I would think that 8 good data points should be good enough to predict where the 9th data point will be in a simple linear extrapolation. I am sure that this scenario requires more than linear extrapolation because of the many numbers of focus points involved. That's most likely where the software/hardware limitations come into play as noted in some earlier posts.

I'll try your other suggestions about limiting the focus area - I did post a series of photos of a diving Osprey earlier with single point focusing and how good they were. Not sure how that would fare against the challenging background like this - it is worth a try. Thanks again.

I never found Wide Area Large to be very effective in situations like that in fact for me it is one of the area modes I have turned off in my rotation. As noted by others a smaller area could be effective ... I use wide area custom at 5x3 and the second wide area custom at 1x1. That shot has a contrast challenge besides the busy background since the bird and background are dipslaying the same shades and contrast. A tough one. As of Z9 FW 5.0 I use AF Area Mode more often than 3d now on BIF.

 

[fcotterill]

As I understand the Z AF system, Wide Area Large / Small and the Custom Area modes are all designed to operate on the same underlying rules, including quasi-Close Focus Priority and they all work with Subject Recognition.

On the subject of the predictive tracking, presumably, the Steady versus Erratic options for Subject motion in the AF custom menu a3 is to inform the predictive tracking algorithm(s)? And Blocked shot AF Response also interfaces with the latter, although AutoAF and 3D work differently, and operate with integrated code.

 

[EricBowles]

Thanks for your thoughts on this, you are absolutely right about the challenging background conditions here. However, before the focus was lost there were 8 frames up to frame #4149 where the focus was maintained even though the bird was diving and coming towards me against the same challenging background. Then on frame #4150 (9th in the series) the focus jumped to the background. In a burst mode I would think that 8 good data points should be good enough to predict where the 9th data point will be in a simple linear extrapolation. I am sure that this scenario requires more than linear extrapolation because of the many numbers of focus points involved. That's most likely where the software/hardware limitations come into play as noted in some earlier posts.

I'll try your other suggestions about limiting the focus area - I did post a series of photos of a diving Osprey earlier with single point focusing and how good they were. Not sure how that would fare against the challenging background like this - it is worth a try. Thanks again.

I suspect that the camera was losing focus earlier in the series, but the lock-on setting kept it from searching for a new subject. Changing the Lock-On setting (by a small amount - it's very sensitive) could help stay on the subject longer.

I had similar experience with the Z7ii and swallow tailed kites. I would repeatedly pick up the subject against a clear blue sky without any issue. As the kite dropped, it crossed a distant treeline and the combination of contrasty detail and change in background caused me to consistently lose the subject. I adjusted the Lock-On setting and focus became very sticky. The subjects were still fast moving and challenging, but the camera held focus. My conclusion was that if lost focus was predictable, a setting or technique could help address the issue.

With the Z8 and Z9, firmware updates have made focus a little stickier, but sticking with focus also means sticking with a focus miss longer. It's a balance. The photographer can apply the degree of control they want, or change backgrounds.

 

[JoelKlein]

There's no on-off switch, it's real time adaptive.

Basically the camera computes a series of AF solutions, if the AF doesn't change at all between computations then there's no predictive delta applied. If each subsequent AF calculation changes in the same direction by a certain amount then a predictive correction factor is applied to the subsequent AF calculation. So if something moved towards or away from you at a steady speed the prediction would compensate based on the delta between each proceeding AF calculation.

But in the general case things don't move at steady speed and they may change direction and stop or start again so a series of AF solutions is fed into an algorithm which is basically some form of filtering that takes into account a series of subsequent AF calculations and comes up with a correction term for the next AF calculation. Again if a bunch of AF calculations in a row are the same (e.g. a static subject) then the correction term is zero and there's no prediction applied.

Nikon doesn't publish the specifics of their AF prediction algorithms nor any specifics on the filtering applied from calculation to calculation but I've worked on many systems like this in various engineering applications and they're all more or less the same. Basically compute a series of single point solutions at each subsequent calculation point, measure the delta (change) between each successive computation and then use that series of time changing (or time invariant) calculations to create a correction factor for the upcoming interval but time filtered in such a way that the algorithm can self adjust as the object you're tracking speeds up, slows down, changes direction or stops moving all together.

There's no perfect tracking algorithm and no perfect predictive tracking algorithms but well designed systems can work pretty well across a wide range of tracking speeds with minimal overshoot or undershoot and can easily handle the cases where there's no motion and there's no real predictive part to the tracking (measurement to measurement delta is zero).

That lack of tracking perfection in every conceivable situation at every conceivable speed or change in speed is part of why sometimes AF misses a bit. For instance in takeoff shots where a bird goes from perched to flying it's not unusual to get some crisp shots but it's also not unusual to have the AF system miss on occasion for some or even all shots in a sequence as predicting motion, speed and relative direction from a standstill is a very hard challenge and sometimes when a bird launches into the air the camera misses and sometimes it nails the entire takeoff sequence. AF systems have gotten a lot better at handling situations like this in recent years but it's still a challenging scenario compared to tracking a subject traveling at a steady speed.

Wooo, I loved your knowledge and explanation. Thank you for taking the time - Delta to explain

(Have you worked in the service as a firing solution officer?)

Back to the topic at hand, The new Sony with the global shutter and high frame rate, would only be effective if the cameras CPU can take in and crunch all those numbers. In other words; It’s not how many pictures the camera can capture per second, but rather the amount of computational data, for example; If the camera can compute AF inputs and solutions at 240 times per second, even if the shutter is only capturing 20fps, each frame has a much higher chance of being accurate?

 

[EricBowles]

Wooo, I loved your knowledge and explanation. Thank you for taking the time - Delta to explain

(Have you worked in the service as a firing solution officer?)

Back to the topic at hand, The new Sony with the global shutter and high frame rate, would only be effective if the cameras CPU can take in and crunch all those numbers. In other words; It’s not how many pictures the camera can capture per second, but rather the amount of computational data, for example; If the camera can compute AF inputs and solutions at 240 times per second, even if the shutter is only capturing 20fps, each frame has a much higher chance of being accurate?

That's generally the right way to think about it. Every part of the equation needs to be able to respond at 240 fps or it becomes a bottleneck. There are different ways to make it easier to process, so all the information does not need to be captured and processed 240 times. For example, you could read the data 120 times per second and interpolate the data for the additional calculations. Or you could limit the amount of data with a separate readout or a portion of the readout. The same ideas are true for the lens - you may not be able to move the lens 240 times per second in different directions but you can refine the direction it is moving by small amounts 240 times per second. There is also a difference between calculations at a rate of 240 times per second and full data readouts. You don't need full data readouts to have enough information for focus or exposure processing.

Nikon's approach with the dual readout for focus and for the EVF was to strip out a small amount of data to create a small image file for the EVF and for focus. It was a way to both reduce the amount of data and to allow less data movement - all permitting the camera to process the data faster.

 

[JoelKlein]

Interesting that Thom Hogen in his book doesn’t mention anything about AF ‘Predictions’. And he loves to dig technical.
This term is new for us here on BCG.
I suggest Steve should write it somewhere in his book, explaining why it’s important to assist the AF

 

[EricBowles]

Interesting that Thom Hogen in his book doesn’t mention anything about AF ‘Predictions’. And he loves to dig technical.
This term is new for us here on BCG.
I suggest Steve should write it somewhere in his book, explaining why it’s important to assist the AF

The biggest issue with AF is having a specific target that can be recognized as THE TARGET. Nikon for years has described the level of contrast required for AF targets. They also have used scene recognition for exposure, and more recently for subject recognition and focus. Through the years the main theme I've seen with missed focus is failure to chose a specific enough focus target in the form the camera can recognize. Smaller is better. Good lighting and contrast vs. the surrounding area is important. And choosing alternate AF targets to help your camera focus is important.

 

[ccirelli]

I thought that's how it was supposed to work, at least in Wide Large / Small, C1 / C2, etc. SD stays in the lines or close to them. And in 3D, it'll grab the face / eye regardless of where the little white box is positioned on the body, and stick to that person / subject even in a group (at least with people). I'll have a closer look at this now.

For OP / Larry, I have my Fn1 button set to AF Area (single point) + AF-ON, as an instant Subject Detection override, for when SD gets confused (rare) or when I just want to target something else. Although that doesn't really address your question and you're probably aware of that too.

This is one thing I wish Nikon would change - give us the option to keep Subject Detection inside the box or not.

 

[John Woodworth]

One hundred and twenty calculations per second may seem like a lot, but consider the speed our subjects might be approaching. Not even getting into airplanes and racing cars it's easy for a bird to be flying at us at 50 kilometers per hour (approximately 31 mph) and that's on the slow side as Ospreys have been clocked at 80 mph, Pronghorn Antelope can run at 60 mph and a Peregrin Falcon in a dive at close to 240 mph but dives aren't generally right towards us. Still, speeds can be fast even for live animals and a lot faster for other photo subjects.

At 50 kph coming directly at us the subject is moving nearly 11.5 cm in 1/120 of a second or roughly 4.5 inches between each AF calculation. That might not seem like much movement but when viewed through a long lens that's easily enough to be out of focus by the time the shutter is released based on the most recent AF calculation. So for many years the camera manufacturers have used predictive AF tracking algorithms to try to stay slightly ahead of subject movement to deliver as many crisp well focused images as possible for fast moving subjects. Even at 120 calculations per second there are many subjects that move fast enough that we'd want predictive AF tracking.

But yes, moving the mass of focusing elements is a small part of the question but not the whole question as even with instantaneous mechanical focus adjustment 120 calculations per second can't keep up with the fastest photography subjects if they're coming right at us.

Some thoughts:

I think a reasonable (and idealized) guestimate for a desirable time between focusing updates might be based on shutter speed. Any filtering algorithm--Kalman or Bayesian predictor--will have assumptions about future motion baked in which will fail for sufficiently erratic motion. At that point, brute force, in the form of rapid updates, will be required to maintain focus.

It is not uncommon to see shutter speeds of 1/3000 s which would imply a >6 kHz sampling rate. Hopefully this is a gross overestimate since the power consumption required would be much larger than the current state. Probably, a better estimate is to ask what shutter speed is required to stop the motion of a bird's eye (the point of focus) in the viewfinder. Maybe that is more like 1/500 s...We are still talking about sampling speeds in the low kiloHertz.

This raises the issue of what should the system be focusing on. The future trajectory of the eye of many diving birds can probably be represented by a 1st or 2nd order extrapolation at much lower sample rates that mentioned here. But, if the detected focus point disappears--as it does when the bird's head enters the water--how is the focus system to behave? The main possibility is for it to freeze until it can detect and reacquire the target. This would be the "Tracking lock-on" option. It could combine this with a "locality search" feature which would calculate an expanding search volume. Hopefully, this expanding front would be slower that the water wave. To my knowledge, this is currently not implemented and maybe just isn't a useful idea. There are also conceivable AI approaches.

Hopefully, I have provided at least some level of amusement.

 

[EricBowles]

Some thoughts:

I think a reasonable (and idealized) guestimate for a desirable time between focusing updates might be based on shutter speed. Any filtering algorithm--Kalman or Bayesian predictor--will have assumptions about future motion baked in which will fail for sufficiently erratic motion. At that point, brute force, in the form of rapid updates, will be required to maintain focus.

It is not uncommon to see shutter speeds of 1/3000 s which would imply a >6 kHz sampling rate. Hopefully this is a gross overestimate since the power consumption required would be much larger than the current state. Probably, a better estimate is to ask what shutter speed is required to stop the motion of a bird's eye (the point of focus) in the viewfinder. Maybe that is more like 1/500 s...We are still talking about sampling speeds in the low kiloHertz.

This raises the issue of what should the system be focusing on. The future trajectory of the eye of many diving birds can probably be represented by a 1st or 2nd order extrapolation at much lower sample rates that mentioned here. But, if the detected focus point disappears--as it does when the bird's head enters the water--how is the focus system to behave? The main possibility is for it to freeze until it can detect and reacquire the target. This would be the "Tracking lock-on" option. It could combine this with a "locality search" feature which would calculate an expanding search volume. Hopefully, this expanding front would be slower that the water wave. To my knowledge, this is currently not implemented and maybe just isn't a useful idea. There are also conceivable AI approaches.

Hopefully, I have provided at least some level of amusement.

The Nikon system is relatively specific about focus and the priority.

It uses scene recognition for exposure and subject recognition. Subject recognition is further supported by the menu and the photographer's selection, but if the camera does not find the desired subject, it reverts to Area AF and looks for the nearest high contrast target near the focus point.

If it does find the subject, the subject is recognized based on the subject type - bird, animal, person, etc. Once the subject is found, it further looks for the head, and then the eye. If it can't find the eye, it falls back to the head and then the body in that order. Of course the assumption is the subject still looks like a bird - and with a two dimensional image there is no certainty that a bird looks like the AI based options for birds. Again - if a bird is not recognized, there is a defined alternative to find the high contrast target near the AF point selected or in use. There is also a general process to look near the focus point in use - touching or near that focus point.

All this means that with a diving bird, there is no scenario where it freezes or pauses instead of searching for a high contrast target. It will take something in the background or simply give up after hunting and finding nothing with contrast in the frame. The hope is that it does all this - searching for subjects or finding an alternate high contrast target - is done within a fraction of a second. The camera's answer to finding nothing - no contrast and no potential target - is to hunt and then quit - but that takes some time.

 

[John Woodworth]

The Nikon system is relatively specific about focus and the priority.

It uses scene recognition for exposure and subject recognition. Subject recognition is further supported by the menu and the photographer's selection, but if the camera does not find the desired subject, it reverts to Area AF and looks for the nearest high contrast target near the focus point.

If it does find the subject, the subject is recognized based on the subject type - bird, animal, person, etc. Once the subject is found, it further looks for the head, and then the eye. If it can't find the eye, it falls back to the head and then the body in that order. Of course the assumption is the subject still looks like a bird - and with a two dimensional image there is no certainty that a bird looks like the AI based options for birds. Again - if a bird is not recognized, there is a defined alternative to find the high contrast target near the AF point selected or in use. There is also a general process to look near the focus point in use - touching or near that focus point.

All this means that with a diving bird, there is no scenario where it freezes or pauses instead of searching for a high contrast target. It will take something in the background or simply give up after hunting and finding nothing with contrast in the frame. The hope is that it does all this - searching for subjects or finding an alternate high contrast target - is done within a fraction of a second. The camera's answer to finding nothing - no contrast and no potential target - is to hunt and then quit - but that takes some time.

Right.

My post mainly addressed, in principle, the sampling frequency required to predict focus of erratically, but detected, subjects. This must be related to shutter speed and for BIF it is generally a lot faster than the current state of the art of today's top-end cameras.

The scenario in this thread is a case where the detected subject disappears for several frames. What does the camera do in the meantime?

I haven't seen a complete explanation for what Nikon's "Tracking with Lock-On" and "Blocked Shot AF Response" actually does. I just assumed that it stops trying to refocus for a set duration until the subject is re-detected and re-acquired. But does it continue to extrapolate a trajectory (Steady), or does it search within a volume (Erratic) while its waiting? Or what?

That is the question.

 

[SCoombs]

Without the aim of getting off topic, this whole very technical discussion increasingly makes me wonder if it may be relevant to another subject tracking phenomenon recently under discussion elsewhere.

The short version is that I had observed win lower light that my Z8's subject tracking in AF-C would frequently miss focus (on people) despite the focus point indicating a lock. Further investigation has revealed that frequently what is happening is it is focusing on the other eye so that if the focus was reported as being on the near eye, in the photo the far eye will be in focus and the near one our, or vice versa. At least one other user has taken to testing this extensively and duplicated my results. He has also found it happens less-so shorter focal lengths, and I've found it happens less beyond 2 or 3 meters. Sometimes nothing apparent is in focus, but very often it looks out of focus because it's swapped eyes.

Of course I also think of the way the system will sometimes have the AF tracking box appear to jitter a bit, as if unsure, or to bounce between eyes, but this happens even when it is steady and not changing eyes at all. In the same situations single point or dynamic modes with no tracking work fine, but with subject tracking modes in this lowering light even in focus shots are often much softer than their single point counterparts.

I wonder: might there be some impact here of these predictive tracking algorithms? As if it is expecting for some reason the eye to be in a different place?

Now that I have experienced this with people I am thinking about whether I have seen anything similar with birds or animals and want to try shooting with subject tracking off a bit to see if I notice the same sort of difference.

 

[EricBowles]

Right.

My post mainly addressed, in principle, the sampling frequency required to predict focus of erratically, but detected, subjects. This must be related to shutter speed and for BIF it is generally a lot faster than the current state of the art of today's top-end cameras.

The scenario in this thread is a case where the detected subject disappears for several frames. What does the camera do in the meantime?

I haven't seen a complete explanation for what Nikon's "Tracking with Lock-On" and "Blocked Shot AF Response" actually does. I just assumed that it stops trying to refocus for a set duration until the subject is re-detected and re-acquired. But does it continue to extrapolate a trajectory (Steady), or does it search within a volume (Erratic) while its waiting? Or what?

That is the question.

The camera stops trying to focus on the original subject if it is no longer visible and identifiable. The time is based on your Lock-On setting (fractions of a second). The Erratic setting does not matter unless you have a subject. When it stops trying to focus on the original subject it does not stop looking for a target - it chooses based on Auto Area a target in the frame that has contrast and is near the AF point. This could be the water, a splash, or the background depending on which AF area mode is being used and the availability of a contrasty target.

The Lock On setting is essentially saying after a specified period after you last had focus, go find something else. That also means the last few frames before losing focus are likely using this time period.

The issue of predicting movement is part of AF-C. It does not affect lock on. Usually the subject is close enough to the AF point being used that it is touching the AF box and the camera will recognize the subject. But if it has moved out of the frame, all you have is the Lock on duration and then the camera finds something else.

To test it, turn off subject detection, use Area or Wide mode, and see what the camera finds. It will likely focus on water using the wide modes, and whatever has contrast and makes a good target in Area mode. With Dynamic, it will likely pick up either water or the background. Obviously with a test like this, you won't have much control and the camera can pick up lots of targets. But the instruction is simple - go find something.

 

[EricBowles]

Without the aim of getting off topic, this whole very technical discussion increasingly makes me wonder if it may be relevant to another subject tracking phenomenon recently under discussion elsewhere.

The short version is that I had observed win lower light that my Z8's subject tracking in AF-C would frequently miss focus (on people) despite the focus point indicating a lock. Further investigation has revealed that frequently what is happening is it is focusing on the other eye so that if the focus was reported as being on the near eye, in the photo the far eye will be in focus and the near one our, or vice versa. At least one other user has taken to testing this extensively and duplicated my results. He has also found it happens less-so shorter focal lengths, and I've found it happens less beyond 2 or 3 meters. Sometimes nothing apparent is in focus, but very often it looks out of focus because it's swapped eyes.

Of course I also think of the way the system will sometimes have the AF tracking box appear to jitter a bit, as if unsure, or to bounce between eyes, but this happens even when it is steady and not changing eyes at all. In the same situations single point or dynamic modes with no tracking work fine, but with subject tracking modes in this lowering light even in focus shots are often much softer than their single point counterparts.

I wonder: might there be some impact here of these predictive tracking algorithms? As if it is expecting for some reason the eye to be in a different place?

Now that I have experienced this with people I am thinking about whether I have seen anything similar with birds or animals and want to try shooting with subject tracking off a bit to see if I notice the same sort of difference.

If your image is under exposed, subject and eye detection is much less effective. Your exposure needs to be accurate or moderately bright. Jittering is a common result of marginal focus - low light being a contributor. It happens with animals and birds as well as people.

It does swap eyes if it does not recognize the eye or believes that distance is the same because of low light. You can switch eyes if that happens. You can also use a smaller AF area mode. You might also simply turn off subject recognition and use another mode if you are fighting focus using SR. I might use Single or Dynamic if it was not working.

 

[John Woodworth]

The camera stops trying to focus on the original subject if it is no longer visible and identifiable. The time is based on your Lock-On setting (fractions of a second). The Erratic setting does not matter unless you have a subject. When it stops trying to focus on the original subject it does not stop looking for a target - it chooses based on Auto Area a target in the frame that has contrast and is near the AF point. This could be the water, a splash, or the background depending on which AF area mode is being used and the availability of a contrasty target.

The Lock On setting is essentially saying after a specified period after you last had focus, go find something else. That also means the last few frames before losing focus are likely using this time period.

The issue of predicting movement is part of AF-C. It does not affect lock on. Usually the subject is close enough to the AF point being used that it is touching the AF box and the camera will recognize the subject. But if it has moved out of the frame, all you have is the Lock on duration and then the camera finds something else.

To test it, turn off subject detection, use Area or Wide mode, and see what the camera finds. It will likely focus on water using the wide modes, and whatever has contrast and makes a good target in Area mode. With Dynamic, it will likely pick up either water or the background. Obviously with a test like this, you won't have much control and the camera can pick up lots of targets. But the instruction is simple - go find something.

So, if a runner goes behind a tree and comes out the other side within the time limit, there is no attempt to extrapolate the previous trajectory and anticipate, for the purposes of detection, when he/she will emerge on the other side?

 

[EricBowles]

So, if a runner goes behind a tree and comes out the other side within the time limit, there is no attempt to extrapolate the previous trajectory and anticipate, for the purposes of detection, when he/she will emerge on the other side?

I don't believe so. Of course, normally the photographer would follow the subject expecting it to come out the other side. But if the time limit was exceeded, the camera would stop looking for the runner/walker and focus on the tree. Once the subject appeared again, the camera would start over with subject detection and focus on the runner. Exactly how long until it drops the person and chooses the tree is based on the Lock On setting - you can make it quicker or slower.

 

[SCoombs]

If your image is under exposed, subject and eye detection is much less effective. Your exposure needs to be accurate or moderately bright. Jittering is a common result of marginal focus - low light being a contributor. It happens with animals and birds as well as people.

It does swap eyes if it does not recognize the eye or believes that distance is the same because of low light. You can switch eyes if that happens. You can also use a smaller AF area mode. You might also simply turn off subject recognition and use another mode if you are fighting focus using SR. I might use Single or Dynamic if it was not working.

Yes, all settings are optimized to maximize evf brightness - eg adjust for easy viewing (even tried starlight). Dynamic or single point mostly yield good focus. The issue being explored has to do with accuracy of focus with SR in lower light. It's not even the actual subject recognition, which the system does perfectly. It's that the subject it has recognized and put the AF indicator on winds up being out of focus and specifically focused on the opposite eye from where the AF box is located - even if you use the option to specify which eye it should focus on. The discussion of predictive tracking made the wheels really start turning to think if there is a connection.

 

[Ken Miracle]

Yes, all settings are optimized to maximize evf brightness - eg adjust for easy viewing (even tried starlight). Dynamic or single point mostly yield good focus. The issue being explored has to do with accuracy of focus with SR in lower light. It's not even the actual subject recognition, which the system does perfectly. It's that the subject it has recognized and put the AF indicator on winds up being out of focus and specifically focused on the opposite eye from where the AF box is located - even if you use the option to specify which eye it should focus on. The discussion of predictive tracking made the wheels really start turning to think if there is a connection.

If you set evf to adjust for easy viewing you are not going to see what your image exposure is going to be. I do not want it artifically adjusting the brightness of the view finder for easy viewing. I have mine set to manual and +1 same as Steve. I use Manual with auto iso and adjust EV exposure compensation and for darker subject af accuracy goes up with proper exposure and I have my histogram visible so I know what exposure I am going to get.

 

[SCoombs]

If you set evf to adjust for easy viewing you are not going to see what your image exposure is going to be. I do not want it artifically adjusting the brightness of the view finder for easy viewing. I have mine set to manual and +1 same as Steve. I use Manual with auto iso and adjust EV exposure compensation and for darker subject af accuracy goes up with proper exposure and I have my histogram visible so I know what exposure I am going to get.

This generally works for wildlife shooting, but when one is in a situation where a flash is going to be used there are two problems with this.

The first is that the camera does not know or properly display in the EVF or the viewfinder what the exposure will look like with the flash. Actually, my understanding is that it might do this for a select handful of Nikon branded speedlights but even then it's pretty iffy as to how accurate it is. Meanwhile, for any 3rd party speedlights, off camera strobes, and even other Nikon branded ones it just doesn't work.

The second is that because of the fact that the evf is not being adjusted according to the anticipated effect of the flash, and because the camera will have it's exposure set to account for the flash, the EVF will display an extremely dark, even pitch black scene, and since this is what the AF system uses to focus, focus will become impossible.