FYI: Z8/Z9 rolling shutter in 60fps/120fps continuous modes

[Horshack]

I just completed my sensor readout measurements of the Z8 for my Crowdsourced Rolling Shutter GitHub project. I previously tested the Z9, whose performance I was expecting the Z8 to match, which it does. But I added a few more tests on the Z8 that I didn't perform on the Z9, specifically 60fps and 120fps continuous shooting modes (manual page).

My 60fps/120fps measurements have revealed the camera significantly drops the readout rate on these modes from the usual Z8/Z9 1/268:

• 60 fps (full-sized DX-Area JPEG) drops to 1/90
• 120 fps (small-sized FX-Area JPEG) drops to 1/173

A common use case for these ultra-fast frame rates is fast moving action, so this drop has implications for rolling shutter artifacts. Here's a demonstration I did of a tripod-mounted pan of my window shutters (puns are fun), comparing 20 fps DX to 60 fps DX:

Animation: Z8 20fps vs 60fps rolling shutter on camera pan

I don't own a Z9 anymore to test but I wouldn't expect it to perform any differently than what I'm observing with the Z8.

 

[Rodenmg]

Not that I ever use those high shutter rates but it's good to know. Great job.

 

[Horshack]

Based on the observed readout rates I'm guessing the camera is using the video logic for the "High-Speed Frame Capture" C30/C60/C120 modes, probably because the stills imaging pipeline on Expeed isn't fast enough to support these rates.

This would explain why the C120 has less rolling shutter than the C60 - that matches the 4K 120fps vs oversampled 60fps video measurements, the former of which is faster due to some measure of line skipping/binning.

 

[maverick256]

Based on the observed readout rates I'm guessing the camera is using the video logic for the "High-Speed Frame Capture" C30/C60/C120 modes, probably because the stills imaging pipeline on Expeed isn't fast enough to support these rates.

If this hypothesis is true, then I guess the prospect of having raw support for DX and 11MP modes might be unlikely? Raw for pre-release capture might also be unlikely as well huh. I was hoping one day DX mode pre-release can get raw support at like 30-40FPS, that would be nice.

 

[Horshack]

Based on the observed readout rates I'm guessing the camera is using the video logic for the "High-Speed Frame Capture" C30/C60/C120 modes, probably because the stills imaging pipeline on Expeed isn't fast enough to support these rates.

If this hypothesis is true, then I guess the prospect of having raw support for DX and 11MP modes might be unlikely? Raw for pre-release capture might also be unlikely as well huh. I was hoping one day DX mode pre-release can get raw support at like 30-40FPS, that would be nice.

Through the stills-imaging pipeline it seems unlikely, considering Expeed on the Z8/Z9 takes 50ms per still frame, as described here, which works out to 20fps.

There's a chance Nikon could offer image raws from the video pipeline though, using the same logic path as N-RAW.

 

[Restoc]

I do not know if there is a hardwired pipeline different in still and video. Sure at the level of coding there is a separation but i coud be by SW only.

When reading the Nikon specifications it appears that the read out speeds should limit only the hrorizintal lines number per sec: not the columns both in still or video:

As example video 8K format is able to deliver 8126 columns ( such as Still FF) up to 60 Fps rather tjat 20 IPS burst only . Max throuput is given by but 8126 *4464 lines ( video only) instead of 5564 (still FF) and probably because of limitation of 16/9 factor it could be a bit higher number of lines.

For the full FF stills which are limited up to now at bursts of 20 ips so it looks like the sensor readout doesnt seems to be the bottleneck.

In addition when looking at the so many frames formats that Nikon Z8/9 manage in Still+ video ( look at (https://onlinemanual.nikonimglib.co...0179f851-6592-5fd1-76fb-afc37e21c6ee_306.html) Nikon is in position to arrange bursts and Pre-release RAW by reducing at a16/9 still format. The critical point is that the image buffer can welcome only 79 uncompressed Still Raw at full speed.Nikon have just to find a coherence ; duration of Pre release / Max burst RAw speed/ Max real time buffering to make it usable.

The latest release 5.0 of the Z9 has been designed for news/sports and direct diffusin in jpeg with some improvment of Jpeg (120 ipS jump to normal Jpeg instad of only basic).

We keep hope to get soon a PR in raw .

 

[Horshack]

I do not know if there is a hardwired pipeline different in still and video. Sure at the level of coding there is a separation but i coud be by SW only.

When reading the Nikon specifications it appears that the read out speeds should limit only the hrorizintal lines number per sec: not the columns both in still or video:

As example video 8K format is able to deliver 8126 columns ( such as Still FF) up to 60 Fps rather tjat 20 IPS burst only . Max throuput is given by but 8126 *4464 lines ( video only) instead of 5564 (still FF) and probably because of limitation of 16/9 factor it could be a bit higher number of lines.

For the full FF stills which are limited up to now at bursts of 20 ips so it looks like the sensor readout doesnt seems to be the bottleneck.

In addition when looking at the so many frames formats that Nikon Z8/9 manage in Still+ video ( look at (https://onlinemanual.nikonimglib.co...0179f851-6592-5fd1-76fb-afc37e21c6ee_306.html) Nikon is in position to arrange bursts and Pre-release RAW by reducing at a16/9 still format. The critical point is that the image buffer can welcome only 79 uncompressed Still Raw at full speed.Nikon have just to find a coherence ; duration of Pre release / Max burst RAw speed/ Max real time buffering to make it usable.

The latest release 5.0 of the Z9 has been designed for news/sports and direct diffusin in jpeg with some improvment of Jpeg (120 ipS jump to normal Jpeg instad of only basic).

We keep hope to get soon a PR in raw .

Sensor readout is on a per-row basis, thus the metrics about readout speeds are oriented around rows, not columns. The sensor readout to stacked memory is clearly not the bottleneck since we measure a readout speed around 1/270. But getting from that initial pixel readout to a finished image ready for writing to media involves a lot of processing steps and a lot of bandwidth, much of which has to occur in regular SDRAM rather than stacked sensor memory. That's where the 20fps stills pipeline bottleneck comes in.

In video mode the sensor doesn't run at 1/270 - it runs at 1/70, as I've measured here. I have an open discussion about why stacked sensors don't or can't seem to run at their native readout speed for video here.

 

[Restoc]

Thanks that a good sumary.

The bottleneck for using the Still "Pipeline" at higher rate ( let say "virtual pipeline" i because we cannot see double wired busses on the Mother Board) is probably somewhere when sensor unmatriced photosites values come into the SDram at Bayer level for elaborating the so called raw . Then photosites data need to be stored, for matrixing, compressed, then, some other Nikon image processes added before intoducing exifs and becoming a true Nef formated file sended to the Cfexpress.

But i cannot imagine why the video coding pipe ( which is comples too), and shares most of the hereunder list of preprocessing in Ram can be up to 3 times faster at the end at full pixels frame formas.

For sure HE* raw format for Stills may eat a lot of computing power if not wired. But Nikon needs now to deliver simple some normal raw in Pre release to match with Competitors doing such for years.

20 FPS still vs 60 Fps video capability remain a question. Except if they are using some dedicated fast Asics for video coding and not the CPU.


Claude.T

 

[Horshack]

Thanks that a good sumary.

The bottleneck for using the Still "Pipeline" at higher rate ( let say "virtual pipeline" i because we cannot see double wired busses on the Mother Board) is probably somewhere when sensor unmatriced photosites values come into the SDram at Bayer level for elaborating the so called raw . Then photosites data need to be stored, for matrixing, compressed, then, some other Nikon image processes added before intoducing exifs and becoming a true Nef formated file sended to the Cfexpress.

But i cannot imagine why the video coding pipe ( which is comples too), and shares most of the hereunder list of preprocessing in Ram can be up to 3 times faster at the end at full pixels frame formas.

For sure HE* raw format for Stills may eat a lot of computing power if not wired. But Nikon needs now to deliver simple some normal raw in Pre release to match with Competitors doing such for years.

20 FPS still vs 60 Fps video capability remain a question. Except if they are using some dedicated fast Asics for video coding and not the CPU.
Claude.T

Agree that the stills vs video bottlenecks are a bit puzzling, since they not only share most if not all of the pipeline steps (demosaicing, WB scaling, NR, picture control application, lens corrections, etc..) but also share the same intoPIX TicoRAW HE encoding. Yet video can do 8K 60fps N-RAW vs only 20fps stills (with modestly higher resolution). Here are the differences that come to mind that might be factors:

• Video uses a 12-bit readout (N-RAW) or 10-bit readout (H.264/H.265) vs 14-bit readout for stills. This difference seems modest compared to the 60fps vs 20fps bandwidth demands.
• Even with its lower readout depth, video uses a much slower sensor readout. The slower readout reduces peak bandwidth demands for the sensor readout -> stacked memory transfer, yet doesn't have an affect on overall bandwidth demands since it's ultimately still doing 60fps, so not clear why this should matter much.
• As you noted, the video pipeline is likely using a mature, full end-to-end pipeline in the ASIC, probably a partial IP block purchased by Nikon rather than developed entirely internally.

Whatever the reason it appears to be a hard technical limit, considering Nikon went through the trouble of implementing these C30/C60/C120 video-based stills image modes but with their obvious resolution limits and rolling-shutter implications. I have to think Nikon would've implemented them as full-resolution stills-pipeline equivalents if it could have.

 

[Rodenmg]

But Nikon needs now to deliver simple some normal raw in Pre release to match with Competitors doing such for years.

Claude.T

I haven’t really been keeping up with this stuff but who has been doing 120fps raw Precapture for years now?